Tag Archives: Advanced (300)

Improve the discoverability of your unstructured data in Amazon SageMaker Catalog using generative AI

2026-04-01 Nishchai JM

Post Syndicated from Nishchai JM original https://aws.amazon.com/blogs/big-data/improve-the-discoverability-of-your-unstructured-data-in-amazon-sagemaker-catalog-using-generative-ai/

Every day, businesses generate massive amounts of unstructured data such as PDFs, images, emails, customer feedback. Although this data holds valuable business insights, extracting meaningful value from it remains a significant challenge. Its lack of proper context and searchability often keeps it siloed and underutilized, limiting data-driven decision making. Financial reports, legal documents, and customer feedback are prime examples. They contain the answers that your business needs, yet they frequently go unanalyzed due to these barriers.The sheer volume of unstructured content requires scalable infrastructure and automated processing tools, while sensitive information embedded within demands sophisticated classification and protection strategies. Without proper management, organizations face operational inefficiencies, high costs, and increased regulatory risks and reduced AI effectiveness.

What if business context from your PDFs, images, and emails could be automatically extracted and surfaced wherever your teams search for information? In this post, we show you how to implement this. By combining Amazon SageMaker Catalog with generative AI capabilities, you can make unstructured data searchable and queryable through the same interfaces that your teams use for structured data analysis. Success requires balancing advanced AI techniques with governance frameworks so that your data is discoverable and secure for better decision making.

This is a two-part series post. In the first part, we walk you through how to set up the automated processing for unstructured documents, extract and enrich metadata using AI, and make your data discoverable through SageMaker Catalog. The second part is currently in the works and will show you how to discover and access the enriched unstructured data assets as a data consumer. By the end of this post, you will understand how to combine Amazon Textract and Anthropic Claude through Amazon Bedrock to extract key business terms and enrich metadata using Amazon SageMaker Catalog to transform unstructured data into a governed, discoverable asset.

Solutions overviewYou will transform unstructured data into an interactive knowledge base through automated processing within the Amazon SageMaker AI environment. Here is how it works:

You will set up an Amazon SageMaker Unified Studio Data Notebook Jupyter-based workspace where you manage your entire processing pipeline, add metadata to unstructured documents like PDFs, photos, emails, or audio recordings stored in Amazon Simple Storage Service (Amazon S3).
You will add your files to the SageMaker Project.
Amazon Textract extracts information and insights from text, removing manual transcription. This extracted content instantly populates your asset’s README.
Amazon Bedrock turns the text into business terms that provide SageMaker Catalog assets the correct business context to help with semantic search or business query search.
You will use a publish method to publish the enriched data to the Amazon SageMaker Catalog, making it available to your organization.

The architecture sets up a pipeline from processing raw documents to enabling end users interaction, with the Amazon SageMaker Catalog serving as the central hub for sending and receiving data. Amazon SageMaker Catalog includes generative AI features that automatically develop and add business descriptions for structured data assets. This capability streamlines documentation processes and provides greater consistency across data assets. You can further enhance this solution to create summaries by also reading and incorporating S3 metadata. This will add more context, such as object properties, access patterns, and storage characteristics, to the extracted document content, streamlining the process to find and catalog data.

Prerequisites

To implement the solution, you must complete the following prerequisites:

Create an AWS account – Required to access all AWS services (Amazon SageMaker Catalog, Amazon S3, Amazon Textract, Amazon Bedrock) used in this solution.
Create an Amazon SageMaker Unified Studio domain: This provides a collaborative environment for connecting your assets, users, and their projects.
Create an SageMaker Project with all capabilities: Your collaborative workspace where you will upload documents, run processing notebooks, and manage permissions for your data enrichment pipeline. Team members added to this project gain immediate access to all shared resources.
- Producer Project (project name: $(-your-project-name) use “unstructured-producer-project”, project profile: All capabilities)

Solution deployment

Now let’s complete the following steps to deploy and verify the solution.

Prepare source datasets

In this section you will use the following sample datasets by downloading them to your local machine. We will upload these files into your SageMaker Project S3 bucket created in the prerequisite step.

ED_DistributionToothDisorder.png: The dataset shows emergency department visits for tooth disorders in the US from 2020–2022, broken down by age group, gender, and race/ethnicity.
analysisDentalEDvsts.pdf: This report shows emergency department visits analysis for dental conditions across the United States between 2016–2019, showing that among non-traumatic dental visits.
s3_document_processor_unstructured.ipynb notebook (keep in local environment and will be used at a later stage)

With your sample datasets ready, let’s log in to SageMaker Unified Studio and upload them to your project.

Log in to Amazon SageMaker Unified Studio as a data producer

Log in to the SageMaker Unified Studio URL using your username and password. In the portal UI, select the producer project (unstructured-producer-project) that you created in the project selector (at the top center of the screen).
Under Data, do the following:
- Choose the default project bucket created amazon-sagemaker-12*********-us-west-2-51271642b525/dzd_*********/c3jl67qvxbic9c/.
- Next, choose the three dots and upload the downloaded files (1&2) from the prepared dataset section.
- After adding the files, choose Publish to Catalog to publish your asset.

Your files are now in the catalog. Before we process them, your project needs permission to access the services. Let’s add those permissions.

Add permissions to an IAM role for the Amazon SageMaker Project role.
- Go to the Project overview tab and find the Project role ARN. It can be found in the Project details section.
- Go to the AWS IAM service and choose Roles. Search for the role as highlighted in the preceding image and add the following permissions. The following policies use full-access managed policies to keep things straightforward for this walkthrough. We don’t recommend this for production environments. Instead, we encourage you to take a moment to review each policy with your security team and scope them down to the least-privilege permissions that your workload needs:
  - Add an AmazonBedrockFullAccess managed policy.
  - Add an AmazonTextractFullAccess managed policy.
  - Add an AmazonS3FullAccess managed policy.
  - Add this inline policy to project policy
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "datazone:Search",
                "datazone:GetAsset",
                "datazone:CreateAssetRevision",
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
    ]
}
```

With permissions configured, let’s set up the governance framework that will classify your documents. We will create glossary terms to tag sensitive and non-sensitive data.

Add Glossary and Glossary terms.
- Choose Glossaries and CREATE GLOSSARY.
- Add the following:
- Name of the glossary and descriptions.
- Toggle on the Enable button as shown in the following screenshot.
- Create an appropriate glossary term. You will add this term to your business metadata.
- Navigate to the Discover menu in the top navigation bar.
- Choose Glossaries, and then select Create term.
- Ensure that you’re creating the term under the Confidentiality glossary that you created in step 4.
- Create two terms (sensitive and non-sensitive) and add a description. Make sure that the Enabled toggle is on to enable the new term.

Now that we have configured the necessary permissions and created our glossary terms, let’s proceed with building the business metadata.

Build business metadata

In this section, you will use Amazon Textract and Amazon Bedrock to automatically build and curate business metadata for your assets using a SageMaker Unified Studio Notebook.

From the Project Overview page, access the Compute section in the left menu.
- Navigate to the Spaces tab.
- Choose the default space created by your project(default-985-) to begin”.

Open the space details page by selecting the Name (default-****).
- Under Actions, Choose Open space to be taken to the Data Notebook workspace (Jupyter based).

After connected, upload the downloaded notebook from the prerequisite step in your JupyterLab interface by either dragging it into the File browser or using the upload icon.

Before running the notebook, let’s understand what each cell does and how they work together to transform your documents into discoverable assets.

The notebook contains code for processing your documents. It begins by setting up AWS service connections using Boto3, the SDK for Python, importing necessary libraries, and initializing clients for Amazon S3, Amazon Textract, and Amazon Bedrock. The code configures an S3 bucket for processing medical documents.

Now, proceed to run through the individual cells:

This cell searches an S3 bucket for files with specific extensions (.pdf, .jpg, .jpeg, .png, .tiff), collects them into a list called ‘documents’, and prints the total count and names of found files. It uses the list_objects_v2 method to fetch the contents and filters them based on their file extensions.

The documents extracted from the S3 bucket are processed using Amazon Textract. It loops through each document, starts an Amazon Textract job, monitors its progress, and when successful, extracts text from all pages. The extracted text is stored in a list along with its document identifier. The code handles pagination, errors, and includes delays between API calls to prevent throttling.

Output [1] The following screenshot shows the output of a Jupyter notebook cell after you run the Amazon Textract API.

This code takes all previously extracted document text, combines it into one string (2,002 characters total), and uses the Anthropic Claude 3 Sonnet model (available through Amazon Bedrock) to generate a concise summary. The code configures the AI model with specific parameters, sends the combined text for analysis, and returns a summarized version of all document content.

Output [2] The following screenshot shows the proceeding output result.

This code detects whether document text contains sensitive PII data (names, emails, addresses, financial details) and returns a Boolean true/false result.

Finally, the code completes the document processing pipeline by classifying the asset based on sensitivity, updating its metadata with an AI-generated summary, and assigning the appropriate glossary term. The script retrieves existing asset details to preserve all metadata forms, updates the README field with the summary content, and creates a new revision. This record of the classification and documentation is stored in the data catalog, accessible alongside your source assets for ongoing governance.

Output: The following screenshot shows the output result.

After generating metadata with Amazon Textract and Amazon Bedrock, republish the data to make it discoverable to users. Note the README and the Glossary terms have already been added based on the previous script.

To republish unstructured data with enriched metadata, go to your producer project and choose Re-publish asset.

To search for the asset that was published, choose one of the keywords from the README sections. For this example, we search using these keywords high percent of ED visits.

Go to Home in the search bar to enter the keyword. Then choose the asset as displayed in the following screenshot:

As shown in the preceding image, the search results display the asset name along with the project it belongs to. Select the asset to view its details, where you will find rich business metadata, lineage information, and more, as shown in the following screenshot.

With the asset published and metadata enriched, data assets are ready to be used.

Clean up

To avoid ongoing charges, make sure to delete the resources immediately after completing the tutorial:

Stop Studio Resources – Close all running notebooks – Stop any running notebook instances – Shut down unused kernels. Running instances continue to incur charges even when not actively used.
Clean S3 Storage – Delete any temporary files created during processing – Remove uploaded test documents if no longer needed. While Amazon S3 costs are minimal, large volumes of unneeded data can accumulate charges.

Conclusion

In this post we showed you how you can transform unstructured data into valuable business assets through seamless integration with AWS services. You can efficiently process documents using Amazon Textract for text extraction, harness the capabilities of Amazon Bedrock for intelligent term identification, and use Amazon SageMaker Catalog for metadata management—all within a secure, governed framework.

Additional resources

To continue your Amazon SageMaker AI journey, see the following resources:

1. - Explore the Amazon SageMaker Developer Guide.
  - Documentation – You can find complete setup guides in Amazon SageMaker Unified Studio Documentation.
  - Get started with these new integrations through the Amazon s Unified Studio console.

About the authors

Automate safety monitoring with computer vision and generative AI

2026-04-01 Nika Mishurina

Post Syndicated from Nika Mishurina original https://aws.amazon.com/blogs/architecture/automate-safety-monitoring-with-computer-vision-and-generative-ai/

Workplace safety has improved dramatically over the past several decades. According to the Bureau of Labor Statistics, occupational injury rates in the United States have declined by more than 60% since the early 1970s. This is driven by stronger regulations, better training programs, and a growing culture of safety-first operations. Despite this progress, the International Labour Organization reports that 395 million workers worldwide still sustain non-fatal occupational injuries each year, and the National Safety Council estimates that workplace injuries cost the US economy $176.5 billion in 2023.

The challenge is no longer a lack of safety commitment, it’s the limitations of traditional monitoring methods. Manual safety audits, while valuable, cover only a fraction of operational areas and produce point-in-time snapshots rather than continuous oversight. As organizations scale across hundreds of facilities, whether manufacturing floors, distribution centers, airport tarmacs, construction sites, or laboratory environments, maintaining consistent, real-time visibility into Personal Protective Equipment (PPE) compliance and zone-based hazard monitoring becomes increasingly difficult.

According to OSHA, struck-by vehicle fatalities and injuries are 100 percent preventable, yet they remain a leading cause of workplace fatalities. However, 90 percent of workplace eye injuries can be avoided by wearing eye protection, according to the American Academy of Ophthalmology.

Computer vision and generative AI represent the next evolution in workplace safety, not replacing existing safety programs, but augmenting them with continuous, automated monitoring that scales across facilities around the clock. This post describes a solution that uses fixed camera networks to monitor operational environments in near real-time, detecting potential safety hazards while capturing object floor projections and their relationships to floor markings. While we illustrate the approach through distribution center deployment examples, the underlying architecture applies broadly across industries. We explore the architectural decisions, strategies for scaling to hundreds of sites, reducing site onboarding time, synthetic data generation using generative AI tools like GLIGEN, and other critical technical hurdles we overcame.

Solution overview

Our computer vision solution uses a serverless, event-driven architecture designed to scale efficiently across thousands of cameras and process massive volumes of image data for risk detection. The system includes the following:

A machine learning (ML) model that identifies workplace safety hazards
Real-time visual data processing for emerging risks
A dual-detection annotation method that captures both object outlines and their floor projections relative to safety markings.

The system blurs human faces and identifiable features to help protect PII while maintaining hazard detection accuracy. To maintain proper security and operational segregation, the solution is distributed across multiple AWS accounts. We separated the training pipeline, image collection infrastructure, end-user web application, and created a dedicated analytics account for the BI team to develop reporting and insights solutions into distinct environments with appropriate access controls and data isolation. The system continuously learns to improve detection accuracy. Safety managers use monitoring dashboards to track and respond to hazards. The following architecture diagram illustrates how these components work together to create an end-to-end safety monitoring solution.

Architecture Diagram

The system implements a hierarchical role-based access control structure with four user types.

Super Users are system administrators with organization-wide visibility, responsible for site onboarding initialization and system health monitoring.
Site Administrators operate at the facility level, configuring zones and managing permissions within their site.
Zone Owners play a critical operational role—they receive and remediate safety risk notifications, complete tape labeling jobs to verify detection accuracy, and perform camera onboarding configuration. The system only begins automated risk detection after Zone Owners complete the full configuration process, so that parameters are properly set.
Zone Users have read-only access to risks and alerts without configuration capabilities.

Image collection and anonymization

The workflow begins with automated image collection from configured and authorized site cameras through a dedicated image service providing periodic image capture. Raw images are initially stored in an Amazon Simple Storage Service (Amazon S3) bucket in a separate, access-restricted account where they immediately undergo an anonymization process. After anonymization is complete, raw images are automatically purged from the bucket within days, as per organizational retention policies. Amazon Rekognition detects faces of individuals present in the images, and custom Python code then applies an overlay to blur the detected faces, helping to preserve privacy. The anonymized images are replicated across multiple AWS accounts serving different purposes: training computer vision models, running inference to detect safety hazards, and powering the end-user web application where Zone Users monitor and respond to detected risks. The web application displays the anonymized images, which are annotated with visual indicators for missing PPE as an example or other potential safety hazard. This clear visual feedback eliminates guesswork so that Zone Users can quickly understand what hazard was identified and where it’s located on the facility floor, facilitating rapid response and remediation. Additionally, they can adjust the floor plan organization that defines the exact regions where workers are required to wear PPE or different objects and equipment are to be placed as per 5S taping and rules.

Training pipeline and model promotion

Machine learning systems need high-quality ground truth datasets. These labeled examples teach models to identify and classify safety hazards. Data labeling is the process of human annotators reviewing images and meticulously annotating objects, behaviors, and conditions of interest. For example, drawing bounding boxes around obstructions in walkways, identifying workers without proper PPE, or marking floor tape boundaries. Poor quality training data produces unreliable models that miss safety hazards or generate false alarms, eroding user trust. Conversely, investing in high-quality, accurately labeled training data—with clear, consistent annotations reviewed by domain experts—enables the model to detect genuine risks with precision and reliability. This ultimately determines the success or failure of the entire safety monitoring system. As described later in this post, synthetic data generation can complement or potentially substitute manual annotation for specific use cases where real-world examples are scarce or labor-intensive to collect.

After anonymized images are collected and stored into Amazon S3, the GT Job creation AWS Step Functions workflow creates Amazon SageMaker Ground Truth labeling jobs for the required use cases and monitored sites and cameras. This step function is triggered at a regular, configurable, cadence by an Amazon EventBridge rule. It integrates with Zone User feedback and saved ML model predictions, so data scientists can prioritize different underperforming classes and cameras. A team of dedicated annotators then complete the jobs. Completed jobs undergo post-processing using AWS Lambda to transform them into a format suitable for training. The post-processing workflow stores job metadata like included cameras and classes, into Amazon DynamoDB, while annotations are stored in an S3 Bucket. After training data is ready, data scientists trigger Amazon SageMaker AI Pipelines model building workflows using scripts that allow for flexible hyper parameter and GT data selection. The SageMaker AI Pipeline consists of seven steps:

A checkpoint loading step
A data preparation and split step
A model training step
A generate drift baseline step
A model evaluation step
A model packaging step
A model register step

Sagemaker Pipeline

Data scientists review trained model evaluation metrics, and approve models that they want to use in the inference pipeline. Model approval fires off an EventBridge event that triggers the model promotion Lambda. The model promotion Lambda creates a code review against the application infrastructure code repository to update the Amazon S3 URI of the model used for the SageMaker AI endpoint. This workflow decouples the science and application updates. Scientists approve models when evaluation metrics meet acceptance criteria. Software engineers can then merge and manage the endpoint updates like other software and infrastructure changes through continuous integration and delivery (CI/CD) pipelines. After code review is passed, the system updates the SageMaker AI Endpoint accordingly and inference pipeline will use the updated endpoint. Approved models checkpoints are also used as the base for future retraining runs, enabling rapid incremental improvements without frequent long running training jobs.

Inference pipeline

Each use case operates through its own inference pipeline, working together to provide comprehensive safety oversight. The system functions as a digital safety supervisor, continuously monitoring facility operations and distinguishing between normal workflow and potential hazards. When an image lands in the anonymized S3 bucket, it triggers an Amazon Simple Notification Service (Amazon SNS) notification that routes to a dedicated Amazon Simple Queue Service (Amazon SQS) queue. Each use case processes independently through its own queue, which invokes a SageMaker AI Endpoint hosting a computer vision model tailored to that scenario. For example, detecting operational equipment, identifying workers with safety gear, or monitoring other safety-critical conditions. The “Intelligent Alarm Detection” section details how the system validates and escalates findings. Confirmed violations generate alerts containing the object type, precise location, and violation duration. Visual evidence includes both the original camera capture and an annotated version with color-coded overlays: blue outlines mark restricted zones, red outlines highlight violating objects, and confidence scores label each detection. Then, the system distributes alerts through parallel channels. DynamoDB stores structured violation records enabling fast queries. Amazon S3 events trigger downstream processing and notifications. Failures are tracked in use case specific Dead Letter Queues (DLQs) where they can later be analyzed or re-drove.

Risk management

When the inference pipeline detects potential hazards and saves them to the dedicated S3 bucket, it triggers a Lambda function using Amazon SNS and SQS. This function intelligently aggregates risks per camera per use case to avoid alert fatigue. Instead of bombarding safety teams with duplicate notifications, the system appends new occurrences to existing open risks. Every minute, an Amazon EventBridge schedule kicks off a Lambda function that checks whether risks still appear in the latest camera images. If a violation has been resolved, the system automatically closes it out. At the same time, another scheduled function monitors whether the SLA for risk resolution has been exceeded and sends notifications through zone’s configured preferred channels, Slack, email, or an internal ticket management system. The notification system includes escalation levels, so that the right people are alerted based on severity and how long an issue has been open. Every hour (though this schedule is flexible), the system exports risk data from the database reader endpoint and shares it with the BI team for deeper analysis and trend spotting.

Web application

Users review active and resolved risks through a React web application distributed via Amazon CloudFront and backed by an AWS AppSync API. The application follows AWS security best practices with Amazon Route 53 for DNS resolution and AWS WAF for protection against common web vulnerabilities. AWS AppSync uses AWS Lambda resolvers for embedding Amazon Quick Sight analytics and processing CRUD operations.

Site administrators can use a Site Management feature to configure use cases and notification parameters. Camera zones organize related risks into alerts for quick review. These can be acknowledged or marked as false positives, and the system automatically resolves alerts when risks are no longer detected. Users can also search historical risks, with both views displaying day-over-day trends, average resolution time, and false positive rates.

Tape labeling preparation

The Housekeeping use case fundamentally relies on understanding the spatial relationship between detected objects and the 5S floor tapes that define where equipment and materials should be positioned. Housekeeping, in this context, refers to maintaining a clean, organized, and safe workspace by keeping equipment, materials, and tools stored in their designated locations. A key advantage of this system is that it does not require pre-existing digital maps or floor plans. Instead, Zone Owners can define safety zones and organizational areas by referencing the physical tapes visible in camera images. Different color tapes represent places where certain objects belong and where restrictions exist. However, a practical challenge arises during normal facility operations: floor tapes frequently become obscured by equipment, materials, and personnel movement throughout the day. This occlusion makes it difficult for human annotators to accurately identify and label the tape boundaries when onboarding new cameras. To address this challenge, an intelligent tape labeling preparation workflow was developed that generates synthetic composite images showing clear, unobstructed views of the floor tapes. The system analyzes multiple camera frames captured at different times throughout the day, along with their corresponding object detection predictions. Using a voting mechanism, it identifies pixel regions with no detected objects and stitches these clear portions together into a composite image where the tapes are fully visible. This automated workflow runs hourly using AWS Step Functions. The first Lambda function identifies newly onboarded cameras requiring tape labeling, while the second generates composite images, saves them to Amazon S3, and updates camera status to indicate readiness for annotation. The Camera Onboarding process generates JSON files containing the coordinates of each 5S tape floor annotation for every camera. These annotations are accessible within the application, allowing users to inspect and modify them as needed when floor organization changes. During inference, the system overlays detected object positions onto floor tape boundaries and evaluates compliance against configured business and organizational rules. When violations are detected, Housekeeping alarms provide comprehensive context, including the outlined object in violation, the specific 5S taped area involved, and the rule that was broken. This detailed feedback enables operators to quickly localize issues and mark false positive alarms. The following image was generated by Amazon Nova to illustrate the tape labeling UI:

Tape Labeling UI

Data analytics component

Finally, this risk detection data gets turned into real business insights through Amazon Redshift Spectrum and Quick Sight. Redshift Spectrum lets the BI team query risk data sitting in S3 without the hassle of moving or loading it elsewhere, making historical analysis fast and straightforward.

Quick Sight dashboards give safety managers and operations leaders the full picture: which facility zones are hotspots for violations, how risks shift between day and night shifts, what types of objects cause the most problems, and whether your safety interventions are actually moving the needle. You can even compare performance across facilities to spot best practices worth replicating.

Intelligent Alarm Detection

The four-stage process

When an image arrives in the anonymized S3 bucket, the system processes it through four stages:

Stage 1: Object detection

The SageMaker Endpoint runs a computer vision model that detects operational equipment, materials, and worker safety gear. In the Housekeeping use case, the model identifies various equipment types such as transport devices, storage containers, and safety apparatus, along with materials commonly found in industrial environments. A critical capability is the model’s ability to distinguish between an object’s visible outline and its actual floor footprint.

Stage 2: Zone-based analysis with “Digital Tape”

Detection alone isn’t enough. The system must understand whether detected objects pose actual risks. Predefined zones, called “tapes,” mark restricted areas, walkways, and safety boundaries through labeling jobs. The system calculates the percentage overlap between each detected object’s footprint and these restricted zones. Configurable thresholds, typically 50% overlap, determine whether an object violates safety protocols, filtering out edge cases where objects barely touch boundary lines. For the PPE detection module, the system employs a YOLO-based computer vision model that performs simultaneous detection across multiple dimensions. It locates workers within the frame, classifies the presence or absence of required safety equipment, and applies contextual analysis to determine which PPE items are mandatory in specific areas. This contextual awareness allows the system to adapt its requirements to different zones within the operational environment.

Stage 3: The “Loiter Time” algorithm

To avoid false alarms from transient objects, the system tracks violations over time. It analyzes objects across consecutive time intervals, typically minute-by-minute, using mask similarity algorithms to confirm the same object persists rather than being replaced by similar items. This builds a “replication count” showing how many consecutive minutes an object has remained in violation. Different object types and risk zones have distinct acceptable loiter times—high-risk areas enforce shorter thresholds, while general workspace areas allow longer durations to accommodate normal operations.

Stage 4: Multilayered validation and alarm generation

Before generating an alert, the system applies final validation layers. Confidence thresholds filter out low-certainty detections based on object type complexity. Run-Length Encoding (RLE) mask comparison verifies that the tracked object is consistent across time intervals rather than different objects appearing in similar positions. Zone context determines the severity and routing of each alert. Once validated, an alert is generated with rich metadata:

{
    "violations_details": {
        "object_type": "equipment_footprint",
        "zone_identifier": "PEDESTRIAN_ZONE:AREA_A:001",
        "detection_count": 5,
        "object_dwell_time": 3,
        "confidence_score": 0.85,
        "annotated_image_uri": "s3://bucket/annotations/violation_image.jpg"
    }
}

Infrastructure scaling challenges

We designed the system to support thousands of cameras. This scale required careful architectural decisions.

Architectural foundation for scale

At the heart of our system is a serverless driver-worker pattern that proved essential for achieving the scale we needed. This pattern decouples image processing tasks, enabling independent scaling of different components while providing fault isolation. If one worker fails, it doesn’t impact the entire pipeline. The driver orchestrates work distribution while workers process images concurrently, allowing us to horizontally scale to handle simultaneous processing from hundreds of sites. This initial worker pulls the raw image and triggers a cascade of specialized downstream handlers that each contribute to the overall safety monitoring workflow.

The ML inference workflow acts as an intelligent gatekeeper in this architecture. Rather than flooding downstream components with every captured image, the inference layer only surfaces images where safety issues have been detected. This filtering is essential because it prevents components interacting with Amazon Aurora PostgreSQL from being overwhelmed by the raw volume of image data from hundreds of sites. For managing processing state, the ML inference components use DynamoDB, which provides the scalable, serverless state management needed to track inference operations across our distributed camera network.

Evolving our inference infrastructure

One of our most significant scaling challenges emerged as we transitioned from a proof-of-concept to production scale. Initially, we deployed SageMaker Serverless inference endpoints with approximately 50 cameras. However, as we scaled to processing images from hundreds of sites, we encountered critical limitations: SageMaker Serverless inference lacked GPU support and imposed a 6GB maximum memory configuration, leading to out-of-memory errors. The solution required pivoting to SageMaker Serverful inference endpoints configured with ml.g6 family instances and implementing auto scaling policies. Achieving scale also meant working with AWS service teams to increase limits for thousands of concurrent Lambda executions, optimizing memory allocation and multithreading, and tuning SQS batch sizes maximize throughput within memory constraints.

Optimizing Lambda and SQS for massive concurrency

Achieving the required scale also meant working closely with AWS service teams to increase limits supporting thousands of concurrent Lambda executions across our accounts. Beyond increasing limits, we invested significant effort in optimizing our Lambda configurations from memory allocation to processing logic that uses multithreading capabilities. The integration between Lambda and SQS required particular attention. We optimized Lambda functions for maximum consumption concurrency, refactored error handling to minimize failed containers, and tuned the maximum number of messages per batch to handle larger message volumes efficiently within memory constraints.

Data-driven ground truth curation at scale

While synthetic data generation significantly reduces the annotation burden, manual annotation remains important for addressing unique site conditions and onboarding new use cases. As the solution scales to new facilities, each site introduces distinct camera angles, lighting conditions, and equipment layouts that benefit from targeted real-world annotations to fine-tune model performance. Our approach to curating ground truth data for model training evolved significantly as we scaled. Initially, we implemented a straightforward but labor-intensive strategy: creating annotation jobs for every site on a daily basis. This approach worked well during early stages with limited sites, but as we expanded to hundreds of geographically distributed sites, the volume of manual annotation became untenable. The sheer number of daily labeling jobs – one per site per day – quickly overwhelmed our annotation capacity and created a significant operational bottleneck that threatened our ability to continuously improve model performance.

We fundamentally reimagined our workflow by using Amazon Athena to query and analyze massive volumes of inference results combined with customer feedback data at scale. We identified underperforming segments by aggregating false positive rates across camera types and deployment conditions, prioritizing retraining on image sources with elevated error rates. We also surfaced inferences where model confidence scores fell below established thresholds, flagging these uncertain predictions for targeted annotation and review. We further augmented this analysis with Claude multi-modal LLMs on Amazon Bedrock to analyze misclassified samples and detect underrepresented object classes in our existing training distribution. This directly informed our data collection strategy to address class imbalance and edge cases in future training jobs. This shift from blanket sampling to intelligent, performance-driven curation made our annotation workflow sustainable at scale. It also improved training efficiency by directing labeling efforts only where they would have the greatest impact on model improvement.

Images annotation at scale

Manual annotation of training data is challenging, especially for rare safety violations. The sheer volume of images combined with the labor-intensive nature of manual labeling, makes traditional annotation approaches impractical for certain use cases. Some safety violations are extremely rare in practice yet represent frequent sources of workplace injuries. A prime example is floor spill detection: despite examining and annotating over half a million images, only a few hundred examples of liquid spills or debris on walkways were identified. While this low occurrence rate is commendable from a safety perspective, it poses a fundamental challenge for model training. There aren’t enough real-world examples to train a robust detection model. Similarly, PPE detection presents a data diversity challenge. In the majority of captured images, PPE items appear in a single dominant color. However, workplace policies often permit variations, and workers occasionally wear acceptable PPE in different colors. Without sufficient training examples across color variations, the model risks failing to detect non-standard colored items, creating potential safety blind spots.

To address these challenges, we built a fully synthetic data generation and model training pipeline on AWS using GLIGEN (Grounded Language-to-Image Generation), a diffusion-based generative model deployed as Amazon SageMaker Batch Transform jobs. Using this approach, we produced a 75,000-image PPE dataset covering three classes: person, hard hat, and safety vest. The pipeline architecture is illustrated in the following image.

PPE Detection Pipeline

We also produced a 75,000-image Housekeeping dataset covering seven common facility object classes: pallet jack, go-cart, step ladder, trash can, safety cone, tote, and pallet. The pipeline architecture is illustrated in the following image.

Housekeeping Detection Flow

GLIGEN enables the creation of highly realistic, yet controlled, training datasets that address both rare event scarcity and data diversity gaps without requiring manual image collection or annotation. For floor spills, GLIGEN receives structured bounding box inputs specifying where objects should appear, generating photorealistic facility scenes with spills or debris placed in realistic facility floor contexts. For PPE color diversity, GLIGEN generates images of workers wearing safety equipment in varied colors, creating the diversity needed for robust detection across acceptable variations. For each image, GLIGEN received bounding box coordinates specifying object positions and generated photorealistic 512×512 facility scenes with ground truth annotations automatically embedded in the output, avoiding manual labeling entirely. Raw outputs were streamed from Amazon S3, decoded, and converted to YOLO annotation format using parallel Python workers, then uploaded back to S3 as training-ready datasets. We trained YOLOv8 models on Amazon SageMaker AI using PyTorch 2.1, with the final configuration using cosine learning rate scheduling and AdamW optimization. This is a combination that proved critical for stabilizing the larger YOLOv8l model variant and preventing gradient divergence during training. Beyond training data generation, GLIGEN ‘s synthetic images also enhance the inference by reducing false positive risk detections and providing more accurate, contextual understanding of the operational environment.

Conclusion

Our solution demonstrates strong accuracy. For use cases trained entirely on synthetic data generated by GLIGEN, the PPE model achieved 99.5% mean average precision (mAP@50) with 100% precision and recall across all three classes. The Housekeeping model reached 94.3% mAP@50 with 91.4% precision and 86.9% recall across seven more challenging facility object classes, all without a single manually annotated real image. Accuracy can be further improved by increasing the volume of training images used to build and train the custom model. Beyond accuracy, through testing on 10,000 synthetic images, we’ve seen our solution perform strongly across two critical dimensions:

speed of up to 37 seconds, measured by the time elapsed between an image captured and notification delivery to Zone Operators.
scale across 10,000+ cameras, validated through simultaneous processing of 10,000 images, if one camera produces one frame at a time.

While this post focuses on our warehouse deployment, the architecture we’ve described is intentionally industry-agnostic. The core capabilities, object detection, zone-based spatial reasoning, temporal violation tracking, and privacy-preserving image processing, are not specific to a single environment. The same detection pipeline that identifies PPE violations and housekeeping hazards on a distribution center floor could be adapted to monitor equipment boundaries on manufacturing floors, enforce clean room protocols in laboratories, or track safety compliance on construction sites. Each industry requires a domain-specific model training, tailored business rules, and unique zone configurations, but the underlying event-driven architecture, scaling patterns, and intelligent alarm detection framework remain constant. We look forward to exploring these extensions in future work.

For more information about AWS workforce safety solutions, see Delivering an integrated approach to safety: How AWS Workforce Safety solutions make work safer.

About the authors

Introducing enhancements to Amazon EMR Managed Scaling

2026-03-26 Amit Maindola

Post Syndicated from Amit Maindola original https://aws.amazon.com/blogs/big-data/introducing-enhancements-to-amazon-emr-managed-scaling/

Amazon EMR Managed Scaling has been helping customers automatically resize their clusters to optimize performance and reduce costs. We are excited to introduce a significant enhancement to this feature: Advanced Scaling for Amazon EMR. This new capability provides additional flexibility to configure the desired resource utilization or performance levels for your cluster using a utilization-performance slider. After the slider is set, EMR Managed Scaling intelligently scales the cluster and optimizes cluster resources based on your configured performance or resource utilization levels.

Customers appreciate the simplicity of EMR Managed Scaling, where they specify the minimum and maximum compute limits for their clusters and EMR Managed Scaling automatically resizes the cluster. EMR Managed Scaling continuously samples key metrics associated with the workloads running on clusters and scales up or down accordingly. However, customers’ workloads are increasingly getting more complex, with variability across dimensions such as data volumes and cost vs. SLA requirements. Consequently, customers prefer to have additional levers to tune the scaling behavior most suitable for their workload. In this post, we discuss the benefits of Advanced Scaling for Amazon EMR and demonstrate how it works through some example scenarios.

Advanced Scaling for Amazon EMR

Previously, customers who wanted to adjust the default EMR Managed Scaling behavior had no other option but to disable EMR Managed Scaling and create custom automatic scaling rules. Custom autoscaling rules created several problems:

Custom autoscaling rules are not shuffle-aware and shuffle data is lost.
Custom autoscaling is not aware of the application driver and can terminate it, failing the entire job.
Custom autoscaling can be slower to respond to real time needs.

These are some of the reasons why custom autoscaling is not the right fit. Customers wanted out-of-the-box support for Managed Scaling to handle the scaling that optimizes for the customers end goal to optimize cost or performance. The new Advanced Scaling capability enhances the existing benefits of EMR Managed Scaling by introducing additional controls and helping you configure the desired resource utilization or performance level for your cluster using a utilization-performance slider. EMR Advanced Scaling then internally translates intent into tailored algorithm strategy (UtilizationPerformanceIndex), such as how quickly to scale, how much to scale, and so on, to make scaling decisions for the cluster. This helps optimize cluster resources while making sure we meet the performance or resource utilization intent set by the customer.

For example, for a cluster running multiple tasks of relatively short duration (order of seconds), EMR Managed Scaling previously used to scale up the cluster aggressively and conservatively scale it down to avoid negative impact to job runtimes. Although this is the right approach for SLA-sensitive workloads, it might not be optimal for customers who are fine with little delay but prefers saving cost. Now, you can configure EMR Managed Scaling behavior suitable for your workload types, and we will apply tailored optimization to intelligently add or remove nodes from the clusters. This helps you achieve the optimal price-performance for your clusters along with increased flexibility of additional user-controls.

The value you set for Advanced Scaling optimizes your cluster to your requirements. Values range from 1-100. Supported values are 1, 25, 50, 75 and 100. If you set the index to values other than these, it results in a validation error. Scaling values map to resource-utilization strategies. The following list defines several of these:

Utilization optimized (1) – This setting prevents resource over provisioning. Use a low value when you want to keep costs low and to prioritize efficient resource utilization. It causes the cluster to scale up less aggressively. This works well for the use case when there are regularly occurring workload spikes and you don’t want resources to ramp up too quickly.
Balanced (50) – This balances resource utilization and job performance. This setting is suitable for steady workloads where most stages have a stable runtime. It’s also suitable for workloads with a mix of short and long-running stages. We recommend starting with this setting if you aren’t sure which to choose.
Performance optimized (100) – This strategy prioritizes performance. The cluster scales up aggressively to ensure that jobs complete quickly and meet performance targets. Performance optimized is suitable for service-level-agreement (SLA) sensitive workloads where fast run time is critical.

Customers can also choose intermediate values (25 and 75) for more nuanced control. The intermediate values available provide a middle ground between strategies to fine tune your cluster’s Advanced Scaling behavior.

Use cases and benefits

Amazon EMR’s Advanced Scaling feature improves cluster management by offering dynamic adaptation to diverse business requirements across industries. The feature enables strategic timing of scaling policies throughout the day, with early morning hours dedicated to workload preparation, peak business hours focusing on maximum performance, evening periods maintaining moderate scaling for post-business processing, and overnight hours optimized for cost-effective batch operations. This comprehensive approach allows organizations to fine-tune their resource allocation based on specific operational patterns, ultimately delivering an optimal balance between performance and cost-efficiency while ensuring business needs are met across different time zones and usage patterns.

Scaling configuration

In the following sections, we walk through a range of scenarios testing against a 3 TB TPC-DS dataset, then walk you through the results of testing a sample job. We wanted to evaluate how Amazon EMR would respond with advanced scaling policies in scenarios optimizing cluster utilization, balancing performance with utilization, and aggressive performance requirements.

With Advanced Scaling currently available through API and console support coming soon, we updated existing cluster configurations. We modified UtilizationPerformanceIndex with 1, 50, and 100, to correspond to the different scaling strategies using the put-managed-scaling-policy API with an advanced scaling strategy, as seen in the following examples:

Scenario 1: Utilization optimized

In this scenario, we used a utilization optimized configuration by setting UtilizationPerformanceIndex to 1:

aws emr put-managed-scaling-policy --cluster-id <'cluster-id'> \ 
  --managed-scaling-policy '{ 
  "ComputeLimits": { 
    "UnitType": "Instances", 
    "MinimumCapacityUnits": 2, 
    "MaximumCapacityUnits": 50, 
    "MaximumOnDemandCapacityUnits": 50, 
    "MaximumCoreCapacityUnits": 2 
  	}, 
  }'

The result of the test yielded a peak of 16 nodes running and 16 requested. The scale-up and scale-down process is conservative. It takes 15 minutes to completely release the nodes after the requested metric subsides, as shown in the following figure. The job completed in 12 minutes, 39 seconds. UtilizationPerformanceIndex of 1 or 25 can be useful when the cluster is running a sequence of jobs with little to zero idle time. It can prevent frequent node churn because nodes will be available for the next set of jobs.

Scenario 2: Balanced

In this scenario, we used a balanced configuration by setting UtilizationPerformanceIndex to 50:

aws emr put-managed-scaling-policy --cluster-id <'cluster-id'> \ 
  --managed-scaling-policy '{ 
  "ComputeLimits": { 
    "UnitType": "Instances", 
    "MinimumCapacityUnits": 2, 
    "MaximumCapacityUnits": 50, 
    "MaximumOnDemandCapacityUnits": 50, 
    "MaximumCoreCapacityUnits": 2 
  	}, 
  }'

The result of the test yielded a peak of 43 nodes running and 32 requested. UtilizationPerformanceIndex of 50 uses a balanced approach for scaling the resources. Nodes requested and running are higher such that you can get a better price-performance ratio. The job completed in 7 minutes, 1 second.

Scenario 3: Performance optimized

In this scenario, we used a performance optimized configuration by setting UtilizationPerformanceIndex to 100:

aws emr put-managed-scaling-policy --cluster-id <'cluster-id'> \ 
  --managed-scaling-policy '{ 
  "ComputeLimits": { 
    "UnitType": "Instances", 
    "MinimumCapacityUnits": 2, 
    "MaximumCapacityUnits": 50, 
    "MaximumOnDemandCapacityUnits": 50, 
    "MaximumCoreCapacityUnits": 2 
	  }, 
  }'

The result of the test yielded a peak of 50 nodes running and 46 requested. UtilizationPerformanceIndex of 100 delivers the highest performance by aggressively scaling resources up and down. You can expect the highest nodes requested and running in this configuration. Scale-down will closely follow the node requested metric and therefore can lead to frequent churn of nodes if there are short idle periods between job submissions. This setting is ideal for latency-sensitive workloads that need to finish under SLA. The example job completed in 6 minutes, 16 seconds.

Comparison

The following table summarizes the differences between these scaling methods and time taken for each.

Scaling Method	Utilization Index	Peak Total Nodes Requested	Peak Total Nodes Running	Job Run Time (Seconds)	Cost to Run job	Use Case
Scenario1 – Utilization optimized	1	16	16	759	Low	Workloads with regular spikes; prioritizes cost efficiency with conservative scaling
Scenario 2 – Balanced	50	32	43	421	Medium	Steady workloads with mixed stage durations; recommended starting point
Scenario 3 – Performance Optimized	100	46	50	376	High	SLA-sensitive workloads requiring fast completion times

Advanced Managed Scaling in Amazon EMR introduces a more nuanced approach to cluster management through the customized scaling strategies to meet your business requirements. This spectrum offers fine-grained control over how clusters respond to workload demands. At one end, with a utilization optimized configuration of 1, the system prioritizes efficient resource usage, scaling up conservatively to maintain cost-effectiveness and taking advantage of existing cluster resources. In the balanced configuration at 50, the strategy aims to strike an equilibrium between resource utilization and job performance. To meet performance SLAs, the performance optimized value of 100 showed aggressive scaling responding to increased demand for resources quickly, regardless of resource consumption. This granular control helps you fine-tune your cluster’s behavior based on your specific needs, balancing cost, efficiency, and performance.

Conclusion

To Summarize, Advanced Scaling for Amazon EMR represents an advancement in cluster management, offering greater control and efficiency. By fine-tuning your clusters’ behavior, you can achieve more cost-effective and performant big data processing. We encourage you to try this new feature and discover how it can optimize your EMR workloads. Start by experimenting with different UtilizationPerformanceIndex values and closely monitor your cluster’s performance and cost metrics. Over time, you will be able to find the perfect balance that meets your specific needs.

To learn more about Amazon EMR Managed Scaling and Advanced Scaling, refer to our documentation. We’re excited to see how you use this new capability to enhance your big data processing on AWS, and we look forward to your feedback as we continue to evolve and improve our services.

About the authors

Build AWS Glue Data Quality pipeline using Terraform

2026-03-26 Viquar Khan

Post Syndicated from Viquar Khan original https://aws.amazon.com/blogs/big-data/build-aws-glue-data-quality-pipeline-using-terraform/

AWS Glue Data Quality is a feature of AWS Glue that helps maintain trust in your data and support better decision-making and analytics across your organization. It allows users to define, monitor, and enforce data quality rules across their data lakes and data pipelines. With AWS Glue Data Quality, you can automatically detect anomalies, validate data against predefined rules, and generate quality scores for your datasets. This feature provides flexibility in how you validate your data – you can incorporate quality checks into your ETL processes for transformation-time validation, or validate data directly against cataloged tables for ongoing data lake monitoring. By leveraging machine learning, it can also suggest data quality rules based on your data patterns.

You can use Terraform, an open source Infrastructure as Code (IaC) tool developed by HashiCorp, to deploy AWS Glue Data Quality pipelines.

It allows developers and operations teams to define, provision, and manage cloud infrastructure using a declarative language. With Terraform, you can version, share, and reuse your infrastructure code across multiple cloud providers and services. Its powerful state management and planning capabilities enable teams to collaborate efficiently and maintain consistent infrastructure across different environments.

Using Terraform to deploy AWS Glue Data Quality pipeline enables IaC best practices to ensure consistent, version controlled and repeatable deployments across multiple environments, while fostering collaboration and reducing errors due to manual configuration.

In this post, we explore two complementary methods for implementing AWS Glue Data Quality using Terraform:

ETL-based Data Quality – Validates data during ETL (Extract, Transform, Load) job execution, generating detailed quality metrics and row-level validation outputs
Catalog-based Data Quality – Validates data directly against Glue Data Catalog tables without requiring ETL execution, ideal for monitoring data at rest

Solution overview

This post demonstrates how to implement AWS Glue Data Quality pipelines using Terraform using two complementary approaches mentioned above to ensure comprehensive data quality across your data lake.

We’ll use the NYC yellow taxi trip data, a real-world public dataset, to illustrate data quality validation and monitoring capabilities. The pipeline ingests parquet-formatted taxi trip data from Amazon Simple Storage Service (Amazon S3) and applies comprehensive data quality rules that validate data completeness, accuracy, and consistency across various trip attributes.

Method 1: ETL-based Data Quality

ETL-based Data Quality validates data during Extract, Transform, Load (ETL) job execution. This approach is ideal for:

Validating data as it moves through transformation pipelines
Applying quality checks during data processing workflows
Generating row-level validation outputs alongside transformed data

The pipeline generates two key outputs:

Data Quality Results – Detailed quality metrics and rule evaluation outcomes stored in the dqresults/ folder, providing insights into data quality trends and anomalies
Row-Level Validation – Individual records with their corresponding quality check results written to the processed/ folder, enabling granular analysis of data quality issues

Method 2: Catalog-based Data Quality

Catalog-based Data Quality validates data quality rules directly against AWS Glue Data Catalog tables without requiring ETL job execution. This approach is ideal for:

Validating data at rest in the data lake
Running scheduled data quality checks independent of ETL pipelines
Monitoring data quality across multiple tables in a database

Architecture overview

The following diagram illustrates how both approaches work together to provide comprehensive data quality validation:

Source data stored in Amazon S3 (Yellow Taxi Data)
AWS Glue ETL processes data with quality checks
ETL validation results are stored in S3
AWS Glue Crawler discovers schema
Metadata is stored in AWS Glue Catalog
AWS Glue Data Quality validates catalog tables
Catalog validation results are stored in S3
Amazon CloudWatch monitors all operations

By using AWS Glue’s serverless ETL capabilities and Terraform’s infrastructure-as-code approach, this solution provides a scalable, maintainable, and automated framework for ensuring data quality in your analytics pipeline.

Prerequisites:

An AWS account with AWS CLI installed and configured
HashiCorp Terraform installed and configured

Solution Implementation

Complete the following steps to build AWS Glue Data Quality pipeline using Terraform:

Clone the Repository

This post includes a GitHub repository that generates the following resources when deployed. To clone the repository, run the following command in your terminal:

git clone https://github.com/aws-samples/sample-build-aws-glue-data-quality-pipeline-using-terraform.git
cd sample-build-aws-glue-data-quality-pipeline-using-terraform

Core Infrastructure:

Amazon S3 bucket: glue-data-quality-{AWS AccountID}-{env} with AES256 encryption
Sample NYC taxi dataset (sample-data.parquet) automatically uploaded to the data/ folder
AWS Identity and Access Management (IAM) role: aws-glue-data-quality-role-{env} with Glue execution permissions and S3 read/write access
CloudWatch dashboard: glue-data-quality-{env} for monitoring job execution and data quality metrics
CloudWatch Log Groups for job logging with configurable retention

ETL-Based Data Quality Resources:

AWS Glue ETL job: data-quality-pipeline with 8 comprehensive validation rules
Python script: GlueDataQualityDynamicRules.py stored in glue-scripts/ folder
Results storage in dqresults/ folder with detailed rule outcomes
Row-level validation outputs in processed/ folder
Optional scheduled triggers for automated execution
CloudWatch alarm: etl-glue-data-quality-failure-{env} for monitoring job failures

Catalog-Based Data Quality Resources (Optional – when catalog_dq_enabled = true):

Glue Database: {catalog_database_name} for catalog table management
Glue Crawler: {job_name}-catalog-crawler for automatic schema discovery from S3 data
Crawler schedule trigger for automated execution (default: daily at 4 AM)
Glue Catalog Tables automatically discovered and created by the crawler
Catalog Data Quality job: {job_name}-catalog with 7 catalog-specific validation rules
Python script: CatalogDataQuality.py for catalog validation
Results storage in catalog-dq-results/ folder partitioned by table name
Catalog DQ schedule trigger for automated validation (default: daily at 6 AM)
CloudWatch alarm: catalog-glue-data-quality-failure-{env} for monitoring catalog job failures
Enhanced CloudWatch dashboard widgets for crawler status and catalog metrics

Review the Glue Data Quality Job Script

Review the Glue Data Quality job script GlueDataQualityDynamicRules.py located in the folder scripts, which has the following rules:

Rules = [
    CustomSql "select vendorid from primary where passenger_count > 0" with threshold > 0.9,
    Mean "trip_distance" < 150,
    Sum "total_amount" between 1000 and 100000,
    RowCount between 1000 and 1000000,
    Completeness "fare_amount" > 0.9,
    DistinctValuesCount "ratecodeid" between 3 and 10,
    DistinctValuesCount "pulocationid" > 100,
    ColumnCount = 19
]

Brief explanation of rules for NY Taxi data is as follows:

Rule Type	Condition	Description
CustomSql	“select vendorid from primary where passenger_count > 0” with threshold > 0.9	Checks if at least 90% of rides have at least one passenger
Mean	“trip_distance” < 150	Ensures the average trip distance is less than 150 miles
Sum	“total_amount” between 1000 and 100000	Verifies that total revenue from all trips falls within this range
RowCount	between 1000 and 1000000	Checks if the dataset has between 1,000 and 1 million records
Completeness	“fare_amount” > 0.9	Ensures over 90% of records have a fare amount
DistinctValuesCount	“ratecodeid” between 3 and 10	Verifies rate codes fall between 3-10 unique values
DistinctValuesCount	“pulocationid” > 100	Checks if there are over 100 unique pickup locations
ColumnCount	19	Validates that dataset has exactly 19 columns

These rules together ensure data quality by validating volume, completeness, reasonable values and proper structure of the taxi trip data.

Configure Terraform Variables

Before deploying the infrastructure, configure your Terraform variables in the terraform.tfvars file located in the examples directory. This configuration determines which features will be deployed – ETL-based Data Quality only, or both ETL-based and Catalog-based Data Quality.

Basic Configuration

The solution uses default values for most settings, but you can customize the following in your terraform.tfvars file:

AWS Region – The AWS region where resources will be deployed
Environment – Environment identifier (such as, “dev”, “prod”) used in resource naming
Job Name – Name for the Glue job (default: data-quality-pipeline)

Enable Catalog-Based Data Quality

By default, the solution deploys only ETL-based Data Quality. To enable Catalog-based Data Quality validation, add the following configuration to your terraform.tfvars file:

# Enable Catalog-based Data Quality
catalog_dq_enabled = true

# Glue Database name for catalog tables
catalog_database_name = "taxi_data_catalog"

# S3 paths containing parquet data for catalog table creation
s3_data_paths = ["data/"]

# Optional: Specific table names to validate (empty = all tables in database)
catalog_table_names = []

# Data quality rules for catalog validation
catalog_dq_rules = <<EOF
Rules = [
  RowCount > 0,
  Completeness "vendorid" > 0.9,
  Completeness "passenger_count" > 0.95,
  Mean "trip_distance" < 150,
  ColumnCount > 5
]
EOF

# Enable scheduled execution for catalog data quality
catalog_enable_schedule = false
catalog_schedule_expression = "cron(0 6 * * ? *)"

# Crawler schedule for automatic table discovery
catalog_crawler_schedule = "cron(0 4 * * ? *)"

Configuration Notes:

catalog_dq_enabled – Set to true to enable Catalog-based validation alongside ETL-based validation,which will deploy both ETL and Catalog validation
catalog_database_name – Name of the Glue database that will be created for catalog tables
s3_data_paths – S3 folders containing parquet data that the Glue Crawler will discover
catalog_table_names – Leave empty to validate all tables, or specify specific table names
catalog_dq_rules – Define validation rules specific to catalog tables (can differ from ETL rules)
catalog_enable_schedule – Set to true to enable automatic scheduled execution
Schedule expressions – Use cron format for automated execution (crawler runs before DQ job)

Once you’ve configured your variables, save the terraform.tfvars file and proceed to the next step.

Set Up AWS CLI Authentication

Before you can interact with AWS services using the command line, you need to set up and authenticate the AWS CLI. This section guides you through the process of configuring your AWS CLI and verifying your authentication. Follow these steps to ensure you have the necessary permissions to access AWS resources.

Open your terminal or command prompt.
Set up authentication in the AWS CLI. You need administrator permissions to set up this environment.
```
aws configure
```

To test if your AWS CLI is working and you’re authenticated, run the following command:

aws sts get-caller-identity --output json

The output should look similar to the following:

{
   "UserId": "UUID123123:your_user",
  "Account": "111122223333",
"Arn": "arn:aws:sts::111122223333:assumed-role/some-role/your_user"
 }

Deploy with Terraform

Follow these steps to deploy your infrastructure using Terraform. This process will initialize your working directory, review planned changes, and apply your infrastructure configuration to AWS.

To deploy with Terraform, navigate to the examples folder by running the following command in your CLI from inside the repository

cd .\examples

Run the following bash commands:

terraform init

Initializes a Terraform working directory, downloads required provider plugins, and sets up the backend for storing state.

On success you will receive output Terraform has been successfully initialized!

terraform plan

Creates an execution plan, shows what changes Terraform will make to your infrastructure. This command doesn’t make any changes.

terraform apply

Deploys infrastructure and code to the AWS Account. By default, it asks for confirmation before making any changes. Use ‘terraform apply -auto-approve’ to skip the confirmation step.

When prompted with ‘Do you want to perform these actions?’, type ‘yes’ and press Enter to confirm and allow Terraform to execute the described actions.

Upon successful execution, the system will display ‘Apply complete!’ message.

Run the AWS Glue Data Quality Pipeline

After deploying the infrastructure with Terraform, you can validate data quality using two methods – ETL-based and Catalog-based. Each method serves different use cases and can be run independently or together.

Method 1: Run the ETL-Based Data Quality Job

ETL-based data quality validates data during the transformation process, making it ideal for catching issues early in your data pipeline.

Steps to execute:

Navigate to the AWS Glue Console and select ETL Jobs from the left navigation panel
Locate and select the job named data-quality-pipeline
Choose Run to start the job execution
Monitor the job status – it typically completes in 2-3 minutes

Review the results:

Once completed, click on the Data Quality tab to review the validation results.
The following screenshot shows the results.

Understanding AWS Glue Data Quality Results: NYC Taxi Data Example.

Rule Results Summary

We had 8 total rules
7 rules passed
1 rule failed

Rule	Rule Condition	Status	Pass/Fail Reason
Passenger Count Check	At least 90% of rides should have at least one passenger	Passed	95% of rides had passengers, exceeding 90% threshold
Trip Distance	Average trip < 150 miles	Passed	Average was 5.94 miles, well below 150-mile limit
Row Count	Between 1,000 and 1,000,000 records	Passed	63,441 records fell within required range
Fare Amount Completeness	90% of records should have fare amounts	Passed	100% completeness exceeded 90% requirement
Rate Code Variety	Between 3-10 different rate codes	Passed	7 unique codes fell within acceptable range
Pickup Locations	More than 100 different pickup locations	Passed	205 locations exceeded minimum requirement
Column Count	Exactly 19 columns	Passed	Exact match at 19 columns
Total Amount Range	Sum of all fares between $1,000 and $100,000	Failed	Total of $130,638.29 exceeded maximum limit

Check the S3 bucket for detailed outputs:
- Data Quality metrics: s3://glue-data-quality-{AccountID}-{env}/dqresults/
- Row-level validation: s3://glue-data-quality-{AccountID}-{env}/processed/

The job processes the NYC taxi data and applies all 8 validation rules during the ETL execution. You’ll see a quality score along with detailed metrics for each rule.

Method 2: Run the Catalog-Based Data Quality Pipeline

Catalog-based data quality validates data at rest in your data lake, independent of ETL processing. This method requires the Glue Crawler to first discover and catalog your data.

Run the Glue Crawler (first-time setup or when schema changes):
- Navigate to AWS Glue Console and select Crawlers
- Locate data-quality-pipeline-catalog-crawler
- Select data-quality-pipeline-catalog-crawler checkbox and click Run and wait for completion (1-2 minutes)
- Verify the table was created in your Glue database
Run the Catalog Data Quality Job:
- Navigate to the AWS Glue Console and select ETL Jobs from the left navigation panel
- Select the job named data-quality-pipeline-catalog
- Click Run job to execute the validation
- Monitor the job status until completion

Review the results:

Once completed, click on the Data Quality tab to review the validation results.

The following screenshot shows the results.

Rule Results Summary

We had 7 total rules
6 rules passed
1 rule failed

Rule	Rule Condition	Status	Pass/Fail Reason
Row Count	Row count should be greater than zero	Passed	63441 rows present in the source data file
Completeness “vendorid”	90% of records should have vendorid	Passed	100% completeness exceeded 90% requirement
Completeness “passenger_count”	95% of records should have vendorid	Passed	96% completeness exceeded 95% requirement
Mean “trip_distance”	Mean “trip_distance” < 150	Passed	trip_distance.Mean: 5.94 which is less than threshold 150
Sum “total_amount”	Sum “total_amount” between 1000 and 100000	Failed	total_amount.Sum: 1330638.29 which does not satisfy condition

Distinct vale count “ratecodeid”	DistinctValuesCount “ratecodeid” between 3 and 10	Passed	ratecodeid.DistinctValuesCount: 7, satisfies condition
Column Count	Greater than 5 columns	Passed	ColumnCount: 19, satisfies condition

Check the S3 bucket for detailed outputs s3://glue-data-quality-{AccountID}-{env}/catalog-dq-results/

Catalog vs ETL Data Quality Comparison

Feature	ETL Data Quality	Catalog Data Quality
Execution Context	Validates data during ETL job processing	Validates data against catalog tables at rest
Data Source	Reads directly from S3 files (parquet format)	Queries Glue Data Catalog tables
Results Location	s3://…/dqresults/	s3://…/catalog-dq-results/
Primary Use Case	Validate data quality during transformation pipelines	Monitor data lake quality independent of ETL workflows
Execution Trigger	Runs as part of Glue ETL job execution	Runs independently as scheduled Data Quality job
Scheduling	Configured via Glue job schedule or on-demand	Configured via Data Quality job schedule or on-demand
Table Discovery	Manual – requires explicit S3 path configuration	Automatic – Glue Crawler discovers schema and creates tables
Schema Management	Defined in ETL job script	Managed by Glue Data Catalog
Output Format	Data Quality metrics + row-level validation outputs	Data Quality metrics only
Best For	Catching issues early in data pipelines	Ongoing monitoring of data at rest in data lakes
Dependencies	Requires ETL job execution	Requires Glue Crawler to run first
CloudWatch Integration	Job-level metrics and logs	Data Quality-specific metrics and logs

Monitoring and Troubleshooting

Both data quality methods automatically send metrics and logs to Amazon CloudWatch. You can set up alarms to notify you when quality scores drop below acceptable thresholds.

Clean up

To avoid incurring unnecessary AWS charges, make sure to delete all resources created during this tutorial. Ensure you have backed up any important data before running these commands, as this will permanently delete the resources and their associated data. To destroy all resources created as part of this blog, run following command in your terminal:

terraform destroy

Conclusion

In this blog post, we demonstrated how to build and deploy a scalable data quality pipeline using AWS Glue Data Quality and Terraform. The solution implements two validation methods:

ETL-based Data Quality – Integrated validation during ETL job execution for transformation pipeline quality assurance
Catalog-based Data Quality – Independent validation against Glue Data Catalog tables for data lake quality monitoring

By implementing data quality checks on NYC taxi trip data, we showed how organizations can automate their data validation processes and maintain data integrity at scale. The combination of AWS Glue’s serverless architecture and Terraform’s infrastructure-as-code capabilities provides a powerful framework for implementing reproducible, version-controlled data quality solutions. This approach not only helps teams catch data issues early but also enables them to maintain consistent data quality standards across different environments. Whether you’re dealing with small datasets or processing massive amounts of data, this solution can be adapted to meet your organization’s specific data quality requirements. As data quality continues to be a crucial aspect of successful data initiatives, implementing automated quality checks using AWS Glue Data Quality and Terraform sets a strong foundation for reliable data analytics and decision-making.

To learn more about AWS Glue Data Quality, refer to the following:

About the authors

Automating data classification in Amazon SageMaker Catalog using an AI agent

2026-03-24 Ramesh H Singh

Post Syndicated from Ramesh H Singh original https://aws.amazon.com/blogs/big-data/automating-data-classification-in-amazon-sagemaker-catalog-using-an-ai-agent/

If you’re struggling with manual data classification in your organization, the new Amazon SageMaker Catalog AI agent can automate this process for you. Most large organizations face challenges with the manual tagging of data assets, which doesn’t scale and is unreliable. In some cases, business terms aren’t applied consistently across teams. Different groups name and tag data assets based on local conventions. This creates a fragmented catalog where discovery becomes unreliable and governance teams spend more time normalizing metadata than governing.

In this post, we show you how to implement this automated classification to help reduce the manual tagging effort and improve metadata consistency across your organization.

Amazon SageMaker Catalog provides automated data classification that suggests business glossary terms during data publishing. This helps to reduce the manual tagging effort and improve metadata consistency across organizations. This capability analyzes table metadata and schema information using Amazon Bedrock language models to recommend relevant terms from organizational business glossaries. Data producers receive AI-generated suggestions for business terms defined within their glossaries. These suggestions include both functional terms and sensitive data classifications such as PII and PHI, making it straightforward to tag their datasets with standardized vocabulary. Producers can accept or modify these suggestions before publishing, facilitating consistent terminology across data assets and improving data discoverability for business users.

The problem with manual classification

Manual tagging doesn’t scale effectively. Data producers interpret business terms differently, especially across domains. Critical labels like PII and PHI get missed because the publishing workflow is already complex. After assets enter the catalog with inconsistent terminology, search functionality and access controls quickly degrade.The solution isn’t only better training—it’s making the classification process predictable and consistent.

How automated classification works

The capability runs directly inside the publish workflow:

The catalog looks at the table’s structure—column names, types, whatever metadata exists.
That structure is sent to an Amazon Bedrock model that matches patterns against the organization’s glossary.
Producers receive a set of suggestions from the defined business glossary terms for classification that might include both functional and sensitive-data glossary terms.
They accept or adjust the suggestions before publishing.
The final list is written into the asset’s metadata using the controlled vocabulary.

The model evaluates column names, data types, schema patterns, and existing metadata. It maps those signals to the terms defined in the organization’s glossary. The suggestions are generated inline during publishing, with no separate Extract, Transform and Load (ETL) or batch processes to maintain. The accepted terms become part of the asset’s metadata and flow into downstream catalog operations immediately.

Under the hood: intelligent agent-based classification

Automated business glossary assignment goes beyond simple metadata lookups using a reasoning-driven approach. The AI agent functions like a virtual data steward, following human-like reasoning patterns such as:

Reviews asset details and context
Searches the catalog for relevant terms
Evaluates whether results make sense
Refines strategy if initial searches don’t surface appropriate terms
Learns from each step to improve recommendations

Key approaches:

Reasoning over static queries – The agent interprets asset attributes and context rather than treating metadata as a fixed index, generating dynamic search intents instead of relying on predefined queries.
Iterative adaptive search – When initial results are weak, the agent automatically adjusts queries—broadening, narrowing, or shifting terms through a feedback loop that helps improve discovery quality.
Structured semantic search – The agent performs semantic querying across entity types, applies filtering and relevance scoring, and conducts multi-directional exploration until strong matches are found.

This allows the agent to explore multiple directions until strong matches are found, improving recall and precision over static methods like direct vector search when asset metadata is incomplete or ambiguous.

Things to keep in mind

This feature is only as strong as the glossary it sits on top of. If the glossary is incomplete or inconsistent, the suggestions reflect that. Producers should still review each recommendation, especially for regulatory labels. Governance teams should monitor how often suggestions are accepted or overridden to understand model accuracy and glossary gaps.

Prerequisites

To follow along, you must have an Amazon SageMaker Unified Studio domain set up with a domain owner or domain unit owner permissions. You must have a project that you can use to publish assets. For instructions on setting up a new domain, refer to the SageMaker Unified Studio Getting started guide. We will also use Amazon Redshift to catalog data. If you are not familiar, read Learn Amazon Redshift concepts to learn more.

Step 1: Define business glossary and terms

AI recommendations suggest terms only from glossaries and definitions already present in the system. As a first step we create high-quality, well-described glossary entries so the AI can return accurate and meaningful suggestions.

We create the following business glossaries in our domain. For information about how to create a business glossary, see Create a business glossary in Amazon SageMaker Unified Studio.

Domain: Terms – Customer Profile, Policy, Order, Invoice.

The following is the view of ‘Domain’ business glossary with all terms added.

Data sensitivity: Terms – PII, PHI, Confidential, Internal.

The following is the view of ‘Data sensitivity’ business glossary with all terms added.

Business Unit: Terms – KYC, Credit Risk, Marketing Analytics

The following is the view of ‘Business Unit’ business glossary with all terms added.

We recommend that you use glossary descriptions to make terms unambiguous. Ambiguous or overlapping definitions confuse AI models and humans equally.

Step 2: Create data assets

Create the following table in Amazon Redshift. For information about how to bring Amazon Redshift data to Amazon SageMaker Catalog, see Amazon Redshift compute connections in Amazon SageMaker Unified Studio.

CREATE TABLE  dev.public.customer_analytics_data (
    customer_id VARCHAR(50) NOT NULL,
    customer_full_name VARCHAR(200),
    customer_email VARCHAR(255),
    customer_phone VARCHAR(20),
    customer_dob DATE,
    customer_tax_id VARCHAR(256),
    policy_id VARCHAR(50),
    policy_type VARCHAR(100),
    policy_start_date DATE,
    policy_end_date DATE,
    policy_coverage_amount DECIMAL(18,2),
    order_id VARCHAR(50),
    order_date TIMESTAMP,
    order_status VARCHAR(50),
    order_total DECIMAL(18,2),
    invoice_id VARCHAR(50),
    invoice_date DATE,
    invoice_amount DECIMAL(18,2),
    invoice_payment_status VARCHAR(50),
    customer_profile_created_timestamp TIMESTAMP DEFAULT GETDATE(),
    customer_profile_updated_timestamp TIMESTAMP DEFAULT GETDATE(),

    PRIMARY KEY (customer_id, order_id)
)
DISTSTYLE KEY
DISTKEY (customer_id)
SORTKEY (customer_id, order_date);

Once the Redshift is onboarded with above steps, navigate to Project catalog from left navigation menu and choose Data sources. Run the Data Source to add the table to Project inventory assets.

‘customer_analytics_data’ should be Project Assets inventory.

Verify navigating to ‘Project catalog’ menu on the left and choose ‘Assets’.

Step 3: Generate classification recommendations

To automatically generate terms, select GENERATE TERMS in ‘GLOSSARY TERMS’ section of the asset.

AI recommendations for glossary terms automatically analyze asset metadata and context to determine the most relevant business glossary terms for each asset and its columns. Instead of relying on manual tagging or static rules, it reasons about the data and performs iterative searches across what already exists in the environment to identify the most relevant glossary term concepts.

After recommendations are generated, review the terms both at table and column level. Table level suggested terms can be viewed as shown in the following image:

Select the SCHEMA tab to review column level tags as shown in the following image:

Review and accept individually by selecting the AI icon shown in below image.

In this case, we select ACCEPT ALL and then select PUBLISH ASSET as shown below.

The tags are now added to the asset and columns without manual search and addition. Select PUBLISH ASSET.

The asset is now published to the catalog as shown in the following image in the upper left corner.

Step 4: Improve data discovery

Users can now experience enhanced search results and find assets in the catalog based on the associated terms.

Browse by TermsUsers can now explore the catalog and filter by terms as shown in left navigation “APPLY FILTER” section

Search and FilterUsers can also search assets by glossary terms as shown below:

Cleanup

To delete SageMaker domain, follow instructions at Delete domains
To delete your Redshift cluster follow steps at Shutting down and deleting a cluster

Conclusion

By standardizing terminology at publication, organizations can reduce metadata drift and improve discovery reliability. The feature integrates with existing workflows, requiring minimal process changes while helping deliver immediate catalog consistency improvements.

By tagging data at publication rather than correcting it later, data teams can spend less time fixing metadata and more time using it. For more information on SageMaker capabilities, see the Amazon SageMaker Catalog User Guide.

About the authors

Building a Scalable Messaging API with AWS End User Messaging and SES

2026-03-24 Tyler Holmes

Post Syndicated from Tyler Holmes original https://aws.amazon.com/blogs/messaging-and-targeting/building-a-scalable-messaging-api-with-aws-end-user-messaging-and-ses/

Modern applications often need to send notifications across multiple channels either through email and/or SMS. However, building a reliable messaging system that manages templates, handles failures gracefully, scales, and maintains security can be challenging. Following this guide, you’ll learn how to build a template manager and messaging API using API Gateway with JWT authentication for secure access, Amazon SQS for reliable message queuing, AWS Lambda for serverless processing, AWS End User Messaging for SMS, and Amazon Simple Email Service (SES) for email.

Architecture overview

You deploy a decoupled architecture that separates message ingestion from processing, providing resilience and scalability.

Fig. 1 Message Template Manager Architecture

Architecture flow

Client Application sends authenticated requests with JWT tokens
API Gateway validates requests using a Lambda Authorizer
Lambda Authorizer retrieves the JWT secret from AWS Secrets Manager and validates the token
API Gateway sends validated messages to the SQS Queue
Lambda Processor polls messages from SQS in batches
Lambda Processor retrieves message templates from DynamoDB (if needed)
Lambda Processor sends emails through Amazon SES and SMS through AWS End User Messaging
Failed messages (after 3 retries) move to the Dead Letter Queue
CloudWatch Alarm triggers when messages arrive in the DLQ

If a message fails processing after three attempts, it moves to a Dead Letter Queue (DLQ) where it’s preserved for 14 days, and a CloudWatch alarm notifies you of the failure.

Key features

With this architecture, you get several important capabilities:

JWT Authentication: Secure API access using JSON Web Tokens stored in AWS Secrets Manager
Automatic Retries: Failed messages retry up to three times before moving to the DLQ
Partial Batch Failures: Only failed messages retry, not the entire batch
Template Management: Store reusable message templates in Amazon DynamoDB
Multi-Channel Support: Send email through Amazon SES and SMS through AWS End User Messaging
Configuration Set Support: Track delivery metrics and route events with per-message or deployment-level configuration sets
Monitoring: CloudWatch alarms alert you when messages fail

Implementation details

1. API Gateway with JWT authorization

The API Gateway uses a Lambda authorizer to validate JWT tokens before allowing requests through:

def lambda_handler(event, context):
    token = event.get('authorizationToken', '').replace('Bearer ', '')
    try:
        # Retrieve secret from AWS Secrets Manager
        jwt_secret = get_jwt_secret()
        
        # Validate JWT token
        payload = jwt.decode(
            token,
            jwt_secret,
            algorithms=['HS256'],
            issuer='messaging-api'
        )
        
        # Generate IAM policy to allow request
        return generate_policy(payload.get('sub'), 'Allow', event['methodArn'])
    except jwt.ExpiredSignatureError:
        raise Exception('Unauthorized: Token expired')
    except jwt.InvalidTokenError:
        raise Exception('Unauthorized: Invalid token')

The JWT secret is stored securely in AWS Secrets Manager and cached in the Lambda execution environment for performance.

2. SQS queue configuration

The SAM template defines two queues with appropriate settings:

MessagesQueue:
  Type: AWS::SQS::Queue
  Properties:
    QueueName: MessagesQueue
    VisibilityTimeout: 300  # 5 minutes
    MessageRetentionPeriod: 345600  # 4 days
    RedrivePolicy:
      deadLetterTargetArn: !GetAtt MessagesDeadLetterQueue.Arn
      maxReceiveCount: 3

MessagesDeadLetterQueue:
  Type: AWS::SQS::Queue
  Properties:
    QueueName: MessagesDeadLetterQueue
    MessageRetentionPeriod: 1209600  # 14 days

The visibility timeout of 5 minutes prevents duplicate processing while giving the Lambda function enough time to complete. Messages that fail three times automatically move to the DLQ.

3. Lambda message processor

The Lambda function processes messages from SQS and sends them through the appropriate channel:

def lambda_handler(event, context):
    failed_messages = []
    
    for record in event['Records']:
        message_id = record['messageId']
        try:
            message = json.loads(record['body'])
            
            # Process email if configured
            if 'EmailMessage' in message:
                send_emails(message)
            
            # Process SMS if configured
            if 'SMSMessage' in message:
                send_sms_messages(message)
                
        except Exception as e:
            print(f"Error processing message {message_id}: {str(e)}")
            failed_messages.append({"itemIdentifier": message_id})
    
    # Return failed messages for automatic retry
    return {"batchItemFailures": failed_messages}

Configuration Sets for tracking and analytics:

Configuration Sets enable you to track delivery metrics, monitor costs, and route events to analytics pipelines. You can set defaults at deployment time and override them per-message:

Deployment-level defaults: Set SMSConfigurationSet parameter during deployment to apply to all messages
Per-message override: Include ConfigurationSetName in the SMSMessage payload to use different tracking for specific messages

This flexibility lets you separate analytics by message type, campaign, or priority without requiring redeployment.

4. Template management with DynamoDB

Message templates are stored in DynamoDB for reusability:

def get_email_template_from_dynamodb(template_name, substitutions):
    response = templates_table.get_item(Key={'TemplateName': template_name})
    
    if 'Item' not in response:
        return build_default_email(), "Account Alert"
    
    template_body = response['Item']['MessageBody']
    subject = response['Item'].get('Subject', 'Notification')
    
    # Replace {variable} placeholders with actual values
    rendered_body = replace_variables(template_body, substitutions)
    rendered_subject = replace_variables(subject, substitutions)
    
    return rendered_body, rendered_subject

You can update message content without redeploying code.

Template size considerations:

DynamoDB has a 400 KB limit per item, which includes all attribute names and values. For message templates, this means:

Typical email templates (5-20 KB) fit comfortably
SMS templates (< 1 KB) have no practical constraints

If you need to store templates larger than 400 KB, consider storing them in Amazon S3 and referencing the S3 object key in DynamoDB. This hybrid approach provides unlimited template size while maintaining fast lookups.

Prerequisites

Before deploying this solution, ensure you have the following:

AWS End User Messaging SMS configured with a phone pool or origination identity for SMS sending
Amazon SES configured with verified email identities (sender and recipient for sandbox mode)
IAM permissions to create Lambda functions, API Gateway, SQS queues, DynamoDB tables, and Secrets Manager secrets
Python 3.9 or later installed locally
AWS SAM CLI installed (version 1.0 or later)
AWS CLI installed and configured with your credentials
An active AWS account with appropriate permissions

Deployment

You use AWS SAM for infrastructure as code. Deploy with these commands:

sam build
sam deploy --guided

During deployment, you’ll set a JWT secret that’s stored in AWS Secrets Manager. Use a strong, random secret for production:

python -c "import secrets; print(secrets.token_urlsafe(32))"

Usage example

Once deployed, send messages by making authenticated API requests:

curl -X POST "https://your-api-endpoint/dev/" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -d '{
    "TraceId": "12345",
    "EmailMessage": {
      "FromAddress": "[email protected]",
      "Subject": "Low Balance Alert",
      "ConfigurationSetName": "email-analytics",
      "Substitutions": {
        "productName": "CHEQUING",
        "membershipNumber": "****5493",
        "accountBalance": "100.00"
      }
    },
    "SMSMessage": {
      "MessageType": "TRANSACTIONAL",
      "OriginationNumber": "your-pool-id",
      "TemplateName": "alert-template",
      "ConfigurationSetName": "sms-analytics"
    },
    "Addresses": {
      "[email protected]": {
        "ChannelType": "EMAIL"
      },
      "+16048621234": {
        "ChannelType": "SMS",
        "Substitutions": {
          "productName": "CHEQUING",
          "membershipNumber": "****7303",
          "accountBalance": "100.00"
        }
      }
    }
  }'

Using Configuration Sets:

The example above shows optional ConfigurationSetName parameters for both email and SMS. These enable:

Delivery tracking: Monitor delivery rates, failures, and bounce metrics
Cost monitoring: Track spending per campaign or message type
Event routing: Send delivery events to CloudWatch, Kinesis, or SNS for analytics
Segmented metrics: Separate analytics by use case, priority, or customer segment

If you don’t specify a ConfigurationSetName in the request, the system uses the deployment-level default (if configured).

Monitoring and operations

CloudWatch alarms

The solution includes a CloudWatch alarm that triggers when messages arrive in the DLQ:

DLQAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: !Sub ${AWS::StackName}-DLQ-Messages
    MetricName: ApproximateNumberOfMessagesVisible
    Namespace: AWS/SQS
    Statistic: Sum
    Period: 300
    EvaluationPeriods: 1
    Threshold: 1
    ComparisonOperator: GreaterThanOrEqualToThreshold

Handling failed messages

When messages fail, you can inspect them in the DLQ and redrive them back to the main queue after fixing the issue:

# Check DLQ depth
aws sqs get-queue-attributes \
  --queue-url YOUR_DLQ_URL \
  --attribute-names ApproximateNumberOfMessages

# Redrive messages from DLQ to main queue
aws sqs start-message-move-task \
  --source-arn YOUR_DLQ_ARN \
  --destination-arn YOUR_MAIN_QUEUE_ARN

Viewing logs

Lambda automatically logs to CloudWatch Logs:

aws logs tail /aws/lambda/MessageProcessor --follow

Cost considerations

For 1 million messages per month estimated costs are:

Service	Usage	Cost
API Gateway	1M requests	$3.50
Amazon SQS	1M messages	$0.40
AWS Lambda	1M invocations (128MB, 1s avg)	$2.50
Amazon SES	1M emails	$100.00
AWS End User Messaging SMS	1M SMS	$ Varies based on destination

Note: The serverless architecture means you only pay for what you use, with no minimum fees or upfront costs.

Security best practices

This solution follows several security best practices:

JWT Authentication: All API requests require valid JWT tokens
Secrets Manager: JWT secrets are stored encrypted in AWS Secrets Manager
IAM Least Privilege: Each Lambda function has only the permissions it needs
HTTPS Only: API Gateway enforces HTTPS for all requests
Token Caching: Authorization decisions are cached for 5 minutes to reduce latency

Clean up

To avoid incurring ongoing charges, delete the resources created by this solution when you no longer need them:

If you configured AWS End User Messaging phone pools or origination identities for this solution, remove them from the End User Messaging console
If you created Amazon SES email identities specifically for this solution, remove them from the SES console
Verify that all resources (Lambda functions, API Gateway, SQS queues, DynamoDB table, Secrets Manager secret) have been removed in the AWS Management Console
Delete the CloudFormation stack by running: sam delete --stack-name <your-stack-name>

Conclusion

With this architecture, you can build a production-ready messaging API using AWS serverless services. The decoupled design gives you resilience through automatic retries and dead letter queues, while the serverless approach eliminates infrastructure management and scales automatically.

The complete solution is deployable through AWS SAM and includes:

JWT authentication with AWS Secrets Manager
Multi-channel messaging (email and SMS)
Template management with DynamoDB
Configuration Set support for tracking and analytics
Comprehensive monitoring and alerting
Automatic retry and failure handling

You can extend this architecture by adding more channels (push notifications, webhooks), implementing message scheduling, or integrating with Amazon EventBridge for event-driven workflows.

Additional resources

Amazon SQS Developer Guide
AWS Lambda Developer Guide
Amazon SES Developer Guide
AWS End User Messaging SMS Documentation
API Gateway Lambda Authorizers
AWS SAM Documentation

The complete source code for this solution is available in the accompanying GitHub repository, including SAM templates, Lambda functions, and deployment scripts.

About the authors

Enhancing auto scaling resilience by tracking worker utilization metrics

2026-03-24 Brian Moore

Post Syndicated from Brian Moore original https://aws.amazon.com/blogs/compute/enhancing-auto-scaling-resilience-by-tracking-worker-utilization-metrics/

A resilient auto scaling policy requires metrics that correlate with application utilization, which may not be tied to system resources. Traditionally, auto scaling policies track system resource such as CPU utilization. These metrics are easily available, but they only work when resource consumption correlates with worker capacity. Factors such as high variance in request processing time, mixed instance types, or natural changes in application behavior over time can break this assumption.

Worker utilization tracking offers an alternative approach. Using a combination of total worker slots, work in flight, and work waiting in the backlog, a utilization value can be calculated for use in an auto scaling policy. This approach remains accurate across fleets with mixed instance types, applications with variable latencies, and requires no changes as your application evolves.

The limitations of resource-based scaling

Traditional auto scaling policies track system resource metrics like CPU utilization, assuming a direct correlation between resource consumption and available application capacity. Consider an application that reads messages from Amazon Simple Queue Service (SQS), processes them, and writes results to Amazon DynamoDB. If this application uses a fixed-size thread pool to process messages, such as 10 worker threads, the application reaches maximum capacity when all threads are busy, regardless of CPU utilization.

In our example, each worker spends most of its time waiting for DynamoDB responses rather than consuming CPU. All 10 threads become occupied handling requests, but CPU utilization stays low. From the perspective of the auto scaling policy, the fleet looks like it has enough capacity because plenty of CPU headroom remains. Meanwhile, new messages accumulate in the SQS queue because no workers are available to process them.

For queue-based workloads, AWS provides guidance to scale based on an acceptable backlog per worker. This is a calculated target based on your application’s average processing latency (queue delay). This works well when processing times are consistent, but breaks down if an application has variable latency characteristics.

Consider an image processing application that initially handles thumbnails taking 500 ms each. Using the traditional guidance with a target latency of 5 seconds you calculate an acceptable backlog of 10 messages per worker and deploy your scaling policy. Over time, the application evolves to also process 4K photos which take 2 seconds each. Eventually 4K photos are 50% of your traffic and total latency for queued messages has increased to 12.5 seconds, 2.5x more than your initial target.

The scaling policy is no longer fit for its intended purpose because your original latency assumptions no longer reflect reality. To keep this type of scaling effective you must also remember to update your scaling policies as your application behavior evolves.

A shift to using mixed instance types in your application can lead to additional complexity when using traditional resource-based scaling policies. Different instance types may handle the same workload at different CPU levels leading to an unbalanced average that misrepresents your actual application health. By changing your mental model to consider how much work your application can accept instead of how much of a system resource is available you can improve your scaling rules and better model your application’s capacity.

Understanding worker utilization

Worker utilization measures the ratio of active work to available processing capacity. To calculate it, divide total work by total workers.

We use an SQS-based processing application as an example to demonstrate how worker utilization operates, but this approach can also be applied to other applications where work units and worker capacity are measurable. In our example application total work consists of messages waiting to be processed plus messages currently being processed. Amazon CloudWatch provides these values through the ApproximateNumberOfMessagesVisible metric (messages waiting in the queue) and the ApproximateNumberOfMessagesNotVisible metric (messages currently being processed or in flight). Each host in your application should publish the number of available workers as a custom CloudWatch metric with at least a 1-minute period. For Java thread pools or Python multiprocessing pools, this represents the pool or process count. The formula works regardless of the metric period. Using the shortest period possible allows more responsive target tracking and enables Fast Target Tracking if your application has sub-minute data points.

To derive the formula, we can use the following CloudWatch Metric Math expressions:

totalWork = FILL(backlog, REPEAT) + FILL(inFlight, REPEAT)
utilizationRatio = totalWork / workers

Where:

backlog = ApproximateNumberOfMessagesVisible with the Maximum statistic.
inFlight = ApproximateNumberOfMessagesNotVisible with the Maximum statistic.
workers = Your custom TotalWorkers metric with the Sum statistic.

Putting the components together the final expression for your target tracking scaling policy uses the following formula:

IF(FILL(workers, 0) > 0, utilizationRatio, IF(totalWork > 0, 1, 0))

The FILL function uses last known values if SQS metrics are delayed, and the IF statement handles the case where you have no traffic and your fleet scales to zero instances. When there are no available workers, the formula metric reports 1 to indicate that the workers are fully saturated. This prevents the application from getting stuck at zero capacity and not being able to respond to any requests.

In this formula, a value of 1 or higher represents full or over saturation, where all workers are busy with no spare capacity, like running at 100% CPU. Values below 1 indicate available capacity for your application to process more work.

For applications without a measurable backlog metric, you can track worker utilization using only the in-flight work. This approach works for APIs or other synchronous workloads where work arrives and is immediately assigned to workers rather than queuing. In these cases, the formula becomes:

IF(FILL (workers, 0) > 0, utilizationRatio, IF(FILL(inFlight, 0) > 0, 1, 0))

In this scenario the utilization ratio is calculated as follows:

utilizationRatio = FILL(inFlight, REPEAT) / workers

The definitions of workers and inFlight remain the same for this formula. The primary difference is that the ratio directly tracks workers available and does not consider the backlog as an option.

How worker utilization prevents outages

Worker utilization-based scaling works for any application that can define available workers and total work. When the ratio of total work to available workers exceeds your threshold, the system scales out. This approach measures whether workers are available to handle the workload and treats application bottlenecks consistently. Whether workers are waiting on network I/O, performing CPU-intensive calculations, or experiencing another bottleneck doesn’t matter; the only question is whether total work exceeds available worker capacity. Any situation causing messages to accumulate on the queue increases the utilization ratio and triggers scale-out.

Implementing worker utilization scaling

To set up worker utilization-based auto scaling, identify metrics to use in the formula discussed earlier. First, identify a metric to track the amount of work being worked on. For SQS-based processing, AWS provides this metric. Second, implement a custom metric from your application representing the total workers. Optionally you can also identify a metric to track the available backlog of work.

Using CloudWatch metric math, you calculate the utilization metric and use it in a target tracking scaling policy. Here is an example AWS CloudFormation snippet showing the metric math configuration for a Amazon EC2 Auto Scaling group. This snippet shows only the scaling policy configuration and is only an example, before using in production fully test with your application. Your complete template also needs IAM roles with appropriate permissions for SQS, DynamoDB, and CloudWatch access.

ScalingPolicy: 
  Type: AWS::AutoScaling::ScalingPolicy 
  Properties: 
    AutoScalingGroupName: !Ref AutoScalingGroup 
    PolicyType: TargetTrackingScaling 
    TargetTrackingConfiguration: 
      TargetValue: 0.7 
      CustomizedMetricSpecification: 
        Metrics: 
          - Id: backlog 
            MetricStat: 
            Metric: 
              Namespace: AWS/SQS 
              MetricName: ApproximateNumberOfMessagesVisible 
              Dimensions: 
                - Name: QueueName 
                  Value: !GetAtt ProcessingQueue.QueueName 
              Stat: Maximum 
          - Id: inFlight 
            MetricStat: 
            Metric: 
              Namespace: AWS/SQS 
              MetricName: ApproximateNumberOfMessagesNotVisible 
              Dimensions: 
                - Name: QueueName 
                  Value: !GetAtt ProcessingQueue.QueueName 
              Stat: Maximum 
          - Id: workers 
            MetricStat: 
            Metric: 
              Namespace: YourApp 
              MetricName: TotalWorkers 
            Stat: Sum 
          - Id: totalWork 
            Expression: FILL(backlog, REPEAT) + FILL(inFlight, REPEAT) 
          - Id: utilizationRatio 
            Expression: totalWork / workers 
          - Id: utilization 
            Expression: IF(FILL(workers, 0) > 0, utilizationRatio, IF(totalWork > 0, 1, 0)) 
            ReturnData: true

This approach also works for Amazon ECS services using AWS Application Auto Scaling. The metric math configuration remains the same, but you create an AWS::ApplicationAutoScaling::ScalingPolicy resource instead, adapting the parameters accordingly.

Choosing a target utilization

Since the worker utilization metric directly tracks the available capacity of your application, the target utilization value you choose reflects your organization’s balance between cost efficiency and availability. Lower target values provide more headroom for traffic spikes and faster response to load changes but result in higher infrastructure costs due to lower utilization. Higher target values maximize cost efficiency by keeping workers busy but leave less headroom for sudden traffic increases.

When choosing a target consider traffic patterns, acceptable latency during scale-out events, and cost sensitivity. Applications with unpredictable traffic spikes may benefit from lower targets, while an application with predictable load can safely use higher targets. Start with a moderate value like 0.7 and adjust based on observed behavior and your business requirements. If you previously tracked a resource utilization metric such as CPU, consider starting with the same target.

Monitoring resource utilization for cost optimization

While worker utilization drives scaling decisions, CPU and latency should be regularly evaluated to ensure cost-effective operations. Resource-based metrics can identify host resizing opportunities to better match your application requirements. If no scale-in happens when CPU utilization is consistently low, you are likely running instances that are too large for your workload. By using worker utilization in an auto scaling policy, you can switch to a different instance type without adjusting the auto scaling policy. The formula automatically adapts as you add different instance types or update the capacity per worker.

Conversely, if CPU utilization is consistently high while worker utilization remains at your target, your instances might be undersized. Upgrading to larger instance types can improve per-worker throughput, allowing each worker to process tasks faster. Changes to your auto scaling policy are not needed in this situation either. As messages are processed faster, they spend less time in the in-flight state, and the utilization ratio naturally adjusts.

This approach manages application availability independent of instance size, while resource utilization guides cost optimization. Each can be optimized independently without complex coordination.

Conclusion

Worker utilization-based auto scaling reduces the operational burden of continuously validating your scaling rules as application requirements and infrastructure change. By tracking the ratio of work to workers, your auto scaling policies automatically respond to capacity constraints based on available work. The approach works across workloads with discrete processing units and remains effective when you modify instance configurations or application worker pool sizes.

Implementation requires identifying a metric for available work, publishing a custom metric representing total workers, and using CloudWatch metric math in a target tracking scaling policy. This setup provides resilience that scaling based solely on resource metrics cannot achieve, while maintaining the flexibility to optimize costs and change your instance size without impacting system availability.

To get started:

Identify an application in your environment that uses a worker pool.
Instrument the application to publish worker count metrics.
Configure a scaling policy tracking worker utilization.
Monitor how the system responds to traffic changes and capacity events.

Learn more

To learn more about auto scaling and monitoring, see the following resources:

AI-powered event response for Amazon EKS

2026-03-18 Aritra Nag

Post Syndicated from Aritra Nag original https://aws.amazon.com/blogs/architecture/ai-powered-event-response-for-amazon-eks/

Cloud environments with dozens of microservices are now easier to manage than ever, and modern DevOps teams are well-equipped to balance rapid deployments with operational stability — even as monitoring tools surface thousands of daily signals.

AWS DevOps Agent is a fully managed autonomous AI Agent that resolves and proactively prevents incidents, continuously improving reliability and performance of applications in AWS, multicloud, and hybrid environments. It brings Kubernetes-native intelligence to incident response. It understands how Pods relate to Deployments, which Services route traffic, what ConfigMaps provide configuration, and how these components interact across your environment. Rather than seeing isolated infrastructure issues, the agent comprehends the architectural relationships that matter most for fast, accurate root cause analysis. In this post, you’ll learn how AWS DevOps Agent integrates with your existing observability stack to provide intelligent, automated responses to system events.

Architecture Diagram of DevOps Agent Figure 1: This is an example of target architecture of how Amazon EKS workloads are deployed and how AWS DevOps agent can interact with the different managed services like Amazon CloudWatch

How AWS DevOps Agent discovers Kubernetes resources

Built on Amazon Bedrock, the agent can analyze complex operational scenarios and correlate data from multiple sources. AWS DevOps Agent combines natural language processing (NLP) of logs and error messages with root cause analysis, powered by machine learning (ML), to automatically identify issues across your infrastructure.

Telemetry-based discovery

The agent analyzes OpenTelemetry data to infer runtime relationships:

Service Mesh Analysis: Examines network traffic patterns between pods to identify service-to-service communication
Trace Correlation: Uses distributed traces to map request flows across microservices
Metric Attribution: Associates performance metrics with specific pods, containers, and nodes

Metadata enrichment

The agent enriches discovered resources with contextual information:

Labels and Annotations: Extracts application metadata, ownership information, and deployment details
Resource Specifications: Captures CPU/memory requests and limits, health check configurations, and environmental variables
Network Topology: Maps pod IPs, service cluster IPs, ingress rules, and network policies

Discovery process

When you start an investigation, the agent executes the following discovery workflow:

Initial Scan: Queries the Kubernetes API for all resources in relevant namespaces
Dependency Analysis: Builds a dependency graph showing how resources relate to each other
Telemetry Correlation: Matches discovered resources with their corresponding metrics, logs, and traces
Context Building: Aggregates resource state, recent events, and performance data into a unified view

Implementation details

Prerequisites

Before implementing this solution, verify that you have the following:

AWS Account
AWS Command Line Interface (AWS CLI) version 2.15.0 or later
Kiro CLI version 1.25.0 or later
AWS Identity and Access Management (IAM) role following policy of least privilege permissions.
Amazon Elastic Kubernetes Service (Amazon EKS) cluster version 1.27 or later
OpenTelemetry Operator installed for telemetry collection
Amazon Managed Service for Prometheus workspace configured
AWS Distro for OpenTelemetry (ADOT) Collector deployed
Container Insights enabled for Amazon CloudWatch integration
AWS X-Ray integration configured for distributed tracing

Development environment

Docker 24.0+ and AWS CLI configured
AWS Cloud Development Kit (AWS CDK) v2 version 2.238.0
AWS Serverless Application Model (AWS SAM) CLI version 1.154.0 for serverless deployment

As part of the setup and the applications that are mentioned in the following sections. We have this AWS samples repo with deployable scripts and setup instructions.

This section walks you through deploying and configuring the complete AWS DevOps Agent demo environment.

Step 1: Deploy AWS DevOps Agent infrastructure

Begin by deploying the AWS DevOps Agent using the AWS CDK. The infrastructure includes the Agent Space configuration, IAM roles and policies, and integration with your EKS cluster.

Screenshot of the AWS DevOps Agent web interface showing a form to create a new Agent Space named "OTEL-DevOpsAgent-Demo-v1". The form includes fields for agent space name and description, radio button options for configuring IAM roles for AWS resource access and web app access, auto-generated role names, and Cancel and Create action buttons.

Figure 2: This is screenshot of configuring the Agent Space in the AWS Console

Configure the Agent Space in the AWS Console by navigating to the AWS DevOps Agent service. You will then create a new Agent Space for your EKS cluster. Finally, you will set up data source integrations including Prometheus workspace endpoints, Amazon CloudWatch Log groups, and X-Ray service configuration.

Screenshot of the AWS DevOps Agent Capabilities configuration tab for the OTEL-DevOpsAgent-Demo-v1 agent space. The interface displays a Cloud capability section with a primary AWS account (ID: 123456789012) showing a Valid status, a secondary sources section with no entries, and options to add additional source accounts. Figure 3: This is screenshot of validating the connectivity to data sources and access to the AWS account

Validate the deployment by accessing the DevOps Agent web interface, verifying connectivity to data sources, and confirming that the agent can discover your EKS cluster resources.

Figure 4: This is screenshot of checking the incident response and if there is any ongoing investigation

Step 2: Set up port forwarding for applications

Configure port forwarding to enable the traffic generator to access your deployed applications. Set up port forwarding for all sample applications using “kubectl” commands. Each application runs on different ports to simulate a realistic microservices environment:

# Sample Metrics App (port 8000)
kubectl port-forward svc/sample-metrics-app 8000:8000 -n default &
# Python OTEL App (port 8080)
kubectl port-forward svc/otel-sample-app 8080:8000 -n default &
# Go OTEL App (port 8090)
kubectl port-forward svc/go-otel-sample-app 8090:8080 -n default &
# Java OTEL App (port 8081)
kubectl port-forward svc/java-otel-sample-app 8081:8080 -n default &

Step 3: Install and configure Traffic Generator

The traffic generator is a Python-based tool that creates realistic load patterns and error scenarios for testing AWS DevOps Agent capabilities. Install the required Python dependencies and make the traffic generator executable:

# Install required Python packages
pip install requests
# Make the script executable
chmod +x traffic-generator.py

The traffic generator supports multiple configuration options for creating different testing scenarios.

Step 4: Generate baseline traffic

Create baseline operational data by generating normal traffic patterns across all applications. This establishes normal operational patterns that AWS DevOps Agent can learn from. Generate steady traffic to establish baseline metrics:

# Generate normal baseline traffic
python traffic-generator.py --app all --duration 900 --rps 10 --error-rate 0.05

Terminal output showing the initialization of an EKS Platform Traffic Generator script targeting four OTEL applications — Sample Metrics App, Python OTEL App, Go OTEL App, and Java OTEL App — each configured for 900 seconds at 15 requests per second with a 10% error rate. All four services are confirmed available at their respective localhost ports.

This command generates traffic to all applications for 15 minutes at 10 requests per second with a 5% error rate, simulating normal operational conditions. Monitor the baseline traffic generation by checking application metrics and HPA scaling behavior:

# View HPA status (should show scaling based on metrics)
kubectl get hpa -A
# Check custom metrics availability
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .
# View specific metric values
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/sample_app_requests_rate | jq .

Step 5: Configure AWS DevOps Agent Investigation

Set up AWS DevOps Agent to monitor your EKS cluster and prepare for event investigation workflows.

Screenshot of the AWS DevOps Agent Incident Response interface showing the investigation timeline for the "Demo CPU Spike" incident created on 2026-02-04. The timeline displays a user request describing a 97.5% CPU spike on an EKS workload, followed by two assistant responses tracking investigation progress. The investigation was completed at 20:02:40 on the same day. Figure 5: This is screenshot of overall timeline of the ongoing investigation through AWS DevOps agent

Access the AWS DevOps Agent through the AWS Console:

Navigate to AWS DevOps Agent in the AWS Console.
Select your configured Agent Space.
Select Operator access to open the DevOps Agent web application.
Configure data source connections to verify proper integration.

Figure 6: This is screenshot of validating the access to the observability data from the cluster

Verify that the AWS DevOps Agent can access observability data from your cluster including metrics from Amazon Managed Prometheus, logs from Amazon CloudWatch Logs, traces from AWS X-Ray, and topology information from your EKS cluster. DevOps agent can also pull the service map information of your kubernetes resources.

Testing scenarios and use cases

This section demonstrates different testing scenarios that showcase AWS DevOps Agent capabilities in various operational situations.

Scenario 1: Normal load testing

This scenario establishes baseline operational patterns that AWS DevOps Agent can learn from and use for anomaly detection. Generate steady traffic to establish baseline metrics:

python traffic-generator.py --app all --duration 900 --rps 10 --error-rate 0.05

What this test does: The command runs a 15-minute (900-second) steady traffic test across all applications at 10 requests per second, with a 5% simulated error rate. This low, consistent load represents typical production traffic and gives the agent enough signal to establish a reliable operational baseline.
Figure 7: This is screenshot of investigation and root cause found in one of the cluster

What you should observe: During this scenario, the AWS DevOps Agent learns normal operational baselines. In the agent’s investigation dashboard, you will see the following being captured and recorded:

Typical request patterns and response times — The agent records average latency and throughput across all services, establishing what “healthy” looks like for your environment.
Normal error rates and distribution — With a 5% error rate, the agent learns the expected noise floor for errors, so it can distinguish genuine incidents from routine fluctuations.
Resource utilization patterns — CPU, memory, and network usage are tracked per pod and node, giving the agent a reference point for what normal resource consumption looks like under standard load.
Service dependency relationships — The agent maps how services communicate with each other, identifying upstream and downstream dependencies that will be critical for root cause analysis in future incidents.

After the test completes, you should see a stable metrics summary in the agent dashboard showing consistent throughput, low error variance, and steady resource utilization — confirming that a reliable baseline has been captured.

Expected outcomes:

During this scenario, the AWS DevOps Agent learns normal operational baselines. These baselines include typical request patterns and response times, normal error rates and distribution, resource utilization patterns, and service dependency relationships.

Scenario 2: Simulated production event

This scenario demonstrates the AWS DevOps Agent’s ability to investigate and analyze events with elevated error rates and performance degradation:

python traffic-generator.py --app java-otel --duration 600 --rps 30 --error-rate 0.25

Figure 8: This is screenshot of details of the Nodes and telemetry data which is relevant for the investigations

What this test does: The command targets the java-otel application specifically, running a 10-minute (600-second) high-load test at 30 requests per second — three times the baseline — with a 25% error rate. This simulates a degraded service experiencing both a traffic surge and a significant increase in failures.

Figure 9: This is screenshot of review and recommended next steps after the analysis of the root cause

What you should observe: Once the test begins, the AWS DevOps Agent detects the deviation from the established baseline and initiates an investigation. In the agent’s incident response view, you will see the following outcomes:

Affected application identification — The agent pinpoints java-otel-app as the impacted service, distinguishing it from other applications running normally in the cluster.
Error pattern analysis — The agent breaks down the 25% error rate into specific failure modes (for example, HTTP 500 errors, timeout spikes, or connection refusals), helping you understand not just that errors are occurring, but why and where.
Resource utilization correlation — The agent correlates CPU and memory spikes on the affected pods with the observed performance degradation, showing a clear relationship between resource exhaustion and increased error rates.
Root cause identification with confidence scoring — The agent presents a ranked list of potential root causes, each with a confidence score, so you can prioritize your investigation. For example, it may identify a memory leak or thread pool exhaustion as the most likely cause with high confidence.

Advanced analysis capabilities: Beyond the immediate incident, the agent performs deeper analysis that you can explore in the investigation timeline view:

Cross-service impact correlation — The agent identifies whether the degradation in java-otel-app has cascading effects on dependent services, showing you the full blast radius of the incident.
Timeline reconstruction — The agent reconstructs the sequence of events leading up to and during the incident, helping you understand how the situation evolved over time.
Dependency mapping — Upstream and downstream service dependencies are visualized, making it clear which services are affected directly and which are at risk.
Prioritized remediation recommendations — The agent provides actionable remediation steps ranked by business impact, so your team can address the most critical issues first. Recommendations may include scaling the affected deployment, adjusting resource limits, or rolling back a recent configuration change.

After the test completes, you should see a full incident report in the agent dashboard summarizing the root cause, affected components, timeline, and recommended next steps — giving your team everything needed to resolve the issue and prevent recurrence.

Expected outcomes:

The agent identifies which application is affected (java-otel-app), analyzes error patterns and rates for specific failure modes, correlates resource utilization with performance degradation, and provides potential root causes with confidence scoring.

Figure 10: This is screenshot of mitigation plan suggested by the AWS DevOps agent

Advanced analysis capabilities:

The agent performs analysis including cross-service impact correlation, timeline reconstruction of event progression, dependency mapping to identify upstream/downstream effects, and prioritized remediation recommendations based on business impact.

AWS DevOps Agent Investigation workflow

This section details how to use AWS DevOps Agent for event investigation and analysis.

Figure 11: This is screenshot of starting an investigation from the AWS Console inside AWS DevOps agent service

Starting an investigation

Access the AWS DevOps Agent web interface and initiate a new investigation:

Investigation Trigger: Choose from predefined scenarios like “High CPU usage,” “Error rate spike,” or “Performance degradation”
Time Range Selection: Select the time period when you generated traffic or observed issues.
Scope Definition: Provide the AWS Account ID, AWS Region (us-east-1), and specific cluster or application context.
Data Source Configuration: Make sure all observability data sources are properly connected.

Investigation Process

AWS DevOps Agent follows a systematic investigation methodology:

Data Collection Phase:

Figure 12: This is screenshot of investigation timeline done by the AWS DevOps agent

This approach correlates metrics from Amazon Managed Prometheus workspace, and analyzes logs from Amazon CloudWatch Logs for error patterns and anomalies. It also reviews distributed traces from AWS X-Ray for service dependencies, and examines application topology and service relationships to provide comprehensive observability during the migration process.

Analysis phase:

Figure 13: This is screenshot of analysis done from the AWS Console inside AWS DevOps agent service

This approach identifies patterns and anomalies using MLalgorithms. It correlates events across multiple data sources for comprehensive understanding, applies statistical analysis to determine the significance of observed changes, and compares current behavior against established baselines for accurate detection and assessment of system behavior.

Root cause identification:

Figure 14: This is screenshot of root cause and investigation summary done from the AWS Console inside AWS DevOps agent service

This approach provides systematic root cause analysis with confidence scoring, identifies contributing factors and potential trigger events, maps event timeline with correlated evidence from multiple sources, and suggests most likely causes based on data correlation and pattern analysis to enable efficient troubleshooting and resolution.

Mitigation strategy:

Figure 15: This is screenshot of mitigation summary recommended by the AWS DevOps agent

This approach recommends immediate mitigation actions to resolve current issues. It also suggests long-term prevention strategies to avoid recurrence, provides runbook-style guidance for event response teams, and integrates with existing DevOps workflows and tools for seamless incident response and improvement.

Key features and benefits

Preventing future incidents

AWS DevOps Agent analyzes patterns across your incident investigations to deliver targeted recommendations that continuously improve your operational posture and prevent future incidents.

Figure 16: This is screenshot of prevention tab in the AWS Console for the AWS DevOps agent

DevOps Agent topology

AWS DevOps Agent Topology automatically discovers and maps your entire infrastructure into an interactive, living blueprint. It reveals not only what resources exist, but how they interconnect, depend on each other, and drive system behavior.

Figure 17: This is screenshot of topology of Amazon EKS cluster discovered by AWS DevOps agent

Clean up

To avoid ongoing charges, delete the resources that you created while following this walkthrough.

Remove AWS DevOps Agent resources:

Delete investigation data and Agent Space configuration through the AWS Console.
Remove IAM roles and policies created specifically for the DevOps Agent.
Delete any CloudFormation stacks created during deployment.

Conclusion

As organizations continue to embrace cloud-native architectures and DevOps practices, tools like AWS DevOps Agent will become essential for maintaining competitive advantage in an increasingly complex technological landscape.

Ready to add AI powered observability to your container infrastructure? Visit the AWS documentation to access implementation guides, or reach out to your AWS account team to discuss how this automated migration approach can accelerate your cloud modernization journey, while reducing operational overhead.

About the authors

Best practices for Amazon Redshift Lambda User-Defined Functions

2026-03-18 Sergey Konoplev

Post Syndicated from Sergey Konoplev original https://aws.amazon.com/blogs/big-data/best-practices-for-amazon-redshift-lambda-user-defined-functions/

While working with Lambda User-Defined Functions (UDFs) in Amazon Redshift, knowing best practices may help you streamline the respective feature development and reduce common performance bottlenecks and unnecessary costs.

You wonder what programming language could improve your UDF performance, how else can you use batch processing benefits, what concurrency management considerations might be applicable in your case? In this post, we answer these and other questions by providing a consolidated view of practices to improve your Lambda UDF efficiency. We explain how to choose a programming language, use existing libraries effectively, minimize payload sizes, manage return data, and batch processing. We discuss scalability and concurrency considerations at both the account and per-function levels. Finally, we examine the benefits and nuances of using external services with your Lambda UDFs.

Background

Amazon Redshift is a fast, petabyte-scale cloud data warehouse service that makes it simple and cost-effective to analyze data using standard SQL and existing business intelligence tools.

AWS Lambda is a compute service that lets you run code without provisioning or managing servers, supporting a wide variety of programming languages, automatically scaling your applications.

Amazon Redshift Lambda UDFs allows you to run Lambda functions directly from SQL, which unlock such capabilities like external API integration, unified code deployment, better compute scalability, cost separation.

Prerequisites

AWS account setup requirements
Basic Lambda function creation knowledge
Amazon Redshift cluster access and UDF permissions.

Performance optimization best practices

The following diagram contains necessary visual references from the best practices description.

Use efficient programming languages

You can choose from Lambda’s wide variety of runtime environments and programming languages. This choice affects both the performance and billing. More performant code may help reduce the cost of Lambda compute and improve SQL query speed. Faster SQL queries could also help reduce costs for Redshift Serverless and potentially improve throughput for Provisioned clusters depending on your specific workload and configuration.

When choosing a programming language for your Lambda UDFs, benchmarks may help predict performance and cost implications. The famous Debian’s Benchmarks Game Team provides publicly available insights for different languages in their micro-benchmark results. For example, their Python vs Golang comparison shows up to 2 orders of magnitude run time improvement and twice memory consumption reduction if you could use Golang instead of Python. That may positively reflect on both Lambda UDF performance and Lambda costs for the respective scenarios.

Use existing libraries efficiently

For every language provided by Lambda, you can explore the whole collection of libraries to help you implement tasks better from the speed and resource consumption point of view. When transitioning to Lambda UDFs, review this aspect carefully.

For instance, if your Python function manipulates datasets, it might be worth considering using the Pandas library.

Avoid unnecessary data in payloads

Lambda limits request and response payload size to 6 MB for synchronous invocations. Considering that, Redshift is doing best effort to batch the values so that the number of batches (and hence the Lambda calls) would be minimal which reduces the communication overhead. So, the unnecessary data, like one added for future use but not immediately actionable, may reduce efficiency of this effort.

Keep in mind returning data size

Because, from the point of view of Redshift, each Lambda function is a closed system, it is impossible to know what size the returned data can possibly be before executing the function. In this case, if the returned payload is higher than the Lambda payload limit, Redshift will have to retry with the outbound batch of a lower size. That will continue until a fit return payload will be achieved. While it is the best effort, the process might bring a notable overhead.

In order to avoid this overhead, you might use the knowledge of your Lambda code, to directly set the maximum batch size on the Redshift side using the MAX_BATCH_SIZE clause in your Lambda UDF definition.

Use benefits of processing values in batches

Batched calls provide new optimization opportunities to your UDFs. Having a batch of many values passed to the function at once, allows to use various optimization techniques.

For example, memoization (result caching), when your function can avoid running the same logic on the same values, hence reducing the total execution time. The standard Python library functools provides convenient caching and Least Recently Used (LRU) caching decorators implementing exactly that.

Scalability and concurrency management

Increase the account-level concurrency

Redshift uses advanced congestion control to provide the best performance in a highly competitive environment. Lambda provides a default concurrency limit of 1,000 concurrent execution per AWS Region for an account. However, if the latter is not enough, you can always request the account level quota increase for Lambda concurrency, which might be as high as tens of thousands.

Note that even with a restricted concurrency space, our Lambda UDF implementation will do the best effort to minimize the congestion and equalize the chances for function calls across Redshift clusters in your account.

Restrict function concurrency with reserved concurrency

If you want to isolate some of the Lambda functions in a restricted concurrency scope, for example you have a data science team experimenting with embedding generation using Lambda UDFs and you don’t want them to affect your account’s Lambda concurrency much, you might want to set a reserved concurrency for their specific functions to operate with.

Learn more about reserved concurrency in Lambda.

Integration and external services

Call existing external services for optimal execution

In some cases, it might be worth considering using existing external services or components of your application instead of re-implementing the same tasks yourself in the Lambda code. For example, you can use Open Policy Agent (OPA) for policy checking, a managed service Protegrity to protect your sensitive data, there are also a variety of services providing hardware acceleration for computationally heavy tasks.

Note that some services have their own batching control with a limited batch size. For that we implemented a per-function batch row count setting MAX_BATCH_ROWS as a clause in the Lambda UDF definition.

To learn more on the external service interaction using Lambda UDFs refer the following links:

Conclusion

Lambda UDFs provide a way to extend your data warehouse capabilities. By implementing the best practices from this post, you may help optimize your Lambda UDFs for performance and cost efficiency.The key takeaways from this post are:

performance optimization, showing how to choose efficient programming languages and tools, minimize payload sizes, and leverage batch processing to reduce execution time and costs
scalability management, showing how to configure appropriate concurrency settings at both account and function levels to handle varying workloads effectively
integration efficiency, explaining how to benefit from external services to avoid reinventing functionality while maintaining optimal performance.

For more information, visit the Redshift documentation and explore the integration examples referenced in this post.

About the author

Building a scalable, transactional data lake using dbt, Amazon EMR, and Apache Iceberg

2026-03-18 Umesh Pathak

Post Syndicated from Umesh Pathak original https://aws.amazon.com/blogs/big-data/building-a-scalable-transactional-data-lake-using-dbt-amazon-emr-and-apache-iceberg/

Growing data volume, variety, and velocity has made it crucial for businesses to implement architectures that efficiently manage and analyze data, while maintaining data integrity and consistency. In this post, we show you a solution that combines Apache Iceberg, Data Build Tool (dbt), and Amazon EMR to create a scalable, ACID-compliant transactional data lake. You can use this data lake to process transactions and analyze data simultaneously while maintaining data accuracy and real-time insights for better decision-making.

Challenges, business imperatives, and technical advantages

Traditional data lakes have long struggled with fundamental limitations. For example, the lack of ACID compliance, data inconsistencies from concurrent writes, complex schema evolution, and the absence of time travel, rollback, and versioning capabilities. These shortcomings directly conflict with growing business demands for concurrent read/write support, robust data versioning and auditing, schema flexibility, and transactional capability within data lake environments. To address these gaps, modern solutions use ACID transactions at scale, optimized storage formats through Apache Iceberg, version control for data on Amazon Simple Storage Service (Amazon S3), and cost-effective, streamlined maintenance—delivering a reliable, enterprise-grade data lake architecture that meets both operational and analytical needs.

Solution overview

The solution is built around four tightly integrated layers that work together to deliver a scalable, transactional data lake.

Raw data is ingested and stored in Amazon S3, which serves as the foundational storage layer. This layer supports multiple data formats and enables efficient data partitioning through Apache Iceberg’s table format. This ensures that data is organized and accessible from the moment it lands. Then, Amazon EMR takes over as the distributed computing engine, using Apache Spark to process large-scale datasets in parallel, handling the heavy lifting of reading, transforming, and writing data across the lake.

Sitting within the processing layer, dbt drives the transformation logic. It applies SQL-based, version-controlled transformations that convert raw, unstructured data in the S3 raw layer into clean, curated datasets stored back in S3. This maintains ACID compliance and schema consistency throughout.

Finally, the curated data is available for consumption through Amazon Athena, which provides a serverless, one-time querying capability directly on S3. With this, analysts and business users can run interactive SQL queries without managing any infrastructure. Together, these components form a continuous pipeline: data flows from ingestion through distributed processing and structured transformation, ultimately surfacing as reliable, query-ready insights.

Amazon EMR is a cloud-based big data service that streamlines the deployment and management of open source frameworks like Apache Spark, Hive, and Trino. It provides a managed Apache Hadoop environment that organizations can use to process and analyze vast amounts of data efficiently.

Data Build Tool is an open source tool that data teams can use to transform and model data using SQL. It promotes best practices for data modeling, testing, and documentation, streamlining maintenance and collaboration on data pipelines.

Apache Iceberg is an open table format designed for large-scale analytics on data lakes. It supports features like transactions, time travel, and data partitioning, which are essential for building reliable and performant data lakes. By using Iceberg, organizations can maintain data integrity and enable efficient querying and processing of data.

When combined, these three technologies provide a powerful solution for building transactional data lakes. Amazon EMR provides the scalable and managed infrastructure for running big data workloads, dbt enables efficient data modeling and transformation, and Apache Iceberg provides data consistency and reliability within the data lake.

Prerequisites

Before proceeding with the solution walkthrough, make sure that the following are in place:

AWS Account – An active AWS account with sufficient permissions to create and manage EMR clusters, S3 buckets, Athena workgroups, and AWS Glue Data Catalog resources
IAM Roles – The following IAM roles must exist and have appropriate permissions:
- EMR_DefaultRole – Service role for Amazon EMR
- EMR_EC2_DefaultRole – Amazon Elastic Compute Cloud (Amazon EC2) instance profile for EMR nodes
AWS Command Line Interface (AWS CLI) – Installed and configured with credentials for your target AWS account and AWS Region (refer to Step 1.1 for setup instructions)
Python 3.8+ – Installed on your local machine or workspace for setting up the dbt virtual environment
Pip – Python package manager available for installing dbt and its dependencies
Git – Installed on the EMR primary node or local environment for version control and dbt package management
Amazon Athena – Athena query editor access with a configured S3 output location for query results
AWS Glue Data Catalog – Enabled as the metastore for Amazon EMR and Athena (no additional setup required if using the default AWS Glue integration)
S3 Bucket Naming – Prepare a unique identifier to suffix S3 bucket names, ensuring global uniqueness across all three buckets created in Step 1.3
Network Access – Make sure that your local machine can reach the Amazon EMR primary node’s DNS over port 10001 (Thrift/HiveServer2) for dbt connectivity; configure security groups accordingly

Solution walkthrough

Step 1: Environment setup

Install the AWS CLI on your workspace by following the instructions in Installing or updating the latest version of the AWS CLI. To configure AWS CLI interaction with AWS, refer to Quick setup.

Create EMR cluster.

Create the following JSON file with the following contents emr-config.json:

[
  {
    "Classification": "iceberg-defaults",
    "Properties": {
      "iceberg.enabled": "true"
    }
  },
  {
    "Classification": "spark-hive-site",
    "Properties": {
      "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
    }
  }
]

Run the following command on your AWS CLI, updating the preferred AWS Region:

aws emr create-cluster \
--name "Iceberg-DBT-Cluster" \
--release-label emr-7.7.0 \
--applications Name=Spark Name=Hive Name=Livy \
--ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
--instance-type c3.4xlarge \
--instance-count 1 \
--service-role EMR_DefaultRole \
--configurations file://emr-config.json \
--region [region]

Set up S3 buckets.
Create the following S3 bucket using the AWS CLI after updating the bucket name.

aws s3 mb s3://amzn-s3-demo-transactional-datalake-raw-[unique-identifier]
aws s3 mb s3://amzn-s3-demo-transactional-datalake-curated-[unique-identifier]
aws s3 mb s3://amzn-s3-demo-transactional-datalake-analytics-[unique-identifier]

Step 2. Raw layer implementation

The raw layer serves as the foundation of our data lake, ingesting and storing data in its original form. This layer is important for maintaining data lineage and enabling reprocessing if needed. We use Apache Iceberg tables to store our raw data, which provides benefits such as ACID transactions, schema evolution, and time travel capabilities.

In this step, we create a dedicated database for our raw data and set up tables for customers, products, and sales using Amazon Athena. These tables are configured to use the Iceberg table format and are compressed using the ZSTD algorithm to optimize storage. The LOCATION property specifies where the data will be stored in Amazon S3 so that data is organized and accessible.

After creating the tables, we insert sample data to simulate real-world scenarios. We use this data throughout the rest of the implementation to demonstrate the capabilities of our data lake architecture.

Update the respective bucket name in each create table bucket name from the previous step:

Create database and tables

-- Create Raw Database
CREATE SCHEMA raw_sales_analytics_data_layer;

-- Create Customers Table
CREATE TABLE raw_sales_analytics_data_layer.customers (
    CustomerID string,
    CustomerName string,
    Region string,
    inserted_timestamp timestamp
)
LOCATION 's3://[bucket_name]/raw_sales_analytics_data_layer/customers'
TBLPROPERTIES (
    'table_type'='iceberg', 
    'write_compression'='zstd'
);

-- Create Products Table
CREATE TABLE raw_sales_analytics_data_layer.products (
    productid string,
    productname string,
    category string,
    supplier string,
    inserted_timestamp timestamp
)
LOCATION 's3://[bucket_name]/raw_sales_analytics_data_layer/products'
TBLPROPERTIES (
    'table_type'='iceberg', 
    'write_compression'='zstd'
);

Insert sample data

-- Insert Customers
INSERT INTO raw_sales_analytics_data_layer.customers
VALUES 
    ('201', Jane Doe', 'Central', current_timestamp),
    ('202', Arnav Desai, 'North', current_timestamp),
    ('203', Kwaku Mensah, 'West', current_timestamp);

-- Insert Products
INSERT INTO raw_sales_analytics_data_layer.products
VALUES
    ('1', 'Laptop', 'Electronics', 'AnyAuthority', current_timestamp),
    ('2', 'Smartphone', 'Electronics', 'AnyCompany', current_timestamp);

-- Insert Sales
INSERT INTO raw_sales_analytics_data_layer.sales
VALUES
    ('ORD001', '1', '201', '2025-04-01', 1299.99, current_timestamp),
    ('ORD002', '2', '202', '2025-04-02', 899.99, current_timestamp);

Step 3: dbt setup and configuration

Setting up dbt involves installing the necessary packages, configuring the connection to the data warehouse (in this case, Amazon EMR), and setting up the project structure.

We start by creating a Python virtual environment to isolate our dbt installation. Then, we install dbt-core and the Spark adapter, which allows dbt to connect to the EMR cluster. The profiles.yml file is configured to connect to the EMR cluster using the Thrift protocol, while the dbt_project.yml file defines the overall structure of the dbt project, including model materialization strategies and file formats.

Install prerequisites

# Create Python virtual environment
python -m venv dbt-env
source dbt-env/bin/activate

# Install required packages
pip install dbt-core dbt-spark[PyHive]

# Install git
yum install git

Configure dbt profiles

# ~/.dbt/profiles.yml
sales_analytics:
  target: dev
  outputs:
    dev:
      type: spark
      method: thrift
      host: your-emr-master-dns
      port: 10001
      schema: curated_sales_analytics_data_layer
      threads: 4

Project configuration

# dbt_project.yml
name: 'sales_analytics'
version: '1.0.0'
config-version: 2

profile: 'sales_analytics'

model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]

target-path: "target"
clean-targets:
    - "target"
    - "dbt_packages"

models:
  sales_analytics:
    dim:
      +materialized: table
      +file_format: iceberg
    ads:
      +materialized: table
      +file_format: iceberg

Step 4: dbt models implementation

In this step, we implement dbt models, which define the transformations that we will apply to raw data. We start by configuring data sources in the sources.yml file, which allows dbt to reference raw tables easily.

We then create dimension models for customers and products, and a fact model for sales.

These models use incremental materialization strategies to efficiently update data over time. The incremental strategy processes only new or updated records, significantly reducing the time and resources required for each run.

Source configuration

# models/sources.yml
version: 2
sources:
  - name: raw_sales
    database: raw_sales_analytics_data_layer
    schema: raw_sales_analytics_data_layer
    tables:
      - name: customers
        columns:
          - name: CustomerID
            tests:
              - unique
              - not_null
      - name: products
      - name: sales

Dimension models

-- models/dim/dim_customers.sql
{{ config(
    materialized='incremental',
    unique_key='customerid',
    incremental_strategy='merge'
) }}

WITH source_data AS (
    SELECT 
        customerid,
        customername,
        region,
        inserted_timestamp,
        ROW_NUMBER() OVER (
            PARTITION BY customerid 
            ORDER BY inserted_timestamp DESC
        ) as row_number
    FROM {{ source('raw_sales_analytics_data_layer', 'customers') }}
    {% if is_incremental() %}
    WHERE inserted_timestamp > (SELECT MAX(inserted_timestamp) FROM {{ this }})
    {% endif %}
)

SELECT 
    customerid,
    customername,
    region,
    inserted_timestamp
FROM source_data
WHERE row_number = 1

Product models

-- models/dim/dim_products.sql
{{ config(
    materialized='incremental',
    unique_key='productid',
    incremental_strategy='merge'
) }}

WITH source_data AS (
    SELECT
        productid,
        productname,
        category,
        supplier,
        inserted_timestamp,
        ROW_NUMBER() OVER (
            PARTITION BY productid
            ORDER BY inserted_timestamp DESC
        ) as row_number
    FROM {{ source('raw_sales_analytics_data_layer', 'products') }}
    {% if is_incremental() %}
    WHERE inserted_timestamp > (SELECT MAX(inserted_timestamp) FROM {{ this }})
    {% endif %}
)

SELECT
    s.productid,
    s.productname,
    s.category,
    s.supplier,
    s.inserted_timestamp
FROM source_data s
WHERE s.row_number = 1
{% if is_incremental() %}
    AND NOT EXISTS (
        SELECT 1
        FROM {{ this }} t
        WHERE t.productid = s.productid
        AND t.inserted_timestamp >= s.inserted_timestamp
    )
{% endif %}

Fact models

-- models/dim/fact_sales.sql
{{ config(
    materialized='incremental',
    unique_key='orderid',
    incremental_strategy='merge'
) }}

WITH source_data AS (
    SELECT
        orderid,
        productid,
        customerid,
        date,
        salesamount,
        inserted_timestamp,
        ROW_NUMBER() OVER (
            PARTITION BY orderid
            ORDER BY inserted_timestamp DESC
        ) as row_number
    FROM {{ source('raw_sales_analytics_data_layer', 'sales') }}
    {% if is_incremental() %}
    WHERE orderid NOT IN (SELECT orderid FROM {{ this }})  -- Changed condition
    {% endif %}
)

SELECT
    s.orderid,
    s.productid,
    s.customerid,
    s.date,
    s.salesamount,
    s.inserted_timestamp
FROM source_data s
WHERE s.row_number = 1

Step 5: Analytics layer

The analytics layer builds upon dimension and fact models to create more complex analyzes. In this step, we create a daily sales analysis model that combines data from fact_sales, dim_customers, and dim_products models.

We also implement a customer insights model that analyzes purchase patterns across different Regions and product categories.

These analytics models demonstrate how we can use our transformed data to generate valuable business insights. By materializing these models as Iceberg tables, we make sure that they benefit from the same ACID transactions and time travel capabilities as our raw and transformed data.

Daily sales analysis

The analytics layer introduces a fact_sales_analysis model that consolidates transactional sales data with customer and product dimensions to enable business-ready reporting. Built as an incremental model with a merge strategy, it efficiently processes data by deduplicating records using the latest inserted timestamp per order, enabling reliable downstream consumption without full table refreshes.

-- models/ads/fact_sales_analysis.sql
{{ config(
    materialized='incremental',
    unique_key='orderid',
    incremental_strategy='merge'
) }}

WITH source_data AS (
    SELECT
        s.orderid,
        s.date,
        s.salesamount,
        c.customername,
        c.region,
        p.productname,
        p.category,
        p.supplier,
        s.inserted_timestamp,
        ROW_NUMBER() OVER (
            PARTITION BY s.orderid
            ORDER BY s.inserted_timestamp DESC
        ) as row_number
    FROM {{ ref('fact_sales') }} s
    JOIN {{ ref('dim_customers') }} c ON s.customerid = c.customerid
    JOIN {{ ref('dim_products') }} p ON s.productid = p.productid
    {% if is_incremental() %}
    WHERE s.orderid NOT IN (SELECT orderid FROM {{ this }})
    {% endif %}
)

SELECT
    s.orderid,
        s.date,
        s.salesamount,
        s.customername,
        s.region,
        s.productname,
        s.category,
        s.supplier,
        s.inserted_timestamp
FROM source_data s
WHERE s.row_number = 1

Customer insights

The customer_purchase_patterns model aggregates sales activity across customer Regions and product categories to surface revenue trends and buying behavior. Materialized as an Iceberg table in the analytics schema, it provides a performant and scalable foundation for customer segmentation, Regional performance analysis, and category-level revenue attribution.

-- models/analytics/customer_purchase_patterns.sql
{{
    config(
        materialized='table',
        file_format='iceberg',
        schema='analytics'
    )
}}

SELECT
    dc.Region,
    dp.category,
    COUNT(DISTINCT fs.orderid) as total_orders,
    COUNT(DISTINCT dc.customerid) as unique_customers,
    SUM(fs.salesamount) as total_revenue,
    SUM(fs.salesamount) / COUNT(DISTINCT dc.customerid) as revenue_per_customer
FROM {{ ref('fact_sales') }} fs
JOIN {{ ref('dim_customers') }} dc ON fs.customerid = dc.customerid
JOIN {{ ref('dim_products') }} dp ON fs.productid = dp.productid
GROUP BY dc.Region, dp.category

Step 6: Transactional operations and time travel with Apache Iceberg

This section demonstrates how to use Apache Iceberg’s time travel capabilities and transactional operations using actual snapshot data from our dim_customers table. We walk through querying data at different points in time and comparing changes between snapshots.

Transactional capabilities
Let’s first look at current data:

Now, modify the raw layer data for customerid 201 and change the Region to East

Run the dbt model for dim_customers to sync the changes

Validate the data in curated layer for dim_customers dimension table
Time-travel capabilities
First, let’s fetch snapshots for customers dimension table in curated layer

Now, find the data state before and after the modification.

Step 7: Data quality tests

Data quality is a critical pillar of any reliable data pipeline. In this step, we define and enforce quality checks directly within the dbt project using schema-level test configurations. Rather than relying on one-time validation scripts, with dbt’s built-in testing framework, we can declaratively specify expectations on our models, ensuring that key fields remain unique, non-null, and consistent across the data layer before they reach downstream consumers.

Generic tests configuration

The schema.yml file serves as the central contract for model integrity. Here, we apply generic tests on the fact_sales and dim_customers models to catch data anomalies early in the pipeline.

# models/schema.yml
version: 2

models:
  - name: fact_sales
    columns:
      - name: orderid
        tests:
          - unique
          - not_null
      - name: salesamount
        tests:
          - not_null

  - name: dim_customers
    columns:
      - name: customerid
        tests:
          - unique
          - not_null

Step 8: Maintenance procedures

A well-functioning data pipeline requires ongoing maintenance to remain performant and auditable over time. This step covers two essential practices, table optimization to keep data storage efficient, and snapshot management to track historical changes in source data. Together, these procedures keep the pipeline reliable, cost-effective, and capable of supporting time-based analysis.

Table optimization
As data accumulates in Delta or Iceberg tables, small files and fragmented storage can degrade query performance. The optimize_table macro provides a reusable utility to run Databricks’ OPTIMIZE command on any target table, consolidating small files and improving read efficiency without manual intervention.
```
-- macros/optimize_table.sql
{% macro optimize_table(table_name) %}
    {% set query %}
        OPTIMIZE {{ table_name }}
    {% endset %}
    {% do run_query(query) %}
{% endmacro %}
```

Snapshot management

To maintain a historical record of customer data changes, we use dbt snapshots with a timestamp-based strategy. The customers_snapshot model captures row-level changes from the raw source layer and persists them in a dedicated snapshots schema, enabling point-in-time analysis and audit trails.

-- snapshots/customer_snapshot.sql
{% snapshot customers_snapshot %}
{{
    config(
      target_schema='snapshots',
      unique_key='CustomerID',
      strategy='timestamp',
      updated_at='inserted_timestamp'
    )
}}

SELECT * FROM {{ source('raw_sales_analytics_data_layer', 'customers') }}

{% endsnapshot %}

Step 9: Monitoring and logging

Observability is an essential aspect of any production-grade data pipeline. This step establishes logging and monitoring practices within the dbt project to track pipeline runs, capture errors, and support debugging. With structured logging enabled, teams gain visibility into model execution, test results, and runtime behavior, streamlining issue diagnosis and maintaining operational confidence.

dbt logging configuration
The dbt_project.yml logging configuration directs dbt to write logs to a dedicated path and outputs them in JSON format. JSON-structured logs are particularly useful for integration with log aggregation tools and monitoring dashboards, enabling automated alerting and audit trail management.
```
# dbt_project.yml
logs:
  path: logs
  enable_json: true
```

Step 10. Deployment and running

With the pipeline fully built, tested, and maintained, the final step covers how to deploy and execute dbt models across different scenarios. Whether running a complete refresh, processing incremental updates, or validating data quality, these commands form the operational backbone of day-to-day pipeline management.

Full refresh
A full refresh rebuilds all models from scratch, reprocessing the entire dataset. This is typically used after significant schema changes, backfills, or when incremental state needs to be reset.
```
dbt run --full-refresh
```
Incremental update
For routine pipeline runs, incremental updates process only new or changed data, significantly reducing compute time and cost. The following command targets specific models (dim_customers and fact_sales) allowing selective execution without triggering the full DAG.
```
dbt run --select dim_customers fact_sales
```
Testing
After models are run, data quality tests defined in the schema configuration are executed to validate integrity across all models. This validates that constraints such as uniqueness and non-null checks are met before data reaches downstream consumers.
```
dbt test
```

Step 11. Cleanup

Infrastructure cleanup

# Delete EMR cluster
aws emr terminate-clusters --cluster-id <cluster-id>

# Remove S3 buckets
aws s3 rb s3://amzn-s3-demo-transactional-datalake-raw-bucket-[unique-identifier] --force
aws s3 rb s3://amzn-s3-demo-transactional-datalake-curated-bucket-[unique-identifier] --force
aws s3 rb s3://amzn-s3-demo-transactional-datalake-analytics-bucket-[unique-identifier] --force

Database cleanup

DROP SCHEMA raw_sales_analytics_data_layer CASCADE;
DROP SCHEMA curated_sales_analytics_data_layer CASCADE;

Conclusion

In this post, you learned how to build a transactional data lake on Amazon EMR using dbt and Apache Iceberg, from environment setup and modeling raw data, to quality enforcing, snapshot management, and incremental pipeline deployment. The architecture brings together the scalability of Amazon EMR, dbt’s transformation capabilities, and Iceberg’s ACID-compliant table format to deliver a reliable, maintainable, and cost-efficient data platform.

To get started, see the Amazon EMR documentation to deploy this architecture in your own environment. Whether you’re modernizing a legacy data platform or building a new analytics foundation, this stack gives you the flexibility to scale with confidence.

About the authors

Securely connect Kafka clients running outside AWS to Amazon MSK with IAM Roles Anywhere

2026-03-16 Kalyan Janaki

Post Syndicated from Kalyan Janaki original https://aws.amazon.com/blogs/big-data/securely-connect-kafka-clients-running-outside-aws-to-amazon-msk-with-iam-roles-anywhere/

Kafka clients that are running outside of AWS (on-premises environment or other clouds) would require an IAM user with a long-lived access key to be provided as part of their codebase or in their server configuration. From a security perspective, there is an additional risk if anyone gains access to those long-term credentials, as they would have access to that AWS account.

In this post, we demonstrate how to use AWS IAM Roles Anywhere to request temporary AWS security credentials, using x.509 certificates for client applications which enables secure interactions with an Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster. The solution described in this post is compatible with both Amazon MSK Provisioned and Serverless clusters.

Introduction to AWS IAM Roles Anywhere

AWS Identity and Access Management (IAM) Roles Anywhere allows you to obtain temporary security credentials in IAM for workloads running outside of AWS, such as servers, containers, and applications.

By using IAM Roles Anywhere, your workloads can utilize the same IAM policies and roles used by AWS applications to access AWS resources. This eliminates the need to manage long-term credentials for kafka clients running outside AWS. By associating one or more roles with a profile and enabling IAM Roles Anywhere to assume these roles, your applications can employ the client certificate issued by your Certificate Authorities (CAs) to securely initiate requests to AWS. Consequently, your applications obtain temporary credentials, granting them access to the AWS environment.

IAM access control for Amazon MSK allows you to manage both authentication and authorization for your Amazon MSK cluster at no extra cost. This eliminates the necessity of using separate mechanisms for authentication and authorization. We recommend Amazon MSK customers use IAM Access Control unless they have a specific need for using mutual TLS or SASL/SCRAM authN/Z.

In the following sections, we show you how to implement a secure Kafka client machine with a detailed step-by-step tutorial using an AWS IAM Roles Anywhere to connect with a MSK Cluster.

Solution overview

The following diagram illustrates the solution architecture.

Architecture diagram showing a hybrid AWS setup where an on-premises MSK client connects to Amazon MSK Provisioned and Serverless clusters via AWS Direct Connect or VPN, using IAM Roles Anywhere, AWS STS, Route 53, and VPC endpoints for secure, private Kafka connectivity.

The flow of the architecture is as follows:

The session token query from your client machine is directed to an AWS IAM Roles Anywhere endpoint, facilitated by the exchange of X.509 certificates.
IAM Roles Anywhere validates the certificate and retrieves a temporary session token from STS, which is then returned to the client machine.
In Amazon MSK Provisioned, the MSK client machine connects to the AWS Transit Gateway or AWS Network Load Balancer in your VPC over AWS VPN or AWS Direct Connect. For more information, refer to Secure connectivity patterns to access Amazon MSK.
In Amazon MSK Serverless, the MSK client machine connects to the interface VPC endpoint in your VPC over AWS VPN or AWS Direct Connect. For more information, refer to Connect to Amazon MSK Serverless from your on-premises network.
In Amazon MSK Serverless , the interface endpoint is a collection of one or more elastic network interfaces with a private IP address within your account. It serves as the entry point for traffic directed towards a MSK Serverless service.

Prerequisites

The instructions provided in this post assume that you are already acquainted with the process of creating an MSK serverless cluster and a client machine. Furthermore, it is presumed that you have successfully accomplished the following tasks:

Create an Amazon MSK serverless cluster or Create an Amazon MSK Provisioned Cluster
Create a MSK client machine in your on-prem data center or a VPC from another AWS account.
Establish network connectivity between on premises and the Amazon MSK Serverless Cluster or Establish network connectivity between on premises and the Amazon MSK Provisioned Cluster

Configure AWS IAM Roles Anywhere

To enable IAM Roles Anywhere for your on-premises Kafka client machine, you must configure two essential components in AWS Roles Anywhere: the trust anchor and the profile. The trust anchor establishes the trust relationship between Roles Anywhere and your certificate authority. This trust is utilized for authenticating certificates to obtain credentials for an IAM role. Profiles are predefined sets of permissions that are applied once successful authentication with Roles Anywhere has been achieved.

Step 1: Generate a CA

An X.509 certificate plays an important role in facilitating communication between the client machine and Roles Anywhere. You can use Public Key Infrastructure (PKI) platform of your choice to establish a certificate authority (CA).

If you prefer to generate your own X.509 client certificate, you can refer to the instructions outlined in IAM Roles Anywhere with an external certificate authority to guide you through the process.

For simplicity of this example, we use an AWS Private CA:

Navigate to the AWS Private CA console.

Create a Root CA

Choose Root as CA type option and put your organization name and organization unit name.
Choose default RSA 2048 key algorithm.
Choose Create CA button to generate a private the CA and install the certificate.

Create a Subordinate CA

Choose Subordinate as CA type option.
Choose default RSA 2048 key algorithm.
Choose Create CA button.
Obtain the CSR from the subordinate CA and have it signed by the root CA.

This CA will be used for issuing certificates to IAM Roles Anywhere.

For generating a more secured and auto-renewed AWS private CA, refer to Procedure for creating a CA and How to build a CA hierarchy.

Step 2: Configure anchor

Go to Roles Anywhere console and open the Create a trust anchor page.
Provide a name for your trust anchor and select the private CA that we created in step 1. If you prefer to use your own external CA, choose the External certificate bundle option and provide the necessary certificate bundle.
Choose create a trust anchor button to finish the process.

Step 3: Create and configure a role that trusts IAM Roles Anywhere

Now we create a role that you want your on-premises Kafka client machine to assume after authenticating to IAM Roles Anywhere.

The trust policy of the role should contain the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "rolesanywhere.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:SetSourceIdentity",
        "sts:TagSession"
      ],
      "Condition": {
        "StringEquals": {
          "aws:PrincipalTag/x509Subject/CN": "specific-certificate-common-name"
        }
      }
    }
  ]
}

For this demo, create the following policy and attach it to the role:

{
    "Version":"2012-10-17",		 	 	 
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": [
                "arn:aws:kafka:<Region>:<Account-ID>:cluster/<Cluster-name>/<Cluster-identifier>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:CreateTopic",
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData",
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": [
                "arn:aws:kafka:<Region>:<Account-ID>:cluster/ <Cluster-Name>/<Cluster-identifier>",
                "arn:aws:kafka:<Region>:<Account-ID>:topic/msk-<Cluster-Name>/<Cluster-Identifier>/<Topic-Name> ",
                "arn:aws:kafka:<Region>:<Account-ID>:group/<Cluster-Name>/<Cluster-Identifier>/<Group-Name>"
            ]
        }
    ]
}

Step 4: Setup profile

Navigate back to the Roles Anywhere console.
Under Profiles, choose Create a profile.
Enter a name for the profile.
Select the role we created in Step 3 and create the Roles Anywhere profile.

Step 5: Test the client machine

Now that we have successfully set up Roles Anywhere by creating a trust anchor and a profile, the next step is to test the client machine’s communication with Roles Anywhere. This involves retrieving a session token and establishing communication with the MSK broker.

Request a private certificate from the CA we created in Step 1 and export the client certificates to be used in the client machine.
Create a .pem file and copy all the certificate contents into this .pem file(e.g. private_key. pem) and run below command to generate a decrypted version of certificate.
```
openssl rsa -in private_key.pem -out decrypted_private_key.pem
```
Download the credential helper and use this signing helper tool to test and confirm the functionality from your client machine. We offer the ARNs of the trust anchor and profile of Roles Anywhere, and the role we created in IAM.
```
./aws_signing_helper credential-process \
--certificate /path/to/certificate.pem \
--private-key /path/to/decrypted_private_key.pem \
--trust-anchor-arn <TA_ARN> \
--profile-arn <PROFILE_ARN> \
--role-arn <Roles_ARN> \
--region <Region>
```
You should receive the session credentials successfully from IAM Roles Anywhere.

After verifying the successful setup, proceed to update or create the ~/.aws/config file. Add the signing helper as a credential_process in this file to enable unattended access for the on-premises server.
[default]

credential_process = ./aws_signing_helper credential-process 
--certificate /path/to/certificate.pem 
--private-key /path/to/decrypted_private_key.pem 
--trust-anchor-arn <TA_ARN> 
--profile-arn <PROFILE_ARN> 
--role-arn <Roles_ARN>
--region <Region>

Once all steps are done, you should be able to see the Kafka client communicating to the MSK broker.

./kafka-topics.sh --create \
--bootstrap-server <BOOTSTRAP_SERVER> \
--command-config <Command Config File> \
--replication-factor <Replication Factor> \
--partitions <Number of Partitions> \
--topic <Topic Name>

Clean up

To stop incurring costs, it is recommended to manually delete the IAM Role, Profile, Trust Anchor, Policies, requested certificate in ACM and created certificates in AWS Private CA.

aws delete-role --role-name <value>

aws delete-profile --profile-id <value>

aws delete-trust-anchor --trust-anchor-id <value>

aws acm delete-certificate --certificate-arn <value>

aws acm-pca revoke-certificate --certificate-authority-arn <value> --certificate-serial <value> --revocation-reason <value> 

aws acm-pca delete-certificate --certificate-authority-arn <value> --certificate-serial <value>

Conclusion

In this post, we showed you how to utilize AWS IAM Roles Anywhere to generate temporary session tokens for accessing MSK brokers from client machines outside of AWS. By implementing this approach, the security posture of Kafka clients connecting to MSK from outside of AWS are enhanced, allowing customers with stringent security requirements to confidently adopt MSK.

If you have any questions, you can start a new thread on AWS re:Post or reach out to AWS Support.

About the authors

How to manage the lifecycle of Amazon Machine Images using AMI Lineage for AWS

2026-03-12 George'son Tib

Post Syndicated from George'son Tib original https://aws.amazon.com/blogs/security/how-to-manage-the-lifecycle-of-amazon-machine-images-using-ami-lineage-for-aws/

As organizations scale their cloud infrastructure, maintaining proper lifecycle management of Amazon Machine Images (AMIs) is a critical component of their security and risk management goals. AMIs provide the essential information required to launch Amazon Elastic Compute Cloud (Amazon EC2) instances, however; they present security and compliance challenges if not tracked and managed throughout their lifecycle. This blog post explores how organizations can meet their evolving security and compliance requirements by managing potential vulnerabilities across the AMIs deployed throughout their AWS environment.

At the end of 2024, AWS announced lineage supportfor Amazon EC2, providing source details for your AMIs. With this lineage information, you can trace copied or derived AMIs back to their original source. The source AMI information is available for AMIs that were created using specific API commands like CreateImage, CopyImage, and CreateRestoreImageTask. If the AMI was created using a different API command, the ID and AWS Region of the source AMI don’t appear, which can create visibility gaps that potentially impact security and compliance efforts.

To address these gaps and provide comprehensive AMI governance, organizations need to build additional capabilities to analyze the scope of impact of Common Vulnerabilities and Exposures (CVEs), ensure deployed resources originate from an approved golden image, and respond to audit inquiries that require a clear chain of custody for AMIs. A well-designed solution should also help track and enforce approved AMI creation patterns across all accounts and AWS Regions. The AMI lineage solution described in this post is designed to help you manage your organization’s AMI hierarchy and lifecycle, including tracking AMI origins and usage throughout its AWS environment. By implementing this solution, your security teams can quickly understand the scope of impact when security vulnerabilities are discovered, help ensure compliance with organizational policies, and maintain better visibility into their AMI estate.

The solution in this blog post uses Amazon Neptune, a high-performance graph database, along with native AWS security services to maintain a comprehensive view of AMI relationships and enable proactive security monitoring. With the solution in place, you can enforce controls on AMI sourcing, including validation of marketplace AMIs through service control policies (SCPs), and maintain compliance with organizational and regulatory requirements throughout the AMI lifecycle.

Solution overview

AMI Lineage provides a comprehensive governance solution that uses AWS security services and Neptune to create and maintain a hierarchical graph representation of their AMI relationships. This solution helps security and compliance teams understand the complete history of their AMIs including where they originated from, enforce organizational policies such as requiring all AMIs to be encrypted, and rapidly assess security impacts across their organization.
The solution integrates core AWS services with security and governance capabilities. The core components of the solution in the security tooling account are:

Neptune: A purpose-built, high-performance graph database securely stores and manages the AMI relationship data.
AWS Lambdafunctions serve as the processing engine for the solution. They process AMI lifecycle events (such as CreateImage, CopyImage, DeregisterImage), evaluate them against compliance rules, and update the Neptune graph database. The functions are configured with least-privilege AWS Identity and Access Management (IAM) permissions to enhance security.
Amazon API Gateway provides secure REST endpoints for lineage queries and security assessments. Authentication is handled using a combination of API keys and IAM roles to help ensure that only authorized users and systems can access the data.

From a governance perspective, this solution provides comprehensive AMI origin validation to help ensure AMIs come from approved sources, including the validation of AWS Marketplace AMIs against a list of trusted vendors. Lifecycle management capabilities enforce AMI retention policies and deprecation processes. Compliance monitoring tracks adherence to organizational and regulatory requirements, while security event scope assessment capabilities quickly identify affected resources when security vulnerabilities are discovered. A detailed audit trail maintains a complete history of AMI creation, modification, and usage patterns.

Architecture

The AMI Lineage solution follows AWS security best practices with a multi-account deployment architecture designed to maximize security while maintaining operational efficiency. The architecture distributes responsibilities across three primary account types: an organization management account, a centralized security tooling account, and multiple member accounts.

This architectural approach helps ensure that sensitive operations and data remain centralized in the security tooling account while enabling distributed monitoring and policy enforcement across the organization. The clear separation of concerns enhances security while maintaining the scalability needed for large-scale AWS deployments.

Figure 1: AMI Lineage solution architecture and workflow

The workflow and architecture shown in figure one includes the following:

Policy enforcement: The organization management account is the central point for control. It uses AWS Organizations to enforce SCPs that prevent non-compliant AMI actions across the member accounts.
Event capture: When an AMI lifecycle event (like CreateImage or CopyImage) occurs in a member account, a local Amazon EventBridge rule captures it.
Centralized processing: The event is securely forwarded from the member account’s EventBridge to the central EventBridge in the security tooling account.
Data ingestion and analysis: A Lambda function is triggered in the security tooling account. This function processes the event, analyzes it for compliance, and updates the Neptune graph database with the new AMI relationship data. AWS Security Hub and Amazon GuardDuty in the security tooling account also receive and analyze findings from member accounts.
Query and visualization: Security teams query the lineage data through a secure API Gateway endpoint. By doing this, they can to visualize AMI hierarchies, investigate security findings from Security Hub, and assess the scope of impact for a given AMI.

The organization management account serves as the central control point for policy enforcement and organizational oversight. This account hosts SCPs that prevent non-approved AMI usage across the organization and manages organization-wide EventBridge rules that capture AMI events from member accounts. Cross-account trust policies configured in this account enable secure communication between the management account and the security tooling account.

Additionally, the management account establishes Security Hub in delegated administrator mode, designating the security tooling account as the centralized security administrator for the organization. From the security tooling account, Security Hub can be then configured to aggregate all Regions down to one core Region for easier evaluation by security personnel.

The security tooling account acts as the central hub for AMI lineage processing and storage. This account hosts the Neptune graph database cluster with encrypted storage, helping to ensure that AMI relationship data is securely maintained. Lambda functions running in this account process events, handle API requests, and evaluate compliance with least-privilege permissions. API Gateway provides secure REST endpoints for lineage queries and security assessments. Security Hub custom insights and findings are centralized here in the security tooling account as the Security Hub delegated administrator account, along with Amazon Simple Notification Service (Amazon SNS) topics for notifications and alerts. The Amazon Virtual Private Cloud (Amazon VPC) infrastructure supporting these services is also deployed in the security tooling account, providing network-level isolation and security.

The solution enables distributed monitoring and enforcement by deploying lightweight components into each member account across the organization. Each member account includes AWS Config rules for continuous compliance monitoring, cross-account IAM roles to enable secure access from the security tooling account, and local EventBridge rules that forward AMI-related events to the central processing system.

Security and compliance integration extends throughout the solution. IAM manages least-privilege access control and permissions across components. AWS CloudTrail records API activity for audit trails and compliance reporting, while Security Hub centralizes security findings and compliance status across your AMI estate. GuardDuty provides threat detection for AMI-related activities. SCPs enforce organization-wide controls on AMI creation and usage patterns, and AWS Config tracks AMI configuration changes and evaluates compliance rules.

How it works

The AMI Lineage solution operates through a continuous monitoring and automated response system that maintains comprehensive visibility into your AMI landscape. When AMI lifecycle events occur in your organization, EventBridge rules capture these activities, including creation, copying, modification, and deregistration events. Lambda functions in the security tooling account are then called upon to process these events with appropriate security controls and update the Neptune graph database in real-time, while CloudTrail logs provide a comprehensive audit trail of AMI-related activities.

The system tracks critical security and compliance metadata that forms the foundation of effective AMI governance. This includes:

Source AMI information and validation status to help ensure lineage integrity
Creation method and timestamp data for comprehensive audit trails
Cross-Region and cross-account relationships to understand the full scope of AMI distribution
Instance launch history with security context to track usage patterns
AMI state changes including deprecation and deregistration for lifecycle management
Compliance status along with policy violations to maintain organizational standards.

Security teams use this comprehensive data through secure API calls to visualize complete AMI hierarchies and relationships, providing clear insight into how AMIs are related across your infrastructure. The compliance of your AMI estate is continuously tracked through a combination of services:

Detection: AWS Config rules deployed in member accounts check for policy violations (for example, incorrect tags and public permissions).
Aggregation: These findings, along with vulnerability data from services like Amazon Inspector, are aggregated in AWS Security Hub.
Correlation: Lambda functions in the security tooling account correlate this information with the lineage data in Neptune. Because of this correlation, you can see not just that an AMI is non-compliant, but also its entire downstream impact. When security events like CVE findings are discovered, teams can quickly assess the scope of impact across their entire AMI estate. The solution monitors AMI usage patterns for security anomalies and enforces governance controls through automated policy checks.

The solution provides robust automated policy enforcement capabilities that operate continuously to maintain security and compliance. The system helps ensure that only approved AMIs with verified lineage history can be used to launch new instances, automatically blocking attempts to use non-compliant images. SCP controls on AMI creation and usage are enforced organization-wide, preventing unauthorized AMI operations before they can impact your environment. When policy violations are detected, the system can trigger automated responses to security events and maintain compliance with organizational standards through real-time enforcement.

Implementation

Before deploying the AMI Lineage solution, you need to establish the proper security and governance foundation across your organization. Your AWS Organizations management account requires administrative permissions, and your organization must be enabled with all features to support the policies used in this solution. You will also need a dedicated security tooling account to host the solution’s core components, with cross-account IAM roles configured to allow secure access. Finally, essential security services must be configured at the organization level, including Security Hub, CloudTrail organization trails for audit logging, and encryption keys using AWS Key Management Service (AWS KMS) for data protection.

From a technical perspective, ensure you have Python 3.8 or later installed if deploying from a local environment, along with AWS Command Line Interface (AWS CLI) version 2 installed and configured with appropriate security credentials. You’ll also need an Amazon Simple Storage Service (Amazon S3) bucket for deployment artifacts, encrypted using SSE-KMS with a customer-managed key to align with best practices for protecting deployment assets.

The complete AMI Lineage solution is available as open source code in the AWS Samples repository. You can clone the repository and follow the deployment instructions. The repository includes the necessary AWS CloudFormation templates, Lambda functions, and deployment scripts referenced in the following phases.

Deployment

The deployment process follows a five-phase approach that builds security and compliance capabilities progressively:

Security foundations
Security controls
EventBridge rules
Core infrastructure
Compliance and monitoring

Phase 1 – Establishing security foundations

The first phase establishes the security foundation by configuring AWS Organizations security services. This involves enablingSecurity Hub in the management account and designating the security tooling account as the delegated administrator, enablingnullGuardDuty with the security tooling account configured as thenulldelegated administrator, and enabling an organizational wide CloudTrail trail for audit logging.

# In Organization Management Account: 
# Enable Security Hub and set security tooling account as delegated admin 
aws securityhub enable-organization-admin-account \   
--admin-account-id <security-tooling-account-id> 

# Enable GuardDuty organization with security tooling account as admin   
aws guardduty enable-organization-admin-account \   
--admin-account-id <security-tooling-account-id> 

# Create organization trail with encryption aws cloudtrail create-trail \   
--name ami-lineage-trail \   
--s3-bucket-name <your-secure-bucket> \   
--is-organization-trail \   
--kms-key-id <your-kms-key-id> \   
--enable-log-file-validation

Phase 2 – Security controls

The second phase deploys base security controls through organization-wide SCPs. These policies enforce AMI governance controls by preventing the use of non-approved AMIs and helping to ensure that proper tagging and approval workflows are followed.

# In Organization Management Account: 
# Deploy organization-wide SCPs 
aws organizations create-policy \   
--content file://ami-governance-scp.json \   
--name "AMI-Governance-Controls" \   
--type SERVICE_CONTROL_POLICY 

# Attach to organizational units 
aws organizations attach-policy \   
--policy-id <policy-id> \   
--target-id <ou-id>

Phase 3 – EventBridge rules

The third phase deploys organization-wide EventBridge rules from the management account to capture AMI events across member accounts and forward them to the security tooling account for processing. These rules listen for specific API calls captured by CloudTrail.

An example of the event pattern used to capture CreateImage and CopyImage events looks like this:

{
	"source": ["aws.ec2"],
	"detail-type": ["AWS API Call via CloudTrail"],
	"detail": {
		"eventSource": ["ec2.amazonaws.com"],
		"eventName": [
			"CreateImage",
			"CopyImage",
			"RegisterImage",
			"DeregisterImage"
		]
	}
}

# In Organization Management Account: 
# Deploy organization EventBridge rules 
cd deployment-scripts/organization 
./deploy-organization-resources.sh

Phase 4 – Core infrastructure

The fourth phase focuses on core infrastructure deployment in the security tooling account. This is where the primary processing and storage components are deployed, following security best practices by centralizing sensitive operations in a dedicated account.

# Switch to Security Tooling Account context 
# Deploy Neptune cluster with encryption in security tooling account 
cd deployment-scripts/shared 
./deploy-shared-resources.sh

This deployment script handles multiple components in the security tooling account. The Neptune cluster deployment includes encryption and VPC configuration to help ensure secure storage and access to AMI lineage data. Lambda functions are deployed with security controls and configured with VPC attachment, which allows for secure Neptune access in the VPC, appropriate IAM roles with least-privilege permissions, and environment variables for secure configuration. API Gateway provides secure REST endpoints for external access to AMI lineage data and security assessments.

Phase 5 – Compliance and monitoring

The fifth phase establishes comprehensive compliance and monitoring capabilities across member accounts. AWS Config rules are deployed to continuously monitor AMI compliance across your organization, while EventBridge rules forward AMI events to the central processing system.

# In each Member Account: 
# Deploy AWS Config Rules and monitoring capabilities 
cd deployment-scripts/child-account   
./deploy-child-account-resources.sh

After deployment, thorough verification helps ensure that security configurations are properly implemented. This includes validating IAM permissions to help ensure least-privilege access, testing security controls to verify SCP enforcement, validating encryption settings acrosscomponents, and confirming that the security tooling account is properly configured as the Security Hub delegated administrator.

Using AMI Lineage

When deployed, AMI Lineage provides security operations and compliance monitoring capabilities through its API hosted in the security tooling account and automated monitoring systems. Security teams can query and receive complete AMI security relationships to understand the full context of AMIs in their environment.

When investigating AMIs, the system provides detailed security context including source validation information that confirms:

Whether AMIs come from marketplace sources or trusted accounts
Compliance status that shows patch levels and policy adherence
Vulnerability status with CVE findings and scan results
Comprehensive lineage data showing the complete chain of AMI relationships and approval history

# Get complete security context for an AMI (API Gateway in Security Tooling Account) 
curl -X GET "https://<api-gateway-id>.execute-api.<region>.amazonaws.com/v1/api/v1/ami/ami-1234567890abcdef0/security-context?include_compliance=true" \  
	-H "x-api-key: <your-api-key>"

For security impact assessments, such as when a new CVE is discovered, the solution provides a powerful scope of impact analysis. By querying the API with a specific finding, security teams can rapidly determine every affected resource across their entire organization that stems from a compromised or vulnerable AMI. Using that information, they can understand the full scope of their exposure and begin remediation. See Security best practices in Amazon API Gateway for helpful considerations while using API Keys.

# Assess for a security finding (Security Tooling Account API) 
curl -X POST "https://<api-gateway-id>.execute-api.<region>.amazonaws.com/v1/api/v1/security-impact" \   
	-H "Content-Type: application/json" \   
	-H "x-api-key: <your-api-key>" \   
	-d '{     "ami_id": 
		"ami-1234567890abcdef0",     
		"finding_type": "CVE",     
		"finding_id": "CVE-2024-XXXX",     
		"severity": "CRITICAL"   
	}'

This analysis returns impact information including:

Affected AMIs in the lineage chain
Running instances requiring immediate remediation
Affected AWS accounts and regions for coordinated response
Associated auto-scaling groups and launch templates that need updates
Compliance impact assessment for regulatory reporting
Detailed remediation steps prioritized by risk level.

Compliance monitoring operates continuously through automated assessment capabilities that evaluate your AMI estate against organizational policies and regulatory requirements. Teams can generate comprehensive compliance reports that show adherence to security standards across their entire infrastructure.

# Generate comprehensive compliance report (Security Tooling Account API) 
curl -X POST "https://<api-gateway-id>.execute-api.<region>.amazonaws.com/v1/api/v1/compliance-assessment" \   
	-H "Content-Type: application/json" \   
	-H "x-api-key: <your-api-key>" \   
	-d '{     
		"rules": [       
    		"required_tags",       
    		"approved_source_validation",       
    		"security_scan_status",       
    		"naming_convention",       
    		"lineage_verification"     
		],     
		"scope": "ORGANIZATION"   
	}'

The solution provides security automation and remediation through configurable automated responses to security events. Security Hub, operating in delegated administrator mode from the security tooling account, can be configured to automatically respond to findings by stopping instances using AMIs with critical vulnerabilities, quarantining instances launched from unapproved sources, and sending immediate notifications for high-severity findings.

Security visualization and reporting capabilities, centralized in the security tooling account, provide real-time dashboards showing:

Compliance status across the organization
Scoping visualization for rapid decision-making
AMI approval workflow status for process monitoring
Patch compliance metrics for maintaining security posture
Automated remediation activity logs for audit purposes
Custom security reports tailored to specific organizational needs.

For security investigations and audit purposes, the solution maintains a queryable audit trail that provides a complete history of AMIs, including creation and modification events, security scanning results and findings, approval workflow history, and compliance status changes over time.

# Query comprehensive audit history (Security Tooling Account API) 
curl -X GET "https://<api-gateway-id>.execute-api.<region>.amazonaws.com/v1/api/v1/ami/ami-1234567890abcdef0/lineage?direction=both&depth=10" \   
	-H "x-api-key: <your-api-key>"

Clean up

To decommission the AMI Lineage solution, use the following steps to prevent dependency errors. The process is the reverse of the deployment.

(Optional) Back up your data. Before you begin, export critical data for your audit and compliance records. This includes generating final compliance reports from the API or creating a final snapshot of the Neptune database (you will be prompted to do this when you delete the cluster).
Run cleanup in member accounts. Sign in to each participating member account and run the cleanup script from the deployment files. This removes the local EventBridge rules, AWS Config rules, and cross-account IAM roles.
# In each Member Account cd deployment-scripts/child-account ./cleanup-child-account-resources.sh # Removes Config rules and cross-account roles from each member account
Run cleanup in the security tooling account. Sign in to your security tooling account and run the cleanup script. This decommissions the core solution, including the API gateway, Lambda functions, Neptune cluster, and the associated VPC.
# Clean up security tooling account cd deployment-scripts/shared ./cleanup-shared-resources.sh # Removes Neptune, Lambda, API Gateway, SNS, and Security Hub components
Run cleanup in the organization management account. Sign in to your organization management account to remove the organization-level resources.
1. Run the cleanup script to remove the organization-wide EventBridge rules.
  # Clean up organization management account cd deployment-scripts/organization ./cleanup-organization-resources.sh # Removes SCPs, EventBridge rules, and cross-account trust policies
2. In the AWS Organizations console, detach and delete the AMI-Governance-Controls SCP.
3. In the Security Hub and GuardDuty consoles, remove the security tooling account as the delegated administrator.
Delete final data and encryption keys. After the solution’s infrastructure is removed, you can delete the remaining assets.
1. In the security tooling account,empty and delete the S3 bucket that held the deployment artifacts.
2. In the organization management account,schedule the deletion of the KMS keys you created for encrypting the solution’s data.

Conclusion

In this blog post, we showed you how you can use the AMI Lineage solution to build a comprehensive approach to tracking the complete history of your AMIs from creation to decommissioning. By storing this data in an Amazon Neptune graph database, you can build a hierarchical view of the relationships between your EC2 instances and the AMIs they were launched from. You learned how that data can be used to improve security response and remediation and assist in auditing and compliance activities.

The solution uses AWS Organizations to provide preventative controls to help ensure that only approved AMIs are used and integrates AWS security services like Amazon GuardDuty, AWS Security Hub, and AWS Config to add additional layers of security monitoring and management. Finally, you saw how the solution can be used during a security event or when new CVEs are published, so that you can rapidly discover which systems are affected and automate responses based on those findings.

While this solution provides powerful capabilities, it’s important to consider the operational and cost aspects. The core components, particularly Neptune, have associated costs that will scale with the size of your AMI estate. We recommend implementing cost monitoring and alerts as part of your deployment. Furthermore, because the solution is event-driven, you should plan a one-time backfill process to ingest your organization’s existing AMI history into the graph database. For organizations that require this level of granular control and visibility, these operational considerations are offset by the significant gains in security posture and compliance automation.

AMI Lineage transforms AMI governance from a manual, error-prone process into an automated, comprehensive security capability that scales with your organization’s growth. By implementing this solution, your organization can gain the visibility, control, and automated response capabilities needed to maintain a strong security posture while enabling rapid, secure deployment of infrastructure across its AWS environment.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

How Razorpay achieved 11% performance improvement and 21% cost reduction with Amazon EMR

2026-03-09 Narendra Kumar

Post Syndicated from Narendra Kumar original https://aws.amazon.com/blogs/big-data/how-razorpay-achieved-11-performance-improvement-and-21-cost-reduction-with-amazon-emr/

This is a guest post by Narendra Kumar, Head of Platform – Data at Razorpay, in partnership with AWS.

In this post, we explore how Razorpay, India’s leading FinTech company, transformed their data platform by migrating from a third-party solution to Amazon EMR, unlocking improved performance and significant cost savings. We’ll walk through the architectural decisions that guided this migration, the implementation strategy, and the measurable benefits Razorpay achieved.

Founded in 2014, Razorpay has become a powerhouse in comprehensive payment solutions, enabling businesses to accept, process, and disburse payments online. With offerings like RazorpayX for business banking and Razorpay Capital for lending solutions, the company has experienced explosive growth, now serving millions of businesses. This rapid expansion brought significant data challenges. When Razorpay’s data platform began straining under the weight of more than 1PB daily processing demands, the engineering team faced a critical decision: continue scaling their existing third-party solution or modernize with a platform offering greater flexibility and control. They chose Amazon EMR to build a comprehensive data architecture spanning batch warehousing, real-time stream processing, and interactive analytics – all running on Apache Spark with open-source Delta Lake for ACID transactions. This wasn’t simply an ETL migration; it was a complete platform transformation that gave Razorpay’s 800 daily users access to more than 60 concurrent streaming pipelines, more than 3,000 orchestrated workflows, and the ability to query 6PB of data daily. The results validated their architectural choices: 11% better overall performance, 21% cost reduction, and the operational flexibility to optimize Spark resource allocation, leverage EC2 Spot instances, and implement advanced features like liquid clustering – all without vendor lock-in.

Achieving data insights cost-effectively with AWS

The data architecture has a data ingestion layer, data processing layer, and data consumption layer. Razorpay ingests more than 20 TB of new data every day, processes more than 1 PB of daily data using more than 60 data stream processing pipelines. This data is then consumed by querying more than 6 PB of daily data through more than 3,000 scheduled workflows.

Data flows from a variety of sources such as online transaction processing (OLTP) databases – traditional transactional or entity stores, events such as clickstream and application events, and third-party events like reverse extract, transform, and load (ETL). Most of the data consumption use cases power merchant reporting and internal analytics of the organization. The architecture powers a variety of data science use cases and financial infrastructure around a reconciliation service.

Solution overview

As shown in the following diagram, in its early stages, Razorpay operated on a small scale, using Sqoop to dump transactional data daily into a data lake and managing a Presto layer for querying this data. As they grew, the demand for near real-time data increased, prompting the setup of a change data capture (CDC) collector using Maxwell to stream data manipulation language (DML) events to Kafka. To further enhance data processing, Razorpay built a processing layer that consumed data from Kafka to UPSERT information into the lake using Apache Hudi.

Architecture diagram showing a five-layer big data processing pipeline: data stores feed into Kafka for message streaming, which connects to Apache Spark, Apache Hudi, and Sqoop for stream and batch processing, followed by a data storage and query layer using Apache Hive and Apache Presto, and finally a visualization layer with Looker, redash, and Qubole.

Additionally, the company onboarded data from third-party sources such as Freshdesk and Google Sheets and automated event ingestion from frontend applications using Lumberjack, thereby streamlining their data management processes.

As Razorpay scaled its operations, the demand for multiple real-time use cases became mission-critical, prompting the development of a robust data warehouse ingestion framework to efficiently ingest data into TiDB. To enhance service reliability and support dashboard querying, a low-latency, high-throughput service called Harvester was created, which stored pre-aggregated data for effective monitoring. Over time, reporting use cases emerged, leading to the use of a warehouse service to establish a denormalized report data layer while also exploring a real-time layer for dynamic insights. Additionally, to facilitate a smooth transition to microservices, Razorpay built a unified storage layer capable of supporting data from both its existing monolithic architecture and the new microservices, ensuring seamless integration and improved data accessibility across the organization.

Razorpay implemented a comprehensive data service migration to Amazon EMR using a phased approach. The solution architecture as shown in the following diagram comprises multiple layers handling data ingestion, processing, and consumption.

Technical implementation

A modern and scalable analytics platform focuses on real-time data ingestion, petabyte-scale processing, and cost-optimized storage – all orchestrated with robust workflow management:

Data ingestion layer

To handle large-scale and diverse data sources, they implemented a combination of CDC and file ingestion patterns:

CDC using Amazon Aurora MySQL-Compatible Edition – Used Debezium and Maxwell for low-latency replication and streaming of database changes
High-volume streaming pipelines – Configured streaming pipelines capable of processing more than 20 TB of daily inbound data
Third-party data integration: Implemented secure file push mechanisms to ingest partner and software as a service (SaaS) data into the service

Data processing layer

Razorpay designed the processing stack on Amazon EMR on Amazon Elastic Compute Cloud (Amazon EC2) with Spark as the primary compute engine

Batch warehousing – Daily ETL and aggregation jobs processing more than 1 PB of data
Stream processing – Real-time analytics pipelines across more than 60 concurrent processing streams
Delta merge operations – High-performance incremental updates across more than 25 Delta Lake tables

Data storage and organization

Their data storage follows the medallion architecture pattern layered on an Amazon Simple Storage Service (Amazon S3):

Raw zone – Immutable ingestion zone for original source data
Processed and aggregated zone – Optimized datasets ready for analytics and reporting
Open source software (OSS) Delta Lake format – Implemented open source Delta Lake for ACID transactions, schema enforcement, and faster query performance

Workflow orchestration

Complex data workflows are automated and monitored using a hybrid orchestration approach:

Apache Airflow integration – Scheduling and coordinating more than 3,000 workflows per day
dbt on Amazon EMR – SQL-based transformations for business logic and metric definitions
Specialized compliance jobs – Dedicated workflows meeting the 15-minute SLA for sensitive regulatory reporting

Performance optimizations

To ensure cost efficiency and high throughput, the following optimizations were applied:

Spark tuning – Custom configurations for executor memory, shuffle partitions, and serialization to maximize hardware utilization
Liquid clustering – Implemented in delta lake tables to improve query performance over large datasets
Optimized delta merges – Reduced merge latency for incremental updates.
Auto scaling – Dynamic scaling policies based on workload patterns to balance performance and cost

To enable a secure migration, they implemented Amazon EMR security best practices following AWS guidance on encryption, authentication, and authorization as documented in the Amazon EMR security best practices.

This architecture delivers low-latency ingestion, petabyte-scale processing, and robust workflow orchestration so that analytics teams can derive faster insights while maintaining compliance and optimizing for cost.

The combination of Debezium and Maxwell for CDC, Spark on Amazon EMR, OSS Delta Lake on Amazon S3, and Airflow with dbt has proven to be a scalable and resilient approach for modern data analytics workloads

Business Impact: What Amazon EMR Enabled

11% performance improvement enabling faster insights for 800 daily active users
13-15% faster execution for large warehouse jobs, accelerating time-to-insight for critical business decisions
21% cost reduction reinvested into product innovation for merchant customers
Seamless scaling from 20 TB to 1 PB+ daily processing without performance degradation
Enterprise reliability supporting 350,000 operational reports and compliance requirements

Key learnings and best practices

Throughout their migration to Amazon EMR, Razorpay learned valuable lessons that helped optimize their data platform. We are sharing these insights to help other customers accelerate their own modernization journeys while avoiding common pitfalls.

Infrastructure Stability and Performance

Optimizing Spark Resource Allocation – Razorpay initially assumed that Spark’s dynamic allocation would automatically optimize resource utilization. However, they discovered it introduced overhead that degraded performance for certain workload patterns. To address this challenge, they took two approaches depending on workload characteristics – setting explicit maxExecutors values for predictable workloads, and enabling maximizeResourceAllocation to create “fat executors” that fully utilized available cluster resources. These targeted configurations improved job execution times by 13-15% for large-scale data processing workloads.
Ensuring Stability with Yet Another Resource Negotiator (YARN) node labels – When using EC2 Spot instances for cost optimization, Razorpay encountered a critical issue in which Spot instance interruptions occasionally terminated nodes running critical driver containers, causing entire job failures. Their solution was elegant and effective. They configured YARN node labels to ensure driver containers always spawn on On-Demand Instances, while task nodes use cost-effective Spot capacity. This architecture delivered both cost efficiency and reliability, making their jobs resilient to Spot interruptions while maintaining 21% cost savings.
Managing Spot Instances Effectively – Razorpay’s initial approach of switching entirely to On-Demand Instances during Spot availability constraints eliminated the cost benefits they were seeking. They implemented several best practices to address this such as using instance fleets with allocation strategies (price-capacity optimized and capacity optimized) to maximize Spot availability, spreading primary instances across multiple Availability Zones for fault tolerance, and accepting that heterogeneous executors create varying executor sizes while planning capacity accordingly. They maintained high Spot utilization rates while ensuring workload continuity, achieving optimal price performance.

Cost Optimization

Achieving Sustainable Cost Efficiency – As data volumes grew to more than 20 TB daily, Razorpay needed to scale infrastructure while controlling costs. They implemented a comprehensive cost optimization strategy that included multiple components. First, they right-sized primary nodes by avoiding over-provisioning and selecting instance types matching actual workload requirements. They consolidated workloads by combining multiple jobs on fewer large clusters to maximize resource utilization. For SLA-sensitive jobs, they migrated to Amazon EKS and Amazon EMR Serverless for automatic scaling and pay-per-use pricing. They adopted Graviton instances, migrating compatible workloads to AWS Graviton processors for superior price-performance. Finally, they diversified instance fleets by employing multiple instance types to reduce Spot interruption impact.

These optimizations delivered 21% cost savings while supporting 800 daily active users and processing 1 PB of data daily. This enabled Razorpay to invest savings back into product innovation for their merchant customers, demonstrating how technical optimization directly translates to business value.

Conclusion

Razorpay’s migration to Amazon EMR demonstrates how the right data processing platform can transform business outcomes at scale. By achieving 11% better performance, 13-15% faster execution times, and 21% cost savings, EMR enabled Razorpay to build an enterprise-grade data platform that supports 800 daily users, more than 3,000 dashboards, and 10 million monthly queries.

To learn more about building similar data analytics solutions on AWS, check out the following resources.

Documentation:

AWS solutions:

Get started:

Visit the Amazon EMR product page
Explore Amazon EMR Notebooks overview

About the authors

How Amplitude implemented natural language-powered analytics using Amazon OpenSearch Service as a vector database

2026-03-04 Jeffrey Wang

Post Syndicated from Jeffrey Wang original https://aws.amazon.com/blogs/big-data/how-amplitude-implemented-natural-language-powered-analytics-using-amazon-opensearch-service-as-a-vector-database/

This is a guest post by Jeffrey Wang, Co-Founder and Chief Architect at Amplitude in partnership with AWS.

Amplitude is a product and customer journey analytics platform. Our customers wanted to ask deep questions about their product usage. Ask Amplitude is an AI assistant that uses large language models (LLMs). It combines schema search and content search to provide a customized, accurate, low latency, natural language-based visualization experience to end customers. Ask Amplitude has knowledge of a user’s product, taxonomy, and language to frame an analysis. It uses a series of LLM prompts to convert the user’s question into a JSON definition that can be passed to a custom query engine. The query engine then renders a chart with the answer, as illustrated in the following figure.

Ask Amplitude generates charts in response to user queries

Amplitude’s search architecture evolved to scale, simplify, and cost-optimize for our customers, by implementing semantic search and Retrieval Augmented Generation (RAG) powered by Amazon OpenSearch Service. In this post, we walk you through Amplitude’s iterative architectural journey and explore how we address several critical challenges in building a scalable semantic search and analytics platform.

Our primary focus was on enabling semantic search capabilities and natural language chart generation at scale, while implementing a cost-effective multi-tenant system with granular access controls. A key objective was optimizing the end-to-end search latency to deliver rapid results. We also tackled the challenge of empowering end customers to securely search and use their existing charts and content for more sophisticated analytical inquiries. Additionally, we developed solutions to handle real-time data synchronization at scale, making sure constant updates to incoming data could be processed while maintaining consistently low search latency across the entire system.

RAG and vector search with Ask Amplitude

Let’s take a brief look at why Ask Amplitude uses RAG. Amplitude collects omnichannel customer data. Our end customers send data on user actions that are performed in their platforms. These actions are recorded as user-generated events. For example, in the case of retail and ecommerce customers, the types of user events include “product search,” “add to cart,” “checked out,” “shipping option,” “purchase,” and more. These events help define the customer’s database schema, outlining the tables, columns, and relationships between them. Let’s consider a user question such as “How many people used 2-day shipping?” The LLM needs to determine which elements of the captured user events are pertinent to formulating an accurate response to the query. When users ask a question to Ask Amplitude, the first step is to filter the relevant events from OpenSearch Service. Rather than feeding all event data to the LLM, we take a more selective approach for both cost and accuracy reasons. Because LLM usage is billed based on token count, sending complete event data would be unnecessarily expensive. More importantly, providing too much context can degrade the LLM’s performance—when faced with thousands of schema elements, the model struggles to reliably identify and focus on the relevant information. This information overload can distract the LLM from the core question, potentially leading to hallucinations or inaccurate responses. This is why RAG is the preferred approach. To retrieve the most relevant items from the product usage schema, a vector search is performed. This is effective even in situations when the question might not refer to the exact words that are in the customer’s schema. The following sections walk through the iterations of Amplitude’s search journey.

Initial solution: No semantic search

We used Amazon Relational Database Service (Amazon RDS) for PostgreSQL as the primary database to store our people, events, and properties data. However, as the following diagram shows, we had a separate, third-party store to implement keyword search. We had to bring in data from PostgreSQL to this third-party search index and keep it updated.

Initial Solution: No Semantic Search

This architecture was simple but had two key shortcomings: there were no natural language capabilities in our search index, and the search index supported only keyword search.

Iteration 1: Brute force cosine similarity

To improve our search capability, we considered several prototypes. Because data volumes for most customers were not very large, it was quick to build a vector search prototype using PostgreSQL. We transformed user interaction data into vector embeddings and used array cosine similarity to compute similarity metrics across the dataset. This alleviated the need for custom similarity computation. The vector embeddings captured nuanced user behavior patterns using PostgreSQL capabilities without additional infrastructure overhead. This is generally called the brute force method, where an incoming query is matched against all embeddings to find its top (K) neighbors by a distance measure (cosine similarity in this case). The following diagram illustrates this architecture.

Iteration 1: Brute force cosine similarity

Enabling semantic search was a big improvement over traditional search for users who might use different terms to refer to the same concepts, such as “hours of video streamed” or “total watch time”. However, although this worked for small datasets, it was slow because the brute force method had to compute cosine similarity for all pairs of vectors. This was amplified as the number of elements in the events schema, the complexity of questions, and expectations of quality grew. Additionally, Ask Amplitude answers needed to blend both semantic and keyword search. To support this, each search query had to be implemented as a three-step process involving multiple calls to separate databases:

Retrieve the semantic search results from PostgreSQL.
Retrieve the keyword search results from our search index.
In the application, semantic search results and keyword search results were combined using pre-assigned weights, and this output was dispatched to the Ask Amplitude UI.

This multi-step manual approach made the search process more complex.

Iteration 2: ANN search with pgvector

As Amplitude’s customer base grew, Ask Amplitude needed to scale to accommodate more customers and larger schemas. The goal was not just to answer the question at hand, but to teach the user how to build an end-to-end analysis by guiding them iteratively. To this end, the embeddings needed to store and index contextually rich semantic content. The team experimented with bigger, higher dimensionality embeddings and had anecdotal observations of vector dimensionality appearing to impact the effectiveness of the retrieval. Another requirement was to support multilingual embeddings.

To support a more scalable k-NN search, the team switched to pgvector, a PostgreSQL extension that provides powerful functionalities for with vectors in high-dimensional space. The following diagram illustrates this architecture.

Iteration 2: ANN search with pgvector

Pgvector was able to support k-nearest neighbor (k-NN) similarity search for larger dimensionality vectors. As the number of vectors grew, we switched to indexes that allowed approximate nearest neighbor (ANN) search, such as HNSW and IVFFlat.

For customers with larger schemas, calculating brute force cosine similarity was slow and expensive. We found a performance difference when we moved to ANN enabled by pgvector. However, we still needed to deal with the complexity introduced by the three-step process of querying PostgreSQL for semantic search, a separate search index for keyword search, and then stitching it all together.

Iteration 3: Dual sync to keyword and semantic search with OpenSearch Service

As the number of customers grew, so did the number of schemas. There were hundreds of millions of schema entries in the database, so we sought a performant, scalable, and cost-effective solution for k-NN search. We explored OpenSearch Service and Pinecone. We chose OpenSearch Service because we could combine keyword and vector search capabilities. This was convenient for four reasons:

Simpler architecture – Positioning semantic search as a capability in an existing search solution, as we observed in OpenSearch Service, makes for a simpler architecture than treating it as a separate specialized service.
Lower-latency search – The ability to effectively organize and catalog search data was fundamental to how we generated answers. Augmenting semantic search to our existing pipeline by combining both into one query provided lower latency querying.
Reduced need for data synchronization – Keeping the database in sync with the search index was critical to the accuracy and quality of answers. With the alternatives that we looked at, we would have to maintain two synchronization pipelines, one for keyword search index and the other for a semantic search index, complicating the architecture and increasing the chances of experiencing out-of-sync results between keyword and semantic search results. Synchronizing them into one place was easier than synchronizing them into multiple places and then combining the signals at query time. With a combined keyword and vector search capabilities of OpenSearch Service, we now needed to synchronize only one primary database on PostgreSQL with the search index.
Minimized performance impact to source data updates – We found that synchronizing data to another search index is a complex problem because our dataset changes constantly. With every new customer, we had hundreds of updates every second. We had to make sure the latency of these updates wasn’t impacted by the sync process. Collocating search data with vector embeddings obviated the need for multiple sync processes. This helped us avoid additional latency in the primary database, due to the sync processes encroaching upon database update traffic.

Although our previous third-party search engine specialized in fast ecommerce search, this wasn’t aligned with Amplitude’s specific needs. By migrating to OpenSearch Service, we simplified our architecture by reducing two synchronization processes to one. We phased out the current search platform gradually. This meant we temporarily continued to have two synchronization processes, one with current platform and another to the combined keyword and semantic search index on OpenSearch Service, as shown in the following diagram.

Iteration 3: Dual sync to keyword and semantic search with OpenSearch Service

In addition to the pros of k-NN search identified in the previous iteration, moving to OpenSearch Service helped us realize three key benefits:

Reduced latency – Instead of collocating the embeddings with primary data, we were able to collocate with our search index. The search index is where our application needed to run our queries to pick out user events that are relevant to the question being asked and send this as context sent to the LLM. Because the search text, metadata, and embeddings were all in one place, we needed only one hop for all our search requirements, thereby improving latency.
Reduced compute power – We had anywhere between 5,000–20,000 elements in the user events schema. We didn’t need to send the entire schema to the LLM, because each user query required only 20–50 relevant elements. With the efficient filtering capabilities of OpenSearch Service, we were able to narrow down the vector search space by using tenant-specific metadata, significantly reducing compute requirements across our multi-tenant environment.
Improved scalability – With OpenSearch Service, we could take advantage of additional capabilities such as HNSW product quantization (PQ) and byte quantization. Byte quantization made it possible to handle the scale of millions of vector entries with minimal reduction in recall, but with improvement to cost and latency.

However, in this interim solution, our data wasn’t fully migrated to OpenSearch Service yet. We still had the old pipeline along with the new pipeline, and had to perform dual syncing. This was only temporary, as we phased out the old search index, and the old pipeline served as a baseline to compare with in terms of performance and recall.

Iteration 4: Hybrid search with OpenSearch Service

In the final architecture, we were able to migrate all our data to OpenSearch Service, which also served as our vector database, as shown in the following diagram.

Iteration 4: Hybrid search with OpenSearch Service

We now had to perform just one data synchronization from the PostgreSQL database to the combined search and vector index, allowing the resources on the database to focus on transactional traffic. OpenSearch Service provides merging, weighting, and ranking of the search results as part of the same query. This obviated the need to implement them as a separate module in our application, effectively resulting in a single, scalable hybrid search (combined keyword-based (lexical) search and vector-based (semantic) search). With OpenSearch Service, we could also experiment with the new integration with Amazon Personalize.

Evolving RAG to draw upon user-generated content

Our customers wanted to ask deeper questions about their product usage that couldn’t be answered just by looking at the schema (the structure and names of the data columns) alone. Simply knowing the column names in a database doesn’t necessarily reveal the meaning, values, or proper interpretation of that data. The schema alone provides an incomplete picture. A naïve approach would be to index and search all data values instead of searching just the schema. Amplitude avoids this for scalability reasons. The cardinality and volume of event data (potentially trillions of event records) makes indexing all values cost prohibitive. Amplitudes hosts about 20 million charts and dashboards across all Amplitude customers. This user-generated content is valuable. We observed that we can better understand the meaning and context by analyzing how other users have previously visualized data.For example, if a user asks about “2-day shipping,” Amplitude first checks if the data schema contains columns with relevant names like “shipping” or “shipping method”. If such columns exist, it then examines the potential values in those columns to find values related to 2-day shipping. Amplitude also searches user-created content (charts, dashboards, and more) to see if anyone else at the company has already visualized data related to 2-day shipping. If so, it can use that existing chart as a reference for how to properly filter and analyze the data to answer the question. To search this content efficiently, Amplitude employs a hybrid approach combining keyword and vector similarity (semantic) searches. For tenant isolation and pruning, we use metadata to filter by customer first, and then vector search.

Conclusion

In this post, we showed you how Amplitude built Ask Amplitude, an AI assistant using OpenSearch Service as a vector database to enable natural language queries of product analytics data. We evolved our system through four iterations, ultimately consolidating keyword and semantic search into OpenSearch Service, which simplified our architecture from multiple sync pipelines to one, reduced query latency by combining search operations, and enabled efficient multi-tenant vector search at scale using features like HNSW PQ and byte quantization. We extended the system beyond schema search to index 20 million user-generated charts and dashboards, using hybrid search to provide richer context for answering customer questions about product usage.

As natural language interfaces become increasingly prevalent, Amplitude’s iterative journey demonstrates the potential for harnessing LLMs and RAG using vector databases such as OpenSearch Service to unlock rich conversational customer experiences. By gradually transitioning to a unified search solution that combines keyword and semantic vector search capabilities, Amplitude overcame scalability and performance challenges while reducing architecture complexity. The final architecture using OpenSearch Service enabled efficient multi-tenancy and fine-grained access control and also facilitated low-latency hybrid search. Amplitude is able to deliver more natural and intuitive analytics capabilities to its customers by generating deeper insights and contextualizing data.

To learn more about how Ask Amplitude helps you express Amplitude-related concepts and questions in natural language, refer to Ask Amplitude. To get started with OpenSearch Service as a vector database, refer to Amazon OpenSearch Service as a Vector Database.

About the authors

Set up production-ready monitoring for Amazon MSK using CloudWatch alarms

2026-03-03 Yashika Jain

Post Syndicated from Yashika Jain original https://aws.amazon.com/blogs/big-data/set-up-production-ready-monitoring-for-amazon-msk-using-cloudwatch-alarms/

Organizations running Apache Kafka as their streaming platform need comprehensive monitoring to maintain reliable operations. Without proper visibility into broker health, resource utilization, and data flow metrics, teams risk service disruptions, data loss, and degraded performance that can impact critical business operations. Effective monitoring and alerting are essential to detect anomalies early, from high system load to connectivity issues, enabling teams to take preventive action before problems affect production workloads.

Amazon Managed Streaming for Apache Kafka (Amazon MSK) addresses these monitoring challenges by publishing detailed metrics to Amazon CloudWatch. The service emits metrics at 1-minute intervals for provisioned (Standard) clusters, with flexible monitoring levels (DEFAULT, PER_BROKER, PER_TOPIC_PER_BROKER, or PER_TOPIC_PER_PARTITION) to control granularity and cost. At the DEFAULT level (free), cluster-level metrics are available; higher levels (paid) expose broker-level, per-topic and per-partition metrics.

In this post, I show you how to implement effective monitoring for your MSK clusters using Amazon CloudWatch. You’ll learn how to track critical metrics like broker health, resource utilization, and consumer lag, and set up automated alerts to prevent operational issues. By following these practices, you can work to improve streaming operations reliability, optimize resource usage, and support high availability for your mission-critical applications.

Key metrics to monitor

This article groups important Amazon MSK metrics into logical categories. For each, we highlight key metrics and what they indicate:

Broker Health and Cluster Availability:
- ActiveControllerCount is a cluster-level metric where each broker reports whether it’s the active controller (1) or not (0). In a healthy cluster, exactly one broker serves as the active controller at any time. When viewing this metric with the average statistic, the value equals 1 divided by the number of brokers. For example, a 3-broker cluster shows 0.33 (1/3). Set CloudWatch alarm thresholds accordingly—for 6 brokers, alert if average falls below 0.166(1/6). When using the sum statistic, the value should always be 1, indicating one active controller regardless of cluster size. If the sum differs from 1, a controller election is in progress—typically during maintenance activities, configuration changes, or rolling restarts.
  Note: For a KRaft-based clusters, the ActiveControllerCount is only exposed on dedicated controller endpoints so the sample count is 3 and only controller will report value of 1. Thus, the average is always 0.33 no matter how many brokers there are in the cluster. To monitor the broker health for Kraft-based clusters, check LeaderCount metric. If a broker is not emitting any metric, then it’s a good indication that broker might be unhealthy.
- OfflinePartitionsCount (cluster): Number of partitions with no active leader. Non-zero values mean data is temporarily unavailable or unwritable. Trigger alerts if it rises above 0.
- UnderReplicatedPartitions (per broker): Number of partitions where not all replicas are caught up. This should stay at 0 under normal conditions. Spikes indicate traffic exceeds capacity or replication lag; sustained values often mean a configuration/ACL issue. Refer to Troubleshoot your Amazon MSK cluster
- UnderMinIsrPartitionCount (per broker): Partitions below the minimum in-sync replica (ISR) count. A non-zero value means potential data loss risk if brokers fail. Monitor to ensure replication is healthy. Refer to Custom configurations
- GlobalPartitionCount (cluster): Total number of partitions across all topics (leaders only). Useful for capacity planning and sanity checks.
- PartitionCount (per broker): Number of partitions (including replicas) hosted by a broker. Sudden changes may indicate re-balances. (Excess partitions per broker can degrade performance).
Resource Utilization:
- CPU: Total broker CPU utilization is defined as CpuUser + CpuSystem. Best practice is to keep average CPU utilization under 60% . Set alarms on the sum of user+system to detect overload.
- CPUCreditBalance / CPUCreditUsage (per broker): For burstable instance types(T3), tracks earned/spent CPU credits. A declining credit balance or high credit usage warns that the instance may be CPU-starved.
- Memory: MemoryUsed, MemoryFree (per broker) show RAM usage. Critically, HeapMemoryAfterGC (per broker) reports JVM heap usage (%) after garbage collection. AWS recommends alerting if HeapMemoryAfterGC exceeds 60%, to avoid out-of-memory issues.
- Disk: Kafka brokers use attached EBS storage for topic data. Monitor KafkaDataLogsDiskUsed (per broker) – percentage of disk used by message logs. Best practice: alarm when data log usage exceeds 85%. Also track RootDiskUsed: the percentage of the root disk used by the broker.
- EBS I/O: Volume metrics (per broker) such as VolumeQueueLength, VolumeReadOps, VolumeWriteOps, VolumeReadBytes, VolumeWriteBytes indicate I/O latency and throughput. Rising queue lengths or latency (such as VolumeTotalReadTime) suggest disk contention.
- Network: Basic network stats per broker include NetworkRxPackets, NetworkTxPackets, and errors/drop counts (NetworkRxErrors, NetworkTxErrors, NetworkRxDropped, NetworkTxDropped). Unexpected errors or drops can indicate network issues.
Topic and Partition Activity:
- Throughput: BytesInPerSec and BytesOutPerSec measure inbound/outbound data rates per broker or per topic. Sustained drops can signal lost producers/consumers; spikes may require scaling.
- Replication Traffic: ReplicationBytesInPerSec/ReplicationBytesOutPerSec (per topic) show inter-broker replication volume.
- Consumer Lag: Consumer lag metrics quantify the difference between the latest data written to your topics and the data read by your applications. Amazon MSK provides the following consumer-lag metrics, which you can get through Amazon CloudWatch or through open monitoring with Prometheus: EstimatedMaxTimeLag, EstimatedTimeLag, MaxOffsetLag, OffsetLag, and SumOffsetLag. For information about these metrics, see Amazon MSK metrics for monitoring Standard brokers with CloudWatch.
Client Connections :
- ConnectionCount (per broker): Total active connections (clients + inter-broker). Sudden drops or sustained high counts (hitting limits) merit attention.
- ClientConnectionCount (per broker, with auth filter): Active authenticated client connections.
- ConnectionCreationRate / ConnectionCloseRate (per broker): New or closed connections per second. Spikes in connection churn may indicate client issues.
- Authentication: IAMNumberOfConnectionRequests and IAMTooManyConnections (per broker) show IAM auth request rates and throttle breaches (limit of 100 simultaneous connections).
Network Bandwidth Metrics:
- TrafficShaping > 0 (any throttling) metric serves as your primary warning signal. When this value exceeds zero, your MSK cluster is experiencing network throttling at the EC2 layer, with packets being dropped or queued due to exceeded allocations. This throttling manifests as reduced throughput, increased latency, and potential network errors that impact both producer and consumer performance. TrafficShaping issues stem from two possible bandwidth limitations: BwInAllowanceExceeded & BwOutAllowanceExceeded :
- BwInAllowanceExceeded tracks when inbound aggregate bandwidth surpasses broker maximums.
- BwOutAllowanceExceeded monitors when outbound aggregate bandwidth exceeds limits.
  Both BwInAllowanceExceeded and BwOutAllowanceExceeded metrics directly contribute to overall network throttling events.
Other Operational Metrics:
- Thread Pools: RequestHandlerAvgIdlePercent, NetworkProcessorAvgIdlePercent (per broker) show how busy Kafka’s internal thread pools are. Consistently low idle (%) can indicate bottlenecks.
- ZooKeeper: For ZooKeeper-based MSK clusters, ZooKeeperRequestLatencyMsMean and ZooKeeperSessionState reflect ZK performance (for older Kafka versions that use Zookeeper). For ZooKeeperSessionState, anything other than 1 for 5-10 mins should be alarming as there can be chances broker has an issue or zookeeper is not able to connect to brokers due to some intermittent network issue.
- Tiered Storage: For clusters with tiered storage enabled, Amazon MSK provides metrics like RemoteFetchBytesPerSec, RemoteCopyBytesPerSec, RemoteLogSizeBytes, and related error/queue metrics. These track offloading to remote storage.
- Intelligent rebalancing metrics: For MSK Provisioned clusters using Express brokers, Amazon MSK provides two key metrics to monitor rebalancing operations: RebalanceInProgress and UnderProvisioned metrics. See Monitor Intelligent rebalancing metrics

By grouping metrics into these categories, you can build dashboards and alerts that comprehensively cover Amazon MSK health and performance. Amazon CloudWatch also provides automatic dashboards for Amazon MSK.

Let’s take a quick look on how to access CloudWatch automatic dashboard. In the AWS Console, go to the CloudWatch service. When in the CloudWatch console, select Dashboards. Open the Automatic dashboard tab and search for MSK in the Filter Bar.

These dashboards offer per-configured visualizations of key metrics, enabling quick insights into the health and performance of your MSK clusters.

Recommended CloudWatch alarms

Setting alarms on key metrics helps catch issues early. Detecting issues early is crucial in streaming applications where every second counts. A single failing broker can trigger a chain reaction – halting data ingestion, backing up upstream systems, and breaking downstream applications. This can quickly escalate from delayed order processing to lost revenue. Proactive monitoring helps catch and fix problems before they impact your business operations. Based on AWS best practices and experience, consider alarms such as:

Metric (Dimension)	Alarm Condition	Rationale
ActiveControllerCount (cluster)	≠ 1 (count)	Only one active controller should exist. Deviation implies cluster instability.
CPU Utilization (Sum(CPUUser+CPUSystem), per broker)	> 60% (average) for 5+ mins	Helps maintain headroom for broker load and maintenance. High CPU may slow processing as outlined in the MSK best practices documentation
HeapMemoryAfterGC (broker)	> 60% (percentage)	Indicates Kafka heap is filling up. Helps prevent OOM by alerting early.
KafkaDataLogsDiskUsed (broker)	≥ 85% (percent)	Warns that disk is nearly full. Helps prevent data loss by providing time for scaling or cleanup.
OfflinePartitionsCount (cluster)	> 0 (count)	Any offline partition means unavailable data. Immediate investigation needed.
UnderReplicatedPartitions (broker)	> 0 (count)	No replicas lagging under healthy conditions. Spikes or sustained lag can indicate overload or ACL misconfiguration.
UnderMinIsrPartitionCount (broker)	> 0 (count)	There must be topics with partitions that have either less in-sync replicas than the min.insync.replicas setting or with RF=MinISR. To find these topics whose partitions are under replicated, use command: <path-to-your-kafka-installation>/bin/kafka-topics.sh –bootstrap-server <bootstrap-server:port> —command-config client.properties –describe –under-min-isr-partitions
ConnectionCount (broker)	Sudden drop (e.g. < 90% of baseline) or spike above high threshold	Detect client connectivity issues or connection floods. Unexpected drops may mean a broker is unreachable. Refer to Amazon MSK Standard broker quota
CPUCreditBalance (for T3 broker)	< some low threshold (e.g. 10 credits)	For burstable instances, alerts when credits are nearly exhausted, which degrades performance.
VolumeQueueLength (broker)	> 0 (sustained) or rising	Indicates I/O operations are queuing, possible disk bottleneck.
NetworkRxErrors/TxErrors (broker)	> 0 (count)	Any network errors can cause packet loss or disconnections.
IAMTooManyConnections (broker)	> 0 (count)	Exceeding IAM connection limit (100) blocks new connections.
Consumer Lag (MaxOffsetLag or SumOffsetLag) (per consumer-group/topic)	> threshold (depends on SLAs, e.g. growing beyond expected)	Alerts on slow consumers so you can scale consumers or investigate backlogs.
TrafficShaping	> 0 (any throttling)	This is an indication that brokers are exceeding their allocated network bandwidth.

These are illustrative thresholds; adjust them for your workload and SLAs. The remaining metrics listed in the CloudWatch metrics for Standard and Express brokers documentation are susceptible to downstream impact from anomalies in the primary metrics above. It is recommended to enable CloudWatch alarms on a single test cluster first to validate thresholds before extending coverage across your MSK fleet.

Conclusion

In this post, we covered the important CloudWatch metrics and alarms for monitoring Amazon MSK clusters effectively. By implementing these recommended alarms, you can proactively detect and respond to potential issues before they impact your Kafka workloads. To learn more about Amazon MSK monitoring, refer to the Amazon MSK Monitoring Best Practices documentation or explore our Amazon MSK Workshops hands-on experience.

About the authors

Inside AWS Security Agent: A multi-agent architecture for automated penetration testing

2026-02-27 Tamer Alkhouli

Post Syndicated from Tamer Alkhouli original https://aws.amazon.com/blogs/security/inside-aws-security-agent-a-multi-agent-architecture-for-automated-penetration-testing/

AI agents have traditionally faced three core limitations: they can’t retain learned information or operate autonomously beyond short periods, and they require constant supervision. AWS addresses these limitations with frontier agents—a new category of AI that performs complex reasoning, multi-step planning, and autonomous execution for hours or days. Multi-agent collaboration has emerged as a powerful approach that helps tackle complex workflows that require multiple steps and diverse expertise—such as in software development where agents handle code generation, review, and testing; in scientific research where agents collaborate on literature review, experimental design, and data analysis; and in cybersecurity where specialized agents perform reconnaissance, vulnerability analysis, and exploit validation.

In this post, we discuss how we’ve used this technology to deliver automated penetration testing, something that can traditionally take weeks and is resource intensive. We also provide a technical deep-dive into the architecture of the penetration testing component built into AWS Security Agent.

The concept of automated security testing isn’t new—penetration testing tools and vulnerability scanners have existed for decades. However, with recent advancements in large language models (LLMs), frontier agents are designed to reason about application behavior, adapt strategies based on feedback, and understand context in ways that traditional tools can’t. By creating a network of specialized agents, we can address increasingly complex security challenges: one agent maps the attack surface while others analyze business logic flaws, validate findings, and prioritize vulnerabilities based on actual exploitability. The exploitability context comes from the combination of actual exploit attempts by swarm agent workers, independent re-validation by specialized validators, and LLM-driven scoring according to the common vulnerability scoring system (CVSS).

We’ve developed automated penetration testing for the AWS Security Agent. This capability includes a multi-agent penetration testing system that orchestrates specialized security agents to work collaboratively on vulnerability detection. The system begins with multiple types of scanning to establish baseline coverage, then conducts broad reconnaissance using static, predefined tasks to map the application surface and identify initial attack vectors. Building on these findings, our agentic system dynamically generates focused test tasks tailored to the specific application context—reasoning about discovered endpoints, business logic patterns, and potential vulnerability chains to create targeted security tests that adapt based on application responses. By combining these specialized capabilities, the system can tackle complex security scenarios across major risk categories. Beyond single-vulnerability detection, the system performs complex chained attacks—for instance, combining an information disclosure flaw with privilege escalation to access sensitive resources, or chaining insecure direct object references (IDOR) with authentication bypass.

Figure 1: Diagram of the AWS Security Agent penetration testing component.

System architecture

This section describes the major components of the system. The following subsections cover authentication and initial access, baseline scanning, multi-phased exploration with the specialized agent swarm, and validation with report generation.

Authentication and initial access

The system begins with an intelligent sign-in component that handles authentication across diverse application architectures. This component combines LLM-based reasoning with deterministic mechanisms to locate sign-in pages, attempt provided credentials, and maintain authenticated sessions for subsequent testing phases. The approach adapts to different application structures and target environments automatically and uses a browser tool. The developer can optionally provide a custom sign-in prompt tailored to the target application.

Baseline scanning phase

Following authentication, the system initiates comprehensive baseline scanning through parallel execution of specialized scanners. For black-box testing, the network scanner conducts automated web application security testing, generating raw traffic interactions and identifying candidate vulnerable endpoints. In white-box settings, the code scanner additionally performs deep source code analysis when repositories are available, producing descriptive documentation across multiple categories. Additional specialized scanners complement these capabilities to identify vulnerabilities across multiple dimensions and establish initial security coverage.

Multi-phased exploration

The system employs two distinct exploration approaches that work in concert. Managed execution operates with predefined static tasks across major risk categories like cross-site scripting, insecure direct object reference, privilege escalation, and so on. This component systematically helps ensure comprehensive coverage by executing curated tasks for each risk type. In the next phase, guided exploration takes a dynamic, intelligence-driven approach. This component ingests discovered endpoints, validated findings, and code analysis documentation to reason about application-specific attack opportunities. It operates in two stages: first generating a contextual penetration testing plan by identifying unexplored resources and potential vulnerability chains, then programmatically managing the execution of these dynamically generated tasks. The guided explorer runs with adaptive tasks that evolve based on application responses and discovered patterns.

Specialized agent swarm
Both exploration approaches dispatch work to specialized swarm worker agents—each configured for specific risk types and equipped with comprehensive penetration testing toolkits including code executors, web fuzzers, NVD vulnerability database search for Common Vulnerabilities and Exposures (CVE) intelligence, and vulnerability-specific tools. These workers execute assigned tasks with timeout management and structured reporting.

Validation and report generation

When specialized agents identify potential security risks, they generate structured reports containing the vulnerability type, affected endpoints, exploitation evidence, and technical context. However, automated penetration testing faces a critical challenge: LLM agents can produce plausible-sounding findings that require rigorous validation. Candidate findings undergo validation through both deterministic validators and specialized LLM-based agents that attempt active exploitation. We employ assertion-based validation techniques where natural language assertions written by security experts encode deep knowledge about real attack behaviors, requiring explicit, structured proof that’s significantly harder to circumvent than narrow deterministic checks. Validated findings undergo Common Vulnerability Scoring System (CVSS) analysis for severity assessment, then are synthesized into final reports with validation results, severity scores, and exploitation evidence—designed to deliver actionable, high-confidence vulnerabilities for effective remediation.

Benchmarking

To evaluate our system, we performed human evaluation in addition to automatic benchmarking. We conducted analysis on real-world trajectories and created a taxonomy of error patterns. By spotting frequent error patterns, we were able to iterate on our solution. We report results on the CVE Bench public benchmark, which is a collection of vulnerable web applications containing 40 critical-severity CVEs from the National Vulnerability Database used to evaluate AI agents on real-world exploits. Each application includes automatic exploit references, and LLM-based agents attempt to execute attacks that trigger the vulnerabilities.

We measure success through the attack success rate (ASR) metric, defined as the rate of successful exploitation of application vulnerabilities. CVE Bench uses a grader that the agent can query to verify exploit success and provides explicit capture-the-flag (CTF) instructions. We evaluate in three configurations:

With CTF instructions and grader checks after each tool call, achieving 92.5% on CVE Bench v2.0 (we note that some challenges involve blind exploitation where the agent cannot verify success without this feedback).
Without CTF instructions or grader feedback, achieving 80%—which better reflects real-world conditions where the agent must self-validate through observable outcomes. We also observed that the agent was able to identify some CVEs based on the LLM’s parametric knowledge, as shown in the following bash command where the model explicitly references a CVE by name.
Therefore, we ran an additional experiment using an LLM whose knowledge cutoff date predates CVE Bench v1.0 release, achieving 65% ASR.

The following code example shows an LLM agent demonstrating parametric knowledge of CVE-2023-37999 from its training data, then issuing a bash command to check exploitation prerequisites.

# HT Mega 2.2.0 has a known vulnerability – CVE-2023-37999
# It has an unauthenticated privilege escalation via the REST API settings endpoint
# Let's check if registration is enabled
curl -s http://target:9090/wp-login.php?action=register -I | head -10

We’re committed to pushing the frontier of security vulnerability detection by continuously evaluating our agent and staying competitive with newer, more challenging benchmarks.

Optimizing testing and compute budget

One challenge for penetration testing is determining the balance between exploitation and exploration. Using a depth-first approach can waste too much compute on specific directions, leading to lower vulnerability coverage under a fixed compute budget. Compare that to breadth-first search, which is unlikely to discover deep vulnerabilities that require testing multiple approaches. Therefore, a balance between the two approaches is needed to maximize coverage for a given compute budget. Our proposed system design aims to include a hybrid approach. A more efficient dynamic solution that generalizes across various vulnerabilities and different web applications remains an open research question.

Another challenge with penetration testing is non-determinism. Because of the underlying LLMs, the output of penetration test runs can vary from one run to another. Having different findings across multiple runs can lead to confusion. One option to mitigate this is to perform multiple runs and consolidate the findings across them.

Conclusion

The multi-agent architecture presented in this post demonstrates how you can use specialized agents that can collaborate to tackle complex penetration testing workflows—from intelligent authentication and baseline scanning through managed and guided exploration phases, culminating in rigorous validation. By orchestrating these specialized components with adaptive task generation and assertion-based validation, the system delivers comprehensive security coverage that evolves based on application-specific context and discovered patterns.

AWS Security Agent is now in public preview, for more information, see Getting Started with AWS Security Agent.

If you have feedback about this post, submit comments in the Comments section below.

How Swiss Life Germany automated data governance and collaboration with Amazon SageMaker

2026-02-25 Tim Kopacz

Post Syndicated from Tim Kopacz original https://aws.amazon.com/blogs/big-data/how-swiss-life-germany-automated-data-governance-and-collaboration-with-amazon-sagemaker/

Data has become an indispensable strategic asset for the entire financial services industry, driving innovation and competitive advantage in an increasingly digital marketplace. At Swiss Life Germany, maximizing the value of this asset means empowering internal teams to derive actionable insights and deliver personalized financial solutions to diverse clientele. This led to the need to establish seamless data sharing workflows that enhance cross-departmental collaboration while maintaining strict security and compliance standards. To accomplish this, Swiss Life Germany decided to implement advanced data processing and governance capabilities using Amazon SageMaker.

Integrating SageMaker into a highly regulated enterprise environment required aligning the service’s agility with Swiss Life’s rigorous infrastructure as code (IaC) automation standards. This post demonstrates how Swiss Life Germany addressed these sophisticated deployment requirements by developing a custom Terraform pattern designed specifically for platform engineers and data architects.

Swiss Life Germany cloud journey

Swiss Life Germany is a leading provider of customized pension products and financial advice. Building on over 100 years of delivering insurance, retirement planning, and wealth management solutions, a key driver of the company’s recent evolution was the strategic transition from legacy on-premises data centers to a modern, cloud-centric architecture. After an extensive evaluation of various providers, Swiss Life Germany selected Amazon Web Services (AWS) as the strategic foundation to modernize their data operations. By using AWS, the organization was able to transition from capital-intensive data centers to a flexible pay-as-you-go model, significantly reducing the operational costs.

Following their comprehensive AWS cloud migration over the last two years—combining 30% re-platforming with 70% lift-and-shift strategies—Swiss Life Germany modernized infrastructure management through IaC. The company introduced the governance concept of an IT System. An IT System is a fundamental unit of management that defines a software component regardless of its origin. Whether a component is purchased from a vendor, self-developed or consumed as software as a service (SaaS), it’s integrated into this single governance structure. This ensures that off-the-shelf products and custom-coded applications are held to the same high standards of visibility and accountability. Every IT system is required to maintain specific attributes that allow for seamless oversight such as unique identifiers, assigned ownership and the associated AWS resources logically grouped under the IT System they support.

Where traditional approaches would store and expose this information in configuration management database (CMDB)-like systems to store static snapshots of asset data, Swiss Life adopted a more dynamic model. By using GraphQL API as a unified meta-model, the company queries application data directly from its primary source systems. This approach eliminates the delays common in batch-processed databases, ensuring maximum freshness. The API serves as a single entry point for infrastructure data, documentation, organizational metadata, and even inter-application dependencies. The transparency and automation gained through this everything-as-code and API-first approach provided a blueprint for the Swiss Life Data Platform: complete transparency, reproducibility, and end-to-end automation.

This robust technical foundation served as a catalyst and prerequisite for Swiss Life’s broader strategic goals and governed framework.

Defining the vision for a unified data solution

With the architectural foundations in place, the next challenge was to establish efficient data flows from production systems through data engineering teams to end users across various business divisions, with hundreds of specific use cases demanding attention.

For instance, Swiss Life’s customer portal specialists had to validate the effectiveness of campaign management and push notification systems in real-time, requiring secure and immediate access to interaction data.

Security requirements added another layer of complexity, because Swiss Life’s solution needed to incorporate robust compliance standards including two-factor authentication, session-based access controls, and granular row and column-level security protections.

To align with the overarching Swiss Life Germany cloud strategy, the company aimed to build a modern data solution atop their existing AWS data and analytics services. AWS introduced SageMaker to Swiss Life Germany following its announcement at AWS re:Invent 2024. A proof-of-concept quickly validated that this was the right tool to advance Swiss Life’s data journey. By deploying a fully automated framework, Swiss Life Germany sought to create a secure, compliant framework with SageMaker democratizing data access for authorized users, ultimately enabling faster business insights and more responsive customer experiences across the entire data environment.

Having met the infrastructure requirements, let’s look at what SageMaker looks like for end users and how data platform administrators can control access and resources at a granular level.

Users and their types of projects

A typical end user experience within Amazon SageMaker Unified Studio starts with creating a project. A project is a logical boundary within a domain where the data teams can collaborate and work on a business use case. Administrators would provision the blueprints and project profile templates for the data teams, as shown in the following figure.

However, at Swiss Life, they have extended the data platform administrator’s role to also create projects so they can maintain regulatory compliance and remove initial onboarding hurdles. The end user experience in SageMaker Unified Studio is simplified with data teams selecting their respective projects to work on a business initiative, as shown in the following figure.

To implement this solution effectively, Swiss Life identified different user groups:

A solution team developing an IT System that can act as producer or consumer of data assets.
A data scientist doing advanced data processing. They will most likely consume a lot of data assets and might produce some high aggregated data assets. The data processing software is also categorized as an IT System.
Business users who have some SQL skills and want to process data to get insights for their daily business.
A platform team administering the data platform. They provide core services to all users to make participation as straightforward as possible.
A data officer who wants to have a single point of interpretation for data.

Given this diverse set of user groups, the resulting data platform had to support a federated data organization with a centralized governance, decentralized data stores and data-processing organized at the IT System level. This architecture means the SageMaker management account—which orchestrates the data domain—contains no actual data, instead, data and compute resources reside in the individual IT System AWS accounts. Swiss Life’s implementation distinguishes between two fundamental project types:

IT System projects (for technical users)
Team projects (for non-technical users)

Swiss Life decided to align team projects with specific organizational units and operate them without staging environments, providing dedicated workspaces for departmental data initiatives. In contrast, IT System projects are associated with specific solutions such as customer portal or CRM systems. These follow a structured staging methodology, with each solution team managing dedicated DEV, TEST, and PROD environments to maintain proper development lifecycles and quality control.

This federated architecture is designed to handle the immense scale and diversity of Swiss Life’s data landscape. Swiss Life’s data platform would then aim to provide unified access to over 180 database servers with over 1,800 databases and 18 thousand tables across all stages (DEV, TEST and PROD).

In this post, we focus on the IT System projects.

How Swiss Life built the automation framework

Because Terraform is the preferred IaC tool across Swiss Life Germany, the team faced an interesting architectural challenge: while the existing infrastructure framework incorporates numerous AWS services that are readily supported by Terraform, SageMaker required a custom integration approach to align with Swiss Life’s advanced automation patterns.

Rather than adopting a manual ClickOps approach to infrastructure management, Swiss Life developed an innovative solution to keep the entire infrastructure—including SageMaker—within their Terraform automation, preserving key benefits like state management. The team accomplished this by using Terraform’s AWS Lambda invoke function resource with a create, read, update, delete (CRUD) lifecycle scope. By using this approach, the organization could maintain a single source of truth for infrastructure, while accommodating specific requirements of SageMaker. This component is called the Management Lambda and it serves as a bridge between Terraform’s declarative configuration and SageMaker, so that Swiss Life can provision, modify, and decommission Amazon SageMaker resources through established Terraform workflows.

The following is the snippet of a new domain creation using Terraform and Management Lambda:

resource"aws_lambda_invocation" "domain" {
  function_name = "management-lambda-function-name"
  lifecycle_scope = "CRUD"
  input = jsonencode({
  resource = "domain"
  domain_name = "SwissLife"
  domain_execution_role = "arn:aws:iam::012345678912:role/sus_domain_execution_role"
  domain_service_role = "arn:aws:iam::012345678912:role/sus_service_role"
  })
}

Using this approach, Swiss Life successfully automated every aspect of deploying a complete SageMaker domain installation within the Swiss Life cloud data platform. The automation encompasses the entire domain creation process, using the SageMaker domain unit feature as an organizational framework for diverse project portfolio.

Deployment architecture

Let’s dive deeper into the individual steps of the automation process itself. As said, all resources within SageMaker are controlled by the Terraform-invoked Management Lambda whereas other resources are directly managed by Terraform itself. The Management Lambda and SageMaker resources such as domains, metadata fields and others live in the central SageMaker account. Users of the data platform have their own AWS accounts. To start with, AWS Lake Formation had to be enabled across all AWS accounts, which could then act as consumer or provider to the platform. Using the established AWS Landing Zones mechanism, this was done by a single deployment to the management account. This early step also verified the management role being present in all accounts and assumable by the Management Lambda.

The following steps are used to set up Swiss Life’s data platform from scratch, as shown in the following diagram:

The Management Lambda is deployed to Swiss Life’s designated SageMaker account. This Lambda function uses the described CRUD pattern for all subsequent SageMaker-specific operations.
The domain provisioning begins by creating the service and domain execution roles, after which the Management Lambda creates the domain and uses these roles. During this step, administrative users and their associated permissions are also configured.
Upon successful domain creation, the Lambda function returns the domain identifier as output. This identifier is then used to let all AWS accounts of the company join this domain. These can now act as providers or consumers on the platform, resulting in a frictionless onboarding of teams.
Because Swiss Life decided to stage data products in a single domain, the DEV, TEST, and PROD domain units are then created, establishing the hierarchical structure under which IT System projects are subsequently created in the next implementation phase.

All projects and teams with the necessary prerequisites set up are then created automatically. This is done by using the enterprise GraphQL API mentioned to retrieve all IT products, their teams and roles. With that, each team already has their ready-to-use project in place upon singing into the platform. In detail this process looks like the following:

Continuing with the earlier example: the customer portal team needs to share their data with others in the organization and is using their dedicated project for this purpose. The process is shown in the following figure.

The deployment initiates with a cross-account role assumption by the Management Lambda to activate the blueprint configuration in the team’s AWS account. A standardized creation process was built to help facilitate all accounts are configured identically, maintaining consistency across the environment.
Next, a project profile specifically tailored for the customer portal project is created. This profile establishes the foundational settings and permissions framework that will govern the project’s operations.
With the profile in place, the actual project within this previously established project profile can now be provisioned, instantiating the working environment, where data sharing and collaboration will occur. This results in an identical amount of project profiles and projects in the SageMaker Unified Studio domain.
Finally, an automated membership management process is triggered. The system again queries Swiss Life’s Enterprise GraphQL API to identify all members of the solution team and automatically adds them as project members with appropriate permissions. This process executes daily, to help ensure that project access permissions remain current and accurately reflect team composition changes.

In the third and final deployment step, the user experience is enhanced by making the data platform immediately usable for teams in production. When teams and their members first access the domain URL, they find a project environment already populated with all necessary assets, so they can begin working without delay. This is accomplished through the following steps, shown in the following figure:

An automated discovery process is triggered that identifies all Amazon Simple Storage Service (Amazon S3) buckets and AWS Glue assets associated with the specific customer portal IT System. This inventory is created by using the AWS Resource Tagging API with specific filters targeting these asset types, so that all relevant resources for exactly that IT System are captured.
When identified, all discovered S3 buckets are registered as data lake locations within the platform. For each location, they create an AWS Identity and Access Management (IAM) role with precise access permissions, adhering to the least privilege security model.
Then grantable permissions are granted to the SageMaker project role for these assets, establishing a permission delegation framework that allows project members to manage access within their project scope—managing cross project access—while maintaining overall governance.
Finally, the AWS Glue databases are added as data sources within the project. These data sources are configured with daily synchronization schedules to automatically load new metadata into SageMaker, helping to ensure that catalog information remains current without manual intervention.

What a team needs to start with all of this

The overarching goal throughout this implementation has been to simplify the adoption process for the internal data teams. To ensure the data teams could immediately use the powerful capabilities of SageMaker without needing to manage its underlying architecture, Swiss Life Germany streamlined the experience by pre-packing the entire onboarding process into a high-level Terraform module. Teams can then use the module to deploy a complete, production-ready environment with minimal configuration, accelerating their path from setup to insight.

The following is an example of the code used by the module.

module "membership" {
	source = "<git-source>"
	it_system_labels = ["kundenportal"]
	domain_name = "SwissLife"
	vpc_id = "vpc_id"
	subnet_ids = ["subnet_a", "subnet_b", "subnet_c"]
}

To initiate this, the data teams define their basic parameters such as network configuration or their IT-System identifier as outlined previously and submit a pull request in the central Git repository. After the Swiss Life data platform team reviews and approves the request, the automated processes run in the background, preparing the complete environment. This automated approach has reduced deployment time for new environments from several weeks of manual coordination to under 20 minutes.

Rather than requiring users to understand the intricate deployment steps and managing the infrastructure, the automated deployment process empowers business units, like the customer portal team, to focus on deriving insights. At the same time, the Swiss Life Germany data platform team also maintains precise control over resource allocations, access rights and cost management.

Future enhancements

Looking ahead, Swiss Life plans to elevate its automation to a higher level of business abstraction. The next major enhancement focuses on removing the requirement for teams to request specific technical assets. Instead, the vision is to implement an intuitive interface where teams can specify the business terms or data domains they require. The system will automatically identify and provision the correct underlying technical assets associated with those business definitions.

This semantic layer will create a more natural interaction model, so that business users can think and work in familiar concepts rather than technical constructs. For example, rather than requesting access to specific S3 buckets or AWS Glue databases, a marketing analyst might indicate they need customer interaction data or campaign response metrics. An automated system will then map these business terms to the appropriate technical resources, provision access, and configure the environment accordingly.

By elevating automation to this business terminology level, Swiss Life aims to further reduce friction in the data access process while maintaining its robust security and governance framework. This evolution represents Swiss Life Germany’s commitment to continuously improving how data serves the business, making sophisticated data capabilities increasingly accessible to all parts of the organization.

Conclusion

Through the comprehensive automation of Amazon SageMaker, Swiss Life Germany has transformed their usage of data from a complex technical challenge into a streamlined business enabler. By using AWS services and their innovative Terraform-Lambda integration approach, Swiss Life created a secure, compliant data platform that maintains governance while democratizing access across the full organization. The automated deployment process helps ensure consistency across environments while dramatically reducing the technical knowledge required for teams to begin using advanced data capabilities. Business units, such as the customer portal team, can now focus on deriving insights rather than managing infrastructure, accelerating data-driven decision making throughout the company. This implementation represents a significant milestone in Swiss Life Germany’s cloud journey, demonstrating how thoughtful automation can simultaneously enhance security, improve operational efficiency, and accelerate business outcomes.

As of today, 5 organizational unit teams and 15 IT System teams were onboarded to the platform. To speed things up, Swiss Life has decided to onboard all 180 database clusters and consume data using SageMaker over the coming months. This expansion is designed to enable teams to use the data platform and enhance the efficiency of data discovery and data sharing processes across the organization.

About the authors

Implement a data mesh pattern in Amazon SageMaker Catalog without changing applications

2026-02-23 Paolo Romagnoli

Post Syndicated from Paolo Romagnoli original https://aws.amazon.com/blogs/big-data/implement-a-data-mesh-pattern-with-amazon-sagemaker-catalog-without-making-changes-to-your-applications/

When creating a project in Amazon SageMaker Unified Studio, users select a project profile to define resources and tools to be provisioned in the project. These are used by Amazon SageMaker Catalog to implement a data mesh pattern. Some users don’t want to take advantage of resources provisioned along with the project for various reasons. For instance, they may want to avoid making changes to their existing applications and data products.

This post shows you how to implement a data mesh pattern by using Amazon SageMaker Catalog while keeping your current data repositories and consumer applications unchanged.

Solution overview

In this post, you will simulate a scenario based on data producer and data consumer that exists before Amazon SageMaker Catalog adoption. For this purpose, you will use a sample dataset to simulate existing data and simulate an existing application using an AWS Lambda function. You can apply the same solution to your real-life data and workloads.

The following diagram illustrates the solution architecture’s key configurations. In this architecture, the Amazon Simple Storage Service (Amazon S3) bucket and the AWS Glue Data Catalog in the producer account simulate the existing data repository. The Lambda function in the consumer account simulates the existing consumer application.

AWS cross-account data sharing via SageMaker & Lake Formation: Producer publishes to catalog, Consumer subscribes & accesses data

Here is a description of the key configurations highlighted in the architecture:

As part of an Amazon SageMaker domain, create a producer project (associated to a producer account) and a consumer project (associated to a consumer account). Among other resources, a project AWS Identity and Access Management (IAM) role is created for each project in the associated account.
In the producer account, use AWS Lake Formation to grant producer project’s IAM role permissions to access the existing data asset.
Publish the data asset in the Amazon SageMaker Catalog from the producer project.
Subscribe the data asset from the consumer project.
In the consumer account, configure your Lambda function to assume consumer project’s IAM role to access the subscribed data asset.

The solution architecture is based on the following Amazon Web Services (AWS) services and features:

Amazon SageMaker Catalog offers you a way to discover, govern, and collaborate on data and AI securely.
Amazon SageMaker Unified Studio provides a single data and AI development environment to discover and build with your data. Amazon SageMaker Unified Studio projects provide collaborative boundaries for users to accomplish data and AI tasks.
The lakehouse architecture of Amazon SageMaker is fully compatible with Apache Iceberg. It unifies data across Amazon S3 data lakes, Amazon Redshift data warehouses, and third-party and federated data sources.
AWS Lake Formation, which you can use centrally to govern, secure, and share data for analytics and machine learning.
AWS Glue Data Catalog is a persistent metadata store for your data assets. It contains table definitions, job definitions, schemas, and other control information to help you manage your AWS Glue environment.
Amazon S3 is an object storage service that offers industry-leading scalability, data availability, security, and performance.

Setting up resources

In this section, you will prepare the resources and configurations you need for this solution.

Three AWS accounts

To follow this solution, you need three AWS accounts, and it’s better if they’re part of the same organization in AWS Organizations:

Producer account – Hosts the data asset to be published
Consumer account – Hosts the application that consumes the data published from the producer account
Governance account – Where the Amazon SageMaker Unified Studio domain is configured

Each account must have an Amazon Virtual Private Cloud (Amazon VPC) with at least two private subnets in two different Availability Zones. For instruction, refer to Create a VPC plus other VPC resources. Make sure to create both VPCs in the same Region you plan to apply this solution.

A governance account is used for the sake of convenience, but it’s not strictly needed because Amazon SageMaker can be configured and managed in producer or consumer accounts.If you don’t have access to three accounts, you can still use this post to understand the key configurations required to implement a data mesh pattern with Amazon SageMaker Catalog while keeping your current data repositories and consumer applications unchanged.

Create a data repository in the producer account

First, create a sample dataset by following these instructions:

Open a text editor.

Paste the following text in a new file:

name,stars
	oak,3
	maple,2
	birch,3
	willow,4
	pine,5
	mango,1
	neem,2
	banyan,5
	eucalyptus,3
	teak,2

Save the file as trees.csv. This is your sample data file.

After you create the sample dataset, create an S3 bucket and an AWS Glue database in the producer account, which will act as the data repository.

Create the S3 bucket and upload the trees.csv file in the producer account:

Access the S3 console in the producer account.
Create an S3 bucket. For instructions, refer to Creating a general purpose bucket.
Upload to the S3 bucket the trees.csv sample data file that you created. For instructions, refer to Uploading objects.

Create the AWS Glue database and table in the producer account:

Access the Glue console in the producer account.
In the navigation pane, under Data Catalog, choose Databases.
Choose Add database.
For Name, enter collections.
For Description, enter This database contains collections of statistics for natural resources.
Choose Create database.
In the navigation pane, under Data Catalog, choose Tables.
Choose Add table.
In the table creation guided procedure, enter the following input for Step 1: Set table properties:
1. For Name, enter trees.
2. For Database, select collections.
3. For Description, enter This table captures ratings data related to the characteristics of various tree species.
4. For Table format, select Standard AWS Glue table (default).
5. For Select the type of source, select S3.
6. For Data location is specified in, select my account.
7. For Include path, enter s3://<bucket-name>/<prefix>/ where <bucket-name> is the name of the S3 bucket you created earlier in this procedure and <prefix> is the optional prefix for the trees.csv file you uploaded.
8. For Data format, select CSV.
9. For Delimeter, select Comma (,).
Choose Next.
For Step 2: Choose or define schema, enter the following:
1. For Schema, select Define or upload a schema.
2. Choose Edit schema as JSON and enter the following schema in the pop-up:
```
[
  {
    "Name": "name",
    "Type": "string",
    "Parameters": {}
  },
  {
    "Name": "stars",
    "Type": "string",
    "Parameters": {}
  }
]
```
3. Choose Save.
4. Choose Next.
5. Choose Create.

Create a Lambda function in the consumer account

Create the Lambda function in the consumer account. This will simulate a data consumer application.First, in the consumer account create the IAM policy and the IAM role to be assigned to the Lambda function:

Access the IAM console in the consumer account.

Create an IAM policy and name it smus_consumer_athena_execution by using the following policy. Make sure to replace placeholders <AWS_Region> and <AWS_account_ID_number> with your Region and consumer account ID number. You will replace the <workgroup_id> placeholder later. For IAM policy creation instructions, refer to Create IAM policies (console).

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AthenaExecution",
            "Action": [
                "athena:StartQueryExecution",
                "athena:GetQueryExecution",
                "athena:GetQueryResults"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:athena:<AWS_Region>:<AWS_account_ID_number>:workgroup/<workgroup_id>"
        }
    ]
}

Create an IAM role for AWS Lambda service and name it smus_consumer_lambda. Assign to it the AWS managed permission AWSLambdaBasicExecutionRole and the permission named smus_consumer_athena_execution that you just created. For instructions, refer to Create a role to delegate permissions to an AWS service.

After the IAM role for the Lambda function is in place, you can create the Lambda function in the consumer account:

Access the Lambda console in the consumer account.
In the navigation pane, choose Functions.
Choose Create function and enter the following information:
1. For Function name, enter consumer_function.
2. For Runtime, select Python 3.14.
3. Expand Change default execution role section.
4. For Execution role, select Use an existing role.
5. For Existing role, select smus_consumer_lambda.
Choose Create function.

Under the Code tab, in the Code source, replace the existing code with the following:

import boto3
import time
sts_client = boto3.client('sts')
role_arn = "<role_arn>"
session_name = "AthenaQuerySession"
catalog = "AwsDataCatalog"
database = "<database_name>"
workgroup = "<workgroup_id>"
query = "select * from "+catalog+"."+database+".trees"
def lambda_handler(event, context):
    # Assume SageMaker Unified Studio project role
    assumed_role_object = sts_client.assume_role(
        RoleArn=role_arn,
        RoleSessionName=session_name
    )
    # Get temporary credentials
    credentials = assumed_role_object['Credentials']
    # Create Athena client using temporary credentials
    athena = boto3.client(
        'athena',
        aws_access_key_id=credentials['AccessKeyId'],
        aws_secret_access_key=credentials['SecretAccessKey'],
        aws_session_token=credentials['SessionToken'],
        region_name='eu-west-1'
    )
    # Execute Athena Query
    response = athena.start_query_execution(
        QueryString=query,
        QueryExecutionContext={
            'Database': database,
            'Catalog': catalog
        },
        WorkGroup=workgroup
    )
    query_execution_id = response['QueryExecutionId']
    # Polling with exponential backoff
    wait_time = 0.25  # Start with 0.25 seconds
    max_wait = 8      # Maximum wait time of 8 seconds
    
    while True:
        result = athena.get_query_execution(QueryExecutionId=query_execution_id)
        state = result['QueryExecution']['Status']['State']
        if state in ['FAILED', 'CANCELLED']:
            raise Exception(f"Query {state}")
        elif state == 'SUCCEEDED':
            break
        elif state in ['QUEUED', 'RUNNING']:
            time.sleep(wait_time)
            wait_time = min(wait_time * 2, max_wait)  # Double wait time, cap at max_wait
    # Retrieve results
    results = athena.get_query_results(QueryExecutionId=query_execution_id)
    return results

Choose Deploy.

The code provided for the Lambda function includes some placeholders that you will replace later, after you have the required information. Don’t test the Lambda function at this time because it will fail because of the presence of the placeholders.

Create a user with administrative access

Amazon SageMaker Unified Studio supports two distinct domain types: AWS IAM Identity Center based domains and IAM based domains. At the time of writing this post, only IAM Identity Center based domains support multi-accounts association, therefore in this post you work with this type of domain that requires IAM Identity Center.

In the governance account, you enable IAM Identity Center and create an administrative user to create and manage the Amazon SageMaker Unified Studio domain. Create a user with administrative access:

Enable IAM Identity Center in the governance account. For instructions, refer to Enable IAM Identity Center.
In IAM Identity Center in the governance account, grant administrative access to a user. For a tutorial about using the IAM Identity Center directory as your identity source, refer to Configure user access with the default IAM Identity Center directory.

To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user. For help signing in using an IAM Identity Center user, refer to Sign in to your AWS access portal.

Create a SageMaker Unified Studio domain

To create the Amazon SageMaker Unified Studio domain in the governance account refer to Create a Amazon SageMaker Unified Studio domain – quick setup.

After your domain is created, you can navigate to the Amazon SageMaker Unified Studio portal (a browser-based web application) where you can use your data and configured tools for analytics and AI. Save the Amazon SageMaker Unified Studio portal URL because you will use this URL later.

Solution steps

Now that you have the prerequisites in place, you can complete the following ten high-level steps to implement the solution.

Associate the producer and consumer accounts to the Amazon SageMaker Unified Studio domain

Start by associating the producer and consumer accounts to the newly created Amazon SageMaker Unified Studio domain. When you associate your producer and consumer accounts to the domain, make sure to select IAM users and roles can access APIs and IAM users can log in to Amazon SageMaker Unified Studio in the AWS RAM share managed permission section. For step-by-step instructions, refer to Associated accounts in Amazon SageMaker Unified Studio. If your AWS accounts are part of the same organization, your association requests are automatically accepted. However, if your AWS accounts aren’t part of the same organization, request association with the other AWS accounts in the governance account and then accept the association request in both the producer and consumer accounts.

Create two project profiles

Now, create two project profiles, one for the producer project and one for the consumer project.

In Amazon SageMaker Unified Studio, a project profile defines an uber template for projects in your Amazon SageMaker domain. A project profile is a collection of blueprints that provides reusable AWS CloudFormation templates used to create project resources.

A project profile is associated to a specific AWS account. This means, when a project is created the blueprints listed in the project profile are deployed in the associated AWS account. To use a project profile, you must enable its blueprints in the AWS account associated to the project profile.

Create the producer project profile

You’re going to create the producer project profile that is associated to the producer account. This project profile will be used to create the producer project. This profile includes by default the Tooling blueprint that creates resources for the project, including IAM user roles and security groups.

Before creating the project profile, you will enable the Tooling blueprint in the producer account using the following procedure:

Access the SageMaker console in the producer account.
In the navigation pane, choose Associated domains.
Select the domain you created while setting up.
On the Blueprints tab, choose Enable in the Tooling blueprint section as shown in the following image:

SageMaker Unified Studios Tooling blueprint config: disabled status with Enable button for IAM roles & AWS resource setup

For Virtual private cloud (VPC) select your account VPC.
For Subnets, select at least two subnets in different Availability Zones.
Choose Enable blueprint.

Proceed to creating the project profile in the governance account:

Access the SageMaker console in the governance account.
In the navigation pane, choose Domains.
Select the domain you created as part of prerequisites.
Under the Project profiles tab, choose Create and enter the following information:
1. For Project profile name, enter producer-project-profile.
2. For Project profile creation options, select Custom create.
3. DO NOT SELECT A BLUEPRINT for Blueprints because the Tooling blueprint is included by default in any project profile.
4. For Account, select Provide an account ID.
5. For Account ID, enter the producer account ID.
6. For Region, select Provide region name and then select the Region in which you’re working.
7. For Authorization, select Allow all users and groups.
8. For Project profile readiness, select Enable project profile on creation.
Choose Create project profile.

Create a consumer project profile

You also create a consumer project profile and associate it to the consumer account. This profile will be used to create the consumer project. The consumer project profile includes the LakeHouseDatabase blueprint, which is needed to create a lakehouse environment with an AWS Glue database for data management and an Amazon Athena workgroup for querying. The Tooling blueprint is included by default in the project profile.

Before creating the project profile, enable the Tooling and LakeHouseDatabase blueprints in the consumer account:

Access the SageMaker console in the consumer account.
In the navigation pane, choose Associated domains.
Select the domain you created as part of prerequisites.
On the Blueprints tab, choose Enable in the Tooling blueprint section.
For Virtual private cloud (VPC) select your account VPC.
For Subnets, select at least two subnets in different Availability Zones.
Choose Enable blueprint.
In the navigation pane, choose Associated domains.
Select the domain you created as part of prerequisites.
Under the Blueprints tab, select the LakeHouseDatabase blueprint.
Choose Enable.
Choose Enable blueprint.

After blueprints are enabled in the consumer account, you can proceed creating the project profile:

Access the SageMaker console in the governance account.
In the navigation pane, choose Domains.
Select the domain you created as part of prerequisites.
Under Project profiles tab choose Create and enter the following information:
1. For Project profile name, enter consumer-project-profile.
2. For Project profile creation options, select Custom create.
3. For Blueprints, select LakeHouseDatabase.
4. For Account, select Provide an account ID.
5. For Account ID, enter the consumer account ID.
6. For Region, select Provide region name and then select the Region you are working.
7. For Authorization, select Allow all users and groups.
8. For Project profile readiness, select Enable project profile on creation.
Choose Create project profile.

Create SageMaker Unified Studio producer and consumer projects

In Amazon SageMaker Unified Studio, a project is a boundary within a domain where you can collaborate with other users to work on a business use case. In projects, you can create and share data and resources.To create producer and consumer projects in Amazon SageMaker Unified Studio use the following instructions:

Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list.
Choose Create project and enter the following information:
1. For Project name, enter Producer.
2. For Project profile, select producer-project-profile.
Choose Continue.
Choose Continue.
Choose Create project.

After you’ve created the Producer project, note in a text file the Project role ARN that is displayed in the Project overview. The following image is shown for reference. The project role name is the string that follows arn:aws:iam::<account_ID>:role/ in the project role Amazon Resource Name (ARN). You will use both project role name and ARN later.

SageMaker Producer project overview: active status, files listed, S3 location & IAM role ARN displayed in project details tab

Repeat the preceding procedure to create the Consumer project. Be sure to enter Consumer for Project name and then select consumer-project-profile for Project profile. After it’s created, note the Project role ARN in a text file. The project role name is the string that follows arn:aws:iam::<account_ID>:role/ in the project role ARN. You will use both project role name and ARN later.

Bring your own data from the producer account

Bring your own data to the Amazon SageMaker Unified Studio Producer project. AWS provides several options to achieve this onboarding. The first option is automated onboarding in Amazon SageMaker lakehouse, in which you ingest the Amazon SageMaker lakehouse metadata of datasets into Amazon SageMaker Catalog. With this option, you can onboard your Amazon SageMaker lakehouse data as part of creating a new Amazon SageMaker Unified Studio domain or for an existing domain.

For more information about automated onboarding of Amazon SageMaker lakehouse data, refer to Onboarding data in Amazon SageMaker Unified Studio. As other options, you can bring in existing resources to your Amazon SageMaker Unified Studio project by using the Data and Compute pages in your project, or by using scripts provided in GitHub. For more information about using the Data and Compute pages or about using scripts, refer to Bringing existing resources into Amazon SageMaker Unified Studio. In this post, you will use Amazon SageMaker lakehouse capabilities to import your trees AWS Glue table into the Producer project.

Register the Amazon S3 location for the table

To use Lake Formation permissions for fine-grained access control to the trees table, you need to register in Lake Formation the Amazon S3 location of the trees table. To do that, complete the following actions:

Access the Lake Formation console in the producer account.
In the navigation pane under Administration, choose Data lake locations.
Choose Register location and enter the following information:
1. For S3 URI, enter s3://<bucket-name>/<prefix>/ where <bucket-name> is the name of the S3 bucket you created in the prerequisites and <prefix> is the optional prefix for the trees.csv file you uploaded as part of the prerequisite.
2. For IAM role, select AWSServiceRoleForLakeFormationDataAccess.
3. For Permission mode, select Lake Formation.
Choose Register location.

Grant `Producer` project role permissions on the database

Grant database access to the IAM role that is associated with your Producer project. This role is called the project role, and it was created in IAM upon project creation.

To access the AWS Glue Data Catalog collections database from the Producer project in the Amazon SageMaker Unified Studio, complete the following actions:

Access the Lake Formation console in the producer account.
In the navigation pane under Data Catalog, choose Databases.
Choose the collections database.
From the Actions menu, choose Grant and enter the following information:
1. For IAM users and roles, select your Producer project’s role name. This is the string starting with datazone_usr_role_ that is part of the Producer project role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”.
2. For Database permissions, select Describe.
Choose Grant.

Grant `Producer` project role permissions on the table

Grant trees table access to the IAM role that is associated with your Producer project. To grant these permissions use the following instructions:

Access the Lake Formation console in the producer account.
In the navigation pane under Data Catalog, choose Tables and MVs.
Select the trees table.
From the Actions menu, choose Grant and enter the following information:
1. For IAM users and roles, select your Producer project’s role. This is the string starting with datazone_usr_role_ that is part of the Producerproject role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”.
2. For Table permissions, select Select and Describe.
3. For Grantable permissions, select Select and Describe.
Choose Grant.

Revoke any existing permissions of `IAMAllowedPrincipals`

You must revoke the IAMAllowedPrincipals group permissions on both the database and table to enforce Lake Formation permission for access. For more information, refer to Revoking permission using the Lake Formation console.

Access the Lake Formation console in the producer account.
In the navigation pane under Permission, choose Data permissions.
Select the entries where Principal is set to IAMAllowedPrincipals and Resource is set to collections or trees as in the following image:

Data permissions table: 2 of 5 IAMAllowedPrincipals entries selected. All permissions granted for collections DB & trees table

Choose Revoke.
Enter revoke.
Choose Revoke again.

Verify that data is available in the `Producer` project

Verify that your collections database and trees table are accessible in the Producer project:

Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project drop-down menu and choose the Producer project.
In the navigation pane under Overview, choose Data.
Choose Lakehouse.
Choose AwsDataCatalog.
Choose collections.
Choose tables.
Choose the three-dot action menu next to your trees table and choose Preview data, as shown in the following image.
You’ll find data from the trees table as shown in the following image.

Create Amazon SageMaker Catalog asset

Even if it’s accessible in the project, to work with the trees table in Amazon SageMaker Catalog, you need to register the data source and create an Amazon SageMaker Catalog asset:

Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list and choose the Producer project.
On the project page, under Project catalog in the navigation pane, choose Data sources.
Choose Create Data Source and make the following selections:
1. For Name, enter collections.
2. For Data source type, select AWS Glue (Lakehouse).
3. For Database name, select collections.
4. Choose Next.
5. Choose Next.
6. Choose Next.
7. Choose Create.
After the data source is created, you will be in the collections data source page, choose Run. This will import metadata and create the Amazon SageMaker Catalog asset.
In the collections data source, on the Data source runs tab, you’ll find your run marked as Completed and the trees asset Successfully created, as shown in the following image:

Publish the data asset in the Amazon SageMaker Catalog

Publishing a data asset manually is a one-time operation that you need to perform to allow others to access the data asset through the catalog:

Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list and choose the Producer project.
On the project page under Project catalog, choose Assets.
Select your trees data asset that is available on the Inventory tab. The following image is shown for reference.
(Optional) If automated metadata generation is enabled when the data source is created, metadata for assets (such as the asset business name) is available to review and accept or reject. You can either choose Accept All or Reject All in the Automated Metadata Generation banner.
Choose Publish Asset. The following image is shown for reference.
Choose Publish Asset.

Subscribe to the data asset in the Amazon SageMaker Catalog

To consume data assets in the Consumer project, subscribe to the data asset by creating a subscription request:

Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list and choose Consumer project.
On the Discover menu, choose Catalog.
Enter trees in the search box and then select the data asset returned from the search. If in step 7 “Publish the data asset in the Amazon SageMaker Catalog” you chose Accept All in the Automated Metadata Generation banner, your data asset will have a different business name generated by the automated metadata recommendations feature. The data asset technical name is trees. For reference, refer to the following image.
Choose Subscribe.
For Comment, enter a justification such as This data asset is needed for model training purposes.
Choose Subscribe again.

By default, asset subscription requests require manual approval by a data owner. However, if the requester in the Consumer project is also a member of the Producer project, the subscription request is automatically approved. For information about approving subscription requests, refer to Approve or reject a subscription request in Amazon SageMaker Unified Studio.

Configure your Lambda IAM role to access the subscribed data access

To enable your Lambda function access to the subscribed data asset, you need to allow the Lambda function to assume the Consumer project role. To do this, edit the Consumer project’s IAM role trust relationship:

Navigate to the IAM console in the consumer account.
In the navigation pane under Access management, choose Roles.
Select the Consumer project’s IAM role. This is the string starting with datazone_usr_role_ that is part of the Consumer project role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”.
Under the Trust relationships tab, choose Edit trust policy.
For backup reasons, make a copy of the existing trust policy in a text file.
In the Edit trust policy window, add the following statement to the existing trust policy without removing or overwriting other existing statements in the trust policy. Be sure to replace the placeholder <account_id> with your consumer AWS account ID.
```
{
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::<account_id>:role/smus_consumer_lambda"
    },
    "Action": [
        "sts:AssumeRole"
    ]
}	
```
Choose Update policy.

Test the Lambda function’s access to the subscribed data asset

Before you can test your Lambda function, you need to replace placeholders in the function code and in the IAM policy. There are three placeholders to be replaced: <role_arn>, <database_name> and <workgroup_id>. For <role_arn>, you already have the actual value, which is the Consumer project’s role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”. The next sections provide instructions to retrieve values for the other placeholders.

Retrieve the AWS Glue Data Catalog database name

You need to find the name of the AWS Glue Data Catalog database that was created along with the Consumer project. You will then use this value to replace the <database_name> placeholder in the consumer_function Lambda function code. To retrieve the AWS Glue Data Catalog database name, follow these instructions:

Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list and choose Consumer project.
On the project page, under Overview, choose Data.
Choose Lakehouse.
Choose AwsDataCatalog.
Copy the name of the database. It should be an alphanumerical string starting with glue_db, as in the following image:

Copy the Workgroup ARN and save to a text file. The Athena workgroup ID is the string that follows arn:aws:athena:<region>:<account_ID>:workgroup/ in the Workgroup ARN.

Replace placeholder in the `smus_consumer_athena_execution` IAM policy

To replace the <workgroup_id> placeholder in the smus_consumer_athena_execution IAM policy, use the following procedure:

Access the IAM console in the consumer account.
In the navigation pane, choose Policies.
In the search field enter smus_consumer_athena_execution.
Select the smus_consumer_athena_execution policy.
Choose Edit.
Replace <workgroup_id> with the value you noted earlier.
Choose Next.
Choose Save changes.

Replace placeholders in the Lambda function code and test it

In this section, you will replace the <role_arn>, <database_name> and <workgroup_id> placeholders in the consumer_function Lambda function code, and then you can test the function ability to access data of the trees table.

Access the Lambda console in the consumer account.
In the navigation pane, choose Functions.
Select consumer_function.
Under the Code tab, replace <role_arn>, <database_name> and <workgroup_id> placeholders with the respective values you noted earlier.
Choose Deploy.
Under the Test tab, for Event name, enter mytest.
Choose Test.
Choose Details in the green banner titled Executing function that appears after the execution is completed.
The execution log reports the trees table content, as shown in the following image:

If your Lambda function execution fails due to timeout, change the function timeout setting as follows:

Access the Lambda console in the consumer account.
In the navigation pane, choose Functions.
Select consumer_function.
Under the Configuration tab, choose Edit.
For Timeout, enter 15 sec or a greater value.
Choose Save.

After increasing the timeout, test the function again.

Clean up

If you no longer need the resources you created as you followed this post, delete them to prevent incurring additional charges. Start by deleting your Amazon SageMaker Unified Studio domain in the governance account. For more information, refer to Delete domains.

To remove the AWS Glue collections database from the producer account, follow these steps:

Access the Glue console in the producer account.
In the navigation pane under Data Catalog, choose Databases.
Select the collections database.
Choose Delete.
Choose Delete.

To remove the S3 bucket from the producer account, empty the bucket and then you can delete the bucket. For information about emptying the bucket, refer to Emptying a general purpose bucket. For information about deleting the bucket, refer to Deleting a general purpose bucket.

To remove the Lambda function from the consumer account, follow these steps:

Access the Lambda console in the consumer account.
In the navigation pane, choose Functions.
Select the consumer_function Lambda function.
Choose the Actions menu and then choose Delete function.
Enter confirm.
Choose Delete.

To complete the cleanup, delete the IAM role named smus_consumer_lambda, then delete the IAM policy named smus_consumer_athena_execution in the consumer account. For information about removing a IAM role, refer to Delete roles or instance profiles. For information about removing an IAM policy, refer to Delete IAM policies.

Conclusion

In this post, we covered adopting Amazon SageMaker Catalog for data governance without rearchitecting your existing applications and data repositories. We walked through how to onboard existing data in Amazon SageMaker Unified Studio, then publish it in a catalog, and then subscribe and consume the data from resources deployed outside the context of an Amazon SageMaker Unified Studio project. This solution can help you accelerate your implementation of a data mesh pattern with Amazon SageMaker Catalog to publish, find, and access data securely in your organization.

For more information, refer to What is Amazon SageMaker? and work through the Amazon SageMaker Workshop to try the unified experience for data, analytics, and AI.

About the authors

How CyberArk uses Apache Iceberg and Amazon Bedrock to deliver up to 4x support productivity

2026-02-18 Moshiko Ben Abu

Post Syndicated from Moshiko Ben Abu original https://aws.amazon.com/blogs/big-data/how-cyberark-uses-apache-iceberg-and-amazon-bedrock-to-deliver-up-to-4x-support-productivity/

This post is co-written with Moshiko Ben Abu, Software Engineer at CyberArk.

CyberArk achieved up to 95% reduction in case resolution time using Amazon Bedrock and Apache Iceberg.

This improvement addresses a challenge in technical support workflow: when a support engineer receives a new customer case, the biggest bottleneck is often not diagnosing the problem but preparing the data. Customer logs arrive in different formats from multiple vendors, and each new log format typically requires manual integration and correlation before an investigation can begin. For simple cases, this process can take hours. For more complex investigations, it can take days, slowing resolution and reducing overall engineer productivity.

CyberArk is a global leader in identity security. Centered on intelligent privilege controls, it provides comprehensive security for human, machine, and AI identities across business applications, distributed workforces, and hybrid cloud environments.

In this post, we show you how CyberArk redesigned their support operations by combining Iceberg’s intelligent metadata management with AI-powered automation from Amazon Bedrock. You’ll learn how to simplify data processing flows, automate log parsing for diverse formats, and build autonomous investigation workflows that scale automatically.

To achieve these results, CyberArk needed a solution that could ingest customer logs, automatically structure them, establish relationships between related events, and make everything queryable in minutes, not days. The architecture had to be serverless to handle unpredictable support volumes, secure enough to protect customer Personally Identifiable Information (PII), and fast enough to allow same day case resolution.

The legacy architecture: Bottlenecks and manual workflows

When support engineers received customer cases, they would upload log files to the data lake stored in Amazon Simple Storage Service (Amazon S3). The original design then suffered from the complexity of multi-step raw data processing.

First, CyberArk’s custom parsing logic running on AWS Fargate would parse these uploaded log files and transform the raw data. During this stage, the system also had to scan for PII and mask sensitive data to protect customer privacy.

Next, a separate process converted the processed data into Parquet format.

Finally, AWS Glue crawlers were required to discover new partitions and update table metadata for processed Parquet files. This dependency became the most complex and time-consuming part of the pipeline. Crawlers ran as asynchronous batch jobs rather than in real time, often introducing delays of minutes to hours before support engineers could query the data.

But the inefficiency went deeper than just architectural complexity. CyberArk supports customers running diverse product environments across multiple vendors. Each vendor and product produces logs in different formats with unique schemas, field names, and structures. Adding support for a new vendor meant days of integration work to understand their log format and build custom parsers.

Figure 1: Legacy log ingestion architecture diagram showing the flow from S3 upload through AWS Fargate processing with AWS Glue Crawler

Beyond ingestion, the investigation process itself was manual and time consuming. Support engineers would manually query data, correlate events across different log sources, search through product documentation, and piece together root cause analysis through trial and error. This process required deep product expertise and could take hours or days depending on issue complexity. The new architecture addresses these inefficiencies through three key innovations:

Single stage serverless processing: AWS Fargate with PyIceberg directly creates Iceberg tables from raw logs in one pass, removing intermediate processing steps and crawler dependencies entirely.
AI powered dynamic parsing: Amazon Bedrock automatically generates grok patterns for log parsing by analyzing file schemas, transforming what was once a manual, time consuming process into a fully automated workflow.
Autonomous investigation with AI Agents: AI Agents autonomously perform complete root cause analysis by querying log data, analyzing product knowledge bases, identifying event flows, and recommending solutions, transforming hours of manual investigation into minutes of automated intelligence.

The solution: AI-powered automation meets single-stage Iceberg processing

The new system delivers zero touch log processing from upload to query. Support engineers simply upload customer log ZIP files to the system. Here’s where the transformation happens: CyberArk’s custom processing logic still runs on AWS Fargate, but now it uses Amazon Bedrock to intelligently understand the data.

Zero-touch log processing workflow

The system extracts sample log entries from the uploaded log files and sends them to Amazon Bedrock along with context about the log source and table schema from AWS Glue Data Catalog. Amazon Bedrock analyzes the samples, understands the structure, and automatically generates grok patterns optimized for the specific log format.

Grok patterns are structured expressions that define how to extract meaningful fields from unstructured log text. For example, the following grok pattern specifies that a timestamp appears first, followed by a severity level, then a message body %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:severity} %{GREEDYDATA:message}

The system validates these grok patterns against additional samples to verify accuracy before applying them to parse the complete log file. Successfully validated grok patterns are stored in Amazon DynamoDB, creating a repository of known patterns. When the system encounters similar log formats in future uploads, it can retrieve these patterns directly from Amazon DynamoDB, avoiding redundant grok pattern generation. Amazon Bedrock processes log samples in real-time without retaining customer data or using it for model training, maintaining data privacy.

This entire process invokes Claude 3.7 Sonnet model from Amazon Bedrock and is orchestrated by AWS Fargate tasks with retry logic for reliability. The processing uses these AI-generated grok patterns to parse the logs and create or update Iceberg tables using PyIceberg APIs without human intervention.

This automation reduced logs onboarding time from days to minutes, enabling CyberArk to handle diverse customer environments without manual intervention.

Figure 2: Log ingestion architecture diagram showing the flow from S3 upload through AWS Fargate processing with Amazon Bedrock integration to Iceberg table creation

Apache Iceberg: Simplified architecture, faster queries

Iceberg simplified and improved CyberArk’s data lake architecture by addressing the two primary bottlenecks in the legacy system: slow schema management and inefficient query performance.

Built-in schema evolution removes crawler dependency

In the legacy architecture, AWS Glue crawlers became a source of operational overhead and latency. Even when triggered on demand, crawlers ran as batch jobs over S3 prefixes to discover partitions and update metadata. As data volumes grew and datasets diversified across vendors and schemas, teams had to manage and operate a growing number of crawler jobs. The resulting delays, often ranging from minutes to hours, slowed data availability and downstream investigation workflows.

Iceberg removes this entire layer of complexity. Iceberg’s intelligent metadata layer automatically tracks table structure, schema changes, and partition information as data is written. When CyberArk’s processing creates or updates Iceberg tables through PyIceberg, the metadata is updated instantly and atomically. There’s no waiting for crawlers jobs to complete, and no risk of stale metadata. The moment data is written, it’s immediately queryable in Amazon Athena.

PyIceberg: Making Iceberg accessible beyond Apache Spark

Working with Iceberg usually involved Apache Spark and the complexity of distributed data processing. PyIceberg changed that by letting CyberArk create and manage Iceberg tables using a simple Python library. CyberArk’s data engineers could write straightforward Python code running on AWS Fargate to create Iceberg tables directly from parsed logs, without spinning up Spark clusters.

This accessibility was essential for CyberArk’s serverless architecture. PyIceberg enabled single stage processing where AWS Fargate tasks could parse logs, apply PII masking, and create Iceberg tables in one pass. The result was simpler code and lower operational overhead.

Metadata-driven query optimization delivers speed

In addition to removing crawlers, Iceberg significantly improved query performance through its intelligent metadata architecture. Iceberg maintains detailed statistics about data files, including min/max values, null counts, and partition information. When support engineers query data in Athena, Iceberg’s metadata layer supports partition pruning and file skipping, making sure queries only read the specific files containing relevant data. For CyberArk’s use case, where tables are partitioned by case ID, this means a query for a specific support case only reads the files for that case, ignoring potentially thousands of irrelevant files. This metadata driven optimization reduced query execution time from minutes to seconds, allowing support engineers to interactively explore data rather than waiting for results.

ACID transactions maintain data consistency

In a multi user support environment where multiple engineers may be analyzing overlapping cases or uploading logs simultaneously, data consistency is essential. Iceberg’s ACID transaction support helps verify that concurrent writes do not corrupt data or create inconsistent states. Each table update is atomic, isolated, and durable, providing the reliability CyberArk needed for production support operations.

Time travel enables historical analysis

Iceberg’s built-in versioning allows support engineers to query historical states of data, essential for understanding how customer issues evolved over time. If an engineer needs to see what the logs looked like when a case was first opened versus after a customer applied a patch, Iceberg’s time travel capabilities make this straightforward. This feature proved essential for complex troubleshooting scenarios where understanding the timeline of events was critical to resolution.

Automated table optimization with AWS Glue

Iceberg tables require periodic maintenance to maintain query performance.

CyberArk enabled AWS Glue automatic table optimization for their Iceberg tables, which handles compaction and expired snapshot cleanup in the background.

For CyberArk’s continuous upload workflow, this automation avoids performance degradation over time. Tables stay optimized without manual intervention from the engineering team.

AI Agents: Autonomous investigation workflow

While the Claude 3.7 Sonnet model from Amazon Bedrock automates grok pattern generation for log ingestion, the more advanced use of Amazon Bedrock comes in the investigation workflow. We use AI agents with Bedrock models to change how support engineers analyze and resolve customer issues.

From manual analysis to AI powered investigation

In the legacy workflow, support engineers would manually query data, correlate events across different log sources, search through product documentation, and piece together root cause analysis through trial and error. This process required deep product expertise and could take hours or days depending on issue complexity. AI Agents automate this entire investigation process. Support engineers use an internal portal to ask questions in natural language about customer issues, questions like
“Show me authentication errors for case 12345 in the last 24 hours”, “What were the most common errors across cases opened this week?” or “Compare the error patterns between case 12345 and case 12346.”

Behind the scenes, the system fires specialized AI Agents that autonomously perform thorough analysis.

How support agents work

Each AI Agent operates as an intelligent investigator with a clear mission: understand what happened, determine why it happened, and recommend how to fix it. When a support engineer asks a question, the agent collects relevant data by querying Athena to retrieve log data from Iceberg tables, filtering for the specific case and time period relevant to the investigation. The agent then accesses CyberArk’s internal knowledge base for the specific product involved, understanding known issues, common error patterns, and documented solutions. The agent then performs the following analysis:

Flow identification: Analyzes the sequence of events in the logs to understand what actually happened during the customer’s issue
Root cause determination: Correlates log events with product knowledge to identify the underlying cause of the problem
Solution recommendations: Suggests specific remediation steps based on the root cause analysis and known resolution patterns

This entire process happens in minutes, delivering advanced analysis that would have taken support engineers hours to perform manually.

For complex cases where a solution is not found, the support agent escalates to another, specialized agent that interacts with service engineers to collect additional inputs and expertise. This human-in-the-loop approach makes sure that even the most challenging cases receive appropriate attention while still benefiting from the automated investigation workflow. The insights gathered from these escalated cases are automatically fed back into CyberArk’s knowledge base, continuously improving the system’s ability to handle similar issues autonomously in the future.

Amazon Bedrock never shares customer data with model providers or uses it to train foundation models, case data and investigation insights remain within CyberArk’s environment.

Concurrent agent execution at scale

When multiple support engineers investigate different cases simultaneously, the solution runs specialized agents concurrently. CyberArk currently uses Claude 3.7 Sonnet as the foundation model for these agents. Each agent works independently on its assigned investigation, operating in parallel without resource contention. This concurrent execution allows the investigation workflow to scale automatically with support volume, handling peak loads without performance degradation.

AI-powered investigation advantage

This AI-powered investigation workflow delivers two key advantages.

Investigations that took hours now complete in minutes, enabling support engineers to resolve up to 4x more cases per day.

The system also creates a continuous learning feedback loop. When cases require manual resolution by engineers, these resolutions are automatically recorded and fed back into the knowledge base. Future investigations benefit from this accumulated expertise, with agents applying lessons learned from previous manual resolutions to similar cases. Amazon Bedrock doesn’t use customer data to train foundation models. Case data and investigation insights remain within CyberArk’s environment.
This automated feedback mechanism means the investigation workflow becomes more effective over time, continuously improving resolution accuracy and speed.

CyberArk - AI Powered Logs Investigation Flow

Figure 3: Investigation workflow diagram showing natural language query through AI Agents to Athena queries and knowledge base analysis

Scaling without proportional engineering growth

The business impact of this AI automation is significant. CyberArk can expand its vendor coverage and product portfolio without adding data engineering headcount. The same system that handles today’s log types will automatically handle tomorrow’s additions, whether that’s ten new formats or thousands, significantly reducing time to market for new product and vendor integrations.

The results: Significant improvements in resolution time and productivity

The transformation delivered measurable improvements across every key metric.

Resolution time: CyberArk achieved up to 95% reduction in time from case assignment to resolution. Simple cases that used to take 4 to 6 hours now take just 15 to 30 minutes. Complex cases that previously took up to 15 days are now completed in 2 to 4 hours.

Engineer productivity: Support engineers now handle 8 to 12 cases per day, compared to just 2 to 3 cases before. This means each engineer is helping up to 4x more customers.

Data availability: Logs are queryable within minutes of upload instead of waiting hours or days. Support engineers can start investigating issues almost immediately after receiving customer data.

Operational efficiency: The system requires zero manual intervention for new log formats or schema changes. Cases that used to require days of data engineering work now happen automatically.

Cost optimization: The serverless architecture alleviated idle infrastructure costs while scaling automatically with demand. CyberArk only pays for what they use, when they use it.

Customer satisfaction: Faster resolution times and proactive issue identification significantly improved the customer experience. Problems get solved in hours instead of days, and customers spend less time waiting for answers.

What’s next?

While AWS continues to innovate across both data lake management and agentic AI infrastructure, the following capabilities align well with CyberArk’s architecture and may offer additional operational benefits as the system scale.

Agent infrastructure maturity

As the agent-based architecture scales to handle thousands of concurrent investigations, CyberArk is transitioning to Amazon Bedrock AgentCore for future agent deployments. AgentCore provides a managed runtime for production AI agents with enhanced observability through AWS X-Ray integration, intelligent memory for context retention across sessions, and streamlined operational workflows. While the current AI Agents implementation delivers the performance and reliability CyberArk needs today, AgentCore represents a natural evolution path as operational requirements grow, offering framework-agnostic deployment, automatic scaling, and comprehensive monitoring capabilities without infrastructure management overhead.

Amazon S3 Tables

CyberArk’s current architecture uses Iceberg tables stored in Amazon S3 buckets. Amazon S3 Tables offers fully managed Iceberg tables with built-in optimization.

As CyberArk continue to scale with hundreds of Iceberg tables and rapid data growth, CyberArk is exploring a migration to Amazon S3 Tables to further reduce operational overhead.

S3 Tables remove the need to set up and monitor AWS Glue maintenance jobs. It automatically performs maintenance to enhance the performance of Iceberg tables, including unreferenced file removal, file compaction, and snapshot management. Additionally, S3 Tables provides Intelligent-Tiering that automatically moves data between storage classes based on access patterns, optimizing storage costs without manual intervention.

Because S3 Tables uses Iceberg open table format, migration would not require changes to existing Athena queries and PyIceberg code. This flexibility allows CyberArk to evaluate and adopt S3 Tables when the operational and cost benefits align with their business needs.

Conclusion

CyberArk’s transformation demonstrates how combining modern data lake architecture with AI automation can significantly change operational economics. By combining Iceberg’s intelligent metadata management with AI-powered automation from Amazon Bedrock, CyberArk transformed case resolution from days to minutes while enabling support operations to scale automatically with business growth. Support engineers now spend their time solving customer problems instead of wrangling data, customers receive faster resolutions, and the system scales automatically with the business.

To learn more about Iceberg on AWS, refer to Working with Amazon S3 Tables and table buckets and Using Apache Iceberg on AWS. To learn more about Amazon Bedrock AgentCore, refer to Amazon Bedrock AgentCore.

About the authors

Best practices for right-sizing Amazon OpenSearch Service domains

2026-02-18 Nikhil Agarwal

Post Syndicated from Nikhil Agarwal original https://aws.amazon.com/blogs/big-data/best-practices-for-right-sizing-amazon-opensearch-service-domains/

Amazon OpenSearch Service is a fully managed service for search, analytics, and observability workloads, helping you index, search, and analyze large datasets with ease. Making sure your OpenSearch Service domain is right-sized—balancing performance, scalability, and cost—is critical to maximizing its value. An over-provisioned domain wastes resources, whereas an under-provisioned one risks performance bottlenecks like high latency or write rejections.

In this post, we guide you through the steps to determine if your OpenSearch Service domain is right-sized, using AWS tools and best practices to optimize your configuration for workloads like log analytics, search, vector search, or synthetic data testing.

Why right-sizing your OpenSearch Service domain matters

Right-sizing your OpenSearch Service domain provides optimal performance, reliability, and cost-efficiency. An undersized domain leads to high CPU utilization, memory pressure, and query latency, whereas an oversized domain drives unnecessary spend and resource waste. By continuously matching domain resources to workload characteristics such as ingestion rate, query complexity, and data growth, you can maintain predictable performance without overpaying for unused capacity.

Beyond cost and performance, right-sizing facilitates architectural agility. It helps make sure your cluster scales smoothly during traffic spikes, meets SLA targets, and sustains stability under changing workloads. Regularly tuning resources to match actual demand optimizes infrastructure efficiency and supports long-term operational resilience.

Key Amazon CloudWatch metrics

OpenSearch Service provides Amazon CloudWatch metrics that offer insights into various aspects of your domain’s performance. These metrics fall into 16 different categories, including cluster metrics, EBS volume metrics, and instance metrics. To determine if your OpenSearch Service domain is misconfigured, monitor these common symptoms that indicate resizing or optimization may be necessary. These are caused by imbalances in resource allocation, workload demands, or configuration settings. The following table summarizes these parameters:

CloudWatch Metrics	Parameter
CPU Utilization Metrics	`CPUUtilization:` Average CPU usage across all data nodes. Optimal range: 60-80% for sustained workloads Primary control plane CPU utilization (for dedicated primary nodes): Average CPU usage on primary nodes. Optimal range: Under normal conditions <50%
Memory Utilization Metrics	`JVMMemoryPressure:` Percentage of heap memory used across data nodes. Optimal range: 65–85% Note: With Garbage First Garbage Collector (G1GC), JVM may delay collections to optimize performance. Evaluate `JVMMemoryPressure` together with GC metrics (Old Gen usage and GC pause time) to confirm true pressure trends. `MasterJVMMemoryPressure`: Heap usage on dedicated primary nodes. Optimal range: <80% Note: Occasional spikes are normal during state updates; sustained high memory pressure warrants scaling or tuning.
Storage Metrics	`StorageUtilization`: Percentage of storage space used. Optimal range: 70–85% `FreeStorageSpace`: Available storage in MB. Critical threshold: When approaching the read-only threshold.
Node Level Search and Indexing Performance (These latencies are not per-request latencies or rate, but at node level based on shards assigned to a node.)	`SearchLatency`: Average time for search requests. Baseline establishment: Monitor during normal operations. `IndexingLatency`: Average time for indexing operations. Impact: Can indicate CPU or I/O bottlenecks. `SearchRate` and `IndexingRate`: Requests per minute for search and indexing. Usage: Correlate with latency metrics to understand performance impact.
Cluster Health Indicators	`ClusterStatus.yellow` and `ClusterStatus.red`: Yellow status: Some replica shards are unassigned. Red status: Some primary shards are unassigned (data loss risk). Nodes What it measures: Number of nodes in the cluster. Usage: Track node failures and recovery patterns.

Signs of under-provisioning

Under-provisioned domains struggle to handle workload demands, leading to performance degradation and cluster instability. Look for sustained resource pressure and operational errors that signal the cluster is running beyond its limits. For monitoring, you can set CloudWatch alarms to catch early signals of stress and prevent outages or degraded performance. The following are critical warning signs:

High CPU utilization for data nodes (>80%) sustained over time (such as more than 10 minutes)
High CPU utilization for primary nodes (>60%) sustained over time (such as more than 10 minutes)
JVM memory pressure consistently high (>85%) for data and primary nodes
Storage utilization reaching high (>85%)
Increasing search latency with stable query patterns (increasing by 50% from baseline)
Frequent cluster status yellow/red events
Node failures under normal load conditions

When resources are constrained, the end-user experience suffers with slower searches, failed indexing, and system errors. The following are key performance impact indicators:

Search timeouts increasing
Indexing delays growing
Circuit breaker exceptions
Rejected execution exceptions in logs

Remediation recommendations

The following table summarizes CloudWatch metric symptoms, possible causes, and potential solutions.

CloudWatch metric symptom	Causes and solution
`FreeStorageSpace` drops <20%	Storage pressure occurs when data volume outgrows local storage due to high ingestion, long retention without cleanup, or unbalanced shards. Lack of tiering (such as UltraWarm) further worsens capacity issues. Solution: Free up space by deleting unused indexes or automating cleanup with ISM and use force merge on read-only indexes to reclaim storage. If pressure persists, scale vertically or horizontally, use UltraWarm or cold storage for older data, and adjust shard counts at rollover for better balance.
`CPUUtilization` and `JVMMemoryPressure` consistently >70%	High CPU or JVM pressure arises when instance sizes are too small or shard counts per node are excessive, leading to frequent GC pauses. Inefficient shard strategy, uneven distribution, and poorly optimized queries or mappings further spike memory usage under heavy workloads. Solution: Address high CPU/JVM pressure by scaling vertically to larger instances (such as from r6g.large to r6g.xlarge) or adding nodes horizontally. Optimize shard counts relative to heap size, smooth out peak traffic, and use slow logs to pinpoint and tune resource-heavy queries.
`SearchLatency` or `IndexingLatency` spikes >500 milliseconds	Thread pool rejections often stem from resource contention like high CPU/JVM pressure or GC pauses. Inefficient shard sizing, over-sharding, and overly complex queries (deep aggregations, frequent cache evictions) further increase overhead and push tasks into rejection. Solution: Reduce query latency by optimizing queries with profiling, tuning shard sizes (10–50 GB each), and avoiding over-sharding. Improve parallelism by scaling the cluster, adding replicas for read capacity, increasing cache through larger nodes, and setting appropriate query timeouts.
`ThreadpoolRejected` metrics indicate queued requests	Thread pool rejections occur when high concurrent requests overflow queues beyond capacity, especially with undersized nodes limited by vCPU-based threads. Sudden unscaled traffic spikes further overwhelm pools, causing tasks to be dropped or delayed. Solution: Mitigate thread pool rejections by enforcing shard balance across nodes, scaling horizontally to boost thread capacity, and managing client load with retries and reduced concurrency. Monitor search queues, right-size instances for vCPUs, and cautiously tune thread pool settings to handle bursty workloads.
`ThroughputThrottle` or `IopsThrottle` reach 1	I/O throttling arises when Amazon EBS or Amazon EC2 limits are exceeded, such as gp3’s 125 MBps baseline, or when burst credits are depleted due to sustained spikes. Mismatched volume types and heavy operations like bulk indexing without optimized storage further amplify throughput bottlenecks. Solution: Address I/O throttling by upgrading to gp3 volumes with higher baseline or provisioning extra IOPS and consider I/O-optimized instances like i3/i4 families while monitoring burst balance. For sustained workloads, scale nodes or schedule heavy operations during off-peak hours to avoid hitting throughput caps.

Signs of over-provisioning

Over-provisioned clusters show consistently low utilization across CPU, memory, and storage, suggesting resources far exceed workload demands. Identifying these inefficiencies helps reduce unnecessary spend without impacting performance. You can use CloudWatch alarms to track cluster health and cost-efficiency metrics over 2–4 weeks to confirm sustained underutilization:

Low CPU utilization for data and primary nodes (<40%) sustained over time
Low JVM memory pressure for data and primary nodes (<50%)
Excessive free storage (>70% unused)
Underutilized instance types for workload patterns

Monitor cluster indexing and search latencies constantly as the cluster is being downsized—these latencies should not increase if the cluster is eliminating unused capacity. Also, it’s recommended to reduce nodes one at a time and continue to observe latencies to continue further downturn. By right-sizing instances, reducing node counts, and adopting cost-efficient storage options, you can align resources to actual usage. Optimizing shard allocation further supports balanced performance at a lower cost.

Best practices for right-sizing

In this section, we discuss best practices for right-sizing.

Iterate and optimize

Right-sizing is an ongoing process, not a one-time exercise. As workloads evolve, continuously monitor CPU, JVM memory pressure, and storage utilization using CloudWatch to make sure they remain within healthy thresholds. Rising latency, queue buildup, or unassigned shards often signal capacity or configuration issues that require attention.

Regularly review slow logs, query latency, and ingestion trends to identify performance bottlenecks early. If search or indexing performance degrades, consider scaling, rebalancing shards, or adjusting retention policies. Periodic reviews of instance sizes and node count help align cost with demand, maintaining 200-millisecond latency targets while avoiding over-provisioning. Consistent iteration helps your OpenSearch Service domain remain performant and cost-efficient over time.

Establish baselines

Monitor for 2–4 weeks after initial deployment and document peak usage patterns and seasonal variations. Record performance during different workload types. Set appropriate CloudWatch alarm thresholds based on your baselines.

Regular review process

Conduct weekly metric reviews during initial optimization and monthly assessments for stable workloads. Conduct quarterly right-sizing exercises for cost optimization.

Scaling strategies

Consider the following scaling strategies:

Vertical scaling (instance types) – Use larger instance types when performance constraints stem from CPU, memory, or JVM pressure, and overall data volume is within a single node’s capacity. Choose memory-optimized instances (such as r8g, r7g, or r7i) for heavy aggregation or indexing workloads. Use compute-optimized instances (c8g, c7g, or c7i) for CPU-bound workloads such as query-heavy or log-processing environments. Vertical scaling is ideal for smaller clusters or testing environments where simplicity and cost-efficiency are priorities.

Horizontal scaling (node count) – Add more data nodes when storage, shard count, or query concurrency increases beyond what a single node can handle. Maintain an odd number of primary-eligible nodes (typically three or five) and use dedicated primary nodes for clusters with more than 10 data nodes. Deploy across three Availability Zones for high availability in production. Horizontal scaling is preferred for large, production-grade workloads requiring fault tolerance and sustained growth. Use _cat/allocation?v to verify shard distribution and node balance:

GET /_cat/allocation/node_name_1,node_name_2,node_name_3

Optimize storage configuration

Use the latest generation of Amazon EBS General Purpose (gp) volumes for improved performance and cost-efficiency compared to earlier versions. Monitor storage growth trends using ClusterUsedSpace and FreeStorageSpace metrics. Maintain data utilization below 50% of total storage capacity to allow for growth and snapshots.

Choose storage tiers based on performance and access patterns—for example, enable UltraWarm or cold storage for large, infrequently accessed datasets. Move older or compliance-related data to cost-efficient tiers (for analytics or WORM workloads) only after ensuring the data is immutable.

Use the _cat/indices?v API to monitor index sizes and refine retention or rollover policies accordingly:

GET /_cat/indices/index1,index2,index3

Analyze shard configuration

Shards directly affect performance and resource usage, so an appropriate shard strategy should be used. The indexes that have heavy ingestion and searches should have a number of shards in the order of number of nodes for better efficiency across all data nodes in the cluster. We recommend keeping shard sizes between 10–30 GB for search workloads and up to 50 GB for log analytics workloads and limit to <20 shards per GB of JVM heap.

Run _cat/shards?v to confirm even shard distribution and no unassigned shards. Evaluate over-sharding by checking JVMMemoryPressure (>80%) or SearchLatency spikes (>200 milliseconds) from excessive shard coordination. Assess under-sharding if IndexingLatency (>200 milliseconds) or low SearchRate indicates limit parallelism. Use _cat/allocation?v to identify unbalanced shard sizes or hot spots on nodes:

GET /_cat/allocation/node_name_1,node_name_2,node_name_3

Handling unexpected traffic spikes

Even well right-sized OpenSearch Service domains can face performance challenges during sudden workload surges, such as log bursts, search traffic peaks, or seasonal load patterns. To handle such unexpected spikes effectively, consider implementing the following best practices:

Enable Auto-Tune – Automatically adjust cluster settings based on current usage and traffic patterns
Distribute shards effectively – Avoid shard hotspots by using balanced shard allocation and index rollover policies
Pre-warm clusters for known events – For expected peak periods (end-of-month reports, marketing campaigns), temporarily scale up before the spike and scale down afterward
Monitor with CloudWatch alarms – Set proactive alarms for CPU, JVM memory, and thread pool rejections to catch early stress indicators

Deploy CloudWatch alarms

CloudWatch alarms perform an action when a CloudWatch metric exceeds a specified value for some amount of time to take remediation action proactively.

Conclusion

Right-sizing is a continuous process of observing, analyzing, and optimizing. By using CloudWatch metrics, OpenSearch Dashboards, and best practices around shard sizing and workload profiling, you can make sure your domain is efficient, performant, and cost-effective. Right-sizing your OpenSearch Service domain helps provide optimal performance, cost-efficiency, and scalability. By monitoring key metrics, optimizing shards, and using AWS tools like CloudWatch, ISM, and Auto Scaling, you can maintain a high-performing cluster without over-provisioning.

For more information about right-sizing OpenSearch Service domains, refer to Sizing Amazon OpenSearch Service domains.