Tag Archives: Customer Solutions

How BigBasket uses the Iceberg based lakehouse architecture on AWS to power lightning-fast grocery delivery across India

Post Syndicated from Annie Mattoo original https://aws.amazon.com/blogs/big-data/how-bigbasket-uses-the-iceberg-based-lakehouse-architecture-on-aws-to-power-lightning-fast-grocery-delivery-across-india/

Delivering fresh groceries to millions of customers across India in a few minutes demands a radically modern data architecture and resilient processes to help the business make faster decisions. This is what BigBasket was able to achieve by building a lakehouse architecture on AWS.

In this post, we demonstrate how BigBasket implemented the lakehouse architecture on AWS, including their architecture decisions, implementation approach, and the measurable business results you can expect from a similar modernization. Whether you’re facing scalability challenges or planning your own lakehouse implementation, this blueprint provides actionable insights you can adapt for your organization.

About BigBasket

BigBasket (Innovative Retail Concepts Private Limited) is India’s largest online supermarket, serving millions of customers across over 60 cities. Founded in 2011, the company offers groceries, fresh produce, household items, and personal care products through its mobile app and website, operating subscription services (BBDaily) and quick commerce (bbnow). For BigBasket, the ability to deliver groceries on time isn’t only a competitive advantage. It’s the foundation of customer trust, where every minute counts.

However, rapid business growth brought significant operational challenges:

  • Inability to consistently meet on-time delivery adherence because of high order volumes, extended travel times, and more, directly impacting key metrics like on-time rate (OTR)-10 mins and OTR-15 mins.
  • Struggling to meet on-time delivery targets because of picking inefficiency, high order volumes, and extended travel times, directly impacting key metrics like OTR-10 mins and OTR-15 mins.
  • Delays in stock availability impacting vendor fill-rates, inter-distribution center orders, and warehouse operations.
  • Inaccurate stock forecasting for top-selling stock keeping units (SKUs), assortment variety, event SKUs, store capacity, and buying cycles.
  • Lower dark store productivity across picking, stacking, order processing, and goods receipt notes (GRN).

Behind these business challenges lay a fundamental technology problem: the existing data infrastructure couldn’t keep pace. The company experienced rapid store growth, expanding 4x in a short timeframe, which exposed several limitations within their existing data architecture that needed attention.

Understanding the technical bottlenecks

BigBasket’s initial architecture relied heavily on a single data warehouse built on Amazon Redshift to meet all reporting and dashboarding needs. While this traditional approach had served them well initially, several important limitations emerged:

  • Stale data: Extract, transform, load (ETL) pipelines delivered only day-old (D-1) data, making near real-time analysis impossible for dashboard requirements.
  • Extended recovery times: Pipeline failure recovery processes took several hours, causing significant delays in data availability for business users.
  • Schema rigidity: Schema changes in source databases frequently triggered pipeline failures because of a lack of schema evolution support.
  • Scalability constraints: The infrastructure struggled to handle the sudden load increase from 13,000 to over 35,000 transactions for reports and dashboards with more than 1,000 dataset refreshes.
  • Cost implications: Increasing data volumes demanded additional compute resources, driving up costs.

Diagram of the scalability and cost limitations of BigBasket’s legacy Amazon Redshift data warehouse

It became clear that the existing data infrastructure wasn’t able to meet the evolving business requirements and a redesign of their data architecture is needed.

Why lakehouse architecture?

A modern data lakehouse architecture addresses these issues with near real-time data processing, flexible schema evolution, and scalable analytics, capabilities necessary for fast-moving commerce operations. The lakehouse approach combines the flexibility and cost-effectiveness of data lakes with the performance and governance features of data warehouses, combining the strengths of both. The design of a data lakehouse provides interoperability across storage systems for combined analytics activities.

Solution overview

BigBasket partnered with AWS to implement a comprehensive lakehouse architecture using a combination of AWS native services and open-source technologies.

The following diagram shows an elaborated view of Bigbasket’s modernized architecture on AWS.

Detailed lakehouse data flow across bronze, silver, and gold medallion layers on AWS

Data ingestion: Enabling continuous replication

AWS Database Migration Service (AWS DMS) ingests data from online transaction processing (OLTP) databases running on Amazon Relational Database Service (Amazon RDS) into the lakehouse on AWS.

This method continuously replicates data with minimal latency, so your analytics reflect near real-time business operations.

Storage and governance: Building a solid foundation

The lakehouse is built on Amazon Simple Storage Service (Amazon S3) and Amazon Redshift, which serve as the centralized data lake and warehouse following a medallion architecture.

The architecture persists all analytical data using Apache Iceberg as the open table format. Iceberg provides a robust foundation for large-scale analytics with the following capabilities:

  • ACID transactions: Guarantees data consistency and correctness across concurrent read and write operations.
  • Time travel: Supports querying historical table versions for auditing, troubleshooting, and recovery.
  • Schema evolution: Allows schema changes without disrupting existing queries or downstream pipelines.

The medallion architecture structures data across three logical layers within the lakehouse:

  • Bronze layer: Implements change data capture (CDC)-based source replication using AWS DMS. Raw change events flow into Amazon S3 as Apache Parquet files in their original format from source systems, preserving the complete change history. The data pipeline processes and deduplicates these events using Apache Spark on Amazon EMR to create and maintain Apache Iceberg tables that act as replicated source tables.
  • Silver layer: Represents the conformed data model, where data is cleansed, standardized, and validated with enforced quality checks. This layer contains core dimension and fact tables, modeled for analytical consistency and reuse across domains. Data is stored as Apache Iceberg tables on Amazon S3, making it reliable and performant for downstream analytics and transformations.
  • Gold layer: Provides business-ready data marts and wide tables optimized for reporting, dashboarding, and domain-specific use cases. These datasets are curated to align with business metrics and key performance indicators (KPIs) and are served from Amazon Redshift, using Iceberg-backed tables to deliver fast, scalable analytics for business intelligence (BI) tools and end users.

This layered approach maintains a clear separation of concerns across raw ingestion, analytical modeling, and business consumption, while supporting scalability and flexibility across the organization. AWS Lake Formation enforces fine-grained data access controls, and the AWS Glue Data Catalog centrally manages metadata across Amazon S3 and Amazon Redshift, ensuring consistent data discovery and governance across the analytics ecosystem.

Data processing: Flexibility and performance

For data processing and transformations, BigBasket uses Amazon EMR with Apache Spark and dbt, orchestrated by Apache Airflow running on Amazon Elastic Kubernetes Service (Amazon EKS) as the core compute layer of the lakehouse. Apache Spark on Amazon EMR handles large-scale distributed processing, including CDC deduplication, incremental transformations, and complex data reshaping. Apache Iceberg serves as the open table format, which provides several critical capabilities.

dbt is used to define and execute transformation logic using SQL, managing the build of data models such as staging, intermediate, and final tables on top of the raw data. dbt uses the dbt-Trino adapter to run these transformations using the Trino engine, materializing the results as Apache Iceberg tables in Amazon S3. This approach provides a simple, modular, and governed way to manage transformations while taking advantage of Iceberg’s transactional guarantees.

These features are necessary for production lakehouse implementations and help you avoid vendor lock-in while maintaining enterprise reliability.

Online analytical processing (OLAP) and analytics: Hybrid approach for cost optimization

The analytics layer uses a hybrid approach that you can adapt based on your query patterns:

  • Amazon Redshift: For querying of active, frequently accessed data from the Gold layer.
  • Amazon Athena: For ad-hoc queries on historical data.
  • Apache Trino: For federated queries across multiple data sources while powering dbt-driven transformations directly on Apache Iceberg tables.

This hybrid strategy optimizes costs by keeping frequently accessed data in Amazon Redshift while querying historical data directly from Iceberg tables in Amazon S3. Amazon Redshift data sharing supports a multi-warehouse architecture for cross-team collaboration, allowing different teams to access shared datasets without data duplication.

Orchestration: Managing complex workflows

Apache Airflow running on Amazon EKS orchestrates and schedules data pipelines across the entire environment, providing visibility and control over complex workflows. This gives you a unified view for monitoring and managing your data operations.

Machine learning integration

Amazon SageMaker AI powers machine learning workloads for predictive analytics and model training directly on lakehouse data, from demand forecasting to delivery optimization. This tight integration means your data scientists can work with the same governed data that powers your analytics.

Visualization: Making insights accessible

Amazon Quick Sight provides data visualization and business intelligence reporting capabilities, making insights accessible to business users across the organization without requiring technical expertise.

Special focus: Clickstream data processing

BigBasket implemented a sophisticated dual-path architecture for processing clickstream data from mobile apps and web interactions:

  • Real-time path: Data flows through Scala stream collectors on Amazon Elastic Compute Cloud (Amazon EC2) (behind Elastic Load Balancing) to Amazon Kinesis Data Streams and Amazon OpenSearch Service for immediate insights into customer behavior. This path is necessary when you need to react to user actions within seconds, for example detecting fraud or personalizing experiences in real time.
  • Batch path: The batch path validates data, stores it in Amazon S3, processes it through Amazon EMR, and loads it into Amazon Redshift for comprehensive historical analysis. This path handles data quality checks, enrichment, and aggregation for long-term analytics.

The trade-off between these approaches is latency versus completeness. Real-time processing gives you speed but may sacrifice some data quality checks, while batch processing provides accuracy but introduces delay. This dual approach achieves both immediate operational insights and deep analytical capabilities, letting you optimize for different use cases.

The following diagram shows how the clickstream data is handled and effectively processed today.

BigBasket’s dual-path clickstream processing architecture with real-time and batch paths on AWS

The results: measurable business impact

The data platform transformation achieved significant results across multiple dimensions:

Technical improvements

  • Near real-time data: Achieved near real-time data availability for dashboards within 3–5 minutes, replacing previously day-old data.
  • Rapid failure recovery: Pipeline failure re-runs now complete in minutes instead of hours.
  • Comprehensive governance: Full control over data governance with robust observability, lineage, data accuracy, and consistency.
  • Enhanced scalability: Successfully handling over 35,000 reports and dashboards with over 1,000 dataset refreshes.

Business outcomes

  • On-time delivery: Improved monitoring with real-time insights on low-performing stores.
  • Stock availability: Reduced operational issues with visibility into key bottlenecks.
  • Stock forecasting: Improved accuracy and availability of top-selling SKUs.
  • Dark store productivity: Enhanced productivity of warehouse executives across all operations.

Key takeaways: lessons for modern data platforms

BigBasket’s journey offers valuable insights for organizations facing similar challenges:

  1. Quick commerce needs quick observability. In the fast-paced world of quick commerce, faster decision-making directly improves business metrics. Real-time data isn’t a luxury. It’s a necessity.
  2. Embrace ELT for real-time needs. Shifting from traditional ETL to an extract, load, transform (ELT) pattern within a lakehouse architecture is important to unlock near real-time analytics capabilities.
  3. A lakehouse delivers speed and governance. Modern lakehouse architectures don’t force trade-offs. You can achieve both fast data availability and comprehensive control, lineage, and accuracy.
  4. Focus on operational resilience. Designing for rapid failure recovery (re-runs in minutes, not hours) is necessary for maintaining data availability and business trust, especially in customer-facing operations.
  5. Incremental migration. You don’t need to rebuild everything. Evolve your current Amazon S3 data lake or reuse your existing investments in Amazon Redshift to build the data lakehouse capabilities.

The road ahead

BigBasket continues to innovate, now moving to adopt Amazon SageMaker Unified Studio to access all lakehouse components in a simplified manner across the enterprise. This next evolution will further streamline data access and accelerate insights across teams.

The company’s transformation demonstrates that with the right architecture and AWS services, organizations can turn data infrastructure challenges into competitive advantages, delivering not only better analytics but better customer experiences.

As you plan your own lakehouse implementation, use these patterns and lessons learned to accelerate your journey and avoid common pitfalls.


About the authors

Naga Sandeep Grandhi

Naga Sandeep Grandhi

Sandeep is an engineering leader at BigBasket, driving data platform and cloud architecture initiatives, including the next-gen data lake built for scale, reliability, and real-time insights.

Vikram Kumar

Vikram Kumar

Vikram is a Principal Engineer at BigBasket, where he leads the data engineering team. He specializes in designing and scaling modern data platforms on AWS, enabling BigBasket to process large-scale data efficiently and power data-driven decision-making across the organization.

Annie Mattoo

Annie Mattoo

Annie is a Sr. Analytics Specialist at AWS, bringing over 15+ years of expertise in helping customers with their DATA & AI journeys. She has successfully led customer teams to seamlessly adopt AWS Data & AI services and has worked with Fortune 500 customers across the globe in her previous roles.

Vineet Thapliyal

Vineet Thapliyal

Vineet is an Enterprise Account Manager at Amazon Web Services (AWS) in Bengaluru, India, where he manages strategic cloud and generative AI engagements across some of India’s largest conglomerates spanning energy, retail, and technology. He is passionate about helping enterprises unlock business value through AI/ML, cloud modernization, and industry-specific innovation — from renewable energy analytics to retail transformation at scale.

Anirudh Chawla

Anirudh Chawla

Anirudh is an Analytics Solution Architect at AWS. He helps organization empowers businesses to harness their data effectively through AWS’s analytics platform. His interest lies in building highly available distributed systems.

Deploy modern data platforms in minutes with MDAA

Post Syndicated from Sudeshna Dash original https://aws.amazon.com/blogs/big-data/deploy-modern-data-platforms-in-minutes-with-mdaa/

Modern Data Architecture Accelerator (MDAA) is an open source framework that replaces infrastructure code with concise YAML configuration, so your team can deploy a governed, production-ready data architecture, reducing deployment time from months to weeks (depending on complexity and team experience).

Organizations building modern data architecture on AWS face a critical challenge: deploying production-ready, governed infrastructure traditionally requires 6–12 months of custom development, thousands of lines of infrastructure code, and continuous remediation cycles to maintain security and compliance. Governance is often added incrementally, treated as an afterthought that creates compliance gaps and engineering rework.

MDAA addresses this by replacing infrastructure code with concise YAML configuration, achieving up to 97.6 percent code reduction (from approximately 1,800 lines of AWS CloudFormation to 45 lines of MDAA YAML) while embedding governance from the start. The complete Governed Lakehouse Starter Kit deploys 491 AWS resources across 12 stacks from approximately 450 lines of YAML configuration, representing a 66x verbosity ratio where each line automatically expands into production-ready infrastructure.

In this post, we explore how MDAA transforms data architecture development from months of manual coding to production-ready deployment through configuration-driven infrastructure and embedded governance, examine a real customer transformation, and provide a clear implementation pathway for your own data modernization journey.

Customer use case and challenge

A university system office needed to modernize its analytics architecture across 17 campuses while managing sensitive educational data. Their third-party dependency created bottlenecks that slowed feature implementation from weeks to months, and their IT team lacked the cloud skillsets to build modern infrastructure independently.

With MDAA, they achieved:

  • 95 percent reduction in time-to-value for dashboard and feature implementation (from weeks to hours).
  • 17 campuses integrated into a unified, secure architecture.
  • 7.2TB of data and over 8,000 dashboards migrated successfully.
  • Significant cost savings by removing third-party dependencies and reducing license costs.
  • Enhanced security posture for external stakeholders accessing sensitive educational data.

The team used MDAA to implement a modernization strategy with continuous integration and continuous delivery (CI/CD) for automated deployment. The architecture now supports rapid response to stakeholder requests while maintaining strict data governance through AWS Lake Formation.

Their transformation demonstrates what becomes possible when governance is embedded from launch rather than added incrementally, moving from months-long manual development to weeks of production-ready deployment through configuration-driven infrastructure.

Solution: MDAA and its value propositions

MDAA’s capabilities stem from its modular, composable architecture. The accelerator provides over 40 pre-built modules that encapsulate AWS best practices for security, governance, and operational excellence. Organizations describe the outcomes they want in MDAA-specific YAML configuration files (not CloudFormation or Terraform YAML) and the accelerator automatically translates these configurations into AWS Cloud Development Kit (AWS CDK) constructs, which then deploy via CloudFormation with embedded governance.

Configuration over code. The MDAA framework takes a fundamentally different approach: describe the outcomes you want in YAML, and the accelerator deploys production-ready infrastructure with embedded governance. Consider deploying a governed data lake where fraud detection teams need write access to transaction data, while marketing analytics teams require read-only access to customer behavior data. Traditional approaches require over 1,800 lines of CloudFormation across Amazon Simple Storage Service (Amazon S3) buckets, AWS Key Management Service (AWS KMS) keys, AWS Identity and Access Management (IAM) policies, and Lake Formation permissions. With MDAA, the same governed data lake is expressed in 45 lines of configuration, a 97.6 percent reduction, while helping you apply encryption, least-privilege access, and cross-account governance as built-in defaults.

The configuration deploys multi-zone S3 storage with KMS encryption, Lake Formation permissions with tag-based access control (TBAC) enabled, Amazon SageMaker Unified Studio for data product discovery, and encrypted AWS Glue Data Catalog with automated crawlers. All permissions flow through Lake Formation rather than individual IAM policies.

Embedded governance from day one. Governance is declared in YAML and deployed alongside infrastructure from the first run. Fine-grained access controls, encrypted data catalogs, data quality validation, audit trails, and sensitive data classification are all part of the same configuration. MDAA’s Governed Lakehouse starter kit defines an entire governed data architecture in roughly 450 lines of YAML, which produces approximately 29,700 lines of CloudFormation across 12 stacks (a 98.5 percent reduction in infrastructure code).

Modular, composable architecture. Each module is purpose-built to handle a specific capability within the data architecture. Modules communicate through AWS Systems Manager Parameter Store, passing resource identifiers (Amazon Resource Names (ARNs), IDs, and names) between stacks. This approach removes hardcoded dependencies. A KMS key created in one module can be referenced by another through parameter resolution, with all dependencies resolved automatically at deployment time.

The diagram illustrates the deployed architecture and team-level access flow that MDAA generates from the 45-line configuration.

Progressive architecture patterns. MDAA provides four reference architecture patterns that align to progressive stages of data infrastructure maturity:

  • Basic Data Lake deploys a governed data lake with built-in security controls, data quality checks, centralized metadata management using AWS Lake Formation and AWS Glue.
  • Data Science Platform extends the data lake with Amazon SageMaker notebooks, feature stores, and machine learning (ML) pipelines so data science teams can experiment and train models on governed data.
  • SageMaker Unified Studio adds a single interface for analytics and ML collaboration, connecting data engineers, analysts, and data scientists in one workspace.
  • Generative AI Platform layers Amazon Bedrock and Retrieval Augmented Generation (RAG) capabilities on top of your existing data foundation, so teams can build generative AI applications grounded in enterprise data.

Each pattern builds the one before it. You can start with the Basic Data Lake and adopt additional patterns as your team’s needs grow. MDAA’s modular design means you add capabilities without rearchitecting what you already deployed.

The infrastructure is versioned through GitHub, repeatable across environments, and auditable through comprehensive AWS CloudTrail logging. Data engineers focus on data pipelines and business logic while MDAA manages infrastructure complexity and governance integration. This represents the fundamental shift: from writing infrastructure code to describing the outcomes you want through configuration, with governance embedded from the start.

Use case of MDAA: Governed data architecture

DataOps teams spend significant time on governance tasks, including permissions management, compliance validation, and access control, rather than building pipelines and analytics. These aren’t data problems, they’re governance problems that consume engineering capacity meant for higher-value work. MDAA addresses this at the architectural level. Governance is declared in YAML and deployed alongside infrastructure from the first run.

The following sections walk through how each governance module works in practice.

Publish, discover, subscribe, and consume data products between business units: SageMaker Unified Studio

Amazon SageMaker Unified Studio provides a governed data catalog where data producers publish data products, and consumers discover and subscribe to them. Your deployment with MDAA includes a pre-configured domain, blueprints (managed and custom), projects, and environment profiles, all defined in a single configuration file:

# sagemaker.yaml --- 16 lines that deploy 114 CloudFormation resources
domains:
  domain1:
    dataAdminRole:
      id: ssm:/{{org}}/govern1/generated-role/data-admin/id
    description: SMUS Domain 1
    userAssignment: MANUAL

    tooling:
      vpcId: '{{context:vpc_id}}'
      subnetIds:
        - '{{context:private_subnet_id1}}'
        - '{{context:private_subnet_id2}}'

    groups:
      team1:
        ssoId: '{{context:team1-group-sso-id}}'
      team2:
        ssoId: '{{context:team2-group-sso-id}}'

Behind this configuration, MDAA deploys an Amazon SageMaker Unified Studio domain with dedicated KMS keys, execution and provisioning roles, and single sign-on group profiles for team access. Data producers tag and publish assets with metadata, ownership, and classification. Consumers browse a searchable catalog, see only authorized assets, and request access through a governed workflow. Cross-account and cross-business-unit data sharing flows through a subscription model, ensuring every access grant is tracked, auditable, and revocable.

Use case of MDAA: Restricting access to cardholder data using Lake Formation

AWS Lake Formation provides fine-grained access control at database and table levels, removing manual IAM policy management. MDAA deploys AWS Lake Formation with pre-configured settings that disable IAMAllowedPrincipals, the critical governance setting that ensures all permissions flow through centralized governance:

# lakeformation-settings.yaml --- 6 lines that deploy 25 CloudFormation resources
lakeFormationAdminRoles:
  - id: generated-role-id:data-admin
createCdkLFAdmin: true
createDataZoneAdminRole: true
iamAllowedPrincipalsDefault: false

That last flag is the single most important governance setting in the platform. Without it, an IAM principal with glue:GetTable can read tables in the catalog, bypassing the entire access control model. Most manual setups miss this or defer it.

With the data lake configuration, you declare roles and access policies in YAML where admins get full control, engineers get read access to curated data, extract, transform, and load (ETL) roles get scoped write access, and MDAA compiles them into the correct S3 bucket policies and Lake Formation registrations.

Use case of MDAA: Ensuring data integrity with AWS Glue Data Quality

AWS Glue Data Quality runs automated validation rulesets continuously as part of the pipeline, not as periodic batch checks. MDAA’s data quality module supports over 15 built-in rule types, from completeness and uniqueness checks to statistical thresholds and data freshness validation:

# data-quality.yaml
projectName: example-project

rulesets:
  customer-data-quality:
    description: Validate customer data completeness and uniqueness
    targetTable:
      databaseName: project:databaseName/customer-data
      tableName: customers
    ruleset:
      - ruleType: IsComplete
        column: customer_id
      - ruleType: Uniqueness
        column: email
        comparisonOperator: ">"
        threshold: 0.95
      - ruleType: RowCount
        comparisonOperator: ">"
        value: 100

Quality metrics flow into Amazon CloudWatch for real-time alerting. If anomalies are detected, automated workflows quarantine affected records and alert data engineering teams before issues reach downstream consumers.

Protecting metadata at rest: AWS Glue Data Catalog encryption

Table schemas, column names, and partition structures can reveal sensitive information about an organization’s data architecture, even without access to the underlying data. AWS Glue Catalog Encryption secures metadata at rest using AWS KMS-managed keys. MDAA configures catalog encryption by default, so schema definitions and connection passwords are encrypted from initial deployment without requiring manual key management setup. Access to catalog metadata follows the same Lake Formation governance controls applied to the data itself, so teams see only the schemas that they’re authorized to query.

Auditing every data access event: CloudTrail integration

Every data access event must be logged and attributable to a specific identity. Without a complete audit trail, demonstrating compliance during a regulatory review becomes a manual, error-prone process. AWS CloudTrail captures API-level activity across the data infrastructure, recording who accesses what data, when, and from which service. MDAA configures CloudTrail integration by default, so audit logging is active from initial deployment rather than added retroactively. Log data flows into a centralized, tamper-resistant store, giving compliance teams a single location to query access history across all business units and accounts.

Identifying sensitive data automatically: Macie integration

In large environments, sensitive information spreads across dozens of S3 buckets through pipelines, transforms, and ad hoc data drops, and self-reporting data owners consistently produce gaps. Amazon Macie uses machine learning to automatically discover and classify sensitive data in S3, surfacing findings at the object level without manual tagging. MDAA configures Macie across your S3 buckets during deployment, routing findings to Amazon EventBridge where automated workflows can alert owners or trigger remediation.

Together, these controls form a layered defense: Lake Formation governs access to cataloged data, Glue Data Quality validates integrity on arrival, and Macie identifies sensitive data that lands outside governed pipelines to reduce compliance risk.

Multi-account data mesh

MDAA provides extensive support for multi-account data mesh setups, with decentralized data ownership across business units and centralized governance. The data mesh starter kit supports cross-account data product publishing and consumption, allowing organizations to scale data sharing while maintaining consistent security and compliance controls.

Technical implementation

Ready to deploy your modern data architecture? Here are the resources to get started:

MDAA Implementation Guide provides detailed instructions for deploying all starter packages, including architecture patterns, configuration examples, security best practices, and troubleshooting guidance.

MDAA Hands-on Workshop offers step-by-step guided implementation with AWS experts. The workshop covers configuration management best practices, implementation patterns, hands-on labs with real-world scenarios, and cleanup instructions.

GitHub Repository and Documentation provide source code, module reference, and comprehensive documentation.

Organizations approach MDAA from different starting points. Some modernize existing data architectures, migrating from on-premises infrastructure or legacy cloud architectures. Others build new architectures for artificial intelligence and machine learning (AI/ML) initiatives or generative AI applications. Financial services organizations require PCI-DSS compliance from day one. Healthcare organizations need controls that can help support HIPAA. Each journey benefits from MDAA’s configuration-driven approach and embedded governance.

Conclusion

MDAA transforms data architecture development from months of manual coding to production-ready deployment. Configuration-driven infrastructure reduces development time by 40–60 percent while embedding governance from the start. The university system’s 95 percent reduction in time-to-value demonstrates the outcome: organizations deploy secure, compliant, governed data architectures in weeks rather than months.

Financial services organizations can deploy architectures to help them align with PCI-DSS compliance requirements using Lake Formation access controls, Glue Data Quality validation, SageMaker Unified Studio data discovery, comprehensive CloudTrail audit trails, and automated Macie data classification, all inherited from configuration rather than built manually.

Data architecture journeys need not follow six-month timelines with governance added incrementally. MDAA provides an alternative: describe the outcomes you want through YAML configuration, inherit pre-validated security controls, and deploy production-ready infrastructure with comprehensive governance from initial deployment.

Security and compliance is a shared responsibility between AWS and the customer. For more information, see the AWS Shared Responsibility Model.

Need help or have questions? Contact AWS ProServe for personalized guidance on selecting the right package and deployment strategy for your organization.


About the author

Sudeshna Dash

Sudeshna Dash

Sudeshna is a Data Scientist at AWS Professional Services based in Berlin, Germany. She specializes in data architecture, generative AI, and agentic AI systems on AWS. Sudeshna is a contributor to the Modern Data Architecture Accelerator (MDAA) open-source project and helps customers design and deploy governed, production-ready data and AI/ML architectures on AWS.

John Reynolds

John Reynolds is a Principal Engineer with AWS Professional Services based in Seattle, Washington. He leads the architecture and development of Modern Data Architecture Accelerator (MDAA), focusing on turning proven delivery patterns into reusable, production-ready foundations that customers can adopt and extend at scale.

Lessons learned from scaling to 1 million Lambda functions

Post Syndicated from Ben Freiberg original https://aws.amazon.com/blogs/architecture/lessons-learned-from-scaling-to-1-million-lambda-functions/

In this post, we share our journey and the lessons learned from building and running a fully serverless, multi-account software as a service (SaaS) platform at scale. We’ll explore why true scale-to-zero is critical, how we handle quota management, why engaging AWS service teams early saved us from outages, and which unexpected practices emerged once we scaled from thousands to over a million functions.

At ProGlove, we build smart wearable barcode scanning solutions that connect frontline workers to digital workflows. Our scanners integrate with Insight, our AWS-based SaaS platform, to provide real-time visibility into processes, helping customers in manufacturing, logistics and retail improve productivity, reduce errors and enhance ergonomics on the shop floor.

We chose a one AWS account per tenant architecture to achieve clearer security boundaries, streamlined ownership of services, and more transparent cost. It is important to focus on efficiency with dedicated tenant resources at scale, because resource wastage will also scale. The ability to scale-to-zero removes this concern.

Phase 1: The “simple” origins (0 to 1,000 Lambda functions)

When you first build a serverless system, you think in single digits. A handful of AWS Lambda functions, maybe a few dozen at most. It’s hard to imagine what changes when your platform operates thousands of AWS accounts and deploys over one million Lambda functions into production, each isolated to a single customer’s account.

We followed standard playbooks, where “scale-to-zero” was merely a nice-to-have. We used serverless best practices like Amazon Simple Queue Service (Amazon SQS) for decoupling and long-polling to keep the application responsive and resilient. At this scale, a few idle functions or a handful of accounts were a negligible expense and the benefits of a high-level managed service like AWS Lambda really showed.

Microservice composition

Each microservice in our platform follows a consistent structure: 5 to 15 Lambda functions coordinated by AWS Step Functions, with Amazon EventBridge handling event routing and Amazon DynamoDB as the primary data store.

Architecture diagram showing a microservice composition with Lambda functions, Step Functions, EventBridge, and DynamoDB

These resources are bundled together into a dedicated AWS CloudFormation stack for deployment.

As we onboarded our first handful of tenants, it quickly became clear that deploying and updating AWS CloudFormation stacks individually per account wouldn’t scale. We adopted AWS CloudFormation StackSets, which let us push infrastructure updates to multiple accounts in parallel from a central management account. At this stage, StackSets felt like a superpower. One deployment operation and many accounts are updated simultaneously. We evaluated building a fully custom replacement later, but ultimately concluded that the maintenance overhead wasn’t worth the marginal control gains and stayed with StackSets as our core mechanism.

Phase 2: The first 50 accounts

Growing to 50 tenant accounts forced us to confront problems that weren’t visible at single-digit scale. Three areas in particular required deliberate architectural decisions: observability, account provisioning, and quota isolation.

Automating account creation

We knew manual provisioning would not scale. Instead we built an automated account factory on top of AWS Organizations: an AWS Step Functions workflow in the management account handles the full provisioning lifecycle: Creating the account, applying baseline service control policies (SCPs), bootstrapping cross-account IAM roles, and triggering the initial CloudFormation StackSet deployment. All done using cross-account AWS Lambda invocations. New tenant accounts go from request to ready in under 15 minutes, at near-zero incremental cost per provisioning run.

Account provisioning workflow using AWS Organizations and Step Functions

The quota isolation benefit

One underappreciated advantage of the account-per-tenant model is quota separation. Each account gets its own Lambda concurrent execution limit, its own Amazon API Gateway throttle, and its own service quotas across the board. In a shared-account SaaS model at this scale, a single noisy tenant could exhaust shared concurrency and cause cascading failures across all other tenants. With account isolation, that class of problem simply doesn’t exist as each tenant’s activity is bound to their own account.

Phase 3: Scaling challenges (the self-DDoS)

As our fleet grew beyond a few hundred accounts, we began to experience the “Physics of Scale”. We discovered that when hundreds of backend service instances simultaneously access other services, the resulting request volume can resemble a coordinated attack, impacting not only our own infrastructure but also AWS.

One time, we faced a massive metric spike where our own functions effectively overwhelmed (similar to a DDoS attack) our internal APIs. The root cause was synchronized schedules: every Lambda was using the same rate(5 minutes) expression, which aligned to the top of the minute across thousands of accounts.

The solution was request scattering. We now use a standardized internal library that enforces jitter, randomized batch offsets, and staggered updates across all scheduled functions.

Rule of Thumb: “Never do the same thing at the same time everywhere”.

Multi-account observability as a cost driver

With several dozen accounts, manual log access per account became unworkable. We adopted a third-party observability platform, forwarding Amazon CloudWatch logs and metrics cross-account to a centralized dashboard. At roughly $3 per account per month, the cost felt insignificant.

That assumption was soon replaced by a very real learning: at thousands of accounts, $3 per account per month becomes an impactful expense that demands active management. We learned to treat per-account observability costs with the same scrutiny you apply to compute costs.

What came as a surprise to us were the actual cost drivers: instead of Lambda compute or storage costs, we found that forwarding all observability data almost doubled our cloud bill. As a result, we had to learn how to differentiate between high and low priority observability data and only move around the priority data.

With all mitigations combined we managed to bring observability costs down to around $0.7 per account. Additionally, we were able to switch accounts to almost 0 after some time of inactivity by only monitoring a small set of very basic metrics.

Phase 4: Rethinking architectural patterns for scale-to-zero

One of the most painful lessons was realizing that traditional Amazon SQS “best practices” increased costs in our use-case and scale.

Replacing SQS and the DLQ dilemma

After we scaled to over a thousand AWS accounts, we understood that “idle” doesn’t necessarily mean there are no costs – even when using Serverless. When Lambda functions consume events from EventBridge through an SQS queue to increase resilience, they constantly make requests to the queue even when there are no messages to process.

To eliminate the cost of continuous polling, we removed Amazon SQS from the path between Amazon EventBridge and AWS Lambda.

  • Metric-Driven Safety: Instead of relying on a queue to buffer requests, we monitor AsyncEventsDropped and ConcurrentExecutions to make sure we stay within our quotas without losing events.
  • The Centralized DLQ: Polling individual Dead Letter Queues (DLQs) in every account reintroduced the same polling cost issues. We solved this by routing failures to a centralized DLQ as shown in the following two diagrams.
  • The Isolation Trade-off: This approach requires extreme discipline to make sure we don’t break our data isolation patterns, as events from different tenants converge in a single location for recovery. Because of cost implications at scale, the use of SQS moved from a silo to a bridged model where the AWS account ID can be treated as a tenant ID.

Individual dead letter queue per queue architecture

Individual DLQ per queue

Centralized dead letter queue polling architecture

Centralized DLQ polling

Phase 5: Industrializing the deployment engine

Serverless architectures grow to large numbers of infrastructure components: where a monolith or Amazon Elastic Compute Cloud (Amazon EC2)-based service might be a handful of resources, a single microservice in our stack spans dozens of Lambda functions, EventBridge rules, DynamoDB tables, and Step Functions state machines. Multiplied across thousands of accounts, deployment complexity compounds quickly.

Initially, we used AWS CloudFormation StackSets to roll out updates in parallel. However, at the scale of 1 million Lambda functions, StackSets hit a performance ceiling and occasionally produced errors that added up significantly at our volume.

From custom engines to collaborative roadmaps

The bottlenecks became such a blocker that we began building our own internal serverless deployment system to replace StackSets. This caught the attention of the AWS CloudFormation service team, who committed to supporting our use case at the scale we required and partnered with us closely from that point on.

By engaging early and often, we were able to:

  • Influence the Roadmap: We provided the scale requirements that helped AWS prioritize StackSet stability and performance improvements.
  • Automate Resiliency: We built a deployment tracking service that aggregates StackSet events through Amazon EventBridge. A central AWS Step Functions state machine now acts as our “single-pane-of-glass,” acting on failures and triggering retries for occasional AWS internal errors.

Phase 6: Mature governance and FinOps

Being able to scale a serverless platform with a small team of engineers requires consistent and efficient governance practices. This applies to both cloud governance topics as well as engineering practices. Otherwise it will be next to impossible to keep software delivery and development performance as well as reliability at a high level over time.

Cost optimization also changes at a higher maturity level: once cost control is tightly monitored and automated, the discipline changes from housekeeping tasks to collect easy cost savings towards increasingly complex architectural changes. For example, if a new feature significantly increases the number of Lambda invocations and drives up cost, you will need to re-think the architecture and include the new focus on cost.

The mono-repo strategy

We consolidated 20 microservices into a single mono-repo. This helped us to:

  • Enforce consistent tooling and security scanning across more than a million functions.
  • Coordinate runtime and library upgrades through a single source of truth for configuration.
  • Make sure every change passes through the same CI/CD chain with guaranteed compatibility.

The “Almost-Zero” Reality

Even with a scale-to-zero mandate, we learned that “zero” is often “almost-zero”.

  • The Monitoring Tax: We avoided services like NAT Gateways, but monitoring introduced additional costs such as CloudWatch Alarms. Aggregating metrics in external observability tools added up quickly.
  • The Optimization Payoff: By aggressively optimizing these costs, we reduced our idle cost for inactive accounts to less than $1 per month.

Think beyond the obvious services

One of the most valuable habits we built was resisting the urge to immediately default to a familiar pattern or write custom code. AWS offers a growing catalog of fully managed, event-driven services such as Amazon EventBridge Pipes, AWS AppSync, Amazon SQS FIFO, and others, that can remove entire categories of custom Lambda code. Before writing a function, ask whether a native service integration already solves the problem.

A deliberate research step of exploring native AWS capabilities before opening an editor consistently paid off. It reduces the surface area you own, eliminates maintenance burden, and builds the team’s instinct for choosing the right service over reinventing it. Serverlessland is an excellent starting point for discovering patterns and service combinations you may not have considered.

Conclusion: Scaling efficiency faster than growth

Scaling from 0 to 1M Lambda functions across thousands of AWS accounts is a question of efficiency not of capacity. Every new account, every new customer, adds potential operational load. The only way to stay ahead is to make sure efficiency scales faster than growth. For us, that means true scale-to-zero, proactive and efficient quota management, tight collaboration with AWS service teams, disciplined developer education, and a mono-repo that enforces consistency.

We’ve learned that the difference between success and failure at this scale lies in unexpected aspects like the hard-learned fact that observability becomes an increasingly complex problem the more distributed your platform becomes.

The benefits are substantial. With the right automation and architectural rigor, a lean team can operate a large-scale infrastructure. Using a cloud-native approach based on serverless services is the most important operational advantage in this case.

To apply these lessons to your own workloads, discover event-driven patterns and service combinations on Serverless Land.


About the authors

Preventing data exfiltration in machine learning environments with Amazon SageMaker AI

Post Syndicated from Ajish Abraham original https://aws.amazon.com/blogs/architecture/preventing-data-exfiltration-in-machine-learning-environments-with-amazon-sagemaker-ai/

If you’re building machine learning solutions with sensitive data, you face a persistent challenge: preventing data exfiltration while enabling data scientists to work productively. iBusiness, an AI-driven fintech organization, needed its data scientists to work with sensitive data to fine-tune and improve machine learning models. As the data science team scaled, traditional air-gapped environments and monitored virtual desktops proved unsustainable, leading to high costs and operational complexity.

In this post, we demonstrate how iBusiness implemented a three-layered security architecture using Amazon SageMaker AI, virtual private cloud (VPC) endpoints, and Amazon WorkSpaces Secure Browser to prevent data exfiltration while maintaining data scientist productivity. You can adapt this approach to build secure machine learning environments that balance strict data protection with team scalability.

Historically, when access to sensitive data was required, iBusiness provided an isolated, air-gapped on-premises environment. However, with the shift to a remote workforce, this approach became impractical. The company locked down secure virtual desktops through device management policies and had them monitored by proctors to prevent inappropriate actions.

As the data science team scaled and expanded machine learning (ML) use cases, this approach proved unsustainable. Each user required a dedicated virtual desktop, even for temporary access, leading to increased costs. Additionally, maintaining ML tools, libraries, and patches in these locked-down environments was time-consuming and operationally complex.

To address these challenges, iBusiness adopted Amazon SageMaker Studio, a fully managed, web-based ML development environment. This removed the need to maintain in-house Jupyter environments while giving data scientists access to up-to-date tools. Furthermore, SageMaker AI’s integration with AWS services provided straightforward data sharing via AWS Lake Formation and Amazon Athena, reducing the need for manual data transfers.

Solution architecture

To achieve this, iBusiness implemented a three-layered security strategy that you can adapt for your own secure ML environments.

Three-layered security architecture for data exfiltration prevention
Figure 1: Three-layered security architecture for data exfiltration prevention

Layer 1: Securing access through WorkSpaces Secure Browser

iBusiness used Amazon WorkSpaces Secure Browser, a managed, locked-down browser environment. This managed service provides a controlled Chromium-based browser, offering a more cost-effective solution for the company’s use case.

The company configured the Secure Browser to run within a dedicated VPC and subnet in its IT infrastructure account, routing outbound traffic through a network address translation (NAT) gateway. In the secure data science account, iBusiness enforced AWS Identity and Access Management (IAM) policies that restrict access to requests originating only from AWS services or from the NAT gateway’s Elastic IP address. This configuration helps validate that access to the environment is only possible through the Secure Browser. It gives you confidence that data scientists cannot bypass security controls when you implement a similar approach.

Additionally, the Secure Browser was configured to disable file downloads and uploads, disable clipboard access, and disable printing. These controls help prevent data from being transferred to local machines.

Key Secure Browser controls configured:

  • Disable file downloads and uploads.
  • Disable clipboard access.
  • Disable printing.

Layer 2: Restricting browser activity and cross-account access

Building on this foundation, iBusiness restricted activity within the Secure Browser itself to address potential exfiltration through web-based channels.

Although the browser provides a temporary working directory, iBusiness prevented its misuse by implementing strict URL allowlisting. Users can only access *.aws.amazon.com and specific SageMaker AI domains. Other websites, including email and external storage platforms, are blocked, preventing users from uploading data to external services.

Permitted URL patterns:

  • *.aws.amazon.com.
  • Specific SageMaker AI domains.

Preventing cross-account data exfiltration

To help verify users cannot move data to other AWS accounts, iBusiness implemented VPC endpoints for AWS Management Console and AWS IAM Identity Center services. These endpoints route traffic privately within the VPC with no internet exposure. They also enforce endpoint policies restricting access to iBusiness’s specific AWS account, giving you control over which accounts data scientists can access.

The company also configured a private Amazon Route 53 hosted zone to redirect console.aws.amazon.com, *.console.aws.amazon.com, and signon.aws.amazon.com to the company’s VPC endpoints instead of public endpoints. To further mitigate DNS-based exfiltration risks, iBusiness configured Amazon Route 53 Resolver DNS Firewall in the SageMaker AI VPC to block DNS queries to non-approved domains, ensuring that only resolution of required AWS service endpoints is permitted.

This configuration helps verify that users can only authenticate into iBusiness’s secured data science account and that access to other AWS accounts is blocked. To further enforce this, iBusiness applied an IAM policy that enhances the IAM policy from Layer 1. This policy helps confirm actions are sourced from an IAM principal originating from a VPC endpoint and denies actions when the target resource belongs to another AWS account, with minimal exceptions for privileged users.

Layer 3: Securing the SageMaker AI environment

As a final layer of defense, iBusiness secured the SageMaker AI environment itself to prevent data exfiltration through the development environment’s terminal and integrated development environment (IDE) access.

Because SageMaker AI provides terminal and IDE access, it could potentially be used to move data externally. To mitigate this risk, the company removed direct internet access from the SageMaker AI VPC with no NAT gateway or internet routes and configured VPC endpoints for the required AWS services.

This configuration confirms that SageMaker AI can access AWS services internally and function normally while simultaneously blocking direct outbound internet traffic. iBusiness further restricted VPC endpoint policies to allow access only to resources within the organization, providing an additional safeguard against cross-account data movement. VPC endpoint policies allow for granular access to specific AWS resources. For example, allowing users restricted access for s3:PutObject API calls to specific Amazon Simple Storage Service (Amazon S3) buckets depending on the use case.

SageMaker AI network configuration:

  • No NAT gateway or internet routes in the SageMaker AI VPC.
  • VPC endpoints configured for all required AWS services.
  • Endpoint policies restricted to organization-owned resources only.

Conclusion

By implementing this three-layered security architecture, iBusiness achieved an 80% cost reduction, from $40+ per user monthly for individual VDI environments to $7 per user with Amazon WorkSpaces Secure Browser. The solution also transformed IT operations, reducing provisioning from a 2-day SLA to automatic setup within minutes while eliminating ongoing desktop maintenance overhead.

For data scientists, the approach improved both productivity and security by streamlining data access without compromising protection. This demonstrates how you can strengthen security controls while reducing costs and operational complexity.

Start by assessing your current data access controls, then progressively implement each security layer based on your organization’s specific compliance requirements and risk tolerance.


About the authors

Automating IT support with AI: How Nexthink uses OpenSearch Service to power self-service issue resolution

Post Syndicated from Rafael Ribeiro, Moe Haidar original https://aws.amazon.com/blogs/big-data/automating-it-support-with-ai-how-nexthink-uses-opensearch-service-to-power-self-service-issue-resolution/

This is a guest post by Rafael Ribeiro and Moe Haidar, at Nexthink, in partnership with AWS.

Nexthink is the leader in digital employee experience, helping enterprises improve how employees interact with technology in the workplace. The company gives IT teams real-time visibility into endpoint performance, application usage, and employee sentiment across millions of devices worldwide.

At the heart of Nexthink’s innovation is Spark, an autonomous artificial intelligence (AI) agent that automates IT support. Spark resolves IT issues for employees, from troubleshooting application crashes to resetting configurations and running remediation scripts. Rather than routing tickets or providing scripted responses, the agent takes direct action, achieving a 77% resolution rate at first contact without human escalation.

Spark operates at enterprise scale, deployed across 12 AWS Regions to serve global customers with low-latency responses.

In this post, we explore how Nexthink combined Amazon OpenSearch Service vector search, Amazon Bedrock, and infrastructure as code to power the Spark agent’s retrieval layer.

The challenge: Why vector search for AI agents?

For an AI agent to autonomously resolve IT issues, it must quickly retrieve the most relevant context from a vast knowledge base. Traditional keyword search falls short because:

  • Semantic understanding matters: An employee asking “my laptop is running slow” should match articles about “system performance optimization” even without exact keyword overlap.
  • Accurate retrieval drives correct outcomes: The quality of an AI agent’s response is only as good as the context it retrieves. When the agent pulls the right documentation, scripts, and historical resolutions, it produces accurate, safe actions. When retrieval is imprecise, the consequences can be severe. An agent acting on the wrong context could run destructive commands like rm -rf *, wipe critical data, or apply an incorrect fix that escalates the problem. Accurate vector search is the guardrail that keeps autonomous agents grounded in verified, relevant knowledge.
  • Speed is critical: Enterprise users expect near-instant responses, so retrieval must run in sub-second time across millions of documents.

This led Nexthink to implement Amazon OpenSearch Service with vector search capabilities, using Amazon Titan Text Embeddings V2 through Amazon Bedrock for embedding generation. With this architecture, Spark performs semantic search across all knowledge sources, retrieving contextually relevant information that drives accurate, autonomous issue resolution.

High-level architecture

The following diagram illustrates the high-level architecture of Nexthink’s Spark agent implementation with Amazon OpenSearch Service.

High-level architecture of Nexthink’s Spark AI agent, showing Amazon Elastic Kubernetes Service hosting the agent, Amazon OpenSearch Service as the vector store, and Amazon Bedrock providing the embedding model

Architecture components

Amazon Elastic Kubernetes Service (Amazon EKS) hosts the Spark agent, which interprets user queries, retrieves relevant context, and runs autonomous resolutions. With container orchestration, the agent scales horizontally across Nexthink’s 12 AWS Regions while maintaining consistent response times. The agent communicates with Amazon OpenSearch Service to perform semantic searches, retrieving the most contextually relevant documentation and automation scripts for each user’s issue.

Amazon OpenSearch Service functions as the central vector store, providing the k-Nearest Neighbors (k-NN) capabilities required for semantic search. OpenSearch Service stores document embeddings (dense vector representations of text content) alongside traditional metadata fields. When the AI agent submits a query, OpenSearch Service performs approximate nearest neighbor (ANN) searches to find documents with semantically similar embeddings, even when exact keywords don’t match. This vector search capability, combined with the proven scalability and managed infrastructure of OpenSearch Service, makes it well suited for AI agent architectures that require fast, accurate context retrieval.

Amazon Bedrock provides the foundation models used to generate text embeddings. Nexthink uses Amazon Titan Text Embeddings V2, hosted on Amazon Bedrock, to convert both documents and queries into dense vector representations. OpenSearch Service integrates natively with Amazon Bedrock through the OpenSearch ML Connector, which handles embedding generation at both index and query time.

Data ingestion pipeline

A critical component of any AI agent architecture is the data ingestion pipeline. This mechanism transforms raw documents into searchable, semantically indexed content in OpenSearch Service. For Spark, the pipeline must handle diverse data sources while automatically generating vector embeddings for semantic search.

Step 1: Staging and preprocessing layer

Staging layer in Amazon S3

Knowledge bases (KBs) are staged in Amazon Simple Storage Service (Amazon S3) before being processed through the ingestion pipeline. Amazon S3 provides durable storage, versioning capabilities, and integration with OpenSearch Service ingestion mechanisms. When documentation updates occur, new versions are uploaded to Amazon S3, which triggers the ingestion pipeline to reprocess and re-embed the content.

Event-driven streaming with Apache Kafka

IT tickets, agent interactions, and remote actions are processed through Apache Kafka for reliable message delivery during traffic spikes. Its consumer group model lets the ingestion pipeline scale horizontally based on event volume.

Step 2: Embedding generation during indexing time

Nexthink uses ingest pipelines inside OpenSearch Service to process the data at ingestion time, including generating text embeddings. When documents are sent to OpenSearch Service, the text_embedding processor inside the ingest pipeline automatically invokes the machine learning (ML) Connector to generate embeddings.

The ML Connector is the OpenSearch Service built-in framework for integrating external ML services. It handles request signing between OpenSearch Service and Amazon Bedrock, parses the Amazon Bedrock response to extract embeddings, maps them to index fields, and manages retries on failure. This eliminated the need for custom integration code and accelerated Nexthink’s time to market.

The following ingestion pipeline configuration demonstrates how to configure the text_embedding processor.

{
  "description": "Embedding ingestion pipeline for Spark AI Agent",
  "processors": [
    {
      "text_embedding": {
        "model_id": "<bedrock-connector-model-id>",
        "field_map": {
          "content": "content_embedding"
        }
      }
    }
  ]
}

In this configuration:

  • model_id: References the registered ML model connected to Amazon Bedrock.
  • field_map: Maps the source text field (content) to the target embedding field (content_embedding).

Step 3: Embeddings and data structure in OpenSearch Service

Nexthink stores embeddings alongside textual and metadata information in their k-NN index. For the vector field, they use Hierarchical Navigable Small World (HNSW) with the Lucene engine, as shown in the following example.

...
"content_embedding": {
  "type": "knn_vector",
  "dimension": 1024,
  "method": {
    "name": "hnsw",
    "space_type": "innerproduct",
    "engine": "lucene"
  }
},
"document_type": {
  "type": "keyword"
},
"tenant_id": {
  "type": "keyword"
}
...

In this configuration:

Multi-tenant search and retrieval

Enterprise AI agent deployments must address a critical challenge: making sure that users only access data they’re authorized to see. For Nexthink, serving multiple enterprise customers from a shared infrastructure requires robust multi-tenant security. Each customer’s knowledge base, automation scripts, and support tickets must remain isolated while the shared vector index continues to perform well.

The following diagram illustrates the search flow from user query to ranked results.

Search flow showing how a user query travels through the Spark agent, OpenSearch Service neural search, the ML Connector to Amazon Bedrock for embedding, and tenant-filtered k-NN retrieval to produce ranked results

Tenant management

Nexthink stores information about each tenant inside the tenant_id field. This design lets permission filters run efficiently alongside vector similarity searches. Additionally, Nexthink stores the tenant_id as a keyword type in the index mapping shared previously, so that filtering runs without the overhead of text analysis. Instead of pre-filtering with k-NN queries through a score script filter, the OpenSearch engine uses an intelligent decision-based approach for k-NN filtering called efficient filtering.

Neural query example with efficient filtering

OpenSearch’s neural search simplifies vector search by handling embedding generation as part of the query itself. Instead of requiring the application to call an embedding model separately and then submit a raw k-NN query with a vector, a neural query accepts plain text and uses the registered ML Connector to generate the embedding on the fly. As a result, the Spark agent can send natural-language queries directly to OpenSearch Service without any client-side embedding logic.

The following query demonstrates how Nexthink combines neural search with tenant isolation through efficient filtering in OpenSearch Service.

{
  "query": {
    "bool": {
      "must": [
        {
          "neural": {
            "content_embedding": {
              "query_text": "laptop running slow",
              "model_id": "<bedrock-connector-model-id>",
              "k": 50
            }
          }
        }
      ],
      "filter": [
        {
          "term": {
            "tenant_id": "customer-123"
          }
        }
      ]
    }
  }
}

In this query structure:

  • bool.must: Contains the neural search clause that performs semantic matching against document embeddings.
  • bool.filter: Applies the tenant isolation constraint, so that only documents belonging to customer-123 are returned.

Nexthink’s contribution to the technical community

A key principle in Nexthink’s architecture is treating infrastructure as code. With deployments spanning 12 AWS Regions, manual provisioning would be error-prone and time-consuming. Therefore, Nexthink uses several infrastructure as code (IaC) technologies, including Terraform, to provision resources.

Although the Terraform provider supports core OpenSearch Service resources like indices and index templates, it lacked support for some of the ML Commons resources required to integrate Amazon Bedrock:

  • ML Connectors: Required to establish connections to external ML services like Amazon Bedrock.
  • ML Model Groups: Needed to organize and manage related models.
  • ML Models: Required to register models that use the connectors.

Without these resources, Nexthink initially relied on workarounds using local-exec provisioners and null_resource blocks to call the OpenSearch Service API directly. This approach was fragile, difficult to maintain, and didn’t integrate well with Terraform’s state management.

Contributing back

Rather than maintaining a private fork indefinitely, Nexthink chose to contribute their custom Terraform resources back to the OpenSearch Project community. This decision aligned with their engineering values to help other organizations implement similar architectures and contribute to the broader community.

Open source contribution links

The Terraform provider contributions are being added to the official OpenSearch project repository:

  • Pull Request: Add support for ML Connector, ML Model Group, and ML Model resources #280.
  • Feature Request: Contribution – Support for ML resources #281.

These contributions let any organization provision OpenSearch Service ML resources with Terraform, which streamlines the deployment of AI agent architectures that integrate with Amazon Bedrock or other ML services.

Conclusion

Nexthink’s implementation of Amazon OpenSearch Service for the Spark agent demonstrates how vector search capabilities can power autonomous IT support at enterprise scale. By combining semantic search with multi-tenant security and infrastructure as code practices, Nexthink achieved a 77% resolution rate at first contact, so that employees can resolve IT issues without human escalation.

Get started

Ready to build your own AI agent with vector search capabilities? Here are your next steps:

  1. Explore Amazon OpenSearch Service vector search features in the OpenSearch Service documentation.
  2. Configure ML Connectors for Amazon Bedrock using the ML Commons plugin guide.
  3. Automate with Terraform using the contributed resources in the terraform-provider-opensearch repository.

The combination of Amazon OpenSearch Service, Amazon Bedrock, and infrastructure as code practices provides a foundation for building intelligent, context-aware AI agents that deliver business value.


About the authors

Rafael Ribeiro

Rafael Ribeiro

Rafael is a Software Engineer at Nexthink, focusing on infrastructure and DevOps for AI teams.

Moe Haidar

Moe Haidar is Head of Agentic AI and Engineering at Nexthink, where he leads AI architecture and strategy alongside the development of Spark, the company’s autonomous personal IT agent that resolves employee issues at scale.

Hajer Bouafif

Hajer Bouafif

Hajer is an Analytics Specialist Solutions Architect at Amazon Web Services. She focuses on Amazon OpenSearch Service and helps customers design and build well-architected analytics workloads in diverse industries. Hajer enjoys spending time outdoors and discovering new cultures.

Luca Perrozzi

Luca Perrozzi

Luca is a Solutions Architect at AWS, based in Switzerland. He focuses on innovation topics at AWS, especially in the area of Artificial Intelligence. Luca holds a PhD in particle physics and has 15 years of hands-on experience as a research scientist and software engineer.

Reducing SMS OTP fraud with Vonage network-powered solutions and Amazon Cognito

Post Syndicated from Tito Milla original https://aws.amazon.com/blogs/architecture/reducing-sms-otp-fraud-with-vonage-network-powered-solutions-and-amazon-cognito/

User authentication remains one of the most targeted touchpoints in application security. With the industrialization of fraud threats by generative AI, cybercrime costs are expected to reach $23 trillion in 2027, an increase of 175 percent from 2022. 20 percent of fraud is attributed to synthetic identity and authentication exploits, with account takeover (ATO) surging 141 percent since 2021.

But the damage goes beyond security. SMS One-time passcodes (OTPs) achieve only approximately 80 percent conversion on authentication flows, meaning 1 in 5 legitimate users is lost at the point of verification. Enterprises absorb hundreds of thousands of password recovery helpdesk tickets annually, representing significant support costs tied to OTP-based verification. Every abandoned authentication attempt today represents an opportunity to maximize your conversion rates across checkout, account recovery, and onboarding flows. The industry has long assumed that stronger security requires more user friction. That isn’t a law of physics. It’s a limitation of the tools available. Mobile operator network data removes that constraint and provides stronger identity assurance and a smoother experience, not one at the expense of the other.

In this post, we show how Vonage network-powered solutions work with Amazon Cognito to enhance many mobile-first use cases with network-level identity verification. Vonage network-powered solutions are a composable stack of real-time mobile operator intelligence, silent authentication, and integrated fraud protection, which uses the CUSTOM_AUTH flow to complete identity verification in under 5 seconds, with zero user interaction.

About Vonage

Vonage, part of Ericsson, is an AWS Partner with multiple AWS Marketplace listings. The company provides enterprise and CIAM deployments with cloud-based access to mobile operator network APIs, including real-time mobile identity and authentication across key regions. These complement Vonage’s global communications, voice, and video APIs backed by Ericsson’s global telecommunications infrastructure.

What network-powered means and why it matters

Before diving into architecture, it’s worth being precise about what separates Vonage’s network-powered solutions from the identity and fraud tools enterprises already have in their stack.

Most identity verification signals today are derived from aggregated, cached, or behavioral data. Traditional phone number lookup services query static databases that may be days or weeks out of date. Device fingerprinting analyzes browser characteristics that might be spoofed. Behavioral biometrics builds models from historical sessions. This is useful, but a lagging indicator by definition.

Enterprise customers who implement Vonage’s network-powered solutions operate from a fundamentally different layer: real-time data sourced directly from mobile network operators (MNOs). When you query whether a SIM was recently swapped, you’re querying the network that performed the swap. When Silent Authentication verifies a user, the proof of possession is the cellular data session itself. This session can’t be phished, intercepted, or socially engineered.

In fraud scenarios where SIM swaps are weaponized for account takeover (ATO), “recently” means minutes or hours, not days. Static databases refreshed weekly are not detecting these events. They’re logging them after the fact. Real-time operator queries close that window entirely.

The three pillars: Identity Insights, Verify, and Fraud Defender

Vonage network-powered solutions combine three API service components into a composable security stack that integrates with Amazon Cognito through the CUSTOM_AUTH flow:

1. Identity Insights: Pre-verification intelligence

Identity Insights runs before verification channels are initiated, surfacing real-time operator signals that are directly actionable in authentication policy decisions. The following list shows a representative set of JSON elements that might be returned by a request. Customers have the option to select which data is most valuable given a specific authentication use case and industry combination.

  • format and network_type: Filters invalid numbers, VoIP, landline, and premium-rate numbers used in synthetic account creation and bot-driven fraud.
  • sim_swap: Detects SIM swaps within a configurable look-back window, a leading indicator of ATO events in progress.
  • subscriber_match: Compares subscriber identity (name, address) against operator Know Your Customer (KYC) records.
  • device_swap: A recent change in the mobile device associated with a phone number signals that a bad actor might have taken control of the SIM card. (coming soon)
  • recycled_number: Numbers previously deactivated and reassigned to a new subscriber can trigger false identity matches in onboarding flows, creating risk in account creation. (coming soon)

These pre-checks trigger your defined risk policy: step-up challenge, hard block, or silent logging. Critically, fraudulent attempts are identified and blocked before a single OTP is sent, before verification costs are incurred, and before fraud processing overhead is generated.

2. Verify with Silent Authentication: Alleviating the friction tax

Every additional step a user must finish during authentication carries a measurable cost: abandoned sign-ups, failed conversions, and support tickets from users who don’t receive or mistyped a code. We call this cumulative loss the friction tax. For SMS OTP flows with approximately 80 percent completion rates, the friction tax means roughly 20 percent of legitimate users drop off before they ever reach your application.

After a number passes the risk pre-checks, the Verify API delivers the authentication challenge. The primary authentication method is Silent Authentication.

When a user initiates sign-in from a mobile device, Vonage routes an HTTP request through the user’s cellular data connection. The mobile operator confirms that the SIM registered to the phone number matches the session making the request. The exchange happens in the background, in seconds. The user doesn’t see, type, copy, or enter any code.

If Silent Authentication can’t finish or is unavailable, Verify automatically falls back to traditional SMS, RCS, Voice, WhatsApp, or email, remaining transparent to the user.

Key benefit: Silent Authentication alleviates the three primary exploit vectors against SMS OTP: SIM swap (bad actor receives the code), SS7 interception (message diverted in transit), and social engineering (user tricked into sharing the code). All without additional input from the end user.

3. Fraud Defender: Protecting the verification channel

Fraud Defender addresses a threat familiar to enterprise finance teams: artificially inflated traffic (AIT) and SMS pumping. Automated systems trigger high volumes of OTPs sent to premium-rate numbers that bad actors control. At enterprise verification volumes, these events can run undetected for extended periods.

Fraud Defender provides real-time traffic monitoring and intelligent blocking at the point of outbound delivery, intercepting these malicious events before costs accumulate. The financial impact is immediate and measurable. Fraud Defender typically absorbs its own cost in toll fraud prevention within the first billing cycle. For most enterprises, it quickly becomes a net revenue-positive investment. Vonage customers have collectively saved over $3M in SMS-related fraud costs since deployment. The savings continue to compound as the blocking algorithm evolves to counter new exploit patterns. For Verify customers, the value is even more compelling: Fraud Defender activates automatically with the Vonage Verify API at no additional cost. This makes it one of the highest-ROI fraud protections available.

Prerequisites

To implement this solution, you need:

  • An AWS account with permissions to create and manage Amazon Cognito, AWS Lambda, AWS Secrets Manager, Amazon CloudWatch, and AWS WAF resources.
  • An Amazon Cognito user pool (existing or new).
  • A Vonage API account with access to Identity Insights and Verify APIs.
  • AWS Command Line Interface (AWS CLI) or AWS Serverless Application Model (AWS SAM) CLI installed and configured.
  • For client integration: the Vonage Silent Authentication SDK for your mobile platform (iOS/Android).

Solution architecture with Amazon Cognito

Enterprise customers that integrate the Vonage solution use the Amazon Cognito CUSTOM_AUTH flow, which uses three AWS Lambda functions that orchestrate the solution stack without changing your existing user pool configuration or downstream service integrations.

Architecture diagram showing the Risk-Adaptive Customer Sign-In flow with layers including user devices, edge protection with Amazon CloudFront and AWS WAF, Amazon API Gateway, identity layer with Amazon Cognito, verification layer with Vonage Identity Insights, Verify API, and Fraud Defender, and the carrier network with Mobile Network Operators.

Architecture components

The solution connects five layers, each handling a distinct step in the authentication flow:

  • Client app (mobile/web) – Initiates the CUSTOM_AUTH flow with the Vonage Silent Authentication SDK, follows check_url redirects over the cellular network, and submits the verification code back to Amazon Cognito.
  • Amazon Cognito user pool – Orchestrates the CUSTOM_AUTH challenge flow and issues JWT tokens upon successful verification.
  • AWS Lambda triggers – Define Auth Challenge (orchestrator), Create Auth Challenge (calls Vonage APIs), and Verify Auth Challenge (validates response).
  • Vonage Network APIs – Identity Insights pre-check, Verify with Silent Auth and OTP (built-in failover), and Fraud Defender (automatic).
  • Mobile network operators – SIM-level identity verification through CAMARA/Open Gateway APIs.

Authentication flow

The following steps represent an authentication workflow sequence between Amazon Cognito and Vonage network-powered solutions:

  1. The client calls InitiateAuth with CUSTOM_AUTH, passing the user’s phone number.
  2. The Define Auth Challenge Lambda function instructs Amazon Cognito to issue a CUSTOM_CHALLENGE.
  3. The Create Auth Challenge Lambda function calls Identity Insights for pre-verification risk assessment. If the number passes pre-checks, Lambda calls Vonage Verify to initiate Silent Authentication and returns the check_url to the client.
  4. Upon receiving the check_url, the client opens an HTTPS connection to it, triggering HTTP redirects to the mobile carrier’s network for direct mobile-device-to-mobile-network-operator verification. Upon completion, the client receives a verification code from the operator.
  5. The client calls RespondToAuthChallenge with the code.
  6. The Verify Auth Challenge Lambda function submits the code to Vonage’s check endpoint. On success, it returns answerCorrect = true and Amazon Cognito issues the appropriate session token.

Sequence diagram showing the User Login flow with SIM-swap pre-check using Vonage Identity Insights and Silent Authentication via Vonage Verify, orchestrated through the Amazon Cognito CUSTOM_AUTH flow with Lambda triggers.

Coexistence and phased rollout

A critical design principle: zero disruption to existing infrastructure. The Vonage Network API plugs into the Amazon Cognito CUSTOM_AUTH flow without changes to your existing user pool, app client configurations, or downstream service integrations. Deployment requires a single sam deploy command.

This design approach allows for a phased rollout. Start with the highest-risk journeys (password recovery, high-value transactions) where security ROI is clearest, then expand to daily login and onboarding as you measure impact. Traditional SMS, RCS, and Voice OTP remain options for lower-risk flows during the transition.

Risk-aware workflows by journey type

The strategic value of combining Vonage’s network-powered solutions with the Amazon Cognito policy-driven CUSTOM_AUTH flow is context-aware authentication calibrated to actual risk. CRITICAL journeys are recommended for the first phase of implementation as they aim to meaningfully mitigate synthetic identity and account takeover. The following table describes risk-aware workflows by journey type.

Journey Risk Vonage Workflow
New account signup CRITICAL Identity Insights filters invalid/non-mobile numbers + Subscriber Match validates KYC → Silent Auth for zero-tap onboarding
Daily login MEDIUM SIM swap recency + device consistency check → Silent Auth passively, step-up only on elevated signals
Password recovery, profile change (contacts), 2FA settings change HIGH Mandatory SIM swap hard-check (tight lookback window) + Subscriber Match → Silent Auth required, no passive bypass
High-value transaction CRITICAL Full signal stack (line type, SIM swap, subscriber match) → Silent Auth + secondary challenge if risk elevated

Low-risk actions (for example, viewing account details, browsing content, or checking order history) generate no friction and no unnecessary verification cost. High-risk actions trigger the full assurance stack. The calibration is policy-driven and configurable per journey.

Implementation considerations

Configuring Amazon Cognito starts with setting up the user pool to allow the CUSTOM_AUTH authentication flow and accept phone numbers as the primary sign-in attribute. After the user pool is in place, associate the three required Lambda functions with their corresponding Amazon Cognito trigger hooks and store your Vonage API credentials in AWS Secrets Manager.

Layer in security from the start, following the AWS Well-Architected Security Pillar. Scope each Lambda function’s AWS Identity and Access Management (IAM) role to only what it needs: Amazon Cognito trigger invocations and AWS Secrets Manager access. Enforce TLS 1.2+ on all communication for encryption in transit. For observability, turn on Amazon CloudWatch logging on each Lambda function and turn on AWS CloudTrail to capture Amazon Cognito API audit trails. Finally, deploy AWS WAF with rate-limiting rules in front of the authentication endpoint to protect against brute-force attempts.

To configure the solution, follow these steps:

  1. Set up the Amazon Cognito user pool to allow the CUSTOM_AUTH authentication flow.
  2. Configure the user pool to accept phone numbers as the primary sign-in attribute.
  3. Associate the three required Lambda functions with their corresponding Amazon Cognito trigger hooks.
  4. Store your Vonage API credentials in AWS Secrets Manager.

Important: This solution creates AWS resources that incur charges. These include Amazon Cognito (per monthly active user), AWS Lambda (per invocation), AWS Secrets Manager (per secret per month), Amazon CloudWatch Logs, AWS CloudTrail, and AWS WAF (per rule and request). See the pricing page for each service and delete resources when no longer needed.

Privacy and compliance

The architecture is designed so that PII doesn’t leave the mobile operator. Subscriber Match performs a comparison within the operator’s environment and returns only a match score. The underlying subscriber data isn’t transmitted. Silent Authentication operates without PII exchange. The cellular session is the credential.

  • GDPR: Only match scores are returned. No subscriber PII is stored or transmitted, supporting GDPR data minimization.
  • PSD2 / Open Banking: Silent Authentication qualifies as a possession-factor for Strong Customer Authentication (SCA).
  • HIPAA: Subscriber Match supports identity assurance for healthcare applications.
  • DORA: Multi-channel fallback achieves > 99.9 percent verification availability.
  • CCPA: Same data-minimization architecture as GDPR.

Production results: Lydia Solutions

Lydia Solutions, one of Europe’s fastest-growing mobile financial services applications, deployed Vonage Verify with Silent Authentication in October 2024. The results demonstrate the real-world impact at scale, including up to 50 percent reduction in latency when compared to Lydia Solutions’s previous authentication services.

“Vonage Verify with Silent Authentication has been a real innovation for us. The solution has elevated our ability to deliver a simpler, seamless and more secure user experience while protecting against increasingly sophisticated threats and fraud patterns.”

— William Brulin, Senior VP, Lydia Solutions

Lydia’s results sit at the high end of outcomes observed. Across deployments in ecommerce, digital banking, and consumer services, conversion improvements of 2–8.5 percent compared to SMS-only are the norm, with authentication journey latency reductions of 50–75 percent.

Conclusion

This is where mobile operator data shifts the approach. Rather than applying identical verification friction to every session, enterprises can use real-time network signals to make adaptive authentication decisions. Verify silently when conditions are right, step up when risk indicators appear, and block when fraud is detected.

Enterprise implementation of the offering makes those risk signals and authentication methods accessible through a composable API layer. The combination of Identity Insights for pre-verification intelligence, Verify for network-layer authentication, and Fraud Defender for channel protection delivers risk-proportionate authentication that’s in production at scale today.

The solution deploys with minimal changes to your existing Amazon Cognito user pool. Start with high-risk journeys, measure impact, and expand. Vonage Verify API is available across over 700 MNOs in over 200 countries and territories, and the integration requires only three Lambda functions.

Next steps

Vonage is an AWS Partner. To learn more, visit the Vonage partner page.

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.


About the authors

How Samsung achieved real-time pricing with AWS Lambda Response Streaming

Post Syndicated from Vijay Naik original https://aws.amazon.com/blogs/architecture/how-samsung-achieved-real-time-pricing-with-aws-lambda-response-streaming/

This post is co-authored with Sathish Kumar and Christopher Chan from Samsung ecommerce.

In high-traffic ecommerce, achieving real-time pricing is critical to prevent price inconsistency. Pricing inconsistency creates cart shock and erodes trust. This isn’t broken software, it’s a symptom of architectural latency that you can address using AWS Lambda Response Streaming and Amazon CloudFront for systems aggregating data from multiple backend sources.

In this post, we walk through the legacy architecture challenges, the stateless streaming solution, key implementation patterns, and performance results—a pattern you can apply if you’re building high-traffic APIs that aggregate data from multiple backend sources.

Samsung.com is Samsung’s primary direct-to-consumer channel, selling smartphones, TVs, appliances, and accessories, each with multiple variants, offers, and regional pricing. This complexity makes real-time price accuracy especially important.

Samsung’s All Deals and Product Finder pages showcase these products during high-traffic events like Black Friday. To maintain low latency for these high-density Product Listing Pages (PLPs) and comparison tables, the legacy infrastructure relied on asynchronous caching, which introduced a desynchronization gap where the cached price drifted from the authoritative pricing engine.

Problem: Legacy middleware caching created a 1-hour desynchronization gap between the authoritative pricing engine and customer-facing pages.

Our approach: We dismantled the stateful Data Aggregation (DA) architecture and built a real-time Bulk Arbitration Engine (a stateless orchestration layer that queries the Pricing Engine directly at request time) using AWS Lambda Response Streaming and Amazon CloudFront edge caching.

Challenge: The Data Aggregation trap

When product listing pages need to display pricing for over 30 item combinations simultaneously, the latency of calling the Pricing Engine for each item combination individually becomes untenable. To solve this, we built a backend for frontend (BFF) service to do a “Data Aggregation. This DA service was designed to decouple the frontend from the heavy Pricing Engine.

It relied on a scheduled Cron Worker that ran hourly to fetch the entire product catalog. The worker would then precompute prices for every possible permutation of products and store them in a local cache.

While this improved read speeds, it created two significant failures:

1. The Permutation Explosion – The DA service had to precompute every combination just in case a customer viewed it.

  • The Math: 30 products × (Variants × Offers × Add-ons) = Thousands of records per page
  • Storage Impact: Cache grew exponentially with each new product variant added
  • Waste: Most precomputed combinations were never requested

2. The Synchronization Lag – Because the Cron job ran only once per hour, price changes (for example, flash sales) lagged significantly. Customers continued to see old prices until the next scheduled sync.

  • Business Impact: Flash sales showed incorrect old pricing until the next run time
  • Customer Trust: Cart shock when checkout price differed from product page price
  • Competitive Disadvantage: Competitors with real-time pricing gained market share

Legacy Data Aggregation architecture

Architecture diagram showing the legacy Data Aggregation layer between the Pricing Engine and CloudFront CDN.

Architecture diagram showing the legacy Data Aggregation layer between the Pricing Engine and CloudFront CDN

Figure 1: Legacy Data Aggregation (DA) Architecture The legacy system relied on a scheduled Cron job, creating a distinct “Desynchronization Layer” between the Authority (Pricing) and the Customer. Precomputation of all product permutations consumed significant storage and compute resources.

The solution: Stateless streaming architecture

Intermediate layers storing data will eventually diverge from the source, so we collaborated with our AWS Technical Account Manager (TAM) and service teams to architect a new solution: the Bulk Arbitration Engine, a stateless orchestration layer that queries the Pricing Engine directly at request time.

The new architecture follows a Pass-Through pattern:

1. Client Request: The browser requests prices for 30 specific SKUs using a single HTTP GET request.

2. Streaming Orchestration: An AWS Lambda function fans out these 30 requests to the Pricing Engine in parallel.

3. Immediate Response: As the Pricing Engine returns data, the Lambda streams it immediately to the client without buffering.

Why Lambda Response Streaming?

We evaluated several alternatives before settling on this approach:

  • Traditional request-response pattern (buffered) – A standard Lambda invocation buffers the full response before returning it to the client, which negates the latency benefit of parallel fan-out. For 30 concurrent SKU lookups, this added seconds of wait time.
  • EC2 with improved caching – This was the legacy approach. Caching layers will eventually drift from the source of truth, which was the core problem we needed to solve.
  • Lambda Response Streaming – This was the only option that let us fan out requests in parallel, stream results as they arrived (reducing time-to-first-byte), and remain fully stateless with no intermediate cache to maintain or invalidate.

New stateless streaming architecture

New stateless Lambda Response Streaming architecture with CloudFront caching layer directly connected to Lambda function, which streams real-time responses from Pricing Engine without intermediate buffering. The diagram shows customer GET request to CloudFront, cache hit/miss routing decision, Lambda orchestration of parallel SKU lookups, and streaming response back through CloudFront edge.

Architecture diagram showing the stateless streaming solution with CloudFront connected directly to Lambda.

Architecture diagram showing the stateless streaming solution with CloudFront connected directly to Lambda

Figure 2: Stateless Streaming Architecture The new architecture eliminates the middleware cache. A high-performance stream connects the user directly to the pricing source of truth. CloudFront edge locations cache the response for 95% of traffic, while remaining requests go directly to Lambda for real-time pricing.

Implementation walkthrough

Transitioning to this new architecture required solving two specific technical constraints regarding CDN behavior and cold starts. We implemented the solution in three steps.

Step 1: Implementing the streaming handler

The core of our solution is the Node.js Lambda handler wrapped in awslambda.streamifyResponse(). This allows us to pipe data through a transformation and compression stream directly to the client as it becomes available.

We used a custom NDJSONTransform to convert pricing objects into newline-delimited JSON (NDJSON), allowing the browser to parse and render each price as it arrives rather than waiting for the complete response.

// Lambda Handler with Response Streaming
// File: lambda-handler.js

import * as awslambda from "aws-lambda";
import * as zlib from "zlib";
import { pipeline } from "stream/promises";
import { NDJSONTransform } from "./transforms/ndjson-transform.js";

// Constants
const PROCESSING_MODE = process.env.PROCESSING_MODE || "MODE_QUERYSTRING";
const MODE_QUERYSTRING = "MODE_QUERYSTRING";

// The Handler MUST be wrapped in streamifyResponse() to enable streaming
export const handler = awslambda.streamifyResponse(
  async (event, responseStream, _context) => {
    try {
      if (PROCESSING_MODE === MODE_QUERYSTRING) {
        // Set response metadata with status code and headers
        const httpResponseMetadata = {
          statusCode: 200,
          headers: {
            "Content-Type": "application/x-ndjson",
            "Content-Encoding": "gzip",
            "Cache-Control": "public, max-age=300", // 5 min cache
            "X-Custom-Header": "lambda-streaming",
          },
        };

        responseStream = awslambda.HttpResponseStream.from(
          responseStream,
          httpResponseMetadata
        );

        const ndjsonTransform = new NDJSONTransform();

        // Create gzip stream with fast compression level (Z_BEST_SPEED = Level 1)
        // Level 1 prioritizes speed over compression ratio for real-time responses
        const gzip = zlib.createGzip({
          level: zlib.constants.Z_BEST_SPEED,
        });

        // Process the request, writing to the gzip stream
        await processRequestUsingQueryString(event, ndjsonTransform, _context);

        // Pipeline: Transform → Compress → Send
        await pipeline(ndjsonTransform, gzip, responseStream);
      }

      // ... error handling
    } catch (error) {
      console.error("Lambda streaming error:", error);
      responseStream.destroy();
    } finally {
      await flushMetrics().catch(console.error);
    }
  }
);

Helper function to fan-out the requests in parallel:

// Process request: fan out SKU lookups in parallel
async function processRequestUsingQueryString(event, ndjsonTransform, context) {
  try {
    // Parse query string to extract SKU list
    const queryString = event.rawQueryString || "";
    const skus = parseCompressedQueryString(queryString);

    // Fetch pricing in parallel using Promise.all
    const pricingPromises = skus.map((sku) =>
      fetchPricingForSKU(sku).catch((err) => ({
        sku: sku.id,
        error: err.message,
      }))
    );

    const pricingResults = await Promise.all(pricingPromises);

    // Emit each result as a separate NDJSON line
    for (const result of pricingResults) {
      ndjsonTransform.write(result);
    }

    ndjsonTransform.end();
  } catch (error) {
    ndjsonTransform.destroy(error);
  }
}

The handler also uses helper functions for parsing the compressed query string (parseCompressedQueryString), fetching individual SKU prices with connection pooling (fetchPricingForSKU), and flushing metrics to Amazon CloudWatch (flushMetrics).

Key implementation details:

  • awslambda.streamifyResponse() wraps the handler so it streams data in real time instead of waiting for the full response from the pricing engine.
  • NDJSONTransform converts objects to newline-delimited JSON (one object per line)
  • GZIP (GNU zip) compression with Z_BEST_SPEED (Level 1) prioritizes speed over compression ratio
  • pipeline() handles error propagation and stream cleanup
  • Response headers include Cache-Control for CloudFront caching

Step 2: Compressing the request data into a GET request

We needed to send complex request data (30 SKUs, context metadata) to the API.

Constraint: CloudFront and standard HTTP specs treat POST requests as non-idempotent, meaning they are not cacheable by default.

Our approach: We developed a dense, compressed query string format to fit the complex request data into a standard GET request. Format: g=group1(p=SKU-A:1:p=SKU-B:2)…

This allowed us to strictly use GET requests, keeping the request URI within standard length limits (~800 bytes) while carrying the same data as a 3-4KB JSON body.

Client-Side Code: Building the Compressed Query String

// Client-side: Building the compressed query string format
// File: pricing-client.js

/**
 * Builds a compressed query string for bulk pricing requests
 * Format: g=group1(p=SKU-A:1:p=SKU-B:2:p=SKU-C:3)
 * @param {Array} skus - Array of SKU objects with { id, variant }
 * @param {Object} context - Customer context { customerId, region, sessionId }
 * @returns {string} Compressed query string
 */
function buildPricingQueryString(skus, context = {}) {
  if (!skus || skus.length === 0) {
    throw new Error("SKUs array cannot be empty");
  }

  if (skus.length > 30) {
    throw new Error("Maximum 30 SKUs per request. Split into multiple batches.");
  }

  // Build SKU portion: p=SKU-001:1:p=SKU-002:2
  const skuParts = skus
    .map((sku) => {
      const variant = sku.variant || 1;
      return `p=${sku.id}:${variant}`;
    })
    .join(":");

  // Build context portion (optional)
  let contextPart = "";
  if (context && Object.keys(context).length > 0) {
    const contextStr = Object.entries(context)
      .map(([key, value]) => `${key}=${value}`)
      .join(":");
    contextPart = `:c=${contextStr}`;
  }

  // Final format: g=group1(p=SKU-A:1:p=SKU-B:2:c=customerId=123:region=US-EAST-1)
  return `g=group1(${skuParts}${contextPart})`;
}

/**
 * Fetches pricing data with streaming NDJSON response parsing
 * Core pattern: read chunks, split by newlines, parse each line as JSON
 */
async function fetchPricingStream(skus, options = {}) {
  const queryString = buildPricingQueryString(skus);
  const url = `${options.baseUrl}?${queryString}`;
  const response = await fetch(url, {
    method: "GET",
    headers: { Accept: "application/x-ndjson" },
  });

  // Stream the response using the ReadableStream API
  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  const pricingData = [];
  let buffer = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    // Decode chunk and append to buffer
    buffer += decoder.decode(value, { stream: true });

    // Split by newlines (NDJSON format: one JSON object per line)
    const lines = buffer.split("\n");

    // Keep the last incomplete line in buffer
    buffer = lines.pop() || "";

    // Parse each complete line as a JSON object
    for (const line of lines) {
      if (line.trim()) {
        const pricingObject = JSON.parse(line);
        pricingData.push(pricingObject);

        // Optional: update UI immediately as each price arrives
        if (options.onChunk) {
          options.onChunk(pricingObject);
        }
      }
    }
  }

  reader.releaseLock();
  return pricingData;
}

On page load, the client calls fetchPricingStream with up to 30 SKUs and an onChunk callback that updates each product’s DOM (Document Object Model) element as pricing chunks arrive. Helper functions handle updating individual price elements, displaying variant information, and gracefully degrading with a user-friendly message if pricing is temporarily unavailable.

Step 3: Configuring CloudFront for uncacheable requests

To allow CloudFront to cache these complex GET requests effectively, we configured a precise Cache Policy that includes all query strings and specific headers.

# Terraform: CloudFront Cache Policy Configuration
# File: cloudfront-cache-policy.tf

resource "aws_cloudfront_cache_policy" "pricing-cache-policy" {
  name = "${var.lambda_function_name}-cache-policy"

  # TTL Configuration
  default_ttl = 300  # 5 minutes
  max_ttl     = 1800 # 30 minutes
  min_ttl     = 300   # 5 minute

  parameters_in_cache_key_and_forwarded_to_origin {
    # Don't cache based on cookies (we don't use them)
    cookies_config {
      cookie_behavior = "none"
    }

    # Allowlist specific headers for cache key
    headers_config {
      header_behavior = "whitelist"
      headers {
        items = [
          "x-ecom-pricing-1",   # Custom headers (anonymized)
          "x-ecom-pricing-2",  
          "x-ecom-pricing-3"    
        ]
      }
    }

    # Include query strings in cache key
    # This verifies different SKU combinations have separate cache entries
    query_strings_config {
      query_string_behavior = "all"
    }

    # Enable automatic GZIP compression
    enable_accept_encoding_gzip = true
  }
}

# CloudFront Origin Request Policy (forward headers to Lambda)
resource "aws_cloudfront_origin_request_policy" "pricing-origin-policy" {
  name = "${var.lambda_function_name}-origin-policy"

  headers_config {
    header_behavior = "whitelist"
    headers {
      items = [
        "x-ecom-pricing-1",
        "x-ecom-pricing-2",
        "x-ecom-pricing-3"
      ]
    }
  }

  query_strings_config {
    query_string_behavior = "all"
  }

  cookies_config {
    cookie_behavior = "none"
  }
}

The CloudFront distribution itself is configured with HTTPS-only viewer protocol, the cache policy and origin request policy as shown in the preceding section, and points to the Lambda as its origin through HTTPS.

Cache policy highlights:

  • 5-minute default TTL balances freshness and cache efficiency
  • Query strings included in cache key (different SKU combos = separate cache entries)
  • Header allowlisting allows custom pricing variants
  • Automatic GZIP compression reduces bandwidth
  • 5–30 minute TTL range provides flexibility for different content

Performance optimization results

We optimized the system through four distinct phases, testing each configuration with K6 load test scripts (500 concurrent users, 30 items per request) to simulate high-traffic events like Black Friday.

Phase 1: The baseline (Global VPN)

We tested our initial proof-of-concept with the default network configuration, where all outbound traffic (including requests to AWS services like Lambda) was routed through a global VPN, forcing traffic onto the public network and back into the AWS backbone, adding unnecessary network hops and delays. The Lambda used a standard buffered response with no compression. The results were suboptimal (4,500ms P90) because connection overhead dominated the request.

  • DNS resolution: approximately 50 ms
  • TCP handshake: approximately 100 ms
  • TLS negotiation: approximately 150 ms
  • Total connection overhead: approximately 300 ms per call

This overhead created a massive bottleneck for latency before business logic even ran.

Phase 2: Amazon VPC Peering and warm starts

To remove the network penalty, we acted on two fronts:

  • First: Moved the Lambda inside an Amazon Virtual Private Cloud (Amazon VPC) peered directly to the pricing origin, cutting DNS and TLS overhead to near zero for internal calls.
  • Second: Enabled Provisioned Concurrency for Lambda to remove the 500–1000 ms cold start latency.

With these changes, P90 latency dropped to 1,000 ms, a 4.5x improvement, but still not real-time enough.

Phase 3: HTTP/2 and GZIP compression

The remaining bottleneck was the sheer size of the data transfer. We targeted two optimizations:

  • HTTP/2 Multiplexing: Enabled HTTP/2 multiplexing to reuse a single TCP connection for the 30 parallel SKU lookups, saving seconds of cumulative handshake time.
  • GZIP Compression: Applied GZIP compression (Level 1 / Z_BEST_SPEED), which reduced the response size by 76 percent (170KB → 40KB).

These two optimizations brought P90 latency down to 218 ms.

Phase 4: Production (edge caching)

In the final phase, we layered CloudFront edge caching on top of the optimized Lambda. Because we had successfully converted our request data to a GET request (Step 2), we could now cache the computed prices for 95 percent of incoming traffic.The final P90 latency landed at 50 ms.In practice, the 95 percent cache hit ratio means only 1 in 20 requests actually invokes the Lambda function; the rest are served directly from CloudFront edge locations closest to the customer. During peak events like Black Friday, this translates to millions of requests served at edge speed without touching the origin, keeping both latency and compute costs minimal.

Performance metrics table

Metric Phase 1 (Baseline) Phase 2 (VPC) Phase 3 (HTTP/2) Phase 4 (Production)
P50 Latency 1,670 ms 501 ms 176 ms 35 ms
P90 Latency 4,500 ms 1,000 ms 218 ms 50 ms
P99 Latency 5,100 ms 2,400 ms 500 ms 150 ms
Cache Hit Ratio <1% <1% <1% 95%
Response Size 170 KB 170 KB 40 KB 40 KB
Concurrent Users 500 500 500 500
P90 Improvement vs Baseline 1x 4.5x 20x 90x

The following chart shows P90 latency improvements across each optimization phase.

Latency improvements across four optimization phases

Latency improvements across four optimization phases.

K6 load test configuration:

// A sample K6 load test script
import http from 'k6/http';
import { check } from 'k6';

export let options = {
  stages: [
    { duration: '2m', target: 100 },  // Ramp up
    { duration: '5m', target: 500 },  // Stay at 500 concurrent users
    { duration: '2m', target: 0 },    // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(90)<100', 'p(99)<500'], // P90 < 100ms, P99 < 500ms
  },
};

export default function () {
  const url = 'https://api.example.com/pricing?g=group1(p=SKU-001:1:p=SKU-002:2:...)';
  const res = http.get(url);
  
  check(res, {
    'status is 200': (r) => r.status === 200,
    'response has content': (r) => r.body.length > 0,
  });
}

Resilience, scale, and security considerations

Beyond latency, we designed the system to handle failure gracefully, scale under load, and protect data in transit.

Batching limits

The 30-item limit per request is intentional. If a page requires more (for example, 50 items), the client logic splits them into multiple parallel batches.–We chose 30 because of the following:

  • Lambda execution time under 5 seconds
  • Prevents timeout issues during high latency
  • Balances parallel requests vs. Lambda concurrency limits
  • Typical product listing pages show 20–30 items
async function fetchLargePricingBatch(skus) {
  const BATCH_SIZE = 30;
  const batches = [];

  // Split into 30-item chunks
  for (let i = 0; i < skus.length; i += BATCH_SIZE) {
    batches.push(skus.slice(i, i + BATCH_SIZE));
  }

  // Fetch all batches in parallel
  const results = await Promise.all(
    batches.map((batch) =>
      fetchPricingStream(batch, { timeout: 30000 })
    )
  );

  // Flatten results
  return results.flat();
}

Partial failures

The streaming architecture is resilient. If pricing for one item fails, the stream doesn’t crash; it continues processing the remaining items, so the user still sees a mostly complete page.

Partial failure handling:

async function processRequestUsingQueryString(event, ndjsonTransform, context) {
  const skus = parseCompressedQueryString(event.rawQueryString);

  for (const sku of skus) {
    try {
      const pricing = await fetchPricingForSKU(sku);
      ndjsonTransform.write(pricing);
    } catch (error) {
      // Emit error object but continue processing other SKUs
      ndjsonTransform.write({
        sku: sku.id,
        variant: sku.variant,
        error: error.message,
        timestamp: new Date().toISOString(),
      });
    }
  }

  ndjsonTransform.end();
}

Data protection

  • While constructing the query string on the client exposes the request structure, this data (SKUs, variants) is already public. The actual pricing logic and business rules remain securely protected within the Pricing Engine.
  • Data in transit encrypted with TLS 1.3.
  • Amazon VPC endpoint connection to pricing engine (no internet exposure).
  • No sensitive data logged (PII, pricing algorithms excluded).
  • CloudTrail logs API calls for audit trail.

Conclusion

Stale pricing forces engineering teams to choose between freshness and scale. With the Data Aggregation pattern, we attempted to maintain both but compromised on data integrity due to the lag inherent in scheduled synchronization.By using AWS Lambda Response Streaming and Amazon CloudFront, we removed the need for a synchronization layer entirely. The result is a system that delivers the 50 ms latency required for a smooth user experience while supporting price consistency between the product page and checkout.

Beyond performance, this architecture significantly reduced operational footprint: compute fleet shrank from over 100 auto-scaled instances during peak events to only 5–10 Lambda functions, lowering maintenance and operational costs. This outcome was the result of close collaboration between Samsung’s ecommerce engineering team, our AWS Technical Account Manager (TAM), and the Lambda and CloudFront service teams, who helped architect the solution, review design decisions, and guide Samsung through production readiness. This technique applies to similar high-traffic data aggregation scenarios: product catalogs, inventory systems, recommendation engines, or services that combine multiple backend responses in real-time.

To get started, identify your highest-latency aggregation endpoints, evaluate whether your request data can be converted to cacheable GET requests and implement Lambda Response Streaming for a single endpoint before migrating your full API

Resources: – AWS Lambda Response Streaming documentation – Lambda Response Streaming tutorial – Amazon CloudFront Developer Guide – CloudFront cache policies

Learn more

For more on the concepts and technologies discussed in this post:


About the authors

Diagnose EKS Node Issues Faster with AWS DevOps Agent and Custom MCP

Post Syndicated from Shyam Kulkarni original https://aws.amazon.com/blogs/devops/diagnose-eks-node-issues-faster-with-aws-devops-agent-and-custom-mcp/

AWS DevOps Agent can investigate a growing range of production incidents autonomously. It diagnoses CrashLoopBackOff failures, traces ConfigMap deletions through audit logs, and correlates Amazon CloudWatch metrics with cluster events — all without human intervention.

But AWS DevOps Agent has a visibility boundary. When the data it needs lives outside its native integrations — on a node’s operating system, inside a third-party monitoring tool, behind a database’s internal diagnostics — the agent stalls. It can describe symptoms, but it can’t reach the evidence needed to identify root causes.

This post shows how to extend AWS DevOps Agent by building a custom Model Context Protocol (MCP) server that bridges that gap. Using a concrete example, we give AWS DevOps Agent structured access to Amazon EKS worker node diagnostics and explain how the same approach applies to data sources the agent can’t natively reach. By the end of this walkthrough, you will have a working MCP server that gives AWS DevOps Agent access to 20+ node-level log sources — providing autonomous investigation capabilities that can assist in root cause analysis compared to manual SSH sessions.

Prerequisites

Before you begin, make sure you have the following:

  • An Amazon EKS cluster with AWS Systems Manager Agent (SSM Agent) running on the worker nodes (included by default on Amazon EKS optimized AMIs)
  • Node.js v18 or later
  • AWS CLI v2
  • AWS CDK v2 installed and bootstrapped in your target account and Region
  • An AWS account with permissions to create IAM roles, Lambda functions, and Amazon S3 buckets
  • Familiarity with Amazon EKS, AWS Systems Manager, and the Model Context Protocol (MCP)

How AWS DevOps Agent discovers custom tools through MCP

MCP is an open standard that defines how AI agents discover and invoke external tools. AWS DevOps Agent supports connecting to custom MCP servers, which means you can expose new capabilities to it without modifying the agent itself. When you connect an MCP server to AWS DevOps Agent, the agent automatically discovers the available tools, understands their schemas, and calls them as part of its investigation workflow. You build and connect the MCP server — the agent handles the rest.

The extensibility model follows three steps: first, identify the data source that AWS DevOps Agent cannot natively access; second, build an MCP server that wraps safe, structured access to that data source; and third, connect the MCP server to AWS DevOps Agent so it can incorporate the new tools into its investigations.

Three design principles make this work. Return structured data, not raw text — pre-index findings with severity levels and stable IDs so the agent can filter, reference, and correlate them. Never give the agent a shell — mediate interactions through a controlled, auditable execution model. Make tools composable — design tool outputs to serve as inputs to other tools, creating a chain of evidence the agent can follow.

Why Amazon EKS node OS visibility matters

AWS DevOps Agent integrates with Amazon EKS to inspect pod status, read container logs, query CloudWatch Container Insights, and correlate cluster events. This covers application crashes, container-level resource exhaustion, and configuration drift.

However, EKS production issues with nodes originate in a layer these tools cannot reach: the node operating system. Artifacts such as iptables rules, full CNI configuration and IPAMD state, route tables, conntrack entries, dmesg kernel messages, containerd runtime logs, sysctl parameters, ENI metadata, and the unfiltered kubelet journal exist exclusively on the node. These artifacts are the primary evidence for diagnosing IP allocation failures, DNS resolution issues, network policy enforcement problems, storage mount timeouts, and node registration failures.

Integrating AWS DevOps Agent with an EKS node diagnostics MCP server

The sample-eks-node-diagnostics-mcp repository (sample-eks-node-diagnostics-mcp repository) demonstrates this pattern. It provides an MCP server that gives AWS DevOps Agent structured access to node-level diagnostic data, backed by AWS Systems Manager (SSM) Automation for safe, auditable execution.

How it works

AWS DevOps Agent connects over MCP/HTTPS to AgentCore Gateway, which authenticates via Amazon Cognito OAuth 2.0 and routes tool calls through a Lambda-based Tool Router to SSM Automation. SSM Automation dispatches runbooks to EKS worker nodes running SSM Agent, which upload collected log archives to a KMS-encrypted S3 bucket. An S3 event triggers a Lambda function that extracts and indexes findings for the agent to query.

Figure 1: End-to-end architecture of the EKS Node Diagnostics MCP server. AWS DevOps Agent discovers and invokes 19 tools through AgentCore Gateway, which dispatches SSM Automation runbooks to worker nodes for log collection and uploads results to Amazon S3 for extraction and indexing.

  1. AWS DevOps Agent calls a collect tool with an instance ID.
  2. The MCP server dispatches an SSM Automation execution to the target node, running the AWS-managed AWSSupport-CollectEKSInstanceLogs runbook.
  3. The runbook collects 20+ log sources — kubelet, containerd, iptables, CNI config, route tables, dmesg, sysctl, ENI metadata, IPAMD logs, and more — packages them into an archive, and uploads it to an Amazon S3 bucket where you configure AWS KMS encryption.
  4. A processing pipeline extracts the archive, pre-indexes errors with severity classification and stable finding IDs, and provides the results to you through additional MCP tools.

The server exposes tools for log collection, pre-indexed error retrieval, cross-file search and correlation, structured network diagnostics, and live packet capture. A typical agent workflow chains these together: collect → status → errors → search → correlate → read → summarize, with each step producing outputs that feed into the next.

AWS DevOps Agent does not get a shell on the node. Every interaction is mediated by SSM Automation — an auditable, IAM-controlled, non-interactive execution model.

Connecting through Amazon Bedrock AgentCore Gateway

The reference implementation uses Amazon Bedrock AgentCore Gateway to expose the Lambda-backed MCP server to AWS DevOps Agent. AgentCore Gateway converts Lambda functions into MCP-compatible tools and handles authentication, protocol translation, and tool discovery through a single managed endpoint.

The integration follows three steps:

Step 1: Create an OAuth authorizer with Amazon Cognito. The CDK stack provisions a Cognito User Pool configured for the OAuth 2.0 client credentials flow. This secures inbound access to the gateway — only clients with valid tokens can invoke tools.

Step 2: Create a gateway and register the Lambda as a target. Register the Lambda function that handles tool invocations as a target on the gateway. AgentCore Gateway automatically discovers the tool schemas from the Lambda and makes them available through the MCP protocol. The gateway endpoint becomes the single MCP URL for AWS DevOps Agent.

Step 3: Connect AWS DevOps Agent. Register the MCP server at the account level in the AWS DevOps Agent console, providing the gateway URL and OAuth configuration. Then allowlist the specific tools each Agent Space needs. AWS DevOps Agent authenticates by obtaining a JWT from the Cognito token endpoint using the client credentials grant and passes it as a Bearer token in requests to the gateway URL.

Deploying the MCP server

Deploy the entire stack using AWS CDK :

git clone https://github.com/aws-samples/sample-eks-node-diagnostics-mcp.git
 cd sample-eks-node-diagnostics-mcp
 chmod +x deploy.sh
 ./deploy.sh

The script walks you through cluster selection and node role configuration. Have the following ready before running the script: your target EKS cluster name, the IAM role ARN you attached to your worker nodes, and the AWS Region where your cluster runs. The script outputs your MCP gateway URL, OAuth credentials, and token endpoint — everything you need to configure the connection in AWS DevOps Agent. See the repository README for detailed deployment instructions, CI/CD mode, and prerequisite details.

Seeing it in action

To demonstrate the MCP server’s capabilities, we walk through a realistic node-level failure scenario on a test EKS cluster. We manually inject a fault that blocks pod DNS resolution at the iptables level — an issue that is invisible from kubectl since pods appear Running — then show how AWS DevOps Agent investigates and identifies the root cause using the MCP server’s tools.

Setting up the scenario

Start with an EKS cluster that has a managed node group with SSM Agent running (included by default on Amazon EKS optimized AMIs). Deploy a sample workload to one of the nodes:

kubectl create namespace demo-app

cat <<EOF | kubectl apply -f -
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: web-frontend
   namespace: demo-app
 spec:
   replicas: 3
   selector:
     matchLabels:
       app: web-frontend
   template:
     metadata:
       labels:
         app: web-frontend
     spec:
       containers:
       - name: nginx
         image: nginx:latest
         ports:
         - containerPort: 80
 EOF

Identify the node and instance ID where the pods are running:

kubectl get pods -n demo-app -o wide

Injecting the fault

⚠ WARNING: The following commands will disrupt DNS resolution for all pods on the target node. Only run these in a non-production test environment. Do not execute on production nodes.

Connect to the target node using SSM Session Manager and run the following commands to block pod DNS traffic at the iptables level. This simulates a subtle networking issue – pods continue running but can’t resolve DNS, and the root cause is only visible in the node’s iptables rules:

# Block pod traffic to kube-dns ClusterIP — pods run but DNS fails
 # Only affects FORWARD chain (pod traffic), not the node's own DNS
 sudo iptables -I FORWARD -d 10.100.0.10/32 -p udp --dport 53 -j DROP
 sudo iptables -I FORWARD -d 10.100.0.10/32 -p tcp --dport 53 -j DROP

Replace 10.100.0.10 with your cluster’s kube-dns ClusterIP (kubectl get svc kube-dns -n kube-system -o jsonpath=’{.spec.clusterIP}’).

This fault is particularly insidious because kubectl get pods shows all pods in Running state. The applications fail with DNS resolution errors, but there is no Kubernetes event or pod status that points to the cause. The iptables DROP rules targeting the kube-dns ClusterIP exist only in the node’s firewall configuration — a layer that no Kubernetes API call can inspect.

Investigating with AWS DevOps Agent

An engineer notices applications reporting DNS failures and asks AWS DevOps Agent to investigate:

“Pods on node i-xxxxxxxxxx in cluster EKS-sample (us-east-1) are running but applications report DNS resolution failures. Collect the node logs and investigate.”

The AWS DevOps Agent "Start an investigation" dialog with the investigation details field populated: "Pods on node i-xxxxxxxxxxxx in cluster EKS-sample (us-east-1) are running but applications report DNS resolution failures. Collect the node logs and investigate." The date and time of incident is set to 2026-03-26T16:55:30.593Z.

Figure 2: Starting an investigation in AWS DevOps Agent. The engineer provides the symptom description and incident timestamp, and the agent autonomously plans and executes the investigation.

AWS DevOps Agent begins the investigation by recording the symptom and launching two parallel actions: collecting node logs via the nodelog_collect tool and checking cluster health. The cluster health check confirms all four nodes are running and SSM-online. The agent then polls the log collection status, tracking progress from 25% through 75% to completion. Once collection finishes, the agent fans out into parallel workstreams — running network diagnostics, performing quick triage, and collecting logs from a healthy node for comparison.

The investigation timeline progresses from "Starting" at 11:59:45 AM through symptom identification at +12 seconds, cluster health check at +33 seconds confirming all four nodes are running, log collection polling at 25% and 75%, to log collection complete at +1 minute 22 seconds. The agent then launches parallel network diagnostics, quick triage, and healthy node comparison.

Figure 3: Investigation timeline showing the initial data collection phase. The agent identifies the symptom, confirms cluster health, collects node logs via SSM Automation, polls for completion, and launches parallel diagnostic workstreams.

With the initial data collected, the agent launches four parallel investigation tasks to maximize coverage and minimize time-to-root-cause: (1) deep-dive-iptables-routes examines the node’s firewall rules and routing table in detail, completing in 1 minute 44 seconds across 8 tool calls; (2) search-network-errors scans the collected logs for network-related error patterns, running 15 tool calls over 7 minutes 51 seconds; (3) collect-healthy-node gathers the same diagnostics from a known-good node for comparison, taking 13 tool calls over 4 minutes 55 seconds; (4) check-oom-and-pod-status investigates kernel OOM kills and pod health, executing 19 tool calls over 8 minutes 12 seconds. Each task produces a structured report that feeds into the final synthesis.

Four parallel investigation tasks execute concurrently: deep-dive-iptables-routes (8 tool calls, 1 minute 44 seconds), search-network-errors (15 tool calls, 7 minutes 51 seconds), collect-healthy-node (13 tool calls, 4 minutes 5 seconds), and check-oom-and-pod-status (19 tool calls, 8 minutes 12 seconds). At +14 minutes 22 seconds, all four tasks complete and the agent begins synthesizing findings.

Figure 4: Parallel investigation phase. The agent runs four concurrent deep-dive tasks — iptables/route analysis, network error search, healthy node comparison, and OOM/pod status check — then synthesizes the findings into a unified report.

The iptables and route table deep-dive reveals the root cause. The agent identifies two CRITICAL findings: a FAULT-INJECT-DROP-POD-TO-POD rule in the FORWARD chain that drops inter-pod traffic, and a FAULT-INJECT-DROP-SERVICE-CIDR rule that drops forwarded traffic to the service CIDR range. It also flags a MEDIUM-severity finding — a blackhole route for 10.96.0.0/12 (the Kubernetes service CIDR) that does not exist on healthy nodes. The remaining checks come back normal: kube-proxy chains are intact, AWS VPC CNI SNAT/CONNMARK chains are properly configured, and the default gateway and ENI route tables are correct. This structured severity classification allows the agent to immediately focus on the critical items.

A severity-classified findings summary table from the deep-dive-iptables-routes task. Two CRITICAL findings: a FAULT-INJECT-DROP-POD-TO-POD rule and a FAULT-INJECT-DROP-SERVICE-CIDR rule, both in the FORWARD chain. One MEDIUM finding about limited pod /32 routes. Six Normal findings confirm kube-proxy chains, AWS VPC CNI SNAT/CONNMARK chains, FORWARD chain policy, per-ENI route table, and default gateway are all properly configured.

Figure 5: Deep-dive findings from the iptables and route table analysis. Two CRITICAL fault-injection DROP rules in the FORWARD chain are identified as the primary issue, while standard networking components — kube-proxy, VPC CNI, and routing — check normal.

The healthy node comparison confirms the diagnosis. The agent compares the unhealthy node against a known-good node across seven dimensions: security groups, ENI count, DNS configuration, iptables rules, route tables, conntrack entries, and IPAMD state. The key differences are definitive: the blackhole route for 10.96.0.0/12 exists only on the unhealthy node, kubelet API server timeout errors appear only on the unhealthy node, conntrack entries are 12x higher (1,962 vs 169), and IPAMD reconciliation errors are 5x more frequent. The iptables FORWARD chain counters show 2.4 billion packets processed on the unhealthy node versus zero on the freshly-started healthy node — confirming sustained traffic disruption.

A comparison table titled "Summary of Key Differences" between the unhealthy and healthy nodes. Five differences are listed: a blackhole route for 10.96.0.0/12 present only on the unhealthy node, kubelet API server timeout errors present only on the unhealthy node, conntrack entries at 1,962 versus 169, IPAMD reconcile errors at 5 versus 1, and iptables FORWARD counters at 2.4 billion packets versus 0 on the fresh healthy node. DNS configuration is identical on both nodes.

Figure 6: Healthy node comparison confirming the diagnosis. The agent compares diagnostics across both nodes and identifies five key differences — the blackhole route, elevated conntrack entries, and high FORWARD chain packet counts exist only on the affected node.

The agent synthesizes the findings into a definitive root cause determination. It identifies a fault-injection namespace on the EKS cluster that is running chaos experiments, introducing three specific network-disrupting modifications on the target node: (1) a FAULT-INJECT-DROP-POD-TO-POD iptables rule in the FORWARD chain that drops inter-pod traffic, (2) a FAULT-INJECT-DROP-SERVICE-CIDR rule that drops forwarded traffic to the Kubernetes service CIDR, and (3) a blackhole route for 10.96.0.0/12 that does not exist on healthy nodes. Together, these three modifications create a multi-vector network disruption — pods appear Running but cannot communicate with each other or reach Kubernetes services, including kube-dns.

The Root causes panel identifies one root cause: "Fault-injection workloads on node i-09ffc4a0ea5da9cb7 causing multi-vector network disruption." The explanation states that a fault-injection namespace is running chaos experiments that introduced two iptables FORWARD chain DROP rules (FAULT-INJECT-DROP-POD-TO-POD and FAULT-INJECT-DROP-SERVICE-CIDR) and a blackhole route for 10.96.0.0/12 that does not exist on healthy nodes.

Figure 7: Root cause determination. The agent traces the multi-vector network disruption to three fault-injection modifications — two iptables DROP rules and a blackhole route — deployed by a chaos experiment namespace on the target node.

Cleaning up the fault

To restore the node after the demo, connect via SSM Session Manager and run:

sudo iptables -D FORWARD -d 10.100.0.10/32 -p udp --dport 53 -j DROP
sudo iptables -D FORWARD -d 10.100.0.10/32 -p tcp --dport 53 -j DROP

Extending this pattern to other data sources

The EKS node diagnostics use case demonstrates the pattern, but the architecture generalizes to systems where the SSM Agent is running and you can define an SSM Automation runbook to collect the data you need.

For example, an EC2 instance with SSM Agent can use this same approach — collect OS-level logs, network configuration, package state, or application diagnostics through a custom or pre-built SSM Automation runbook, upload results to S3, and expose them through MCP tools. The same applies to ECS container instances (Docker daemon logs, ECS agent state, iptables), on-premises servers registered via SSM Hybrid Activations, or managed nodes in your fleet.

The pattern also extends beyond SSM-managed hosts. Network devices can be reached through API calls to their management planes, databases through read-only diagnostic queries, and third-party APM tools through vendor API integrations. In each case, the same three-step approach holds: identify the unreachable data, build an MCP server that wraps safe access to it, and connect it to AWS DevOps Agent.

When to use this approach
This pattern works well for incident response where diagnostic data lives outside AWS DevOps Agent’s native reach, fleet-wide triage where manual access to individual systems is impractical, and cross-source correlation where evidence spans multiple log sources.

It is not a replacement for continuous monitoring (use CloudWatch Container Insights or Prometheus for real-time alerting), log shipping (if you have compliance requirements for continuous retention), or native integrations where the agent already has access to the data source.

The reference implementation requires SSM Agent running on the nodes with appropriate IAM permissions. It is a proof of concept — validate it in non-production environments before using it with production workloads.

Clean up

Cost considerations: This solution uses AWS Lambda, Amazon S3, AWS KMS, Amazon Cognito, and Amazon Bedrock AgentCore Gateway. Costs vary based on usage. Lambda charges apply per invocation and duration. S3 charges apply for log storage. KMS charges a per-key monthly fee plus per-request charges. Cognito charges per monthly active user. AgentCore Gateway pricing is based on API calls. For current pricing details, see the AWS Pricing page for each service. To minimize costs during evaluation, delete the stack when not in use.

Remove the deployed resources by running cdk destroy from the repository root. The S3 log bucket uses a RETAIN removal policy — delete it manually after stack destruction if needed.

Conclusion

MCP provides a standardized extensibility mechanism that lets you bridge visibility gaps in AWS DevOps Agent without modifying the agent itself. The pattern is straightforward: identify the unreachable data source, build an MCP server that wraps safe and structured access to it, and connect it to AWS DevOps Agent through Amazon Bedrock AgentCore Gateway. The agent handles the reasoning. The MCP server handles the data access.

To get started:

  • Deploy the reference implementation (sample-eks-node-diagnostics-mcp repository) in a non-production environment.
  • Review the MCP specification (MCP specification).
  • Explore the Amazon EKS troubleshooting documentation (Amazon EKS troubleshooting documentation).
  • Connect custom MCP servers to AWS DevOps Agent — see the Connecting MCP Servers guide in the AWS DevOps Agent documentation.
  • Set up AgentCore Gateway — see the Amazon Bedrock AgentCore Gateway quick start guide.

About the author

Shyam Kulkarni

Shyam Kulkarni

Shyam Kulkarni is a Sr. Technical Account Manager at AWS, where he helps enterprise customers design and implement cloud-native architectures with a focus on container orchestration, platform engineering, and observability at scale. He advises organizations on strategic modernization initiatives and is passionate about architecting AI-native systems, including agentic AI platforms and scalable AI infrastructure. Outside of work, Shyam is an avid travel and landscape photographer who enjoys exploring new destinations and capturing dramatic natural scenery. He’s also an enthusiastic home cook and baker who loves experimenting with new recipes, flavors, and techniques in the kitchen. When not behind a camera or in the kitchen, you’ll find him hiking remote trails.

Building highly available Oracle databases with Amazon FSx for NetApp ONTAP

Post Syndicated from Vignyanand Penumatcha original https://aws.amazon.com/blogs/architecture/building-highly-available-oracle-databases-with-amazon-fsx-for-netapp-ontap/

Oracle databases power mission-critical enterprise applications, making their continuous availability essential for business operations. Traditional Oracle high availability (HA) solutions require complex clustering software, expensive shared storage arrays, and specialized database administration teams. These conventional approaches often introduce single points of failure while demanding significant operational overhead.

Modern cloud architectures offer a transformative approach that combines Amazon FSx for NetApp ONTAP (FSxN) with Amazon EC2 Auto Scaling groups, automated AMI creation, AWS Lambda-driven orchestration, and AWS Systems Manager Parameter Store (SSM Parameter). This solution removes traditional Oracle HA complexities while delivering enterprise-grade availability, automated recovery, and makes sure new instances launch with the latest Oracle configuration.

This post shows how to build a highly available Oracle database architecture using FSxN shared storage, Auto Scaling groups with dynamic AMI updates, and serverless orchestration to help reduce recovery times with current configurations.

Solution overview

The solution uses multiple AWS services working together to create a comprehensive high availability architecture. FSxN Multi-AZ provides persistent shared storage spanning availability zones for Oracle database files, software, and configurations, so that data remains accessible when EC2 instances are replaced. Auto Scaling groups deliver automated instance lifecycle management with the latest AMI configurations, so failed instances are quickly replaced with identical configurations that can immediately access the existing Oracle database files on FSxN. AWS Backup creates AMIs that capture the latest Oracle host configurations including patches and settings, preserving the complete server state for consistent deployments. AWS Lambda extracts the AMI ID from backup recovery points and updates the SSM Parameter, orchestrating the entire configuration management workflow. Systems Manager Parameter Store stores the current AMI ID for Auto Scaling group launch templates, so new instances always launch with the most recent configuration and can immediately connect to the Oracle database on shared storage.

The following diagram shows the complete architecture with all AWS services and their interactions:

AWS architecture diagram showing Oracle Database disaster recovery across two Availability Zones using FSx for ONTAP synchronous replication, AWS Backup automation with EventBridge and Lambda, and Auto Scaling group with SSM Parameter Store for AMI management.

Key benefits include:

  • Recovery Time Objective (RTO): Can help achieve 2–5 minutes with latest Oracle configuration
  • Recovery Point Objective (RPO): Near-zero through synchronous Multi-AZ replication
  • Configuration consistency: New instances launch with identical Oracle host setup
  • Automated AMI management: Scheduled AMI creation with Parameter Store updates

Walkthrough

This walkthrough demonstrates implementing Oracle HA using Amazon FSx for NetApp ONTAP shared storage, AWS Backup-driven AMI creation, Lambda orchestration, and Auto Scaling groups with Parameter Store integration for configuration consistency and automated failover.

Prerequisites

For this walkthrough, you should have the following prerequisites:

  • An AWS account with appropriate permissions for Amazon FSx, Auto Scaling, EC2, Lambda, and Systems Manager
  • A VPC with subnets in at least two Availability Zones
  • Oracle database software

Keep in mind that customers are responsible for their own Oracle licensing compliance.

  • An EC2 instance with Oracle database installed and configured
  • AWS Identity and Access Management (IAM) roles for AMI creation and cross-service communication
  • Basic knowledge of Oracle database administration and AWS automation

Assumptions

This post is a conceptual illustration of the architecture. Your specific implementation will vary based on your VPC layout, Oracle version, storage requirements, and organizational security policies.

We assume the reader is familiar with:

  • Creating and configuring Amazon FSx for NetApp ONTAP file systems through the AWS console
  • iSCSI concepts including initiators, targets, and multipath I/O
  • Oracle database startup and shutdown procedures
  • AWS Backup, Lambda, and Auto Scaling group fundamentals

For detailed step-by-step instructions on specific AWS services, refer to the additional resources section.

Step 1: Create an Amazon FSx for NetApp ONTAP file system

FSxN Multi-AZ provides the persistent shared storage foundation for this architecture. Unlike Amazon Elastic Block Store (Amazon EBS) volumes, which are bound to a single AZ, FSxN Multi-AZ replicates data synchronously across two AZs with automatic failover. This means that when an EC2 instance is replaced (whether in the same AZ or a different one), the new instance can immediately access the existing Oracle database files without restoring from backup.

To create the file system, navigate to the Amazon FSx console and select Amazon FSx for NetApp ONTAP as the file system type.

The critical configuration choice is selecting Multi-AZ deployment, which places an active file server in one AZ and a standby in another.

Amazon FSx console showing oracle-fsxn-multi-az file system configuration with ONTAP Multi-AZ 1 deployment, 1024 GiB SSD storage, 512 MB/s throughput, spanning us-east-1a preferred and us-east-1b standby subnets.

FSxN console showing Multi-AZ deployment type selection with preferred and standby subnets in separate availability zones.

After the file system is created, you need to set up a Storage Virtual Machine (SVM), which acts as a logical storage container providing data access to your Oracle instances. The SVM creation is done from the FSx console under your file system’s details.With the SVM in place, the next step is configuring iSCSI access. FSxN exposes iSCSI endpoints—these are IP addresses (one per AZ) that your EC2 instances use to connect to the storage over the iSCSI protocol. You can find these endpoint addresses in the FSx console under your SVM’s Endpoints tab.

Amazon FSx Storage Virtual Machine configuration page showing oracle-svm with Created lifecycle state, NFS, iSCSI, and management endpoints for Oracle Database storage connectivity.

SVM Endpoints tab showing iSCSI endpoint IP addresses for each availability zone. These addresses are used in the EC2 instance’s iSCSI discovery configuration.

The iSCSI setup involves creating iGroups (which define which EC2 instances can access the storage) and LUNs (logical storage units mapped to those groups) through the NetApp ONTAP CLI. On the EC2 side, you configure the iSCSI initiator to discover and connect to the FSxN endpoints, then mount the resulting block devices. Using multipath I/O with both endpoints makes sure that Oracle data remains accessible even during an AZ failover. For detailed iSCSI configuration steps, see mounting iSCSI LUNs on Linux clients.

A dedicated security group is required for FSxN access. At minimum, the security group must allow inbound traffic on ports 111 (NFS portmapper), 635 (NFS mountd), 2049 (NFS), 3260 (iSCSI), 4045–4046 (NFS lock), 443 (HTTPS for management), and 22 (SSH for ONTAP CLI). Restrict the source to only your Oracle EC2 instances’ security group.

Step 2: Set up AWS Backup for EC2 instance protection

AWS Backup captures the complete state of your Oracle EC2 instance. The key design choice here is using tag-based resource selection rather than specifying instance IDs directly. Because Auto Scaling groups replace instances (and generate new instance IDs), tag-based selection makes sure that any new instance with the correct tags are automatically included in the backup plan.Configure a backup plan with a frequency appropriate for your environment and set the resource assignment to select EC2 instances matching your application tag (for example, ‘Application: Oracle’).

AWS Backup console showing blog-test backup plan with hourly backup rule targeting Oracle EC2 instances identified by the Application:oracle-db tag.

AWS Backup resource assignment configured with tag-based selection. Any EC2 instances tagged with the application tag are automatically included in the backup plan.

Step 3: Configure Lambda for AMI management

When AWS Backup completes an EC2 backup, it creates an AMI as the recovery point. An Amazon EventBridge rule detects this completion event and triggers a Lambda function. The function extracts the AMI ID from the backup recovery point, updates the SSM Parameter Store parameter with the new AMI ID, and cleans up older AMIs to control storage costs.

AWS Lambda function configuration for oracle-backup-handler showing Python 3.11 runtime, EventBridge trigger, and description indicating it processes AWS Backup completion events and updates AMI in SSM.

Lambda function overview showing the EventBridge trigger, Python 3.11 runtime, and function description indicating its role in processing backup completions and updating AMI references in SSM.

This event-driven approach means the latest AMI is available without manual intervention. The Lambda function needs IAM permissions for EC2 (to manage AMIs), SSM (to update the parameter), and Backup (to read recovery point metadata).

Amazon EventBridge rule oracle-backup-completion configured to trigger the oracle-backup-handler Lambda function when AWS Backup completes an EC2 backup job, with event pattern filtering for COMPLETED state.

EventBridge rule configured to match AWS Backup job completion events for EC2 resources, with the Lambda function as the target.

Step 4: Configure the Systems Manager Parameter Store

The SSM Parameter Store holds the current AMI ID that the Auto Scaling group’s launch template references. The parameter is created with the aws:ec2:image data type, which enables the launch template’s resolve:ssm: functionality, a feature that allows the launch template to dynamically resolve the AMI ID at instance launch time without requiring a template version update.

AWS Systems Manager Parameter Store showing /oracle/ec2/ami-id parameter with AMI value ami-0a705a7d5523c555, version 857, last modified by the oracle-backup-lambda-role on April 25, 2026.

SSM Parameter Store showing the /oracle/ec2/ami-id parameter with aws:ec2:image data type. The “Last modified user” confirms the Lambda function is automatically updating this parameter after each backup cycle.

When Lambda updates this parameter after each backup cycle, the next instance launched by the Auto Scaling group will automatically use the latest AMI. This removes the operational burden of manually updating launch template versions.

Step 5: Set up an Auto Scaling Group with dynamic AMI

The launch template references the SSM parameter using the resolve:ssm: prefix for the AMI ID field. This is the mechanism that ties the entire automation pipeline together. The mechanism backups trigger AMI creation, AMI IDs flow into Parameter Store, and the launch template resolves the latest AMI at launch time.

EC2 Launch Template oracle-db-launch-template version 75 showing AMI ID resolved from SSM parameter resolve:ssm:/oracle/ec2/ami-id with r7i.large instance type for Oracle Database deployment.

Launch template AMI configuration showing the ‘resolve:ssm:’ prefix, which dynamically retrieves the latest AMI ID from Parameter Store at instance launch time.

The Auto Scaling group is configured with minimum, maximum, and desired capacity all set to 1. This is not traditional auto-scaling, it’s a self-healing pattern. The sole purpose is to detect when the Oracle instance becomes unhealthy and automatically launch a replacement. The health check grace period should be set to at least 300 seconds (5 minutes) to allow Oracle sufficient time to start before health checks begin evaluating the new instance.

The launch template also includes a User Data script that runs on each new instance. This script configures the iSCSI initiator, discovers and connects to the FSxN endpoints, mounts the Oracle data volumes, and starts the Oracle database through a systemd service. This automation makes sure that a replacement instance is fully operational without manual intervention.

EC2 Auto Scaling group oracle-db-asg configuration showing desired capacity of 1, scaling limits 1-1, r7i.large instance type, oracle-db-launch-template with Latest version, spanning two availability zone subnets.

Auto Scaling group configured with min=max=desired=1 across two availability zones, providing self-healing capability.

Test the complete workflow

To validate the architecture, simulate an instance failure by terminating the current Oracle EC2 instance.

The expected sequence is:

  1. The Auto Scaling group detects the instance is unhealthy (within approximately 30 seconds)
  2. A new instance launches from the latest AMI resolved from Parameter Store (approximately 2 minutes)
  3. The User Data script connects to FSxN using iSCSI and starts Oracle (approximately 2–3 minutes)
  4. The Oracle database is available and accepting connections (total elapsed: approximately 5 minutes)

Auto Scaling group Activity History showing the self-healing sequence — the unhealthy instance is terminated, and a replacement is launched automatically within seconds.

The new instance automatically inherits the application tags from the Auto Scaling group, which means AWS Backup includes it in the next backup cycle without manual configuration.

Cleaning up

To avoid incurring future charges, delete the resources:

  • Delete Lambda functions and EventBridge rules
  • Remove Parameters from Systems Manager Parameter Store
  • Delete AWS Backup plans and backup vault
  • Deregister created AMIs
  • Terminate Auto Scaling group instances
  • Delete the Amazon FSx for NetApp ONTAP file system

Conclusion

This architecture facilitates Oracle high availability with configuration consistency by combining FSxN persistent shared storage with automated AMI management and AWS Backup protection. The Lambda-driven AMI management from backup recovery points and Parameter Store integration helps make sure that replacement instances launched by Auto Scaling groups always use the latest Oracle host configuration and can immediately connect to the existing Oracle database files stored on FSxN. Replacements occur only when health checks fail. Organizations can target high availability while maintaining configuration consistency across instance replacements. The automated AMI management alleviates configuration drift and makes sure that disaster recovery scenarios restore Oracle instances with identical host-level configurations that can immediately access the persistent Oracle database on shared storage. Healthy instances continue running unchanged, with replacements occurring only, when necessary, because of health check failures.Next steps include implementing cross-Region AMI replication, adding AMI validation testing, and developing custom health checks that verify both Oracle database and host configuration consistency.

Additional resources

Automating contract intelligence with Doczy.ai™ on AWS

Post Syndicated from Sanket Nasre original https://aws.amazon.com/blogs/architecture/automating-contract-intelligence-with-doczy-ai-on-aws/

Extracting actionable insights from thousands of contracts and legal documents remains a challenge. For organizations, critical business information is locked in unstructured documents such as contracts, legal agreements, provider arrangements, and vendor invoices. Extracting and operationalizing this information has traditionally been a manual, error-prone, and resource-intensive process. This leads to missed savings opportunities, costly delays, and significant inefficiencies across the enterprise.

AArete, a global management and technology consulting firm specializing in healthcare, recognized this challenge and developed Doczy.ai™, an intelligent contract interpretation solution powered by generative AI on Amazon Web Services (AWS).

In this post, we show you how Doczy.ai™ uses generative AI on AWS to automate contract intelligence at scale, transforming unstructured documents into structured, actionable insights, so organizations can automate critical business processes and unlock the full value of their data.

The challenge: Data trapped in documents

For healthcare organizations, managing and interpreting contracts and documents represents a major operational bottleneck. Manual review processes require deploying teams to extract data from thousands of documents. This is an approach that is neither scalable nor sustainable, highly prone to error, and costly. Organizations relying on institutional knowledge face additional risks: critical information resides with a few key individuals, creating knowledge silos and succession planning challenges. Existing Contract Lifecycle Management (CLM) systems often prove inadequate for capturing the nuanced and complex terms unique to each agreement. These legacy systems can only configure predefined fields, missing the rich detail and contextual information that distinguishes contracts. The downstream impact is substantial: in healthcare, reimbursement terms must be manually translated into claims systems—a slow, error-prone process. Similarly, verifying vendor invoices against contract terms often requires manual effort, leading to payment processing delays and missed contractual savings opportunities. These inefficiencies ultimately leave significant value on the table.

This is where Doczy.ai™ provides significant value.

Doczy.ai™: An intelligent contract interpretation solution

Doczy.ai™ directly addresses these challenges using advanced AI and scalability on AWS. Developed by AArete, Doczy.ai™ pushes the boundaries of document intelligence. The solution automatically interprets complex documents and converts them into a structured, queryable information repository that allows organizations to unlock the full value of their data and drive smarter decisions.The evolution of Doczy.ai™ reflects rapid AI advancement. Prior to 2020, document processing required manual effort, with individuals processing approximately 100 documents per week. Between 2020–2023, the firm implemented rules-based contract processing, achieving approximately 55% accuracy. The breakthrough came in 2024 with an AI-based processing built on AWS achieved 99% accuracy—a dramatic improvement over the 55% accuracy of traditional rules-based systems.

Doczy.ai™ architecture

Doczy.ai™ is built on a comprehensive AWS architecture designed to handle the entire document processing lifecycle: from the moment a file enters the system to the moment it generates actionable business intelligence.

Doczy.ai is built on a comprehensive AWS architecture designed to handle the entire document processing lifecycle: from the moment a file enters the system to the moment it generates actionable business intelligence.

Architecture of Doczy.ai™

External users access the platform through a secure Next.js frontend, with Amazon Cognito managing authentication and authorization behind the scenes. After authentication, users upload documents directly to Amazon Simple Storage Service (Amazon S3), where durable, scalable object storage ensures nothing is lost and everything is accessible at scale. From there, the real intelligence begins.

An AWS Lambda function triggers Amazon Textract to extract text and metadata from documents in various formats. What sets Doczy.ai™ apart at this stage is its patented “smart chunking” algorithm, a proprietary approach that goes far beyond pulling words off a page. Rather than treating a document as a flat sequence of text, smart chunking preserves hierarchical structure and one-to-many relationships within documents. It uses a combination of semantic and keyword search to decompose text into meaningful, context-aware chunks, applying dynamic parameters to maintain logical relationships throughout. Sequential identifiers and metadata-driven grouping organize these chunks into field groups, detecting overlaps and removing duplications while keeping the document’s natural flow intact.

After chunking, the document enters the dual clustering engine of Doczy.ai™. This two-lens methodology analyzes every contract simultaneously from both a semantic and a structural perspective. On the semantic side, extracted text is converted into embeddings, numerical representations of meaning, and similar ideas are grouped together even when they’re expressed in different words. On the structural side, pattern-recognition algorithms identify clause types, formatting conventions, table layouts, and hierarchical organization, understanding. For example, that a three-nested-level exhibit carries fundamentally different implications than a straightforward attached schedule.These two analyses don’t operate in isolation. Projection algorithms compare the semantic and structural clusters side by side, synthesizing them into a unified, enriched document model that captures both meaning and context. It’s this convergence that drives the 99% accuracy rate of Doczy.ai™. The system doesn’t just read the words, it understands the contract. Advanced large language models (LLMs) then generate structured output grounded in this dual-clustered intelligence.Before output is finalized, the system determines each document’s file class and generates prompts tailored to the extracted text, cluster classification, and domain context. Through few-shot and multi-shot prompting, the platform continuously edits the prompt on domain-specific examples and based on real outputs, creating a feedback loop that compounds accuracy improvements over time.

The resulting structured data flows into Snowflake, forming a centralized repository that powers intelligent dashboards with actionable insights and visualizations. Throughout the entire pipeline, Amazon CloudWatch monitors performance in real time and proactively surfaces issues before they escalate, while AWS Secrets Manager safeguards sensitive information, ensuring that security is not an afterthought, but a foundational layer woven into every stage of the system.

The transformative impact of Doczy.ai™

The results of this AI-powered approach are transformative and measurable. By automating contract interpretation and document processing, Doczy.ai™ has demonstrated significant impact at scale for multiple organizations across healthcare and financial services. The scale of operations over the last 22 months demonstrates the maturity and production readiness of Doczy.ai™. This solution has processed 2.5 million contract documents (50 million pages) with 137 million API calls to Amazon Bedrock and 442 billion tokens—a level of automation and accuracy previously unattainable through manual or traditional document processing approaches. Over this same period, Doczy.ai™ has helped clients achieve approximately 330 million dollars in cumulative direct and indirect savings.The 99% accuracy rate represents significant improvement over the approximately 55% accuracy of rules-based systems and far exceeds manual processing, which is typically affected by fatigue and human error. The 97% reduction in manual processing time translates directly to cost savings and enables organizations to reallocate human resources to higher-value activities that require judgment and strategic thinking.

A use case in action: Business process automation for health plans

For health plans, Doczy.ai™ provides a powerful solution to automate and improve contract management across the entire lifecycle. It ingests existing contracts in both paper and digital formats, integrates with contract management systems such as Coupa and Icertis, and processes new contracts and amendments as they’re executed. It then creates a centralized metadata repository that feeds directly into downstream systems, enabling end-to-end business process automation.This automation unlocks critical capabilities: Organizations can continuously analyze and improve contract terms, identifying opportunities to improve financial performance and operational efficiency. The architecture feeds accurate, up-to-date contract data directly into claims systems, automating the configuration process that previously required manual translation of reimbursement terms and removing manual data entry, configuration errors, and delays. Additionally, the platform helps maintain claim payment accuracy by assessing payments against contract terms, identifying discrepancies, and flagging potential overpayments or underpayments before they occur.By automating manual processes, health plans can adapt quickly to new contract terms and regulatory requirements. The intelligent dashboards and actionable insights provided by Doczy.ai™ enable decision-makers to understand contract performance, identify trends, and take proactive action to optimize financial outcomes.

Getting started with Doczy.ai™

Organizations interested in using Doczy.ai™ to transform document processing and contract management can engage with AArete to discuss their specific use cases and requirements. AArete offers the platform as a Software as a Service (SaaS) solution, enabling rapid deployment without significant infrastructure investment. AArete’s team of experts will configure this solution for your specific document types, domain terminology, and business processes, supporting maximum value from day one.

Conclusion

The challenge of unlocking data from unstructured documents is a major hurdle for many businesses, particularly in healthcare and financial services where contracts and agreements govern critical operational and financial relationships. By embracing intelligent document intelligence on AWS, organizations can solve this long-standing operational challenge and unlock a new frontier of strategic advantage, turning their data into their most valuable asset.

Built on a sophisticated architecture that orchestrates Amazon Cognito, Amazon S3, AWS Lambda, Amazon Textract, Amazon Elastic Container Service (Amazon ECS), Amazon Bedrock, Amazon CloudWatch, and AWS Secrets Manager, Doczy.ai™ demonstrates how modern cloud services can solve complex document-heavy business problems. Its advanced hybrid smart chunking, dual clustering, and prompt optimization techniques form the core of a patented contract intelligence engine.

Doczy.ai™ delivers tangible impact, processing up to 250,000 contract documents per week with 99% accuracy, reducing manual processing time by 97%, and helping clients unlock roughly 330 million dollars in cumulative savings over 22 months. By embracing this intelligent document processing, organizations can turn contracts into a strategic data asset, improving efficiency, accuracy, and profitability while freeing teams to focus on higher-value work.

To learn more about how AArete and Doczy.ai™ can help your organization transform document processing and unlock the value of your unstructured data, visit the AArete website.


About the authors

Scaling oncology patient support: How New York Cancer and Blood Specialists transformed customer experience with AWS and Pronetx, now part of Caylent

Post Syndicated from Muni T. Bondu original https://aws.amazon.com/blogs/architecture/scaling-oncology-patient-support-how-new-york-cancer-and-blood-specialists-transformed-customer-experience-with-aws-and-pronetx-now-part-of-caylent/

As one of the United States’ leading oncology and hematology providers, the goal of New York Cancer and Blood Specialists (NYCBS) is to provide comprehensive and compassionate care to patients. The organization handles more than 250,000 patient calls every year across over 100 specialized queues and wanted to optimize its manual call handling process.

This post details how NYCBS partnered with Amazon Web Services (AWS) and AWS partner Pronetx (now part of Caylent) to migrate to Amazon Connect Customer, the AWS cloud contact center service. The migration delivered a 54 percent improvement in patient enrollment and transformed the way NYCBS connects with the patients who need them most.

Solution overview

NYCBS chose to migrate to a dedicated Amazon Connect Customer instance with the help of Pronetx, an AWS partner specializing in Amazon Connect Customer. The 13-week engagement covered three phases: discovery and foundation (2 weeks), build and implementation (8 weeks), and acceptance testing and go-live (3 weeks).

This solution uses Amazon Connect Customer as the core contact center service. At the heart of the customer experience infrastructure lies the call routing and patient communication logic. Within this infrastructure, several key components work together to deliver HIPAA-compliant patient care.

Architecture

The architecture has three main layers, as shown in the following diagram. Each layer is designed to handle a specific aspect of the customer experience operations of NYCBS while maintaining HIPAA compliance throughout.

Architecture diagram showing the three-layer NYCBS contact center solution built on Amazon Connect. The diagram illustrates: (1) CTR management microservice with Amazon API Gateway, AWS Lambda, and Amazon DynamoDB; (2) core contact center services with Amazon Connect instance, AWS Secrets Manager, Amazon Polly, and IVR Lambda functions; (3) call recording and AI/ML pipeline with Amazon S3, Amazon Transcribe, and Amazon Lex. Shared AWS services include AWS CloudFormation, AWS IAM, AWS KMS, Amazon CloudWatch, and Amazon SNS. External integrations include Microsoft Intune ID and NYCBS on-premises systems.

Figure 1: NYCBS customer experience solution architecture on Amazon Connect, showing the CTR management microservice, core contact center services, and call recording and AI/ML pipeline layers, integrated with shared AWS services and external systems.

CTR management microservice

This layer handles contact trace record (CTR) processing through Amazon API Gateway as the entry point. The microservice uses AWS Lambda functions (Get Disposition Code and Update CTRs) to retrieve and process call disposition codes, with Amazon DynamoDB storing disposition code data for quick retrieval.

Core contact center services

The central Amazon Connect Customer instance manages incoming agent flows and call routing across more than 100 specialized queues. AWS Secrets Manager securely stores credentials and sensitive configuration. Amazon Polly provides text-to-speech capabilities for automated voice responses. Configuration is managed through DynamoDB configuration tables and configuration Lambda functions, with IVR Integration Lambda functions handling IVR workflows.

Call recording and AI/ML pipeline

Call recordings and artifacts are stored in Amazon Simple Storage Service (Amazon S3). A dedicated Lambda function handles voicemail processing, with Amazon Transcribe converting voicemails to text. A subsequent Lambda function then creates cases from transcribed voicemails. AI capabilities are powered by Amazon Lex for conversational chatbots and other AWS AI services for intelligent automation, with OMS Lookup providing Order Management System integration.

Shared services and integrations

The architecture integrates with external systems including Microsoft Intune ID for identity management and the on-premises systems of NYCBS. Shared AWS services provide the foundation:

Together, these components helped NYCBS reduce patient enrollment time by 54 percent. The reduction came through automated multi-language routing in English, Spanish, Russian, and Mandarin. NYCBS also added specialty-based queue prioritization for urgent cases and real-time agent monitoring across the more than 250,000 annual patient calls. The implementation maintains HIPAA compliance through role-based access controls, encryption with AWS KMS, and secure credential storage.

Technical highlights

A key engineering decision was moving from a shared multi-tenant environment to a dedicated Amazon Connect Customer instance. This architectural shift helped NYCBS implement several capabilities that had previously been unavailable:

  • Multi-language routing (English, Spanish, Russian, and Mandarin) through language-detection flows.
  • Specialty-based queue prioritization for urgent oncology cases.
  • After-hours coverage logic with automated callback options.
  • HIPAA-compliant call recording with role-based access controls.
  • Real-time agent monitoring and performance dashboards.

Key takeaways

A dedicated Amazon Connect Customer instance gives healthcare organizations the flexibility, security, and native feature access that multi-tenant environments can’t match, including the following:

  • Multi-language, specialty-based routing (English, Spanish, Russian, Mandarin) directly improved patient access to the right care team.
  • Migrating from manual workflows to automated IaC-driven continuous integration and continuous delivery (CI/CD) deployment removed third-party management fees and reduced operational costs.
  • HIPAA compliance was maintained throughout using AWS KMS encryption, role-based access controls, and AWS Secrets Manager for credential storage.
  • A 54 percent improvement in patient enrollment shows that modernizing contact center operations delivers measurable clinical and business outcomes.

Results

Within the 13-week implementation window, NYCBS achieved a 54 percent improvement in patient enrollment, which reflects not only operational efficiency, but also improved access to care. The organization removed third-party management fees through direct AWS consumption, reducing operational costs. NYCBS gained real-time visibility into call quality through Operata and the conversational analytics capabilities of Amazon Connect Customer. In addition, the team established a rapid CI/CD pipeline that deploys new features safely and quickly without disrupting patient services.

Conclusion

If your healthcare organization manages complex, high-volume contact center operations, you can apply what NYCBS achieved. A dedicated Amazon Connect Customer instance provides the flexibility, security, and native feature access that healthcare-specific workflows demand. We recommend the following next steps to get started:

Learn more

AWS Partner spotlight

Pronetx, now part of Caylent, an AWS Premier Tier Services Partner, specializes in Amazon Connect. Pronetx helps organizations design, migrate, and operate customer engagement systems on AWS, with a focus on resilience and AI-driven customer experience.

Contact Pronetx | Partner Overview | AWS Marketplace

Multi-Region event-driven failover architecture with Amazon EventBridge and Route 53

Post Syndicated from Napoleone Capasso original https://aws.amazon.com/blogs/compute/multi-region-event-driven-failover-architecture-with-amazon-eventbridge-and-route-53/

Multi-Region Event-Driven Failover Architecture with Amazon EventBridge and Route 53

Event-driven architectures enable applications to respond to events in real-time, providing scalability and loose coupling between components. However, ensuring high availability across multiple AWS regions requires careful design of failover mechanisms. This post demonstrates how to build a resilient multi-region event-driven architecture using Amazon EventBridge, Amazon API Gateway, and Amazon Route 53 health-based failover.

Overview

Organizations building event-driven applications need to achieve high availability and disaster recovery capabilities. This architecture provides automatic failover between AWS regions while maintaining regional independence for event processing. The solution uses Amazon Route 53 health checks to monitor regional Amazon API Gateway endpoints and automatically routes traffic to healthy regions without manual intervention.

The architecture delivers several key benefits. Regional independence reduces latency by processing events in the same region where they originate. Amazon DynamoDB global tables provide automatic data replication across regions, ensuring data availability during regional failures. The solution provides robust failover capabilities while maintaining architectural simplicity.

Organizations with strict availability requirements can find this solution particularly valuable. All event processing remains within AWS regions, and failover occurs automatically based on health check results. The architecture supports both planned maintenance windows and unplanned regional outages, providing flexibility for operational needs.

Solution overview

The solution implements an active-passive multi-region architecture where events flow through Amazon API Gateway to regional Amazon EventBridge buses. Amazon Route 53 health checks monitor the primary region and automatically route traffic to the secondary region during failures. Each region processes events independently, while Amazon DynamoDB Global Tables replicate data across regions.

The following diagram provides an overview of the solution:

The above diagram depicts the multi-region architecture running across two AWS regions. The Route 53 DNS service serves as the main entry point for the application, with health checks monitoring both regions. Each region contains an identical stack with Amazon API Gateway, Amazon EventBridge, Amazon SQS, and AWS Lambda. The Amazon DynamoDB Global Table replicates data between regions automatically.

Solution deployment

To deploy this solution, follow the instructions in the GitHub repository and clone the repository. The solution deploys in two AWS regions. Ensure valid SSL certificates exist in AWS Certificate Manager (ACM) in both regions for the custom domain.

Prerequisites

For this walkthrough, the following resources are needed:

  • AWS Account: An AWS account with permissions to create and manage Amazon API Gateway, Amazon EventBridge, Amazon SQS, AWS Lambda, Amazon DynamoDB, Amazon Route 53, AWS IAM, and AWS CloudFormation resources
  • AWS Serverless Application Model (SAM): The AWS SAM CLI installed, as the templates use the SAM transform for Lambda and API Gateway resource definitions
  • Domain Name: A registered domain with a Route 53 hosted zone- SSL Certificates: ACM certificates for the custom domain in both deployment regions
  • AWS CLI: The AWS CLI installed and configured with credentials for the target AWS account
  • Region Selection: Two AWS regions for deployment

Walkthrough

The AWS CloudFormation templates from the sample GitHub repository create a secure, multi-region architecture that provides automatic failover for event-driven applications. The templates provision regional API Gateway endpoints, EventBridge buses, SQS queues, Lambda functions, and an Amazon DynamoDB Global Table. The solution establishes health monitoring through Route 53 health checks and configures DNS failover routing. The templates use AWS Serverless Application Model (SAM) transform to simplify Lambda and API Gateway resource definitions.

Step 1: Deploy the primary stack

The primary stack creates the foundational resources in the primary region. This includes the Amazon EventBridge bus, Amazon API Gateway with custom domain, health check, AWS Lambda function, Amazon SQS queue, and Amazon DynamoDB Global Table. The stack creates an EventBridge bus that receives events from API Gateway:

EventBus: 
Type: AWS::Events::EventBus 
Properties: 
Name: !Ref EventBusName

The API Gateway uses AWS service integration to forward events directly to EventBridge:

x-amazon-apigateway-integration: 
type: "aws" 
uri: !Sub "arn:aws:apigateway:${AWS::Region}:events:path//" 
credentials: !GetAtt ApiGatewayEventBridgeRole.Arn 
httpMethod: "POST"

The health check monitors the API Gateway endpoint to determine regional availability:

DomainHealthCheck: 
Type: AWS::Route53::HealthCheck 
Properties: 
HealthCheckConfig: 
Type: HTTPS 
ResourcePath: /Prod/health FullyQualified
DomainName: !Sub ${Api}.execute-api.${AWS::Region}.amazonaws.com 
Port: 443 
RequestInterval: 30 
FailureThreshold: 3

The Route 53 DNS record configures failover routing with the PRIMARY designation:

ApiDnsRecord:
Type: AWS::Route53::RecordSet
Properties:
HostedZoneId: !Ref HostedZoneId
Name: !Ref CustomDomainName
Type: A
SetIdentifier: primary-region
Failover: PRIMARY
HealthCheckId: !Ref DomainHealthCheck

The DynamoDB Global Table creates replicas in both regions:

DataTable: 
Type: AWS::DynamoDB::GlobalTable 
Properties: 
BillingMode: PAY_PER_REQUEST 
Replicas: 
- Region: !Ref AWS::Region 
- Region: !Ref SecondaryRegion

Note the `DataTableName` output value for use in the secondary stack deployment. The `CustomDomainURL` output provides the endpoint to invoke the solution.

Step 2: Deploy the secondary stack

The secondary stack creates identical resources in the secondary region , except for the Amazon DynamoDB table which references the existing Global Table. The secondary stack creates its own Amazon EventBridge bus, Amazon API Gateway, health check, AWS Lambda function, and Amazon SQS queue. The Route 53 DNS record uses the SECONDARY designation

Step 3: Event processing flow

Events flow through the processing pipeline in each region. API Gateway receives events and forwards them to EventBridge using the PutEvents API. EventBridge evaluates event rules and routes matching events to SQS queues. Lambda functions poll the SQS queues and process events in batches. AWS Lambda writes processed data to the DynamoDB Global Table, which replicates across regions.

The Lambda function processes events from the queue and writes to DynamoDB:

def handler(event, context): 
for record in event.get('Records', []): 
body = json.loads(record['body']) 
detail = body.get('detail', {}) 
event_id = body.get('id', '') 
item = { 'id': event_id, 'detail': detail, 'timestamp': datetime.utcnow().isoformat() } 
table.put_item(Item=item)

Testing

Fetch the custom domain URL and test it by sending an event:

curl -X POST https://api.example.com \-H "Content-Type: application/json" \ -d '{ "Detail": { "IsHelloWorldExample": "true" }, "DetailType": "POSTED", "Source": "demo.event" }' -v

The response includes an `X-Region` header indicating which region processed the request. Under normal conditions, this shows the primary region.

To test failover:

  1. Remove the base path mapping for the primary region:
aws apigateway delete-base-path-mapping \ --domain-name api.example.com \ --base-path '(none)' \ --region {primary-region}
  1. Delete the primary API Gateway stage:

aws apigateway delete-stage \ --rest-api-id <primary-api-id> \ --stage-name Prod \ --region {primary-region}

  1. Wait 2-3 minutes for the health check to fail. The Route 53 health check performs checks every 30 seconds with a failure threshold of 3, requiring 90 seconds to detect the failure.
  2. Send another request to the API endpoint:
curl -X POST https://api.example.com \-H "Content-Type: application/json" \ -d '{ "Detail": { "IsHelloWorldExample": "true" }, "DetailType": "POSTED", "Source": "demo.event" }' -v
  1. Verify the failover: The `X-Region` header now shows the secondary region, confirming successful failover.

Verify event processing in the secondary region:

  1. Check the Lambda logs for successful processing:

aws logs tail /aws/lambda/<secondary-lambda-name> --region {secondary region}

You should see log entries similar to:

Processing message: 
{"version":"0",
"id":"abc12345-...",
"source":"demo.event",
"detail-type":"POSTED",...} 
Event Source: demo.event
Detail Type: POSTED
Successfully wrote item to DynamoDB: abc12345-... 
Successfully read item from DynamoDB: 
{'id': 'abc12345-...', 
'source': 'demo.event', 
'detailType': 'POSTED', 
'detail': 
{'data': {'IsHelloWorldExample': 'true'}, 
...}, 
'timestamp': '2025-01-15T18:30:00.000000', 
'processed': True}
  1. Verify the data in Amazon DynamoDB:

aws dynamodb scan \ --table-name <table-name> \ --region {secondary region}```

The scan results should include items with the event details:

{ "Items": 
[ { "id": {"S": "abc12345-..."}, 
"source": {"S": "demo.event"}, 
"detailType": {"S": "POSTED"},
"detail": 
{"M": {"data": 
{"M": 
{"IsHelloWorldExample": 
{"S": "true"}}}}}, 
"timestamp": {"S": "2025-01-15T18:30:00.000000"},
"processed": {"BOOL": true} } ], 
"Count": 1 }
  1. Restore the primary region – recreate the stage:

aws apigateway create-stage \ --rest-api-id <primary-api-id> \ --stage-name Prod \ --deployment-id <deployment-id> \ --region {primary region}

  1. Restore the primary region – recreate the base path mapping:

aws apigateway create-base-path-mapping \ --domain-name api.example.com \ --rest-api-id <primary-api-id> \ --stage Prod \ --region {primary region}

You can find the “deployment-id” by running: aws apigateway get-deployments \ --rest-api-id <primary-api-id> \ --region {primary region}

After 2-3 minutes, the health check passes and Route 53 routes traffic back to the primary region.

Cleanup

To remove the solution and avoid ongoing charges, delete the CloudFormation stacks in the correct order. Delete the secondary stack first, then the primary stack. This order is important because the Amazon DynamoDB Global Table is owned by the primary stack. Warning: Deleting these stacks permanently removes all resources including the Amazon DynamoDB global table and any event data stored in it. Back up any data you need before proceeding. This action cannot be undone. The following resources incur costs while deployed:

  • Amazon API Gateway (REST API)
  • Amazon Route 53 health checks and DNS records
  • Amazon DynamoDB global table (with cross-region replication)
  • AWS Lambda function invocations and duration
  • Amazon SQS queue operations
  • Amazon CloudWatch Logs storage

Delete the secondary stack:

aws cloudformation delete-stack --stack-name secondary-stack --region {secondary region}

Wait for the secondary stack deletion to complete:

aws cloudformation wait stack-delete-complete --stack-name secondary-stack --region {secondary region}

Delete the primary stack:

aws cloudformation delete-stack --stack-name primary-stack --region {primary region}

Wait for the primary stack deletion to complete:

aws cloudformation wait stack-delete-complete --stack-name primary-stack --region {primary region}

This removes all resources including the Amazon EventBridge buses, Amazon API Gateways, AWS Lambda functions, Amazon SQS queues, Amazon DynamoDB Global Table, Amazon Route 53 health checks, DNS records and IAM roles.

Conclusion

This post demonstrates how to establish a resilient multi-region architecture for event-driven applications using Amazon EventBridge, Amazon API Gateway, and Amazon Route 53. The solution uses Route 53 health-based failover, a powerful capability that automatically routes traffic to healthy regions based on health check results. This architecture significantly enhances application availability by providing automatic failover during regional outages while maintaining regional independence for event processing.

How Buildkite Operates Test Analytics at Massive Scale with Amazon MSK and Amazon Managed Service for Apache Flink

Post Syndicated from James Hill original https://aws.amazon.com/blogs/big-data/how-buildkite-operates-test-analytics-at-massive-scale-with-amazon-msk-and-amazon-managed-service-for-apache-flink/

When engineering teams at Slack, Reddit, Canva, Airbnb, Shopify, and Uber need to ship code with confidence, they rely on Buildkite. As a CI/CD platform, Buildkite orchestrates complex build, test, and deployment pipelines for some of the most demanding engineering organizations in the world. It handles everything from routine code commits to artificial intelligence (AI) model-training workloads, processing over 50 billion requests per month.

At the heart of Buildkite’s test orchestration portfolio is Test Engine, a specialized analytics product designed to help engineering teams understand and optimize their test suites at scale. Test Engine aggregates results across thousands of builds, flags flaky tests, runs parallel test execution across machine fleets, and delivers interactive analytics on test execution data. It supports arbitrary metadata tagging for dimensions like instance type, architecture, language version, cloud provider, and feature flags.

The challenge? Delivering all of this in real time, across multiple enterprise tenants, at a volume that would stress even the most robust data infrastructure. In this post, we explore how Buildkite uses Amazon Managed Streaming for Apache Kafka (Amazon MSK) and Amazon Managed Service for Apache Flink to power Test Engine’s streaming-first analytics architecture at scale.

The problem: When scale breaks traditional architectures

Buildkite’s Test Engine must ingest and serve analytics on test telemetry from thousands of distributed pipelines simultaneously, for multiple enterprise customers. The scale is unforgiving: 50 billion test executions per month, 500K events per second at peak ingestion, and webhook payloads reaching 21 MB.

The architectural evolution and its limits

The original Rails and PostgreSQL stack couldn’t sustain this growth. In 2024, the team re-architected around a distributed streaming layer, a stateful stream processor for pre-aggregations, and multiple specialized stores: a key-value store for fast lookups, a relational database for pre-computed aggregates, and an open table format (Iceberg) with a distributed query engine (Trino) for flexible querying.

Yet the core tension remained unsolved. Enterprise customers demanded interactive, arbitrary slicing of billions of records across high-cardinality dimensions, not canned reports. The stream processor couldn’t handle ad hoc aggregations at query time. The key-value store was blind to analytical queries. The distributed query engine offered flexibility but was too slow for interactive use.

The result was a system that was expensive and operationally complex. It included nine relational database clusters, sprawling ETL pipelines, and 24/7 pre-aggregation jobs running regardless of demand. It still couldn’t deliver the one thing customers needed most: fast, flexible, interactive analytics at scale.

Architecture and implementation: MSK and Amazon Managed Service for Apache Flink as the streaming backbone

The solution Buildkite arrived at centers on Amazon Managed Streaming for Apache Kafka (Amazon MSK) and Amazon Managed Service for Apache Flink as the real-time data streaming and processing layers, decoupling high-throughput ingestion from downstream analytics.

The data pipeline

The following diagram shows the end-to-end data flow from CI/CD agents through Amazon MSK and Amazon Managed Service for Apache Flink to the analytics layer.

Architecture diagram showing the Buildkite data pipeline from CI/CD agents through Amazon MSK and Amazon Managed Service for Apache Flink to ClickHouse analytics

Amazon MSK sits at the critical junction between data producers (the distributed CI/CD agents and test collectors running across customer infrastructure) and the downstream processing and analytics layers. Amazon Managed Service for Apache Flink then transforms those raw event streams into enriched, queryable data before it reaches the analytics store.

High-throughput ingestion from CI/CD pipelines

Amazon MSK’s role begins at ingestion. Test collectors embedded in CI/CD pipelines publish test execution events directly to Kafka topics. The existing Amazon MSK cluster handles between 5 MB/sec and 100 MB/sec of inbound data under normal operating conditions. The architecture is designed to absorb the significant variance inherent in CI/CD workloads, where pipeline activity is bursty and correlated with engineering team working hours across global time zones.

When the Buildkite project was initiated, MSK Express Brokers were not yet available, leading the team to adopt MSK Tiered Storage as the primary mechanism for scaling and recovery. With MSK Express Brokers now generally available, the team is evaluating a migration of its most critical log ingestion workload, which sustains up to 1 GB/s at peak ingestion. MSK Express Brokers bring automatic storage scaling with zero storage management overhead, up to 20x faster scaling and 90% faster broker recovery, 3x higher per-broker throughput, 5x more partitions per broker, and built-in Intelligent Rebalancing.

Real-time stream processing with Amazon Managed Service for Apache Flink

Sitting between Amazon MSK and the analytics layer, Amazon Managed Service for Apache Flink acts as the stateful stream processing engine that transforms raw event streams before they reach downstream systems. Buildkite selected Flink for its exactly-once processing, mature stateful computation model, and deep Kafka integration. Handling sustained peaks of over 25,000 events per second, Amazon Managed Service for Apache Flink eliminates the operational overhead of cluster provisioning, version upgrades, checkpointing, and job recovery. This frees engineering teams to focus on application logic.

Amazon Managed Service for Apache Flink powers key stateful processing tasks, including flaky test detection through time-windowed pattern matching, enriching execution events with pipeline and customer metadata, and routing processed data to downstream systems such as ClickHouse for analytics, PostgreSQL for operational workloads, and Amazon Simple Storage Service (Amazon S3) for long-term archival.

Reliability and fault tolerance

Amazon MSK’s three-replica configuration ensures that no single broker failure can cause data loss or ingestion interruption. Combined with flexible data retention, the architecture provides a meaningful replay window. If a downstream consumer (Amazon Managed Service for Apache Flink, ClickHouse, or another service) experiences an outage, it can resume processing from its last committed offset without data loss.

During the migration to the current architecture, Buildkite employed a dual-write strategy: simultaneously writing to both the existing PostgreSQL pipeline and the new Amazon MSK/ClickHouse path. This approach allowed the team to validate data consistency and gradually shift traffic without risking customer-facing disruption. This pattern speaks to the operational maturity Amazon MSK provides.

Operational efficiency gains

The shift to a streaming-first architecture, combined with the downstream simplification of the analytics engine, produced significant operational improvements:

  • Flink workloads reduced by 60%+: Eliminating pre-aggregation jobs that ran continuously regardless of demand.
  • Key/value store completely retired: Amazon MSK’s buffering capability, combined with ClickHouse’s query performance, eliminated the need for a separate fast-lookup store.
  • PostgreSQL capacity cut in half: Nine separate database clusters consolidated and right-sized.
  • Thousands of lines of application code deleted: Simpler architecture means less ETL code, fewer failure modes, and faster onboarding for new engineers.

Platform performance at a glance

Metric Value
Monthly test executions (for test engine platform) 50 billion (4x growth from 3B)
Sustained peak ingestion 500K events/second
Total records in analytics store 200 billion
Log ingestion requests 70,000+ per second
Peak webhook throughput 1.7 GB/second
MSK inbound throughput range 5 MB/sec – 100 MB/sec

Business and developer impact

The technical architecture ultimately exists to serve one purpose: helping developers ship better software faster. The streaming-first architecture built on Amazon MSK and Amazon Managed Service for Apache Flink delivers on that promise across four dimensions.

On-demand analytics replaced pre-computed reports. Customers can now interactively slice and dice 70 billion records across arbitrary metadata dimensions. They get answers to queries like “Show me P50 test durations by instance type and architecture for the last 30 days” in seconds, not hours. Real-time log streaming through the “live tail” feature means developers no longer wait for a build to complete before diagnosing failures. At 25,000 events per second, this experience scales across thousands of concurrent enterprise pipelines without degradation.

Smarter test intelligence comes from Amazon Managed Service for Apache Flink’s stateful flaky test detection: when a test begins exhibiting intermittent failure patterns, Amazon Managed Service for Apache Flink identifies it as it happens, not after the fact. This is what separates a proactive analytics platform from a reactive one. It requires publishing data to Kafka, processing with Flink, and letting ClickHouse handle the complex read requests.

Conclusion: Streaming as a strategic foundation

Buildkite’s journey from a Rails/Postgres monolith to a streaming-first analytics platform reflects a pattern increasingly common among enterprise SaaS companies: a reliable, high-throughput streaming and processing layer is not an optimization. It is a prerequisite for operating at scale.

Amazon MSK and Amazon Managed Service for Apache Flink form the backbone that helps Buildkite ingest 50 billion test executions per month, serve real-time interactive analytics to enterprise customers, and do so at lower cost than the more complex architecture it replaced. Amazon MSK handles durable, elastic event buffering. Amazon Managed Service for Apache Flink transforms raw streams into enriched, queryable data. Together they absorb the operational complexity that would otherwise consume engineering capacity.

For platform engineers evaluating streaming infrastructure for multi-tenant SaaS workloads, the signal is clear: invest in the streaming backbone early, and let managed services handle the operational complexity.

To learn more about Amazon MSK and Amazon Managed Service for Apache Flink, visit aws.amazon.com/msk and aws.amazon.com/managed-service-apache-flink.


About the authors

James Hill

James Hill

James has been building and scaling software systems for more than 25 years, from early web applications to platforms that now process millions of builds every day. Starting his career as a software engineer, James has led teams across Australia, the UK, and globally, solving problems in performance, reliability, and delivery speed at massive scale. Today, he works with some of the world’s largest engineering organizations to help them ship faster and with greater confidence, drawing on deep, hands-on experience in both engineering and product leadership. James is passionate about turning testing from a bottleneck into a feedback engine that accelerates learning across an organization.

Mitch James

Mitch James

Mitch is a Brand and Marketing Strategist with deep expertise crafting end-to-end brand experiences and fostering engaged communities around technical tooling. He brings 15+ years of Brand, Design, and Marketing leadership across devtools, consumer product, and B2B enterprise. Previously, Mitch has built and led creative teams at Adobe, IBM, Salesforce, George P Johnson, Wunderman Thompson, and VML. Today, he leads global marketing and design for Buildkite, working with engineering teams who set the pace at the frontier of software delivery.

Masudur Rahaman Sayem

Masudur Rahaman Sayem

Masudur is a Streaming Data Architect at AWS with over 25 years of experience in the IT industry. He collaborates with AWS customers worldwide to architect and implement data streaming solutions that address complex business challenges. As an expert in distributed computing, Sayem specializes in designing large-scale distributed systems architecture for maximum performance and scalability. He has a keen interest and passion for distributed architecture, which he applies to designing production-ready solutions at internet scale.

Miranda Li

Miranda Li

Miranda is a Senior Solutions Architect at AWS, specializing in Independent Software Vendor (ISV) and cloud-native architectures. With four years dedicated to helping software partners innovate and scale on AWS, she focuses on helping ISVs build and optimize their solutions for the cloud. She brings deep technical expertise in cloud infrastructure and data analytics, with a strong focus on supporting technical customers in areas such as Infrastructure as a Service (IaaS), network architecture, and security. Outside of work, she is an avid badminton player and enjoys staying active through jogging and outdoor adventures.

How Zynga scaled multi-warehouse data governance with Amazon Redshift federated permissions

Post Syndicated from Johan Eklund, Matthew Wongkee, Noelia Tardón original https://aws.amazon.com/blogs/big-data/how-zynga-scaled-multi-warehouse-data-governance-with-amazon-redshift-federated-permissions/

Zynga, a global leader in interactive entertainment operates a portfolio of mobile game studios including Socialpoint, the creators of Dragon City and Monster Legends. Zynga’s analytics platform processes telemetry and revenue data across studios using Amazon Redshift as its central data warehouse.

As Zynga expanded its analytics architecture to include individual studios with their own compute environments, the team faced a challenge: how to maintain centralized data governance while granting studios independent query capacity. Their existing approach to permission management introduced lag and required custom infrastructure if scaled to multiple warehouses.

In this post, we walk through how Zynga adopted Amazon Redshift federated permissions and AWS IAM Identity Center to enforce consistent, tiered data access across provisioned and serverless Amazon Redshift environments without building custom synchronization pipelines.

The challenge

Zynga needed to onboard Socialpoint’s current Amazon Redshift workloads and make Zynga’s central data available to them. Zynga’s existing production cluster would house the Socialpoint raw data, but the compute would come from another warehouse set up as a consumer. At the same time, Zynga’s data access control policies would need to be enforced across all warehouses. Zynga uses a tiered access control policy which would need to be synced across all consumers with no permission lag or manual grant synchronization.

During the migration, Socialpoint’s specific extract, transform, and load (ETL) processes would be included in Zynga’s central ETLs and their data ingestion pipeline would be replaced by Zynga’s latest generation of data ingestion infrastructure. Because the migration process happens in stages, Amazon Redshift sizing would also gradually need to increase.

The team evaluated two alternatives before arriving at a solution:

  • AWS Lake Formation couldn’t manage local and cross-cluster permissions using the same interface, and required AWS Access and Identity Management (IAM) or IAM Identity Center authentication for all users including service accounts.
  • Manual grants on consumer clusters introduced a delay between when permissions were updated on the producer and when they took effect on the consumer. This approach would also require an external job that synced permissions and would be unlikely to scale well beyond 2–3 consumers.

Solution overview

Zynga implemented a solution using three AWS services working together:

  1. Amazon Redshift federated permissions enabled cross-cluster queries without explicit data shares. Permissions granted on the producer cluster propagate immediately to consumer workgroups through AWS Glue Data Catalog registration.
  2. AWS IAM Identity Center provides unified authentication through federation with Okta. When users sign in, their Okta group memberships are provisioned through a System for Cross-domain Identity Management (SCIM) and automatically map to Amazon Redshift roles, removing the need for external synchronization jobs.
  3. Amazon Redshift Serverless provides the compute layer for Socialpoint, scaling to zero when idle and avoiding the need to pre-size a provisioned cluster during the migration period.

The architecture uses a dual-grant approach where every permission is granted to both an IAM Identity Center group (for users) and a federated IAM role (for service accounts). This gives both authentication paths the same access.

How it works

Authentication with IAM Identity Center

Zynga’s existing Okta directory syncs to IAM Identity Center, which is connected to the Amazon Redshift Serverless workgroup. When a user authenticates, Amazon Redshift automatically creates a user mapped to their email address and assigns them to roles based on their Okta group membership.

For example, an analyst in the Gamma Tier group signs in and is automatically assigned the AWSIDC:role.sso.gamma role in Amazon Redshift. No manual role assignment or synchronization job is required.

Service accounts, used for programmatic access, authenticate differently. Either using their IAM role and calling the get-credentials API, or by using the new federated permissions feature. Each service account assumes a federated IAM role, which creates a corresponding federated user in Amazon Redshift (for example, IAMR:role_iam_gamma).

The identity layer shows two authentication paths. Interactive users authenticate through Okta, which syncs group memberships to AWS IAM Identity Center. On login, IAM Identity Center automatically maps users to tiered Amazon Redshift roles (AWSIDC:role.sso.gamma, beta, alpha). Service accounts take a separate path, assuming a cluster IAM role that creates a corresponding federated user (IAMR:role_iam_gamma) in Amazon Redshift.

Figure 1 – Identity layer

The dual-grant approach

To ensure that both users and service accounts can access the same data, every read permission is granted to both the IAM Identity Center group and the federated IAM role in a single statement:

GRANT SELECT ON schema.table TO
	'IAMR:role_iam_gamma',
	ROLE 'AWSIDC:role.sso.gamma';

Transitioning the producer cluster

The shared provisioned cluster already had active users with local grants. To avoid disruption, Zynga implemented a tri-grant approach on the producer during the transition period. Existing stored procedures were modified to grant permissions to three targets: the legacy local role, the IAM Identity Center group, and the federated IAM role.

GRANT SELECT ON schema.table TO
	ROLE role_rs_gamma,
	ROLE 'AWSIDC:role.sso.gamma','IAMR:role_iam_gamma';

This approach maintains backward compatibility for existing users on the producer while enabling immediate access from the new serverless workgroup. The long-term plan includes migration of all producer users to IAM Identity Center and retire the legacy local grants.

Stored procedures for consistent governance

Rather than requiring users to construct dual-grant statements manually, Zynga created stored procedures that encapsulate the grant logic:

  • grant_read accepts a table name and access tier, then issues the appropriate dual-grant for both the IAM Identity Center group and the federated IAM role.
  • grant_write grants data definition language (DDL) and data manipulation language (DML) permissions to the appropriate team-based role.

This provides a consistent interface for permission management regardless of which cluster or workgroup that the user is on.

Architecture diagram in the AWS Cloud showing a federated Amazon Redshift setup. On the left, a Zynga provisioned Amazon Redshift cluster (producer) holds Zynga central data — game telemetry and revenue data — and stored procedures grant_read(table, tier) and grant_write(table, team). On the right, a Socialpoint Amazon Redshift Serverless workgroup (consumer) holds Socialpoint ETL data and the same two stored procedures. Bidirectional arrows between the two environments label the top connection as "Federated queries (cross-cluster reads)" between the data stores, and the bottom connection as "Federated permissions" between the stored procedures.

Figure 2 – Data and compute layer

Results

The migration delivered measurable improvements:

  • Immediate permission propagation – Grants on the producer cluster took effect on the consumer workgroup instantly, replacing a process that previously required manual intervention and introduced lag.
  • Zero additional infrastructure cost – Federated permissions, federated queries, and IAM Identity Center added no incremental costs to the architecture.
  • Removed custom synchronization – The team removed the need for Lake Formation configurations, external AWS Lambda functions, and Airflow workflows for permission management.
  • Scalable pattern – The same architecture can be extended to additional studio workgroups without duplicating permission management logic.

Lessons learned

Zynga adopted federated permissions shortly after the feature launched in US West Oregon (us-west-2) in January 2026. The team shared several observations from their early adoption:

  • Run a proof of concept first. The team validated the full permission model in a test environment before deploying to production, including testing that existing data shares to other clusters were not disrupted.
  • Plan for the dual-grant requirement. Because IAM Identity Center users and federated IAM roles are distinct identity types, every read permission requires two grants. Encapsulating this in stored procedures prevents errors and reduces cognitive overhead.
  • Start with serverless for new workloads. With Amazon Redshift Serverless, the team can avoid sizing decisions during the migration period. If usage patterns later justify it, they can migrate to a provisioned cluster from a serverless snapshot with minimal downtime.
  • Engage with AWS. As an early adopter of a new feature, Zynga maintained regular contact with the Amazon Redshift team through their AWS Technical Account Manager to report issues and request enhancements.

Conclusion

Zynga’s adoption of Amazon Redshift federated permissions demonstrates how organizations with multi-cluster Amazon Redshift architectures can enforce centralized data governance without building custom synchronization infrastructure. By combining federated permissions with IAM Identity Center and Amazon Redshift Serverless, the team established a pattern that scales to additional studios while maintaining consistent access controls and reducing operational overhead.

To learn more about the services used in this post, see the following resources:


About the authors

Johan Eklund

Johan Eklund is a Principal Software Engineer on the Data Engineering team at Zynga, focused on building and operating large-scale data platforms on AWS. Day-to-day, his work spans Redshift architecture, ETL orchestration, platform operations, and security and access control. Outside of work, Johan enjoys exploring art and music through creative coding and DIY hardware synthesizers.

Matthew Wongkee

Matthew Wongkee is a Principal Software Engineer at Zynga, where he builds scalable and high-performance data platforms and infrastructure, including Amazon Redshift. He focuses on enabling teams to leverage their data through self-service capabilities, including data discovery, query optimization, curated datasets, and data quality. In his spare time, he practices short track speed skating.

Noelia Tardón

Noelia Tardón is a Principal Software Engineer at Zynga. Based in Barcelona, she works on large-scale cloud data platforms with a focus on performance optimization and cost efficiency. She designs and builds data lake and data warehouse solutions while ensuring compliance with data protection regulations. Outside of work, she enjoys spending time with her family and running.

Sandeep Adwankar

Sandeep is a Senior Technical Product Manager at AWS. Based in the California Bay Area, he works with customers around the globe to translate business and technical requirements into products that enable customers to improve how they manage, secure, and access data.

Satesh Sonti

Satesh is a Principal Analytics Specialist Solutions Architect based out of Atlanta, specializing in building enterprise data platforms, data warehousing, and analytics solutions. He has over 19 years of experience in building data assets and leading complex data platform programs for banking and insurance clients across the globe.

Photo of AUthor - Steve Phillips

Steve Phillips

Steve is a Principal Technical Account Manager and Analytics specialist at AWS in the North America region. Steve currently focuses on data warehouse architectural design, data lakes, data ingestion pipelines, and cloud distributed architectures.

How Amazon is moving to integrate catalogs to improve data discovery with Amazon SageMaker

Post Syndicated from Pradeep Misra original https://aws.amazon.com/blogs/big-data/how-amazon-is-moving-to-integrate-catalogs-to-improve-data-discovery-with-amazon-sagemaker/

Enterprises face challenges when teams create data assets outside of central data catalogs. It adds overhead for discovery, and limits collaboration. Amazon’s Business Data Technologies (BDT) team has built an enterprise data catalog (Andes) for sharing datasets under well-defined policies. However, teams created catalog of local datasets and other non-tabular assets such as dashboards and metrics, outside Andes. This made it difficult to discover all assets in a consolidated way.

In this post, we share how Amazon.com is working to integrate catalogs by extending enterprise data catalog Andes with Amazon SageMaker.

Need for expanding catalog and governance from datasets to data assets

Without a single solution, users had to search multiple catalogs depending upon the asset type. Teams spent considerable time indexing the different catalogs and identifying the right one for their task. This slowed them down and took time away from solving the business problems.

To address these challenges, BDT team identified four critical capabilities needed:

  1. Multimodal catalog – Data consumers required the ability to blend enterprise data with local datasets and use them together for specific use cases. Teams sought to discover not only datasets, but also assets such as metrics, dashboards, and business files, to obtain a complete view of available resources. This necessitated a catalog that consolidates datasets and data assets in one location.
  2. Uniform governance and enforcement – To maintain best data protection practices and support business goals, teams need consistent enterprise-wide data governance where they request access once and the system enforces that access uniformly across all compute engines, alleviating fragmented or redundant access management. For internal systems, there was need for trusted identity propagation so user identity is preserved and used across AWS and internal systems for consistent enforcing.
  3. Multi-approval workflows – The solution supports multiple approval workflows within a single system, using Andes for dataset approvals and a custom workflow for dashboard approvals to maintain total governance and visibility across data assets.
  4. Delegated ownership – While enterprise teams retain overarching governance responsibility, business-specific data stewards required the ability to modify select attributes and apply appropriate tags to assets produced by their respective producers and consumers.

Solution: Unify datasets and data assets with Amazon SageMaker

Amazon chose to extend Andes with Amazon SageMaker to enhance the discovery experience. SageMaker offers native support for multimodal catalogs, and integrated with enterprise identity management, making it the ideal foundation for extending Andes’ governance model.

Rather than broadcasting assets across multiple domains, a single enterprise-wide domain standardizes and synchronizes data assets in one place. This domain is associated with AWS IAM Identity Center, which is connected to Amazon’s corporate identity system to maintain best data protection practices by limiting direct permissions and using corporate identity and group-based permissions.

Architecture diagram showing how Amazon SageMaker integrates with enterprise data catalog Andes and AWS IAM Identity Center

This integrated architecture directly addresses the identified challenges:

  • Single-pane asset discovery – Datasets and data assets are accessible through a single, consolidated view, avoiding the need to navigate across disparate systems or domains. This simplifies discovery and reduces the time to insight for teams across the organization.
  • Extended governance – Governance of both enterprise-wide and local datasets is orchestrated through a single system.
  • Extended observability – Trusted Identity Propagation (TIP) through AWS IAM Identity Center allows human users to access data interactively using their corporate identities. This provides audit-trail visibility into who is accessing what data for audits and organization’s observability requirements.
  • Amazon tool integration – Integration with Git and other internal systems automates management of accounts, permissions, and approvals. This reduces manual overhead and helps maintain that access controls remain tightly aligned with existing business workflows.

Design overview

This section describes the key features and design of the Amazon SageMaker integration. The technical implementation consists of three core components:

1) Catalog connectors

Amazon built connectors and ingestion paths to bring data assets into Amazon SageMaker while maintaining business continuity and preserving existing governance:

  • Andes integration: SageMaker provides APIs to synchronize assets from external catalogs. BDT extended this to bring Andes datasets (with their sophisticated metadata, business context) into the integrated experience. The integration preserves Andes’ permission model and governance workflows, to maintain existing security standards and best practices intact.
  • Account onboarding: Teams self-serve onboard their AWS accounts through an AWS Lambda-based integration. When creating projects, SageMaker queries this service to determine which accounts a user’s identity can access.

2) Delegated ownership

When data systems scale across business units, centralized governance teams need to delegate permissions for catalog enrichment, policy enforcement, and metadata management.

  • Catalog enhancement allows business teams to define and publish their own business glossaries, curated vocabularies of domain-specific terms, definitions, and relationships, directly within the catalog. Allowing business owners to author and maintain these glossaries increased accuracy and discoverability of catalog assets. Data consumers across the enterprise benefit from clearer, more consistent terminology.

3) Integration with consumption and access tooling

Teams discover data in SageMaker Unified Studio and consume it through both SageMaker Unified Studio and internal tooling:

  • Data discovery: SageMaker Unified Studio integrates with Amazon-wide Identity Center allowing almost all Amazon users to authenticate and search for cataloged assets. This integration addresses the data discovery problem by providing enterprise-wide visibility into available data resources.
  • Integrated development environment: SageMaker Unified Studio provides built-in tooling out of the box including a Query Editor for SQL analytics and Amazon SageMaker AI for machine learning (ML), which helps teams access data, build models, and collaborate across organizational boundaries.
  • Code repository integration: Manage code with full Git operations supported from SageMaker Unified Studio. Query code and notebook code persist to GitFarm (Amazon’s internal Git system), allowing teams to view and manage their work through Amazon’s standard version control system.
  • Native analytics integration: Projects directly connect to AWS analytics engines including Amazon Athena for SQL, AWS Glue and Amazon EMR for Apache Spark, and Amazon Redshift for data warehousing. User-authored jobs use Andes governance and permissions across engines for consistent access control.

SageMaker implementation results

SageMaker catalog now encompasses various types of data assets from across the organization, representing an expansion from datasets alone to a complete inventory of data, dashboards, metrics, models, and other data assets, all while maintaining best practices and appropriate access and use guardrails.

“SageMaker provides a unified catalog that makes discovery and sharing of data assets, metrics and dashboards across teams straightforward, with direct integration to Andes datasets. SageMaker delivers deep integration through Git repository connections and enterprise identity management that aligns with existing Amazon workflows.”

– Gerry Moses, Sr. Principal TPM, Amazon

  • Faster data discovery – Data consumers can go to one place to locate trusted, high-quality assets with significantly less friction, which reduces the time from question to insight. By surfacing well-documented, governed assets through an enriched catalog, teams can confidently identify the right data for their use cases without navigating sprawling, inconsistent inventories or relying on tribal knowledge.
  • Improved collaboration – Breaks down data silos by making curated assets discoverable and reusable across Amazon. When teams can build on shared, authoritative datasets rather than creating redundant copies, data proliferation is reduced.

Conclusion

By integrating their existing governance tooling with Amazon SageMaker to build a centralized data catalog, BDT is creating a foundation for faster, more efficient data discovery across teams. Amazon SageMaker helped unify diverse data types with their existing catalog and enabled collaboration across teams to help them find the right data. By integrating with existing governance frameworks, BDT demonstrates how organizations can expand their catalog capabilities while preserving existing enterprise investments.

To learn more and get started with Amazon SageMaker Unified Studio, visit aws.amazon.com/sagemaker/unified-studio or the AWS console.


About the authors

Matt David

Matt David

Matt is a Sr PMM, specializing in helping data teams with AI-powered analytics. His areas of interest include self-service analytics, data democratization, and preparing organizations for the age of AI agents. He brings extensive experience from his roles at Atlassian, Hex, and DataCamp.

Gerry Moses

Gerry Moses

Gerry is a Senior Principal Technical Program Manager in Business Data Technologies where he leads joint Amazon/AWS programs. His work improved data governance for Amazon’s Andes data lake, enabled broader AWS technology adoption by data lake users, and influenced product improvements that benefited all AWS customers.

Ramesh Singh

Ramesh Singh

Ramesh is a Senior Product Manager Technical at AWS in Seattle, Washington, currently with the Amazon SageMaker team. He is passionate about building high-performance ML/AI and analytics products that help enterprise customers achieve their critical goals using cutting-edge technology.

Pradeep Misra

Pradeep Misra

Pradeep is a Principal Analytics and Applied AI leader at AWS. He is passionate about solving customer challenges using data, analytics, and AI/ML. Outside of work, he likes exploring new places, trying new cuisines, and playing badminton with his family. He also likes doing science experiments, building LEGOs, and watching movies with his daughters.

Eunji Kang

Eunji Kang

Eunji is a Principal Product Manager Technical focusing on democratizing data across Amazon teams for fast data-driven business decisions without compromising security and compliance.

Trevor Gasdaska

Trevor Gasdaska

Trevor is a Principal Engineer focusing on data compliance and agentic AI workflows for Big Data Technologies at Amazon. He builds tools that help teams govern and use data at scale.

Brad Porter

Brad Porter

Brad is a Principal Business Development Manager at Amazon Web Services. He works with Amazon.com and enterprise customers to define and accelerate go-to-market strategies across Data Analytics, AI/ML, and Generative AI. He has over 20 years of experience in cloud strategy, enterprise infrastructure, and technology leadership.

How ALS GeoAnalytics LITHOLENS ™ revolutionizes core logging through machine learning with Amazon EKS

Post Syndicated from Saransh Burman original https://aws.amazon.com/blogs/architecture/how-als-geoanalytics-litholens-revolutionizes-core-logging-through-machine-learning-with-amazon-eks/

In the mining industry, accurate geological analysis is required for improving mine design and development. Traditionally, this involved labor-intensive and time-consuming on-site inspections of drill core samples, often conducted in remote and challenging environments. ALS GeoAnalytics has streamlined this process through its LITHOLENS ™ platform, a machine learning (ML)-powered system that uses deep learning and machine vision to automate core logging. LITHOLENS ™ significantly enhances data consistency, operational efficiency, and scalability while significantly reducing logging-related costs and lowering greenhouse gas emissions to support sustainable mineral extraction.

This post explores how ALS GeoAnalytics successfully deployed LITHOLENS ™ with Amazon Elastic Kubernetes Service (Amazon EKS) to scale model training and inference while minimizing cost.

The challenge

Development of a new mine involves the creation of a 3D map of the ore body, known as a geological or resource model. This model drives all future design decisions and creating it requires drilling thousands of holes throughout the ore body to examine the structure and composition of the samples extracted. This process is subject to numerous challenges that affect both active and historical drilling campaigns. Challenges such as:

  • Remote site access requiring geologists to travel long distances to visually inspect physical core boxes
  • Subjective interpretations led to inconsistencies, with different experts often producing varying geological logs
  • Underutilized historical imagery from past campaigns lacked standardized tools for meaningful analysis
  • Lost or degraded physical samples made it difficult to revisit legacy data or validate past interpretations
  • Limited transparency in logging and decision-making processes hindered collaboration and accountability
  • Scheduling bottlenecks arose from reliance on a small pool of qualified experts
  • Non-standardized data collection methods prevented effective scaling and cross-project comparison

These limitations not only delayed project timelines but also restricted the ability to generate reliable, high-resolution geological insights—ultimately impeding the speed and effectiveness of exploration strategies.

Machine learning at geological scale

ALS GeoAnalytics developed a comprehensive suite of machine learning and computer vision models to automate geological and geotechnical logging, transforming raw core imagery and data into actionable insights.

A machine learning pipeline formed the foundation for high-resolution visual analysis. It begins with the Color Extraction module, which scans each image to identify unique pixel colors and store the results in Amazon Simple Storage Service (Amazon S3). This data is fed into the Color Clustering module, where users can specify clustering parameters and choose from algorithms such as K-Means, which assigns pixels to clusters based on proximity to centroids, or the Gaussian Mixture Model (GMM), which uses probabilistic distributions to capture more complex variance structures within the color data. These methods effectively reduced image complexity and helped highlight mineralogical variation.

To quantify color composition along the core, the Percentage Report module was introduced. It segmented each image into user-defined sections (for example, 20 cm intervals) and calculated the proportional distribution of each color cluster, enabling spatial analysis of lithological patterns.

On the deep learning front, the team developed and deployed an advanced suite of models tailored for geological and geotechnical analysis. A highlight of this work was the development of RoQE Net, a state-of-the-art neural network designed for geotechnical parameter extraction. RoQE Net demonstrated exceptional accuracy in computing Rock Quality Designation (RQD) and extracting alpha angles, key metrics for assessing core integrity and rock mass quality. In parallel, VeinNet and CobbleNet were engineered to identify and map complex geological features such as veins, cobbles, and lithological structures with high precision. These models were benchmarked against industry standards and consistently outperformed traditional methods in terms of accuracy, reliability, and scalability.Together, these machine learning and deep learning components form the backbone of the LITHOLENS ™ platform—delivering automated, scalable, and highly accurate geological intelligence that accelerates decision-making and enhances the efficiency of exploration and resource modeling workflows.

Solution architecture

ALS GeoAnalytics built LITHOLENS ™ on AWS using a hybrid architecture that combines containerized workloads with serverless components. The system uses Amazon EKS for compute-intensive machine learning tasks, AWS Lambda for API operations, Amazon S3 for data storage, and Amazon Relational Database Service (Amazon RDS) for structured data management.

Figure 1: Architecture Diagram

LITHOLENS ™ uses a unified API model to drive next-generation rock and mineral data analysis. This unified API created is a unified application programming interface that combines multiple services, data streams, and analytic capabilities into a single, powerful access point. Unlike traditional APIs—which might deliver basic, one-dimensional data—you can use the unified API to connect, analyze, and automate complex workflows across departments, vendors, and a wide variety of data sources all at once. With the unified REST API, users can submit geological analysis jobs, monitor progress, and retrieve results through a single interface. This API combines multiple services and data streams into one access point, so users can automate complex workflows across departments and data sources.

Architecture flow:

  1. Request Intake – Jobs are submitted through Amazon API Gateway with a payload specifying job parameters and EKS configuration.
  2. Job Orchestration – The API backend, running on AWS Lambda, provisions EKS containers with the appropriate configuration. User data scripts bootstrap each instance with required setup and execution commands.
  3. Execution and Data Flow
    • Input data is retrieved from Amazon S3.
    • Computation is performed on EKS pods using G6 instances.
    • Logs and intermediate results are continuously tracked.
    • Results are stored in S3 or persisted into RDS through dedicated API calls.
  4. Resource Management – Upon job completion, EKS containers instances automatically shut down, reducing costs.

Architecting for scale and efficiency

To handle variable workloads efficiently, ALS GeoAnalytics implemented a hybrid architecture that’s designed for both performance and cost. The system uses Amazon EKS for compute-intensive ML tasks while using AWS Lambda for lightweight API operations and job orchestration.

Key architectural decisions:

  • Amazon EKS for ML Workloads – Deep learning model training and inference require sustained compute power with GPU acceleration. EKS provides the container orchestration needed to manage these workloads across G6 instances, with automatic scaling based on job queue depth.
  • Lambda for API Gateway – Job submission, status checking, and result retrieval are handled through serverless functions. This removes the overhead of maintaining always-on API servers for sporadic client requests, reducing costs during low-usage periods.
  • Pre-configured AMIs – Custom Amazon Machine Images contain all required dependencies and model artifacts, reducing container startup time from several minutes to under 30 seconds. This approach improves job throughput and reduces compute costs by minimizing idle time.
  • Automated Resource Management -–EKS clusters scale down to zero when no jobs are queued, so compute resources are only consumed during active processing. Combined with S3 for data persistence and RDS for metadata, this creates a cost-effective architecture that scales with actual usage.

This design addresses the geological industry’s unpredictable workload patterns while maintaining the performance needed for complex computer vision tasks.

Business impact and results

LITHOLENS ™ has seen success with 10 different mining companies on over 40 active projects, with substantially accelerated project completion and a standard analysis process used across all projects. This new approach has made mineral detection and classification more accurate while reducing the need for experts to visit sites. Teams can now trace how analysis decisions are made, grade minerals more consistently, and plan projects and assign resources more effectively. Real-time monitoring and reporting give managers up-to-the-minute information on how projects are progressing.

Conclusion

The massive scalability of Amazon EKS has allowed ALS GeoAnalytics to fundamentally transform how core logging and analysis is conducted. AWS suite of services enables LITHOLENS ™ to efficiently implement computer vision and machine learning, bringing new operational capabilities to our customers and opening business opportunities throughout the mining industry. The success of LITHOLENS ™ demonstrates how cloud computing and AI can help modernize a long-standing industry like mining, creating value through improved operational efficiency, accuracy, and scalability. ALS GeoAnalytics continues to evolve its platform on AWS, using cloud computing to push the boundaries of what’s possible, and looking to grow LITHOLENS ™ in to promising applications in oil and gas, civil engineering, and even space exploration.


About the authors

How Synthesia optimizes generative AI video inference on Amazon EC2 G7e instances

Post Syndicated from Moises Hernandez original https://aws.amazon.com/blogs/architecture/how-synthesia-optimizes-generative-ai-video-inference-on-amazon-ec2-g7e-instances/

Synthesia, an enterprise-focused AI video platform, has transformed content creation, helping everyone to create video content without cameras or microphones. To achieve this, Synthesia allows its users to create video avatars that synthesize the likeness and voice of real people. Synthesia achieves this through a series of in-house developed models based on various architectures, including latent diffusion video generation models.
Customers like Synthesia often choose to host their models on Amazon Elastic Compute Cloud (Amazon EC2) instances because of the flexibility and control over the underlying hardware that the service provides, and among them they find the G7e instance family to be a cost-efficient option to serve GPU-memory intensive generative AI video models, giving customers access to NVIDIA RTX PRO 6000 Blackwell GPUs, with 96GB of GPU memory.

When generating videos using AI models with a Variational Auto Encoder (VAE) Decoder in the architecture, customers often find the GPU utilization is bottlenecked by the saving rate of the video frames to a file held on storage. This causes GPU stalls and reduces average GPU kernel utilization, which is the percentage of time the GPU is actively executing compute kernels rather than idle.

This post introduces a video decoding optimization technique that we have ideated in collaboration with Synthesia Research Engineering team, which we call Asynchronous Frame Generation Pipeline. Adopting this technique allows you to overlap GPU compute, device-to-host (D2H) data transfer, and host-side post-processing. In this post, we apply this technique to the VAE decoder of a Wan video generation model as an example, where our benchmarks on G7e show increased GPU kernel utilization from 82% to 99.9%, in turn leading to an 8.2% decrease in latency (and increase in throughput) for video decoding. We expect this technique to benefit any customer with a chunked video generation pipeline that transfers frames to host memory.

You can find an end-to-end example implementation of the Asynchronous Generation Pipeline applied to the Hugging Face Diffusers format of the Wan 2.2 14B Model in the associated GitHub repository.

In the following section, you will see how latent diffusion models generate video and why the traditional sequential decoding approach can lead to inefficient GPU utilization.

Understanding the sequential decoding bottleneck

Latent diffusion video generation models have emerged as powerful tools for generating temporally coherent video sequences. To reduce compute and memory requirements, these models perform the diffusion process in a compressed latent space of a variational auto-encoder (VAE), as depicted in the below figure, which has generally a much lower dimensionality than the originating video pixel space.

Fig. 1 High-level architecture of a VAE model. The depicted frames are taken from the Wan 2.2 repository sample video used in the example implementation in the next sections.

At inference time, a latent representation is initialized with noise and iteratively denoised via a Diffusion Process. To make sure that the output follows the text prompt, every denoising step is conditioned on the text input if present. After the final denoising step, the Diffusion Process yields a generated video that is still represented in the latent space of the VAE. The last step therefore consists in decoding the latent video back into a human-readable pixel video using the decoder part of the VAE.

Processing the whole latent or pixel video at once is usually too resource-intensive, even on large GPUs. It is therefore common to split the video along the temporal dimension and to decode video frames one latent frame at a time, resulting in a chunk of, for example, 4 consecutive pixel frames, as shown in Fig 2.

Fig.2 Decoding one latent results in a chunk of 4 time-consecutive pixel frames which are transferred from the GPU to the host.

Once a chunk has been decoded and processed on the GPU, the corresponding pixel frames must be transferred back to host (CPU) memory with a D2H transfer, so they can be written to a file or further processed. If you wait to transfer the video frames once the entire video has been decoded, the full decoded video must fit in GPU memory before being transferred, making this strategy hard to scale for arbitrarily long videos. A more memory-efficient alternative is to transfer the decoded frames every time a chunk is decoded, so that the GPU memory footprint of the decoder scales with the size of the chunk instead of the full video.

Traditionally, a newly generated set of frames in a chunk N is passed from GPU memory to CPU RAM synchronously, and it is committed to storage before the CPU can launch the CUDA kernels (referred to as kernels from now on) that process the N+1 chunk. This leads to systematic GPU stalls between chunks, as the copy from device to host prevents the GPU from immediately starting work on the next chunk, reducing overall hardware utilization, and in turn increasing processing time. We will refer to this synchronous process as Sequential Frame Generation Pipeline, which is depicted in Fig. 3.


Fig.3 Schematic representation of the Sequential Frame Generation Pipeline. Launching the kernels processing Chunk N+1 needs to wait for the full copy and storage of the frames in Chunk N.

In the following section, you will learn how to overcome this bottleneck using an asynchronous pipeline that overlaps GPU computation with data transfers and host-side processing.

Asynchronous Frame Generation Pipeline

To minimize GPU stalling, and in turn increase GPU utilization, you need to modify the Sequential Frame Generation Pipeline introduced in the previous section, so that all host-side CPU work (such as appending decoded frames to a file) runs in parallel with a stream of uninterrupted device-side kernels. This section shows you how to achieve this using as an example the implementation of an Asynchronous Wan VAE Decoder, which is implemented with PyTorch and you can find in the associated GitHub repository.

By default, PyTorch schedules work on a single (default) CUDA stream per device which executes the operations in issue order unless additional streams are explicitly created. To decouple compute and D2H copies, the implementation uses two CUDA streams so that the GPU is allowed to overlap compute kernels, which are enqueued on the default stream, and D2H copies on a dedicated copy stream. Compute kernels are enqueued on the default stream, which from now on can be referred to as Compute Stream, and D2H copies on a dedicated copy stream, namely the Copy Stream. In Fig. 4 you can find the full setup on a GPU accelerated instance (G7e in our example) depicted.


Fig. 4 High level diagram of the key components in our implementation of the Asynchronous Frame Generation Pipeline.

On the host side, kernel submissions are interleaved with D2H transfers and post-transfer processing and, to avoid host-side blocking calls and maximize GPU utilization the Asynchronous Frame Generation Pipeline introduces two mechanics:

  • A dedicated Worker CPU thread responsible for reading chunks from Host Memory (RAM), and writing them to file, leaving the main Python thread to focus on launching kernels and scheduling D2H transfers.
  • Two in-memory Buffers on the GPU Memory (VRAM), and on the Host Memory (RAM), and page-lock the required Host Memory buffers to make sure D2H copies are performed fully asynchronously.

Using a double-buffer strategy makes sure that for adjacent chunks the compute, D2H transfer, and host processing can overlap safely as they operate on distinct memory buffers.


Fig. 5 Schematic representation of the interplay between Events, Streams, Buffers, and Worker component in the Asynchronous Frame Generation Pipeline implementation. Events are depicted as left and right conditions, expressing what the process WAITs for, and ISSUEs when it has completed. The size of the boxes representing operations has been scaled to fit the text, and therefore not representative of the actual processing time.

Since buffers can be accessed concurrently by different components, the implementation introduces a synchronization mechanism to avoid different kernels and threads corrupting data held in the buffer. This can be achieved using CUDA Events which are used as a barrier that clears if it can answer closed questions such as: Has decoding of chunk N completed? You can find the interplay between Streams, Threads and Events depicted in Fig. 5.

G7e benchmark results

To quantify the impact of the Asynchronous Frame Generation pipeline, you can benchmark the decoding of a single 41 latent-frames test video for both the Synchronous and Asynchronous pipeline on a g7e.2xlarge EC2 instance using this sample notebook. To have a common baseline, the benchmarks are based on the unoptimized Hugging Face Diffusers Wan 2.2 14B model, and, in both cases, the benchmarking loop performs an initial full decoding cycle to let CUDA and PyTorch initialize and allocate memory pools and cache. After warmup, the benchmarking session carries out 10 consecutive full video decoding cycles. We have run an end-to-end run as an example and present the results in Table 1.

Metric Synchronous (time s /video) Asynchronous (time s / video)
min 21.98 20.16
mean 21.99 20.17
P99 22.01 20.20

Table. 1 Benchmark results for 10 consecutive decoding runs for the Synchronous and Asynchronous pipelines.

The example results show a speed gain of up to 8.2% on this benchmark, decreasing the Real Time Factor of the decoder from 3.21 to 2.95. Considering the g7e.2xlarge pricing as of writing this post ($3.36 per GPU on-demand in the Ohio region), this leads to an average saving of $896 for decoding 1,000 hours of video on a single GPU. This is a theoretical saving, calculated assuming that the underlying model operates at full computational efficiency without bottlenecks.

To have a qualitative understanding of how this performance boost is achieved, you can profile both the Synchronous and Asynchronous pipelines using this sample notebook. The results for an example run are reported in Fig. 6 and Fig. 7, showing the absence of GPU stalls in the Asynchronous pipeline as opposed to the Synchronous pipeline.


Fig. 6 Profile of the Synchronous pipeline single thread and stream. As the main CPU thread is writing Chunk N frames to disk, the GPU stream stalls, waiting for the CPU to launch the kernels needed to process Chunk N+1


Fig.7 In the Asynchronous pipeline, the Compute Stream is not interrupted by the copying and writing of the frames to disk. The Main Thread is free to launch the processing kernels. This figure is displayed with a different zoom level than Fig. 6 to highlight the qualitative differences.

By focusing on the temporal axis for two consecutive chunks in the steady state, you can quantify the GPU kernel utilization increase, which, in our example run was 82% for the Synchronous case, increasing to 99.9% for the Asynchronous case.

Conclusion

In this post, we demonstrated how the Asynchronous Frame Generation Pipeline can improve GPU utilization when decoding latent videos on Amazon EC2 G7e instances. By decoupling GPU compute from device-to-host transfers and host-side I/O using dual CUDA streams, pinned memory buffers, and a dedicated worker thread, we eliminated the GPU stalls inherent in traditional synchronous decoding pipelines.

The sample benchmarks on the Wan 2.2 14B VAE decoder show an 8.2% reduction in decoding latency, translating to approximately $896 in savings per 1,000 hours of decoded video on a single GPU when considering the g7e.2xlarge pricing as of writing this post ($3.36 per GPU on-demand in the Ohio region). These gains come without any changes to model weights or inference quality, purely from better hardware utilization. We expect the kernel utilization gain to be even more impactful on optimized and compiled models, which make more efficient use of the GPU.

The techniques presented here are not specific to the Wan architecture, nor to the specific GPU utilized. Any chunked video generation pipeline that transfers frames to host memory can benefit from this approach. We encourage you to explore the associated sample implementation to experiment with how the Asynchronous Frame Generation Pipeline could be integrated into your own video generation workloads.

To get started with G7e instances for your generative AI video workloads, visit the Amazon EC2 G7e instance page.


About the authors

How Smartsheet built Real-time Dynamic Filtering on Apache Flink reducing $40K/month in messaging costs

Post Syndicated from Emre Kartoglu original https://aws.amazon.com/blogs/big-data/how-smartsheet-built-real-time-dynamic-filtering-on-apache-flink-reducing-40k-month-in-messaging-costs/

Processing hundreds of thousands of events per second while maintaining sub-second latency is a challenge many organizations face when building real-time data-driven applications. When filter policy changes propagate in up to 15 minutes, dynamic event routing becomes impractical, forcing teams to over-consume events and discard over 90% after costly per-event lookups. Smartsheet, a work management solution serving millions of users and processing hundreds of thousands of events per second to power features like live collaboration, workflows, and real-time notifications, faced exactly this problem.

In this post, you learn how Smartsheet built a Real-time Dynamic Filtering (RDF) system on Amazon Managed Service for Apache Flink, cutting messaging costs by over $40,000 per month and improving live collaboration latency by 1.8x.

The challenge: Static filter policies in a dynamic world

The Smartsheet event-driven architecture publishes hundreds of thousands of events per second to an Amazon Simple Notification Service (Amazon SNS) topic. Internal teams subscribe to this topic, typically by creating an Amazon Simple Queue Service (Amazon SQS) queue with an associated SNS filter policy defined through infrastructure as code (IaC). These filter policies are typically static and specify the types of events a consumer wants to receive, such as “sheet row created,” “sheet row updated,” or “sheet row deleted.”

Although SNS supports programmatic changes to filter policies, the SNS documentation notes that changes can take up to 15 minutes to take effect. This eventual consistency window created a significant problem for Smartsheet live collaboration feature.

Live collaboration requires knowing, in real time, which sheets have active collaborators. When a user opens a sheet, the system needs to immediately start receiving events for that sheet. When they close it, the system should stop. With a 15-minute propagation delay on filter policy changes, dynamic per-sheet filtering through SNS was impractical.

The workaround was brute force: subscribe to all events (hundreds of thousands per second), pull them into an SQS queue, and use compute to check each event against Amazon DynamoDB to determine whether the sheet had active collaborators. Over 90% of events were discarded after this lookup.

Architecture diagram showing events flowing from SNS to SQS with per-event DynamoDB lookups before RDF

Figure 1: Before RDF — all events flow through SNS to SQS, with per-event DynamoDB lookups to filter. Over 90% of events are discarded after processing.

  1. Every event published to the SNS topic is delivered to the SQS queue, regardless of whether any consumer needs it.
  2. The consumer AWS Lambda reads every message from the SQS queue and must evaluate each one individually.
  3. For each event, the consumer queries DynamoDB to check whether the sheet has active collaborators. This per-event lookup adds latency and DynamoDB read costs on the hot path.
  4. After the DynamoDB lookup, over 90% of events are found to have no active collaborators and are discarded.

This approach had three compounding cost and performance problems:

  • SNS-to-SQS data transfer costs: approximately $10,000 per month to deliver all events to the queue
  • SQS costs: approximately $30,000 per month to receive, process, and delete the full event volume
  • DynamoDB costs and latency: per-event lookups to check collaborator status added load to DynamoDB and increased end-to-end data delivery latency

The solution: Real-time Dynamic Filtering with Apache Flink

To solve this, Smartsheet built a system called Real-time Dynamic Filtering (RDF) on Amazon Managed Service for Apache Flink. The core insight was to move the filtering logic into the stream processing layer itself, using Flink’s KeyedCoProcessFunction, a feature that joins and processes multiple streams by a shared key, to maintain dynamic filter policies in Flink state (RocksDB).

How it works

The RDF Flink application reads from two streams:

  1. Filter policy stream, sourced from Amazon DynamoDB Streams. When a team calls the RDF client to change their filter policy (for example, “start receiving events for sheet X”), the change is written to a DynamoDB table and propagated through DynamoDB Streams to the Flink application.
  2. Data stream, the stream of sheet events (creates, updates, deletes) that were previously delivered through SNS.

One challenge remained: some consumers need every event, regardless of sheet. When a consumer subscribes to all events, the system needs every parallel Flink task to know about it. The team solved this using Flink’s broadcast state, which replicates a small set of “subscribe to everything” policies across all tasks. Because only a handful of consumers use this mode, the memory overhead stays negligible.

Architecture diagram showing the RDF system with DynamoDB Streams feeding filter policies to the Flink application

Figure 2: After RDF — consumer teams update filter policies via client libraries. DynamoDB Streams propagates changes to the Flink application, which filters the data stream in real time using keyed state (RocksDB) for specific sheet subscriptions and broadcast state for “all sheets” subscriptions.

  1. When a consumer team wants to start or stop receiving events for a specific sheet, it calls the RDF client, a thin wrapper over the DynamoDB SDK. The filter policy change is written to that consumer’s dedicated DynamoDB table. Each consumer has its own table, providing isolated permissions and preventing noisy neighbor issues.
  2. DynamoDB Streams captures every filter policy change as a change data capture (CDC) record and streams it to the Flink application in real time.
  3. Filter policy records
    1. Filter policy records for specific sheets are routed to the KeyedCoProcessFunction, keyed by SheetID. This makes sure that filter state and event data for the same sheet are co-located in the same Flink parallel task. State is stored in the RocksDB backend, which uses memory when available and spills to disk when necessary, so the system to scale without JVM heap constraints.
    2. Filter policy records where a consumer has called listenToAllEvents() are broadcast to all parallel Flink tasks via Flink’s broadcast state. Because broadcast state lives in JVM heap, it is used exclusively for these “all sheets” records (of which there are very few), keeping the heap footprint small.
  4. The full stream of CDC events flows into the KeyedCoProcessFunction, partitioned by SheetID. Each parallel task receives only the events for the sheets it is responsible for and applies the corresponding filter state to decide whether to forward or drop each event.
  5. The broadcast state (containing “all sheets” subscriptions) is made available to all parallel instances of the KeyedCoProcessFunction, so that consumers subscribed to all events are never filtered out regardless of which task processes their events.
  6. Only events that match an active filter policy are forwarded to the consumer’s SQS queue. The result: sub-second filter policy propagation (p95 ≤1s), elimination of per-event DynamoDB lookups, and over $40,000/month in cost savings.

Critically, because the filter policy state is persisted in Flink’s RocksDB state backend, the application does not need to perform a DynamoDB lookup for every event. Within 1 second of a filter policy change, the Flink application reads the change from the DynamoDB Streams source, updates its internal state, and begins filtering the data stream accordingly.

Results

The impact of RDF was immediate and measurable across multiple dimensions:

Cost reduction

Cost category Before RDF After RDF Monthly savings
SNS → SQS Data Transfer ~$10K/month Eliminated ~$10K
SQS Event Ingestion ~$30K/month ~$2K ~$28K
DynamoDB Collaborator Lookups Significant load Eliminated (state in Flink) Included in total
AWS Lambda ~$12K/month ~$5K/month ~$7K
Total ~$45K/month

Latency improvement

  • 1.8x improvement in live collaboration data delivery latency. Users see changes from collaborators faster than before.
  • Filter policy propagation reduced from up to 15 minutes to a p95 of under 1 second

If your architecture follows a similar fan-out pattern where consumers discard a large percentage of events after per-event lookups, you could achieve comparable cost reductions by moving filtering into the stream processing layer. The savings scale with your event volume and the percentage of events currently discarded.

Key design decisions

Several architectural choices were critical to the success of this solution:

  1. Keyed state with selective broadcast: Specific sheet subscriptions are stored in keyed state using the RocksDB state backend. The system scales to a large number of filter policies without JVM heap constraints. Flink’s broadcast state is used only for the small number of “all sheets” subscriptions, where every parallel task needs visibility. Because broadcast state is stored in JVM heap, limiting its use to these few records keeps the heap footprint manageable.
  2. DynamoDB Streams as the filter policy source: Rather than building a custom control plane, the team used DynamoDB Streams to propagate filter policy changes. DynamoDB Streams gave the team durability, ordering guarantees, and a native Flink source connector integration.
  3. RocksDB state backend: Persisting filter state in RocksDB eliminated the need for external lookups on the hot path, keeping per-event processing latency low even as the number of active filter policies grows.
  4. Client library abstraction: Publishing internal Golang and Java clients lowered the adoption barrier. The client is a thin abstraction on top of the DynamoDB SDK. Each consumer has its own dedicated DynamoDB table and corresponding filter stream, which provides two benefits: it allows fine-grained AWS Identity and Access Management (AWS IAM) permissions per client, and it mitigates the noisy neighbor problem by isolating each consumer’s filter policy traffic. Teams don’t need to understand Flink internals. They interact with a simple API to manage their subscriptions.

Next steps

The live collaboration team was the first adopter of RDF, but the architecture was designed as a shared platform. Smartsheet is now expanding RDF to additional internal teams, including workflow automation and notification routing, where similar fan-out patterns exist. The team is also exploring automatic scaling policies to optimize Flink cluster costs during off-peak hours.

Conclusion

Smartsheet Real-time Dynamic Filtering system demonstrates how Amazon Managed Service for Apache Flink can solve problems that go beyond stream processing. By combining Flink’s broadcast state pattern with CoProcessFunction, Smartsheet replaced a costly and latency-bound SNS/SQS fan-out architecture with a sub-second dynamic filtering platform. The result: over $40,000 per month in savings, 1.8x improvement in live collaboration latency, and a reusable platform that multiple teams are now adopting.

If you process high-volume event streams and need to dynamically control which events reach specific consumers, this pattern can help you reduce costs and latency, whether for live collaboration, workflow automation, notification routing, or multi-tenant event delivery.

To learn more about the services used in this post, visit:


About the authors

Emre Kartoglu

Emre is a Principal Engineer at Smartsheet, where he led the adoption of Apache Flink to power features such as sheet history, live collaboration, and automations. Previously a software engineer at AWS helping build Amazon Managed Service for Apache Flink, he is an active open-source contributor to the Apache Flink project, including FLIP-418 for data skew calculation and visualization and the AWS connectors for Apache Flink. He is based in London, UK, and is passionate about operational excellence, building products, establishing mechanisms, and using data to drive positive culture change.

Rony Blum

Rony Blum

Rony Blum is a Senior Solutions Architect at AWS based in Seattle, working with ISV customers to design and implement advanced cloud architectures, specializing in SaaS solutions, multi-tenant systems, and Generative AI applications. Outside of work, Rony enjoys exploring the Pacific Northwest trails on foot and hitting the slopes during ski season.

Francisco Morillo

Francisco Morillo

Francisco Morillo is a Sr. Streaming Solutions Architect at AWS, specializing in real-time analytics architectures. With over five years in the streaming data space, Francisco has worked as a data analyst for startups and as a big data engineer for consultancies, building streaming data pipelines. He has deep expertise in Amazon Managed Streaming for Apache Kafka (Amazon MSK) and Amazon Managed Service for Apache Flink.

Modernizing Excel VBA to Python at Scale with AWS Transform custom

Post Syndicated from Somnath Chatterjee original https://aws.amazon.com/blogs/devops/modernizing-excel-vba-to-python-at-scale-with-aws-transform-custom/

Learn how AWS Transform custom can help migrate Excel VBA applications to modern Python code while overcoming context window limitations, preserving functional equivalence, and enabling cloud-native deployment—turning weeks of manual rewriting into hours of AI-guided transformation.

Introduction

Many organizations maintain dozens of Excel VBA applications built over decades, containing business-critical logic trapped in workbooks—budget planning tools, demand planning, inventory management, financial modeling, and engineering calculations. Manual migration typically costs thousands of dollars per workbook and takes weeks, while traditional AI tools fail on large codebases that exceed context windows.

This post demonstrates how you can migrate VBA to Python using AWS Transform custom, addressing three key challenges: processing large codebases through intelligent chunking, converting legacy code to maintainable Python while preserving functionality, and validating equivalence through automated testing. You can reuse the transformation across similar projects or apply it to entire portfolios.

With AWS Transform custom, you can accelerate migration timelines, eliminate transcription errors, and scale from single applications to enterprise portfolios.

Solution Overview

The following diagram illustrates how AWS Transform custom migrates VBA source code to Python output through a four-step process powered by an AI agentic system.

Architecture diagram showing how AWS Transform custom migrates VBA source code to Python output through a four-step process powered by an AI agentic system

Figure1:AWS Transform custom VBA to Python migration architecture

AWS Transform custom provides an interactive workflow where you describe your migration requirements. The system interprets your intent and iteratively refines the transformation definition until it meets your specifications. As the system processes your code, it improves the quality of each subsequent run. Once you finalize a transformation, you can publish it to a registry so your team can reuse it across multiple projects without starting from scratch.

Migration Approach

The migration follows a three-phase process that takes you from defining your transformation through execution and validation and finally scaling across your portfolio.

In the first phase, transformation definition creation, you start an interactive session with AWS Transform custom to describe your migration requirements. You can reference your VBA code, API documentation, and target framework guides as context. The system uses these inputs to automatically generate transformation rules and patterns, which you can iteratively refine through build validation until the definition accurately captures your migration logic.

In the second phase, execution and validation, AWS Transform custom applies the transformation to your codebase. It intelligently chunks large codebases into logical modules and processes them in dependency-aware order, so cross-module references remain intact. Throughout execution, the system continuously validates builds and tests, automatically detecting and correcting errors as they arise.

In the third phase, scale and reuse, you publish your finalized transformation to the registry, making it available for your team to apply across similar projects. You can run campaign-based bulk executions across multiple repositories, extract knowledge items for continuous improvement, and integrate the transformation into your CI/CD pipelines.

Key capabilities

AWS Transform custom addresses the core challenges you face when migrating large VBA codebases to Python.

For context window management, the system automatically segments your codebase into logical modules, tracks cross-module dependencies, preserves interface contracts across chunks, and maintains state throughout the transformation. This means you can process codebases that far exceed standard AI context window limits without losing coherence between modules.

For intelligent code restructuring, AWS Transform custom recognizes VBA idioms and maps them to Python equivalents. It refactors procedural code into object-oriented designs, replaces Windows-specific APIs with cross-platform libraries, and applies proper encapsulation and separation of concerns. The result is a clean, maintainable Python that follows modern coding standards.

For functional equivalence preservation, the system generates automated tests based on the original VBA behavior and runs regression testing after each transformation step. It benchmarks performance metrics such as timing and resource usage and validates edge cases to confirm that your transformed Python code produces the same results as the original VBA application.

Prerequisites

Before you begin, you need an AWS account with the appropriate permissions to use AWS Transform custom. Start by configuring authentication and setting up the AWS CLI for your environment.

To create an IAM user for AWS Transform, follow the step-by-step instructions for creating an IAM user and managing IAM policies in the IAM User Guide.

Setup using AWS CLI:

# Create policy
cat > transform-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["transform-custom:*"],
      "Resource": "*"
    }
  ]
}
EOF

aws iam create-policy \
  --policy-name AWSTransformCustomPolicy \
  --policy-document file://transform-policy.json

# Attach to your IAM user (replace with your username and account ID)
aws iam attach-user-policy \
  --user-name YOUR_USERNAME \
  --policy-arn arn:aws:iam::YOUR_ACCOUNT_ID:policy/AWSTransformCustomPolicy

Local Environment Setup

  • Operating System: Linux, macOS, or WSL (Windows Subsystem for Linux)
  • Node.js: Version 20 or higher (required for AWS Transform CLI installation via npm)
  • Git: Required for all target repositories
  • Internet Access: Required for AWS Transform service communication

AWS Transform CLI Installation

Follow the official AWS Transform Custom Getting Started Guide for complete installation and setup instructions, including:

  • Platform requirements (Linux, macOS, or WSL)
  • Installation script usage
  • Authentication configuration
  • Network requirements and firewall rules

Source Code Repository

  • VBA application source code (Excel workbook with macros or exported .bas files)
  • Git repository initialized in source directory
  • Build/test commands defined (if applicable)

Target Environment Setup (for Python migration)

  • Python 3.8 or higher installed
  • Virtual environment tool (venv or virtualenv)
  • Target framework dependencies (e.g., pygame for game applications, tkinter/PyQt6 for GUI, pandas for data processing, openpyxl for spreadsheet operations)

Walkthrough: VBA to Python Migration

Animated demonstration of the VBA to Python migration workflow using AWS Transform custom

Figure2:AWS Transform custom CLI executing a VBA to Python transformation

Step 1: Prepare Your VBA Application

Initialize Git Repository

AWS Transform Custom requires your code to be in a Git repository. If your code isn’t already in Git:

cd your-vba-project
git init
git add .
git commit -m "Initial VBA code"

Step 2: Create Transformation Definition

Start AWS Transform Interactive Session

Provide Migration Context

Screenshot showing the AWS Transform custom interactive session interface for providing migration context

Figure3:AWS Transform custom interactive session with migration context

When you start the interactive session, provide a natural language description of your migration goal, such as “Migrate VBA application to Python.” Specify the target framework you want to use, for example, pygame for GUI applications or pandas for data processing. You should also include document references to give the system the context it needs — this includes your VBA code files, Python framework documentation, and any API migration guides relevant to your project.

Example conversation:

User: I want to migrate a VBA application to Python.
AWS Transform: I’ll help you create a transformation definition. Let me analyze the VBA patterns and generate Python equivalents…

Define Scope and Entry Criteria

Before creating your transformation, review your existing codebase to understand its structure, patterns, and dependencies. Identify the code type and technology stack you are working with, such as an Excel VBA game that uses Windows API calls. Check the transformation registry for existing similar transformations to avoid duplicating work. From there, define a clear objective and scope for your transformation, and document the entry criteria that determine what code qualifies. Outline detailed implementation steps with specific technical mappings, establish validation and exit criteria for measuring success, and name your transformation appropriately so your team can discover and reuse it.

Iterative Refinement

After you provide your migration context, AWS Transform custom generates an initial transformation definition.You can review this definition at: ~/.aws/atx/custom/<session-id>/artifacts/tp-staging/transformation_definition.md.

Examine the generated rules and provide feedback on patterns, edge cases, or missing scenarios. AWS Transform custom incorporates your feedback and regenerates the definition, allowing you to iterate until the transformation accurately captures your migration logic.

Key Patterns to Address

During the transformation definition process, you should account for the common patterns that differ between VBA and Python. AWS Transform custom maps VBA ColorIndex values to standard RGB tuples and converts Range objects to Python data structures such as lists and arrays. Do While loops in VBA translate to event-driven loops in Python. The transformation replaces Windows API calls with cross-platform libraries that provide portable alternatives. VBA UserForms map to Python GUI frameworks like tkinter or PyQt6, where form controls become Python widgets. VBA Property Get/Let constructs convert to Python @property and @setter decorators. Finally, VBA’s 1-based array indexing must be adjusted to Python’s 0-based indexing throughout your codebase.

Step 3: Execute Transformation

Apply Transformation to Codebase

# Execute transformation interactively
atx custom def exec \
  --transformation-name "VBA-to-Python-Migration" \
  --code-repository-path "./your-vba-project" \
  --build-command "python3 -m py_compile **/*.py"

# Execute transformation non-interactively
atx custom def exec \
  --transformation-name "VBA-to-Python-Migration" \
  --code-repository-path "./your-vba-project" \
  --build-command "python3 -m py_compile **/*.py" \
  --trust-all-tools \
  --non-interactive

What Happens During Execution

When you run the transformation, AWS Transform custom begins with agent planning. It analyzes your codebase structure, identifies all VBA components and their dependencies, and generates a step-by-step transformation plan with logical ordering — for example, data models first, then business logic, then UI. Each step in the plan has a clear scope and validation criteria.

Next, the system performs automatic code chunking. It analyzes your codebase size and complexity, then segments the code into logical modules such as constants, game logic, rendering, and controller. These chunks are processed in dependency order based on the plan, and the system manages the context window by focusing on one module at a time.

Throughout execution, AWS Transform custom tracks dependencies by mapping relationships between VBA subroutines, verifying that dependent code references remain valid, and maintaining interface contracts across modules.

The transformation proceeds incrementally, converting one module at a time. After each module is transformed, the system validates the build. If a failure occurs, it automatically rolls back and retries the transformation for that module before moving on.

Sample Output:

Analyzing codebase structure...
Identified 4 logical modules: constants, core_logic, rendering, main_controller
Transforming module 1/4: constants
 - Converting VBA Enums to Python IntEnum
 - Mapping ColorIndex values to RGB tuples
 - Generating color constants
 ✓ Build validation passed
Transforming module 2/4: core_logic
 - Converting collision detection subroutines
 - Refactoring global variables to class attributes
 - Translating VBA arrays to Python lists
 ✓ Build validation passed
Transforming module 3/4: rendering
 - Replacing Range object manipulation with pygame rendering
 - Converting cell-based drawing to pixel-based graphics
 - Implementing screen update logic
 ✓ Build validation passed
Transforming module 4/4: main_controller
 - Converting Do While loop to pygame event loop
 - Replacing GetAsyncKeyState with pygame.event.get()
 - Implementing game state management
 ✓ Build validation passed
Transformation complete! Generated 5 Python modules.

Review Transformed Code

After the transformation completes, check the generated Python files to verify the output. You can review the transformation logs at ~/.aws/atx/custom/<conversation-id>/logs/ to understand the decisions the system made during each step. Examine the build validation results to confirm that the transformed code compiles and passes all checks.

Step 4: Validate Functional Equivalence

Automated Test Generation

AWS Transform generates validation tests based on original VBA behavior:

test_validation.py – Verifies core logic:

# Example generated test (conceptual)
def test_collision_detection():
 """Verify collision logic matches VBA behavior"""
 # Test cases extracted from VBA code analysis

def test_scoring_calculation():
 """Ensure scoring algorithm is preserved"""

def test_state_transitions():
 """Validate game state changes"""

test_performance.py – Benchmarks non-functional requirements:

def test_frame_rate():
 """Verify rendering meets 60 FPS target"""

def test_response_time():
 """Ensure input handling latency matches VBA"""

Run Validation Suite

# Execute all tests
python3 -m pytest tests/

# Run with coverage
python3 -m pytest --cov=. tests/

Manual Validation Checklist

  • Application launches without errors
  • UI renders correctly (layout, colors, sizing)
  • User interactions work as expected (keyboard, mouse)
  • Core functionality produces correct results
  • Performance meets requirements (no lag, smooth rendering)
  • Edge cases handled properly (boundary conditions, invalid inputs)

Step 5: Refine and Iterate

As you refine your transformation, you may encounter a few common issues. Here is how you can address them and provide feedback to AWS Transform custom.

Timing and performance differences can occur when the Python application runs faster or slower than the VBA original. To fix this, adjust timing constants and frame rate limiters. For example, if you notice the game runs too fast because VBA used Sleep(500) for piece drops, you can provide that feedback. AWS Transform custom corrects this by replacing time.sleep() with pygame.time.Clock.tick(60).

Color rendering mismatches happen when colors in the Python output don’t match the VBA version. This is typically caused by incorrect RGB mappings for VBA ColorIndex values. If you notice that ColorIndex 3 should be pure red (255,0,0) but is showing as dark red, provide that feedback. AWS Transform custom updates the COLOR_MAP dictionary with accurate RGB values.

Collision detection bugs may appear when pieces move through walls or other pieces. This requires refining boundary checking and collision logic. For instance, if rotation near walls allows pieces to go out of bounds, you can report this issue. AWS Transform custom adds boundary validation before rotation commits to resolve it.

Continual learning in action

Each time you provide a correction, AWS Transform custom captures it as a knowledge item. Future transformations automatically incorporate these fixes, so the same issues don’t recur. With each execution, the system improves quality for similar migrations across your portfolio.

Step 6: Publish and Scale

Save Transformation as Draft (for testing)

atx custom def save-draft \
  --name "VBA-to-Python-Migration" \
  --source-directory "~/.aws/atx/custom/<session-id>/artifacts/tp-staging"

Publish to Registry (for team-wide use)

atx custom def publish \
  --name "VBA-to-Python-Migration" \
  --description "Migrate Excel VBA applications to Python with pygame rendering" \
  --source-directory "~/.aws/atx/custom/<session-id>/artifacts/tp-staging"

Apply to Multiple Projects

# Apply to each project individually
atx custom def exec -n "VBA-to-Python-Migration" -p ./project1 -c "python3 -m py_compile **/*.py" --trust-all-tools --non-interactive
atx custom def exec -n "VBA-to-Python-Migration" -p ./project2 -c "python3 -m py_compile **/*.py" --trust-all-tools --non-interactive

CI/CD Integration

# Example GitHub Actions workflow
name: VBA Modernization
on:
  push:
    branches: [main]
jobs:
  transform:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Run VBA to Python transformation
        run: |
          atx custom def exec \
            --transformation-name "VBA-to-Python-Migration" \
            --code-repository-path "." \
            --build-command "python3 -m py_compile **/*.py" \
            --non-interactive \
            --trust-all-tools

Benefits

By using AWS Transform custom, you can reduce tech debt and accelerate enterprise modernization at scale. The service supports diverse transformation use cases beyond VBA to Python, adapting to the specific needs of your codebase and target platform. With continual learning and improvement, each transformation builds on the corrections and knowledge items from previous runs, delivering higher quality results over time. Once you define a transformation, you can apply it everywhere — across similar projects, teams, and entire application portfolios — without recreating the migration logic from scratch.

Architecture Evolution

Before migration, your legacy VBA application is constrained to a Windows-only execution environment, tightly coupled to the Excel runtime. The code follows a procedural style with a global state, and testing and deployment are manual processes.

After migration with AWS Transform custom, your application follows a modular architecture with clean separation of concerns across constants, logic, rendering, and controller layers. The code uses object-oriented design with proper encapsulation and runs cross-platform on Windows, macOS, and Linux. The transformed application is cloud-ready, with a structure that supports containerization through Dockerfile generation, serverless adaptation with decoupled logic and rendering, and standard Python packaging through requirements.txt and setup.py. It also integrates with modern DevOps workflows, including automated testing with pytest, CI/CD pipeline compatibility, and version control through Git.

You can deploy the transformed application in several ways: containerized with Docker and Kubernetes for scalable web applications, serverless with AWS Lambda for event-driven processing, as a standalone desktop executable with PyInstaller, or as a web application with FastAPI or Flask wrappers for browser access.

The following screenshot shows the transformed Tetris application running as a Python pygame application.

Screenshot of the transformed Tetris application running as a Python pygame application

Figure4:Transformed Tetris game running as a Python pygame application

Transformation definition creation

When creating your transformation definition, provide comprehensive context by including your VBA code, target framework documentation, and migration guides. Focus on documenting application behavior by describing what the application does functionally, not just its code structure. Identify platform-specific dependencies such as Windows APIs, Excel features, and ActiveX controls that require special handling during migration. Define clear success criteria including performance targets, functional requirements, and test coverage expectations. Be explicit about construct mappings in your transformation definition — the more specific you are, the more consistent the output.

Execution strategy

Start with a pilot project by testing your transformation on a smaller, representative application before applying it broadly. Validate incrementally by reviewing each module transformation before proceeding to the next. Capture corrections as feedback by documenting issues and fixes so the system can incorporate them through continual learning. Iterate on edge cases to refine the transformation for corner cases and error conditions that may not surface during initial runs.

Quality assurance

Generate comprehensive automated test suites from the original VBA behavior to verify functional equivalence. Run performance benchmarking to confirm that non-functional requirements match the original application. Conduct manual validation through user acceptance testing to verify UI and UX consistency. After any refinements, run the full test suite for regression prevention to ensure that fixes in one area haven’t introduced issues elsewhere.

Scaling across your portfolio

When you are ready to scale, categorize your applications by similarity — for example, data processing, UI-heavy, or calculation-focused workbooks. Create transformation variants with customized definitions for each category. Use campaigns for bulk execution to process multiple applications in parallel. Monitor and aggregate results across your portfolio to track success rates, identify common issues, and measure time savings.

Cleanup

Archive Conversation Logs

# Conversation logs are in ~/.aws/atx/custom/<conversation-id>/
# Archive for future reference before cleanup
tar -czf vba-migration-logs.tar.gz ~/.aws/atx/custom/<conversation-id>/

Remove Temporary Files

# Remove conversation data and transformation session artifacts (kept for 30 days automatically)
rm -rf ~/.aws/atx/custom/<conversation-id>

# Remove draft transformations (if not needed)
atx custom def delete --name "VBA-to-Python-Migration-Draft"

Manage Published Transformations

# List your transformations
atx custom def list

# Delete transformations no longer needed
atx custom def delete --name "VBA-to-Python-Migration-Old-Version"

# Update transformation tags for organization
atx custom def tag --transformation-name "VBA-to-Python-Migration" \
  --tags "team:platform-engineering,language:python,status:production"

Knowledge Items Management

# List knowledge items for transformation
atx custom def list-ki --transformation-name "VBA-to-Python-Migration"

# Review and approve valuable knowledge items
atx custom def get-ki --transformation-name "VBA-to-Python-Migration" --id <ki-id>

# Enable approved knowledge items
atx custom def update-ki-status \
  --transformation-name "VBA-to-Python-Migration" \
  --id <ki-id> \
  --status ENABLED

# Delete low-quality knowledge items
atx custom def delete-ki --transformation-name "VBA-to-Python-Migration" --id <ki-id>

Conclusion

In this post, you learned how AWS Transform custom can help you migrate Excel VBA applications to Python at scale. With AWS Transform custom, you can reduce weeks of manual migration work to hours. Intelligent chunking handles large codebases that exceed standard AI context window limits, while continuous validation and automated testing preserve functional equivalence throughout the process. Because transformations are reusable, and the system learns from each execution, you gain compound efficiency with every subsequent migration. The transformed applications are cloud-ready, supporting containerization, serverless deployment, and modern DevOps workflows.

AWS Transform custom is a strong fit when you are modernizing a portfolio of dozens or hundreds of similar legacy applications, when functional equivalence must be guaranteed for quality-critical migrations, or when you have repeatable migration patterns across multiple projects.

Next Steps

The transformation demonstrated in this post — VBA to Python migration with context window management, automated restructuring, and functional validation — is a template you can apply to other modernization challenges, such as mainframe COBOL to Java, Progress ABL to Spring Boot, or .NET Framework to .NET Core. The principles remain consistent: intelligent code analysis, dependency-aware processing, continuous validation, and transformation reuse.

Call to Action

Get Started with AWS Transform Custom

Documentation and Resources:

Explore AWS-Managed Transformations: Access pre-built transformations for common migration patterns:

  • AWS SDK Java v1 to v2: AWS/java-aws-sdk-v1-to-v2
  • Python Version Upgrades: AWS/python-version-upgrade
  • Node.js Version Upgrades: AWS/nodejs-version-upgrade
  • Java Version Upgrades: AWS/java-version-upgrade

Estimate Your Migration: Use the AWS Pricing Calculator to estimate costs for your specific modernization project.


Ankit Srivastava

is a Strategic Technical Account Manager at Amazon Web Services (AWS), where he serves as a trusted advisor to global enterprise customers. With over 15 years of experience in cloud architecture, DevOps and distributed systems, Ankit helps organizations navigate cloud transformation, architecture modernization, and harness the power of Generative AI on AWS.

Somnath Chatterjee

is an accomplished Senior Technical Account Manager at Amazon Web Services (AWS), Somnath is dedicated to guiding customers in crafting and implementing their cloud solutions on AWS. He collaborates strategically with customers to help them run cost-optimized and resilient workloads in the cloud. Beyond his primary role, Somnath holds specialization in the compute, SAP and Developer Experience technical field community. With over 14 years of experience in the information technology industry, he excels in cloud architecture and helps customers achieve their desired outcomes on AWS.

Detect and resolve HBase inconsistencies faster with AI on Amazon EMR

Post Syndicated from Yu-Ting Su original https://aws.amazon.com/blogs/big-data/detect-and-resolve-hbase-inconsistencies-faster-with-ai-on-amazon-emr/

HBase operations teams spend hours manually correlating logs, metadata, and consistency reports to identify root causes. Traditional approaches require deep expertise and extensive investigation across scattered data sources, directly impacting MTTR and operational efficiency. As HBase deployments scale and expertise becomes increasingly scarce, organizations face mounting pressure to maintain service reliability while managing growing operational complexity. The manual nature of troubleshooting creates bottlenecks that delay incident resolution, increase operational costs, and risk service degradation during critical business periods.

In this post, we show you how to build an AI-powered troubleshooting solution using Amazon OpenSearch Service vector search and intelligent analysis. This solution reduces HBase inconsistency resolution from hours to minutes and root cause identification from days to hours through natural language queries over operational data. This democratizes HBase troubleshooting capabilities across teams and reducing dependency on specialized expertise.

Solution overview

The solution addresses HBase troubleshooting challenges through data processing, vector search, and AI-powered analysis. It processes operational data from Amazon EMR clusters, generates semantic vector embeddings, and enables natural language queries for intelligent troubleshooting.
Key components include:

  • Amazon EMR HBase: Runs HBase workloads with Amazon S3 as the HBase rootdir for durable, scalable storage
  • Data Processing: Extracts and processes HBase logs, HBCK reports, and metadata with vector embeddings
  • Amazon OpenSearch Service: Provides vector search capabilities with k-NN algorithms for semantic analysis
  • AI Analysis Interface: Enables natural language queries with context-aware recommendations
  • Custom Knowledge Base: Supports organization-specific runbooks and troubleshooting procedures by ingesting Git repositories via Kiro CLI‘s /knowledge add command, enabling the AI assistant to reference custom operational guides alongside HBase source code and operational tools

AWS cloud architecture diagram showing an HBase log analysis system with EMR cluster, VPC networking, IAM roles, Lambda functions, OpenSearch domain, and supporting services for scalable log processing and analytics.

The preceding diagram illustrates how the HBase log analysis system troubleshoots inconsistencies through automated workflows across AWS services.

When an operations team needs to investigate HBase issues, the engineer connects over SSH to the Amazon EMR primary node and runs the error collection script, which gathers logs from HBase master and RegionServer nodes and uploads them to Amazon S3. Next, the engineer connects to the Analytics Amazon Elastic Compute Cloud (Amazon EC2) instance and executes the automated processing script, which downloads logs from Amazon S3, generates semantic vector embeddings, and injects them into Amazon OpenSearch Service for k-NN-based semantic search. The engineer then queries the Kiro CLI AI Assistant using natural language to investigate. Kiro searches Amazon OpenSearch Service for relevant log entries and uses Amazon Bedrock to analyze patterns, correlate errors across components, and provide actionable recommendations. This reduces troubleshooting time from hours to minutes. The system operates within an Amazon Virtual Private Cloud (Amazon VPC) with private subnets for Amazon EMR and Analytics Amazon EC2, AWS Identity and Access Management (AWS IAM) roles for access control, Parameter Store for configuration, and Amazon CloudWatch for monitoring.

Prerequisites

For this walkthrough, you need the following prerequisites:

AWS account setup

  • An AWS account with administrative access for initial deployment
  • AWS Command Line Interface (AWS CLI) configured with administrative credentials

Required AWS IAM permissions

For infrastructure deployment

Your deployment user or role needs the following permissions:

  • Your deployment user or role requires sufficient access to AWS CloudFormation, Amazon S3, AWS IAM, and AWS System Manager.
  • The user or role must have the ability to create AWS CloudFormation stacks.

Infrastructure deployment:

  • For infrastructure deployment, you need AWS CloudFormation stack management permissions.
  • You also require sufficient access to create and manage the following resources:
    • Amazon OpenSearch Service domains
    • Amazon EC2 instances, Amazon VPCs, security groups, and networking components
    • AWS IAM roles and policies
    • AWS Systems Manager Parameter Store entries
    • Amazon CloudWatch Logs groups
    • Amazon S3 bucket for access logs and session logs

Runtime service roles

The AWS CloudFormation stack automatically creates two specialized AWS IAM roles designed with least-privilege access principles.

The first role is the Amazon OpenSearch Service Role, which manages Amazon VPC networking and Amazon CloudWatch logging for the Amazon OpenSearch Service domain.

The second role is the Application Role, which provides minimal Amazon OpenSearch Service and Amazon S3 access specifically for log processing applications and secure log ingestion operations.

Network requirements

  • Amazon VPC with private subnets for secure Amazon OpenSearch Service deployment
  • NAT Gateway for outbound internet access from private subnets
  • Security groups configured for HTTPS-only communication

Running Kiro CLI on Amazon EC2

Kiro platform requirements:

Kiro subscription

  • Active Kiro License: Valid subscription to Kiro platform
  • User Account: Registered Kiro user account with appropriate permissions
  • API Access: Kiro API keys or authentication tokens for CLI access

AWS Identity Center integration

  • AWS IAM Identity Center Setup: AWS IAM Identity Center enabled in your AWS organization
  • Permission Sets: Configured permission sets for Kiro users with appropriate AWS access
  • User Assignment: Users assigned to relevant AWS accounts and permission sets
  • SAML/OIDC Configuration: Identity provider integration if using external identity systems

Additional prerequisites

  • Python 3.7+ and Node.js installed locally
  • Python 3.11+ for AWS Lambda runtime environment (required for OpenSearch MCP server compatibility)
  • Sufficient service quotas for Amazon OpenSearch Service instances and Amazon EC2 resources
  • Recommended access to the analysis instance via AWS Systems Manager Session Manager (recommended). Amazon EMR clusters running HBase workloads
  • EMR_EC2_Default_Role of Amazon EMR EC2 instance profile can execute describe-stacks on AWS CloudFormation stacks in us-east-1
  • Basic familiarity with HBase operations

The deployment follows AWS security best practices with resource-specific permissions, regional restrictions, and encrypted data storage. All AWS IAM policies implement least-privilege access patterns to help secure operation of the log analysis pipeline.

Walkthrough

This walkthrough demonstrates deploying and configuring the AI-powered HBase troubleshooting solution in five key steps:

  1. Deploy AWS infrastructure using AWS CloudFormation
  2. Configure Amazon EMR analysis log collection
  3. Process and index HBase data
  4. Enable AI-powered analysis
  5. Add custom knowledge base (optional)

The complete solution is available in our GitHub repository.

Step 1: Deploy the infrastructure

Deploy the required AWS infrastructure including Amazon OpenSearch Service domain, Amazon EC2 instances, and AWS IAM roles.

To deploy the infrastructure

  1. Deploy AWS CloudFormation stack. Please update [email protected] to an email address for security alerts and Advanced Intrusion Detection Environment (AIDE) reports:
# Deploy to development environment
aws cloudformation create-stack \
  --stack-name dev-hbase-log-analysis \
  --template-body file://cloudformation/hbase-log-analysis-simple.yaml \
  --parameters \
    ParameterKey=EnvironmentName,ParameterValue=dev \
    ParameterKey=EC2InstanceType,ParameterValue=m7g.xlarge \
    ParameterKey=SecurityAlertEmail,[email protected] \
  --capabilities CAPABILITY_IAM \
  --region us-east-1
# Wait for deployment to complete (~15-20 minutes)
aws cloudformation wait stack-create-complete \
  --stack-name dev-hbase-log-analysis \
  --region us-east-1
  1. Note the deployment outputs including Amazon OpenSearch Service endpoint and Amazon EC2 instance details in the AWS CloudFormation console.

AWS CloudFormation stack outputs table displaying infrastructure resource identifiers including IAM roles, EC2 instances, security groups, S3 buckets, OpenSearch domain configuration, and VPC details for an HBase log analysis application in the development environment.

The deployment creates:

  • Amazon OpenSearch Service domain with vector search capabilities
  • Amazon EC2 instance for data processing and AI analysis
  • AWS IAM roles with appropriate permissions
  • Security groups and Amazon VPC configuration

Step 2: Connect to Amazon EC2 instance and set up system

Connect to the Amazon EC2 instance using AWS Systems Manager (SSM) and set up the required components.

To connect and set up the system

  1. Run the following commands to get the instance ID from AWS CloudFormation outputs and connect via AWS Systems Manager (SSM):
# Get instance ID
INSTANCE_ID=$(aws cloudformation describe-stacks \
  --stack-name dev-hbase-log-analysis \
  --query 'Stacks[0].Outputs[?OutputKey==`EC2InstanceId`].OutputValue' \
  --output text \
  --region us-east-1)
# Connect via SSM
aws ssm start-session --target $INSTANCE_ID --region us-east-1

Terminal screenshot showing AWS CLI commands to retrieve an EC2 instance ID from CloudFormation stack outputs and establish an AWS Systems Manager Session Manager connection to the instance in the us-east-1 region.

  1. Clone the repository and run automated setup:
# On EC2 instance
sudo su - ec2-user

# Re-install aws cli
sudo dnf remove awscli -y

# For ARM64 (Graviton instances - default)
curl "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip" -o "awscliv2.zip"

# For x86_64 (if using non-Graviton instances)
# curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

unzip awscliv2.zip
sudo ./aws/install

# update $PATH in ~/.bashrc
echo 'export PATH=$PATH:/usr/local/bin/' >> ~/.bashrc

# Reload ~/.bashrc
source ~/.bashrc

# Fork and clone the source code repository on GitHub: sample-emr-hbase-inconsistencies-detection-recovery-mcp-kiro
git clone https://github.com/YOUR_USERNAME/sample-emr-hbase-inconsistencies-detection-recovery-mcp-kiro.git hbase-analysis
cd hbase-analysis

# Run automated setup
chmod +x ./scripts/setup/automated-system-setup.sh
./scripts/setup/automated-system-setup.sh \
  --emr-version emr-7.12.0 \
  --stack-name dev-hbase-log-analysis \
  --region us-east-1

The automated setup script installs:

  • System dependencies (awscli, git, unzip)
  • uv package manager and OpenSearch MCP Server
  • Kiro CLI and configuration with AWS IAM Identity Center authentication. The script will automatically add Apache HBase open source repo and Apache HBase open source operational tools to knowledge bases
  • HBase source repositories for your Amazon EMR version
  • Python dependencies and MCP server configuration
  1. Add your own knowledge base to Kiro CLI

To enhance Kiro CLI’s analysis capabilities with Apache HBase open-source repositories, your organization’s HBase runbooks and troubleshooting guides, you can add your own knowledge base repositories. Here are the commands. Please periodically validate and maintain your runbook contents so that they remain accurate and up-to-date, reflecting any changes in your HBase environment, configurations, or operational procedures.:

# Navigate to the HBase repositories directory
cd /opt/hbase-repositories
# Clone your organization's HBase runbook repository
git clone <runbook-repository-url> <your-own-runbook-repo>
# Example:
# git clone https://github.com/your-org/hbase-runbooks.git hbase-runbooks
# git clone https://gitlab.company.com/ops/hbase-troubleshooting.git hbase-troubleshooting
# Add your custom repositories to Kiro CLI knowledge base manually (run these commands inside kiro-cli):
echo "/knowledge add --name \"Your custom HBase knowledge base\" --path /opt/hbase-repositories/<your-own-runbook-repo>" | kiro-cli
# Example:
# echo "/knowledge add --name \"Company HBase runbooks\" --path /opt/hbase-repositories/hbase-runbooks" | kiro-cli
# echo "/knowledge add --name \"HBase troubleshooting guides\" --path /opt/hbase-repositories/hbase-troubleshooting" | kiro-cli

Step 3: Configure Amazon EMR log analysis collection

Set up data collection from your Amazon EMR clusters to gather HBase logs, metadata, and consistency reports using the recommended direct collection method.
To configure Amazon EMR log analysis collection

  1. On your Amazon EMR cluster primary node, run the following commands to download the collection scripts:
# On EMR primary node
sudo su - hadoop

# Fork and clone the source code repository on GitHub: sample-emr-hbase-inconsistencies-detection-recovery-mcp-kiro
git clone https://github.com/YOUR_USERNAME/sample-emr-hbase-inconsistencies-detection-recovery-mcp-kiro.git hbase-analysis
cd hbase-analysis
  1. Run the interactive collection wizard:
# Run collection wizard
python3 scripts/utilities/emr_log_collection/emr_cluster_wizard_v2.py

Input the parameters like the EMR cluster’s jobflow ID, the log analysis Amazon S3 bucket name, and the lookback hours. The default value of the lookback hours is 4 hours.

Terminal screenshot of EMR Cluster Log Collection Wizard V2 showing an interactive command-line interface for configuring HBase diagnostic log collection from Amazon EMR clusters, with step indicators, input fields for job flow ID and S3 bucket, validation confirmations, and lookback hour configuration.

  1. The collection wizard performs these actions:
  • Collects HBase logs from local filesystem. Please reference to prerequisites for the access permission.
  • Runs sudo -u hbase hbase hbck -details (or hbck2 for HBase 2.x)
  • Runs hdfs dfs -ls -R /hbase or aws s3 ls <hbase-root-dir> –recursive
  • Runs hbase shell <<< 'scan "hbase:meta"'
  • Creates properly named files matching analysis system requirements
  • Uploads to Amazon S3 with correct naming conventions

Here’s the data collection summary:

Terminal screenshot showing EMR Cluster Log Collection Wizard V2 completion summary with job flow ID, S3 bucket location, 4-hour lookback period, green success confirmation message, S3 file path, and detailed listing of seven collected diagnostic files including HBCK reports, HBase meta table scans, root directory paths, process information, log collection summary, node logs from all servers, and collection metadata in JSON format.

You can check the uploaded contents through AWS CLI.

aws s3 ls s3://<log-path> --recursive

Here’s a screenshot of the outputs.

Terminal screenshot showing AWS CLI command output listing HBase diagnostic files and logs collected from an EMR cluster and stored in Amazon S3, displaying timestamps, file sizes, and complete S3 object paths including diagnostics directory with HBCK reports, meta table scans, root directory listings, process information, and logs directory with compressed application logs from HBase master and regionserver nodes.

  1. On the Analysis Amazon EC2 instance, download collected files to the Analysis Amazon EC2 instance.
# On analytics EC2 instance
sudo su - ec2-user

# Download logs from S3
mkdir -p /tmp/hbase-log-analysis
cd /tmp/hbase-log-analysis
aws s3 sync s3://<S3-BUCKET-NAME>/emr-logs/<EMR-JOBFLOW-ID>/ .

You can get your jobflow ID from Amazon EMR console:

Amazon EMR clusters management dashboard displaying a table with clusters, showing one cluster entry named "test" in waiting status with green indicator, creation time, elapsed time, normalized instances, along with filter controls, search functionality, pagination showing page 1, and action buttons for View details, Terminate, Clone, and Create cluster operations.

The generated files (hbase-hbase-master-ip-xxx-xxx-xxx-xxx.ec2.internal.log.gz, hbase-hbase-regionserver-ip-xxx-xxx-xxx-xxx.ec2.internal.log.gz, hbck_report.txt, hbase_rootdir_paths.txt, hbase_meta.txt, hbase_processes.txt, log_copy_summary.txt) should be aligned with the automated processing script requirements as following.

Terminal screenshot showing recursive ls -lRt command output listing HBase diagnostic files and logs in /tmp/hbase-log-analysis/ directory, displaying file permissions, ownership by ec2-user, file sizes, timestamps, and complete directory structure including diagnostics directory with text files (manifest.json, HBCK report, meta table scan, process information, root directory paths, log copy summary), logs directory with nested nodes subdirectory containing redacted instance IDs, and applications/hbase subdirectories with compressed RegionServer and Master log files.

Step 4: Process and index data

Process the collected HBase data and create vector embeddings for intelligent search capabilities.To process and index the data, please navigate to the project directory on the Analysis EC2 instance, and run automated-log-processing.sh:

sudo su – ec2-user
cd ~/hbase-analysis
chmod +x ./scripts/processing/automated-log-processing.sh
./scripts/processing/automated-log-processing.sh \
  --job-flow-id j-YOUR-JOB-FLOW-ID \
  --stack-name dev-hbase-log-analysis

The processing scripts extract and parse HBase logs and generate dimensional vector embeddings from HBase log messages using sentence transformer models to enable semantic search beyond keyword matching. The system uses the all-MiniLM-L6-v2 model by default (producing 384-dimensional embeddings), but supports configurable models with different embedding dimensions, automatically adapting the OpenSearch vector index to match the chosen model’s output. The system processes comprehensive HBase operational data including region operations, compaction activities, Write-Ahead Log events, memstore operations, and cluster management information from HMaster and RegionServer logs. Vector embeddings capture error messages, exception stack traces, performance warnings, and multi-line log entries through intelligent text preprocessing. This semantic representation enables advanced troubleshooting where users can query conceptually for “region server performance issues” or “memory pressure” and receive contextually relevant results across different log files and time periods. The vector search capabilities support error correlation by grouping similar exceptions, performance analysis by identifying related bottlenecks, and operational pattern recognition. Each log entry is stored in Amazon OpenSearch Service with original metadata (timestamp, log level, source file, job flow ID) alongside the embedding vector, enabling both structured queries and AI-powered semantic analysis. This approach transforms raw HBase logs into a searchable knowledge base supporting anomaly detection, trend analysis, and predictive insights for proactive cluster management and troubleshooting.

All scripts use AWS IAM authentication automatically. Here’s a screenshot of the data processing outputs.

Terminal screenshot showing successful completion of HBase log analysis processing, green checkmark, confirmation message "Successfully processed 4 file(s)", and next steps section displaying three numbered instructions with redacted URLs for accessing OpenSearch Dashboards, starting Kiro CLI for AI-powered analysis, and querying data using job flow ID, followed by troubleshooting documentation references for HBase inconsistency analysis and log analysis guides.

Step 5: Enable AI-powered analysis

Configure the AI analysis interface to enable natural language queries against your HBase operational data.

To set up AI-powered analysis

  1. Launch Kiro CLI (already configured by automated setup):

kiro-cliCheck mcp and knowledge bases. /mcp list

Terminal screenshot showing MCP list command output displaying one configured MCP server named "opensearch-mcp-server" with command "uvx" in green and white text on dark background with pink shell prompt, featuring a purple "Configured MCP Servers" header with checkbox icon and green horizontal separator line.

/knowledge show

Terminal screenshot showing "/knowledge show" command output displaying Agent kiro_default's knowledge base with repositories: Apache HBase source code, and HBase operational tools

If you cannot see these 2 knowledge bases, you can manually add them through the following commands:

# Note: Large repositories (~500MB) may take a while to index. Check progress with: /knowledge show
/knowledge add --name "HBase operational tools" --path /opt/hbase-repositories/hbase-operator-tools"
/knowledge add --name "Apache HBase source code" --path /opt/hbase-repositories/hbase"
  1. Use natural language queries to analyze your HBase data. The AI analysis uses both the OpenSearch MCP Server for querying indexed data and the Filesystem knowledge bases for accessing HBase source code. You can add your custom runbooks for Kiro’s reference as well.

For HBase inconsistency analysis:

# HBase Inconsistency Detection and Remediation Guidelines
## Search Strategy
- Use fuzzy search for case variations/typos, term query for exact region IDs, match_phrase for paths, query_string for logs
- Always use .keyword subfields for exact text matching
- Cross-reference filesystem (wildcard: {"wildcard": {"path": "*<region_id>*"}}) with hbase:meta (match: {"match": {"row_key": "<region_id>"}})
- The total region count in hbase meta must match the total matched document count of wildcard path like "*/.regioninfo" in hbase rootdir path.  
- All terms of region_name.keyword for a region encoded name must match a wildcard path like "*/.regioninfo"
- All terms of table_name.keyword for a table must match a wildcard path like "*/.tabledesc*"
- 1595e783b53d99cd5eef43b6debb2682 is the master store region that will locate in <hbase-root-dir>/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/
- May cross check with the raw logs in /tmp/hbase-log-analysis/
## Issue Types
Orphan regions, missing .regioninfo, missing/extra regions in hbase:meta, rowkey holes, stuck RIT, master initialization failures
## Analysis Steps
### 1. Cross-Reference Meta vs Filesystem
- Filesystem regions NOT in hbase:meta → ORPHAN REGION
- Meta regions NOT in filesystem → MISSING REGION
### 2. Validate Region Chain Continuity
- Sort regions by STARTKEY, verify region[i].ENDKEY == region[i+1].STARTKEY
- First STARTKEY must be '', last ENDKEY must be ''
- Gaps → ROWKEY HOLE
### 3. Check Region States
- state != 'OPEN' → Check RIT
- Missing server assignment → UNASSIGNED
- Multiple servers → SPLIT BRAIN
- "deployed_servers" field must have only one region server address like "ip-xxx-xxx-xxx-xxx.ec2.internal,16020,1770781485397" . The value should not be null or have multiple values. 
### 4. Validate .regioninfo Files
- Missing .regioninfo in region directory → CORRUPT REGION
### 5. Cross-Check HBCK Report
- Compare orphan counts, RIT regions, filesystem vs meta region counts
### 6. Analyze Logs
- Search: "updating hbase:meta row=<region>", "STUCK", "RIT", "Failed" + "<region>", "Split"/"Merge" + "<region>"
## Remediation
- Reference knowledge bases: "Apache HBase source code", "HBase operational tools"
- Use hbck2: /usr/lib/hbase-operator-tools/hbase-hbck2.jar
- Prefix commands with sudo -u hbase
- Use aws s3 for S3-based rootdir
- Wait 300s after creating holes before hbck fixMeta (catalog janitor cycle)
- Use unassign instead of deprecated close_region
- If the region does not have .regioninfo in  <hbase-root-dir>/data/<namespace>/<table-name>/<region-encoded-name>/ but hbase:meta has that region's information and that region has been deployed on a healthy region server, you can use hbase shell to unassign and assign the region to re-generate .regioninfo
- Always add "sudo -u hbase hbase" before "hbase shell" and "hbase hbck" commands
## Job flow
Target: <your-job-flow-id>
Inconsistency to detect: All kinds of inconsistencies

You can trust or input “y” or “t” to grant Kiro to search through mcp and knowledge bases.

Terminal screenshot showing MCP tool execution authorization prompt.

You may get some outputs like this: Kiro checked for any HBase issue.

Terminal screenshot showing HBase database query results for user table entries with server configuration details and an HBase Inconsistency Detection Framework analysis report

Kiro summarized the examination results.

Terminal screenshot displaying HBase inconsistency detection analysis results for job flow, showing one critical missing .regioninfo file issue for HBase region in a HBase table, with cluster health metrics, risk assessment, recommended fixes, and generated diagnostic reports.

Kiro provided mitigation commands after Kiro summarized the issue.

Terminal screenshot displaying a structured HBase quick fix guide with three sections: recommended fix procedure with sequential steps for region reassignment, verification steps using AWS S3 and HBCK2 tools, and impact assessment showing 30-60 second downtime, zero data loss risk, and isolated region scope for fixing missing .regioninfo file in HBase region.

Cleaning up

To avoid incurring future charges, delete the resources created during this walkthrough.

To clean up the resources

  1. Delete the AWS CloudFormation stack from AWS Management Console:

AWS CloudFormation Stacks management console displaying a list view with stacks, showing the "dev-hbase-log-analysis" stack with CREATE_COMPLETE status, along with action buttons for Delete, Update stack, Stack actions, and Create stack.

  1. Clean up Amazon EMR cluster resources (if created only for this walkthrough):
AWS EMR Clusters management console showing page clusters with a cluster in "Waiting" status
  1. Verify resource cleanup in the AWS Console to verify that all resources are deleted and review your AWS bill to confirm no unexpected charges.

Important considerations:

  • Amazon OpenSearch Service domains take several minutes to fully delete
  • Amazon S3 buckets with versioning retain object versions
  • Use smaller instance types for development to optimize costs
  • Monitor usage with AWS Cost Explorer

Conclusion

In this post, we showed you how to build an AI-powered HBase troubleshooting solution that transforms manual log analysis into an automated workflow. By combining Amazon OpenSearch Service vector search with Amazon Bedrock-powered analysis through the Kiro CLI, operations teams can resolve complex HBase inconsistencies faster and gain deeper operational insights. The solution demonstrates how AI augments human expertise to improve operational efficiency, reducing HBase inconsistency resolution from hours to minutes and root cause identification from days to hours. Ready to transform your HBase operations? Get started with the GitHub repository and explore the Amazon OpenSearch Service documentation for additional guidance on vector search capabilities.

Acknowledgments

The author would like to thank Xi Yang, Anirudh Chawla, and Sasidhar Puthambakkam for their contributions to developing the technical solution. Xi Yang is a Senior Hadoop System Engineer and Amazon EMR subject matter expert at AWS. Anirudh Chawla is an AWS Analytics Specialist Solution Architect who helps organizations empower businesses to harness their data effectively through AWS’s analytics platform. Sasidhar Puthambakkam is a Senior Hadoop Systems Engineer and Amazon EMR Subject Matter Expert who provides architectural guidance for complex BigData workloads.


About the authors

Yu-Ting Su

Yu-ting Su, Sr. Hadoop System Engineer, AWS Support Engineering. Yu-Ting is a Sr. Hadoop Systems Engineer at Amazon Web Services (AWS). Her expertise is in Amazon EMR and Amazon OpenSearch Service. She’s passionate about distributing computation and helping people to bring their ideas to life.