This post is cowritten by Manikandan Vasu, Vanji Sivajothy, and Ramchandra Koty from Equinix.
Equinix is the world’s digital infrastructure company, operating over 260 data centers across more than 70 metros globally. To address the operational sprawl that had grown from its earlier self-managed Kubernetes environment, Equinix built a shared services architecture on Amazon EKS. Previously, Equinix operated a self-managed Kubernetes environment on Amazon EC2 instances. In this model, they provisioned EC2 instances serving as etcd, control plane, and worker nodes, relying on open-source tooling to automate the installation and configuration of Kubernetes components. While this approach provided control and flexibility in the early stages of their Kubernetes journey, it also introduced a structural problem: individual application teams independently provisioning and managing their own clusters, each with its own lifecycle, configuration, and operational patterns.
Over time, this decentralized ownership model created significant operational sprawl. With no shared infrastructure layer, the cloud operations team had no consistent mechanism to enforce governance, standardize configurations, or provide common infrastructure services across the organization. Each application team operated in isolation, making decisions about networking, observability, security policies, and deployment pipelines independently. This resulted in a fragmented environment that was increasingly difficult to manage, secure, and scale.
Key operational challenges included:
Operational sprawl – Independently managed clusters led to duplicated infrastructure and no unified operational baseline.
No centralized governance – The cloud operations team had no mechanism to enforce network isolation, security policies, or deployment standards across team-owned clusters.
Lack of shared services model – Common needs (CI/CD, observability, data services, networking) were solved differently by each team, creating redundancy.
Cluster lifecycle complexity – Upgrading and patching control planes across multiple self-managed clusters introduced compounding risk with every cycle.
These challenges made it clear that Equinix needed a fundamental architectural shift, from a model where every team managed its own cluster. They moved to a shared services architecture where the cloud operations team centrally owns and governs the infrastructure while application teams focus purely on their business services. This led to their migration to Amazon EKS with a multi-account strategy designed for scalability, security, and centralized control.
In this post, we walk through how Equinix designed and implemented their shared services architecture on Amazon EKS after migrating from their self-managed Kubernetes environment. We also cover the operational results they achieved, including 4x faster deployments and significantly reduced operational overhead.
The solution: A shared services architecture on Amazon EKS (North Star architecture)
Figure 1: Equinix North Star architecture, a multi-account Amazon EKS architecture with workload isolation, centralized shared services, and hybrid connectivity through AWS Transit Gateway and AWS Direct Connect
Key architecture capabilities
The North Star architecture implements a multi-account, shared services model on Amazon EKS that cleanly separates concerns between application teams and the cloud operations team. The diagram illustrates the user acceptance testing (UAT) environment, with identical patterns replicated across non-production (system integration testing and development) and production, built around three core AWS accounts:
Workloads Account (Workloads-VPC): The Workloads account hosts all application team services running on Amazon EKS in a dedicated VPC deployed across two Availability Zones (AZ-1 and AZ-2) in us-west-1 for high availability. Traffic ingress is managed through Application Load Balancer (ALB) with Kubernetes Gateway API (GatewayClass and Gateway resources) providing precise routing to application namespaces. Cilium serves as the Container Network Interface (CNI), enforcing network policies that isolate workloads at the pod level, while Hubble provides real-time network observability across all application traffic flows.
Platform Account (Platform-VPC): The Platform account is owned and operated by the cloud operations team, housing all shared infrastructure services in a separate VPC. This includes:
CI/CD infrastructure: GitHub Runners deployed as EKS workloads, providing a centralized, self-service pipeline for all application teams.
Shared services: Capabilities managed centrally and accessible to all application teams.
Traffic ingress to platform services is handled through Network Load Balancer (NLB), with the same Cilium/Hubble networking and observability stack as the Workloads account.
Network Account (Network-VPC): A dedicated Network account acts as the connectivity hub, implementing an AWS Transit Gateway architecture that spans two Regions (us-west-1 and us-east-2) with Transit Gateway peering between them. Key networking capabilities include:
Cross-account connectivity: AWS Transit Gateway attachments connect both the Workloads and Platform VPCs to the central hub, enabling controlled communication between accounts.
DNS resolutionAmazon Route 53 Resolver endpoints (inbound and outbound) in each Availability Zone, with a private hosted zone providing service discovery across the platform.
Hybrid connectivity: AWS Direct Connect Gateway with dual circuits connecting back to on-premises Equinix border routers through network firewalls for security enforcement.
Results: Measurable impact across the organization
The migration to Amazon EKS delivered clear, measurable outcomes that validated Equinix’s North Star architecture strategy:
40%
Reduction in operational overhead by eliminating the need to manage Kubernetes infrastructure. AWS now handles upgrades, patching, and high availability automatically.
4x
Increase in deployment frequency, enabling engineering teams to ship features and updates faster, freed from the constraints of infrastructure bottlenecks.
100%
Unified architecture adopted across multiple business organizations, establishing a single, consistent operating model for all containerized workloads.
Additional operational improvements include:
Enhanced developer productivity: Standardized CI/CD workflows through centralized GitHub Runners and self-service namespace provisioning reduced friction across the development lifecycle, allowing application teams to deploy independently without cloud operations team intervention.
Improved observability: Hubble provides unified network flow visibility across both clusters, replacing fragmented, team-specific monitoring.
Strengthened security posture: Multi-account isolation between application and shared services workloads, combined with Cilium network policies for pod-level segmentation, reduced the scope of potential security incidents and simplified compliance enforcement.
Accelerated onboarding: New application teams onboard to the North Star architecture in days rather than weeks, deploying into pre-configured namespaces with access to shared data services, CI/CD pipelines, and observability. They do this without provisioning or managing cluster infrastructure.
“Amazon EKS gave us the foundation we needed to establish our North Star architecture – a scalable, standardized infrastructure that lets our engineers focus on what matters most: delivering innovation for our customers.”
Conclusion
Equinix’s migration to Amazon EKS from a self-managed environment demonstrates how leading digital infrastructure companies are using managed services from AWS to eliminate undifferentiated operational work and redeploy engineering talent toward higher-value innovation.
The North Star architecture now serves as Equinix’s blueprint for scaling containerized workloads across additional business organizations and geographies. It is an architecture that grows with the company while maintaining the governance, security, and operational consistency that enterprise-scale infrastructure demands.
If you are managing complex, distributed infrastructure, the broader takeaway is this: when you consolidate operational ownership onto a well-architected infrastructure and remove the burden of cluster management from your application teams, you can achieve a markedly different pace of innovation.
With the advent of generative AI, organizations across industries face a common challenge: how do you move from the experimentation and ideation phase to production-ready workloads quickly and confidently? Many teams get stuck in a cycle of proofs of concept that never ship. DHI Group, a leader in talent acquisition services, was evaluating options to accelerate its generative AI adoption in an effort to roll out features at an accelerated pace. The traditional software development lifecycle (SDLC) approach involved months of requirements gathering, architecture reviews, and phased development that wouldn’t deliver the speed DHI needed. They needed a mechanism that would simultaneously validate technical feasibility, build organizational AI literacy, and produce shippable code.
In this post, explore how AWS partnered with DHI Group using a structured Hackathon Acceleration Package (HAP) to quickly generate production-grade artifacts, accelerate organizational AI confidence, and create a repeatable framework for innovation.
Hackathon Acceleration Package
In this section, review how DHI and AWS collaborated to plan and host hackathons to achieve the key business outcomes defined by DHI leadership. The entire process can be split into four phases:
Phase 1: Preparation
In the initial phase, the AWS team and DHI leadership collaborated to define the key outcomes the participants would work toward. The hackathon themes included:
Interpreting Job Descriptions Better: Enhancing the system’s parsing and presentation of job requirements.
Premium Candidate Experience: Defining what “Premium” means from the candidate’s perspective.
Onboarding That Sticks: Guiding new users through uncertainty to realize value sooner.
Candidate Engagement & Stickiness: Sustaining candidate engagement and return visits.
AgileATS Network: Streamlining the ClearanceJobs–AgileATS integration.
Streamlining Recruiter Experience: Reducing friction across the recruiter workflow.
Phase 2: Enablement
To support these outcomes, the AWS team curated and delivered training sessions and hands-on workshops covering generative AI concepts across Amazon Bedrock AgentCore and the AI-driven development lifecycle (AI-DLC). DHI has embraced Kiro as its productivity tool of choice, so AWS tailored the workshops around Kiro, giving participants prescriptive guidance on applying it across the full software development lifecycle.
Phase 3: Hackathon
The three-day hackathon was hosted by DHI at their headquarters in Des Moines, Iowa, and was attended by 20 DHI participants split across 3 teams. The key objective was to build a prototype that could then be accelerated to production. An AWS team of Solutions Architects (SAs) was present on-site to provide technical guidance to the participants. On the final day, a panel of judges comprising senior DHI leadership evaluated the teams to identify the winner. The three use cases the teams worked on:
Real-time Employer Analytics Dashboard: Addressing the Streamlining Recruiter Experience theme, this team built a real-time Employer Analytics Dashboard powered by Amazon Bedrock AgentCore and the Strands framework. The solution automates Quarterly Business Review (QBR) reporting for ClearanceJobs’ employer customers, replacing a manual process that currently demands 3+ QBRs per week across 250 customers.
Intelligent Candidate Matching: Addressing the Interpreting Job Descriptions Better and Premium Candidate Experience themes, this team built an intelligent candidate matching system with a real-time analytics dashboard. The solution combines Amazon OpenSearch Service for semantic search, Amazon Bedrock for matching intelligence, and Kiro for rapid frontend development.
ClearanceJobs MCP Server + AgileATS: Addressing the AgileATS Network and Streamlining Recruiter Experience themes, this team built a unified talent marketplace that connects ClearanceJobs and AgileATS through an agentic AI layer. By creating a single intelligent interface spanning both systems, the solution significantly boosts recruiter efficiency.
Phase 4: Path to production
DHI leadership was committed to advancing all three hackathon use cases to production, a strong signal of the value each prototype demonstrated. Building on the hackathon’s momentum, DHI and AWS aligned on a roadmap to harden each solution, address scalability and security requirements, and integrate them into DHI’s existing system.
In the next section, we focus on the winning hackathon use case, ClearanceJobs MCP Server + AgileATS, and dive deeper into the architecture.
ClearanceJobs MCP Server + AgileATS
Figure 1: High-level overview of the unified ClearanceJobs and AgileATS solution
The winning team’s solution represents a modern agentic AI architecture pattern that’s broadly applicable to organizations looking to unify disparate systems through intelligent automation. The architecture uses the Model Context Protocol (MCP) to expose system capabilities as tools that an AI agent can orchestrate.
Figure 2: Agentic architecture for the unified ClearanceJobs and AgileATS talent marketplace
How it works
The solution creates a unified recruiter experience by exposing ClearanceJobs capabilities through an MCP server, orchestrated by an intelligent agent built on Amazon Bedrock AgentCore. A separate ProfileLookup AWS Lambda function provides GitHub profile enrichment for candidates.
The problem it solves: Recruiters on ClearanceJobs currently lack an intelligent interface that can search candidates, retrieve profiles, and enrich them with external data such as GitHub profiles in a single conversational flow. This gap requires manual cross-referencing across systems.
The solution: The team built a single agentic interface where recruiters can issue natural-language commands, such as “Find top cleared software engineers with strong GitHub profiles and add them to my pipeline.” The agent handles the multi-step orchestration automatically, with session memory preserving context and preferences across interactions.
Architecture components
Amazon Bedrock AgentCore (orchestration layer) AgentCore provides the full agent infrastructure: Agent Runtime for session management and reasoning loops, Gateway (an MCP gateway with AWS Identity and Access Management (IAM) authentication and semantic search) for tool discovery and routing, and McpBearerToken for secure authentication to downstream MCP servers. An IAM role scopes the agent’s permissions.
MCP Server Lambda (tool layer) The ClearanceJobs MCP Server Lambda function, deployed in a private subnet within a virtual private cloud (VPC), exposes system capabilities as discrete tools:
search_candidates performs candidate search with clearance and skills filtering.
The Lambda function connects to the ClearanceJobs pilot environment through a NAT gateway with a WAF-allowlisted egress IP address, making sure only authorized traffic reaches the production APIs. Credentials and base URLs are stored in AWS Systems Manager Parameter Store.
ProfileLookup Lambda (external enrichment) A separate Lambda function (find_github_profile) enriches candidate data with external GitHub profiles, routed through an internet gateway to the GitHub Users API.
Foundation model (reasoning layer) Anthropic’s Claude 3.5 Haiku in Amazon Bedrock provides the agent’s reasoning capabilities. It interprets recruiter intent, decomposes complex requests into tool calls, and synthesizes results into actionable responses.
CJRecruiterAgent memory (context layer) AgentCore memory, a capability of Amazon Bedrock AgentCore, persists session state and recruiter preferences across conversations. This context lets the agent recall past searches, preferred candidates, and workflow patterns.
Security and networking The architecture spans two AWS accounts:
AgileATS account houses the AgentCore components, the foundation model, and a Bedrock Adapter Lambda function that provides an alternate MCP JSON-RPC path for classic Amazon Bedrock agent integration.
ClearanceJobs account houses the MCP Server and ProfileLookup Lambda functions within a VPC (with private and public subnets), a NAT gateway for controlled egress, and Amazon CloudWatch Logs for structured observability.
Communication between AgentCore and the ClearanceJobs account uses MCP over HTTPS with bearer authentication and custom headers for tenant identification.
Results
The hackathon delivered measurable outcomes across multiple dimensions:
Technical acceleration
Teams delivered functioning agentic AI features using Amazon Bedrock AgentCore and MCP servers in three days, compressing what would typically take more than three months.
The teams validated a production-ready architecture during the hackathon itself, which reduced post-event rework.
Kiro served as more than a coding assistant, driving both new code creation and deep analysis of existing systems to accelerate development velocity.
Organizational transformation
Kiro usage across product and engineering teams increased 84% following the hackathon, with more unique daily users each week and adoption continuing to grow.
33% of developers reported increased interest in the AI-enabled SDLC.
Delivery velocity rose across teams that fully adopted the AI-enabled software development lifecycle, marking a sustained step change rather than a short-term spike.
As the second successful hackathon with AWS, and with DHI leadership committing to make it an annual event, the engagement reflects a sustained, deepening partnership.
Kiro has become ClearanceJobs’ productivity tool of choice, with adoption expanding beyond developers to product managers. This accelerates product development and lets product managers self-serve on code base analysis and feature scoping.
“Participating for the second straight year as a judge, this hackathon only deepened my appreciation for the AWS team’s partnership, the ambition our teams brought, and what AI makes possible when you clear the runway. The problems they tackled were real, the solutions were creative, and the energy was contagious. It’s given us a fresh lens on how we build.”
– Alex Schildt, President of ClearanceJobs, DHI Group, Inc.
“Our second hackathon with AWS was even more successful than the first. We walked away with deeper confidence and more excitement about AI, all backed by hands-on experience with AWS’s latest capabilities. Post-hackathon, it’s been great to see our teams continue to lean into AI to accelerate how we ship. I think the hackathon was a real catalyst for that. I can’t wait to see these features get into the hands of our users.”
– Rose Fan, Sr. Director of Product, DHI Group, Inc.
Lessons learned: Making hackathons production-ready
Based on our experience hosting multiple hackathons with customers like DHI, here are key principles for hackathons that ship:
Set production-grade success criteria upfront: Prototypes must be sprint-ready, not only demo-ready.
Put decision-makers on the judging panel: Production go/no-go decisions happen on the final day of the hackathon, not weeks later.
Invest in pre-enablement: Workshops before the event mean teams build on day 1 instead of spending it learning.
Use cross-functional teams: Product, go-to-market (GTM), and subject matter experts (SMEs) alongside engineering make sure real business problems get solved.
Build relationships: On-site AWS presence helps build relationships that accelerate delivery long after the event.
Make it repeatable: DHI’s second hackathon planned faster and set higher expectations because the first one shipped to production.
Hackathons as a production accelerator
Hackathons are often dismissed as team-building exercises or limited to generating ideas that never ship. When structured correctly, they become a powerful production acceleration mechanism. Here’s why:
Time-boxed intensity drives decisions. A time-bound constraint (typically one to three days) forces teams to make architectural choices quickly, which alleviates analysis paralysis. Teams can’t over-engineer when the clock is ticking.
Cross-functional alignment happens naturally. When engineering, product, sales, and executives work side by side for several days, alignment that typically takes weeks of meetings happens organically.
Executive visibility de-risks production decisions: When leadership sees a working demo, not a slide deck, they can make go/no-go decisions with confidence. At DHI, the President and Head of Product & Engineering served as judges, giving them firsthand visibility into feasibility.
Real code beats theoretical architecture. Hackathon prototypes aren’t wireframes. They’re functioning applications built on production-grade services, making the path to production shorter and more predictable.
Conclusion
DHI Group’s experience across its annual hackathons shows that structured hackathons are one of the fastest paths from generative AI experimentation to deployed workloads. Their first hackathon shipped two features to production. Their second is on track to deliver three more, including an agentic AI system that unifies two systems through MCP servers and Amazon Bedrock AgentCore.
The takeaway is that hackathons aren’t only idea generators. They compress the entire innovation lifecycle (ideation, architecture, prototyping, executive alignment, and production planning) into a single high-intensity event. Paired with proper preparation and a clear path to production, they become a strategic tool for digital transformation and workforce enablement.
If your organization is looking to accelerate generative AI adoption, consider whether a structured hackathon could compress months of planning into days of building. To get started:
This post was co-written with Ankit Aggarwal and Raja Kalluri from United Airlines.
United Airlines processes billions of events daily across its data platform, which spans Amazon Redshift and Databricks with Unity Catalog. To bridge these platforms without duplicating data, the team turned to AWS Glue Data Catalog federation.
In this post, we walk through how to configure AWS Glue Data Catalog federation to connect with Databricks Unity Catalog, so you can run live SQL queries from Amazon Redshift without moving or duplicating data.
Why United Airlines needed catalog federation
United Airlines curates petabytes of data through a medallion architecture (bronze to silver to gold) on Amazon Simple Storage Service (Amazon S3). The airline user interaction data layer alone is several double-digit terabytes of near real-time streamed data. Teams use it to measure customer engagement patterns, feature adoption, and conversion behavior across web and mobile touchpoints. Analysts need to query this curated data through Amazon Redshift Serverless. As part of the existing data platform architecture these data tables are cataloged in Databricks Unity Catalog, not in the AWS Glue Data Catalog. As a result, Amazon Redshift has no native visibility into them. Without catalog federation, the only way to make this data queryable from Amazon Redshift would have been to duplicate it into Amazon Redshift Managed Storage (RMS) and build pipelines to keep it in sync.
AWS Glue Data Catalog federation removed this need. Amazon Redshift users now query the gold layer stored in Amazon S3 directly, with Iceberg metadata resolved from Unity Catalog at query time and no data movement. AWS Glue Data Catalog federation connects Amazon Redshift to external catalogs like Unity Catalog, so analysts query cross-platform data without building sync pipelines or duplicating storage.
Amazon Redshift Serverless is powered by the same Graviton-based query engine used in the new RG instance family, which delivers up to 2x faster data lake query performance compared to prior generations. This engine is purpose-built for reading Apache Iceberg tables directly from Amazon S3, making it well-suited for such federated query workloads.
United Airlines is taking a phased approach to adopting AWS Glue Data Catalog federation across its data platform. The initial focus is the most heavily used user interaction data tables, with 30 tables currently federated in production and 70 more in active rollout. Several hundred additional tables across different business domains are planned for production in the coming months.
Solution overview
AWS Glue Data Catalog federation bridges these platforms at the metadata layer. Here’s how the architecture works.
The architecture follows a four-layer federation chain:
Databricks Unity Catalog exposes tables through its Iceberg REST API endpoint. For Delta tables, you can turn on UniForm format to make them Iceberg compatible.
AWS Glue Data Catalog creates a federated catalog that connects to Databricks Unity Catalog, making metadata visible within AWS without data movement.
A resource link database in the default AWS Glue catalog acts as a bridge, pointing to the federated catalog database. This is required for Amazon Redshift compute.
Amazon Redshift Serverless references the resource link database through an external schema. When a query runs, Amazon Redshift traverses the link, calls AWS Glue Federation, and reads the Iceberg data through the Databricks Unity Catalog REST API. AWS Lake Formation governs permissions throughout this chain.
Key services or service features used in this solution:
Federation chain from Databricks Unity Catalog through AWS Glue and Lake Formation to Amazon Redshift Serverless
Figure 1: Federation chain from Databricks Unity Catalog to Amazon Redshift Serverless through AWS Glue and Lake Formation
The architecture follows a six-step flow:
A SQL analyst submits a query to Amazon Redshift Serverless.
Amazon Redshift resolves the external schema through the AWS Glue Data Catalog (resource link to federated catalog).
The AWS Glue federated catalog calls the Databricks Unity Catalog Iceberg REST API to retrieve current table metadata.
The namespace IAM role calls AWS Lake Formation GetDataAccess to obtain scoped, temporary S3 credentials.
Lake Formation evaluates fine-grained access policies and vends credentials for the authorized data files.
Amazon Redshift Serverless reads the Iceberg data files directly from S3 and returns results to the analyst.
Prerequisites
Before you begin, make sure the following are in place:
A Databricks workspace with Unity Catalog enabled and at least one catalog, schema, and table. Databricks uses UniForm to generate Iceberg metadata on Delta Lake tables on Amazon S3.
An AWS account with permissions to manage AWS Glue, AWS Lake Formation, Amazon Redshift Serverless, and IAM.
An Amazon Redshift Serverless workgroup and namespace already provisioned.
AWS Lake Formation set up with a data lake administrator.
AWS Command Line Interface (AWS CLI) configured with appropriate credentials.
Familiarity with Amazon Redshift Query Editor v2 or a SQL client.
The walkthrough is organized into six steps covering Lake Formation configuration, the resource link pattern, IAM role setup, and querying Databricks tables from Amazon Redshift.
Step 1: Configure AWS Lake Formation
1a. Add a data lake administrator
In Lake Formation, choose Administration, then choose Administrators and add your admin IAM user or role.
1b. Confirm the federated catalog is registered
Choose Data Catalog, then Catalogs and verify that databricks-federated-catalog is visible and registered.
Step 2: Create a resource link in the default AWS Glue catalog
This step is the key architectural detail in the walkthrough. Amazon Redshift resolves CREATE EXTERNAL SCHEMA only against the default AWS Glue Data Catalog. The federated catalog (databricks-federated-catalog) is a separate, non-default catalog object. To give Amazon Redshift a path to the federated data, you create a resource link database in the default catalog that points to the federated catalog’s database.
A resource link does not copy data or metadata. It’s a pointer that Lake Formation resolves at query time.
To create the resource link in the Lake Formation console:
Choose Data Catalog, Databases, Create database. Then select Resource link.
For Resource link name, enter databricks_federated_db_link.
For Target catalog, enter databricks-federated-catalog.
For Target database, enter the database name that was discovered by the AWS Glue crawler (for example, databricks_federated_db).
Step 3: Configure the Amazon Redshift Serverless namespace IAM role
When Amazon Redshift queries through the resource link, it uses the IAM role attached to the Amazon Redshift Serverless namespace to call the Lake Formation GetDataAccess API. Lake Formation permissions must be granted to this namespace role.
Choose one of these two approaches:
Option A – Update your existing namespace role by adding the following policy inline.
Option B – Create a new dedicated role (named RedshiftServerlessNamespaceRole) and attach it to the namespace alongside existing roles.
Note: The Resource: “*” in this policy is shown for simplicity. In production, scope resources to specific AWS Glue catalog ARNs, database ARNs, and table ARNs based on your use case.*
After creating or updating the role, associate it with your Amazon Redshift Serverless namespace:
In the Amazon Redshift Serverless console, choose Namespaces, select [your namespace], then choose Security and encryption, then Manage IAM roles.
If you use Option A, the existing role already has the new permissions, so no change is needed.
If you use Option B, add the new role alongside the existing roles.
Step 4: Grant Lake Formation permissions to the Amazon Redshift namespace role
4a. Grant DESCRIBE on the resource link database (default catalog)
In Lake Formation, choose Permissions,Data lake permissions, then Grant.
Principal: RedshiftServerlessNamespaceRole.
Resources: Named Data Catalog resources,Default catalog,databricks_federated_db_link (resouce link).
Database permissions: DESCRIBE.
4b. Grant SELECT and DESCRIBE on the target tables (Grant on Target)
Resource links permit only DESCRIBE and DROP permissions on the link itself. To allow Amazon Redshift to actually read data, you must separately grant SELECT on the target tables in the federated catalog. This is the Lake Formation Grant on Target pattern.
Principal: RedshiftServerlessNamespaceRole.
Resources: Named Data Catalog resources,databricks-federated-catalog,databricks_federated_db, then Tables.
Table permissions: SELECT, DESCRIBE.
Catalog permission: DESCRIBE.
Important: SELECT must be granted on the TARGET tables in the federated catalog, not on the resource link. Granting SELECT only on the resource link won’t work. This is a common configuration error.
Step 5: Create an external schema in Amazon Redshift
With the resource link in place and permissions granted, you can now create an external schema in Amazon Redshift that points to the resource link database. The external schema is the query interface. When a user runs SQL against it, Amazon Redshift traverses the link to the federated catalog and retrieves metadata and data from Databricks Unity Catalog.
The DATABASE parameter must reference the resource link database name in the default AWS Glue catalog (databricks_federated_db_link), not the federated catalog name directly. The CATALOG_ARN parameter isn’t required here because the resource link lives in the default catalog and Amazon Redshift resolves it automatically.
Connect to your Amazon Redshift cluster as a superuser (for example, using Amazon Redshift Query Editor v2) and run:
CREATE EXTERNAL SCHEMA databricks_schema
FROM DATA CATALOG
DATABASE 'databricks_federated_db_link'
IAM_ROLE '<iam-role-arn>'
REGION '<region>';
A key design principle in this architecture is the clear separation between data physically stored in Amazon Redshift and data accessed externally through federation. External schemas provide a transparent abstraction layer, so Amazon Redshift users can query data stored in S3 without ingestion. For consistency and clarity, United Airlines follows a standard naming convention for all federated schemas in Amazon Redshift: {domain}_iceberg. This convention makes it immediately clear that the data isn’t natively stored within Amazon Redshift but is accessed by using federation through AWS Glue and Lake Formation. This distinction is critical for analysts and engineers, because it improves discoverability, avoids ambiguity between storage layers, and reinforces architectural discipline when working across hybrid data environments.
The User Interactions domain exposes curated datasets representing customer interaction activity, engagement behavior, and channel usage patterns. Operational datasets follow the same pattern, providing governed access to supporting business events and reference information through a common federation framework.
You create a view layer over each external schema using WITH NO SCHEMA BINDING, so that analysts always resolve the freshest schema on each query execution. For example:
CREATE VIEW analytics.clickstream_events AS
SELECT * FROM {domain}_iceberg.interaction_events
WITH NO SCHEMA BINDING;
Step 6: Verify and query Databricks tables from Amazon Redshift
After creating the external schema, verify that the Databricks tables are visible and run a test query.
Verify table visibility
-- Confirm federated tables are visible in Redshift
SELECT * FROM SVV_EXTERNAL_TABLES
WHERE schemaname = 'databricks_schema';
Query a Databricks Unity Catalog table
-- Query a Databricks Unity Catalog table via the federated catalog
SELECT *
FROM databricks_schema.<table_name>
LIMIT 10;
When a query runs, Amazon Redshift calls Lake Formation GetDataAccess using the namespace IAM role to obtain temporary credentials. It then contacts the AWS Glue federated catalog, which in turn calls the Databricks Unity Catalog Iceberg REST API to retrieve metadata and read table data. The result is returned to the Amazon Redshift user transparently.
For SAML-authenticated users, connect using your IdP JDBC plugin:
The Amazon Redshift JDBC driver handles authentication automatically. It authenticates with your IdP, receives a SAML assertion, and calls sts:AssumeRoleWithSAML for temporary IAM credentials. It then calls redshift-serverless:GetCredentials to connect as the mapped database user.
Business impact
AWS Glue Data Catalog federation delivered measurable architectural and operational improvements for United Airlines:
Area
Before
After
Impact
Data access
Delta Lake and Amazon Redshift data were completely siloed, so Amazon Redshift users had no access to curated datasets on Databricks-managed S3 data
Amazon Redshift users get real-time access to Databricks-managed data through AWS Glue Data Catalog federation
~100 analysts gained access to user interaction data tables in the first phase without adding new pipelines.
Disaster recovery
Cross-Region DR relied on Amazon Redshift snapshots every 3 hours (recovery point objective, or RPO, of 3 hours or more)
Amazon S3 cross-Region replication on the Delta Lake provides a near-continuous RPO. A new Amazon Redshift Serverless workgroup in the DR Region can federate to the same S3 data
More resilient architecture. Reduces cost for Amazon Redshift snapshot and copy maintenance across Regions
Architecture simplification
Data processing happened in both Databricks and Amazon Redshift, requiring manual catalog synchronization between the two platforms which was operationally expensive and prone to drift
With the federated architecture, data processing is consolidated in Databricks, and Amazon Redshift acts solely as a query engine powering user queries and dashboards through catalog federation
Single processing platform, zero sync pipelines, single source of truth
Infrastructure cost
Running dedicated Amazon Redshift ETL cluster with RMS storage, snapshots, and compute for data processing
For this use case with federation, Amazon Redshift is not needed for ETL but only as a query engine. No RMS storage duplication, no snapshot replication required
~$30K/month in redundant ETL infrastructure cost reduced
Security considerations
At United Airlines, identity governance is unified through Azure Active Directory groups. On the AWS consumption side, users authenticate to Amazon Redshift Serverless through SAML federation. AD group membership determines database-level access to federated schemas. On the Databricks side, the same AD groups govern access to Unity Catalog schemas. This single-identity model provides consistent access control across both platforms without requiring separate user provisioning. Lake Formation handles credential vending for S3 data access during federated queries, while schema-level access decisions are managed through the AD group mappings on each platform.
The architecture also provides multiple layers of security controls built into the federation chain:
AWS Lake Formation governs fine-grained access control throughout the federation chain, so that principals can only access authorized databases, tables, and columns.
IAM roles follow least-privilege principles. The Amazon Redshift namespace role is scoped only to AWS Glue metadata operations and Lake Formation GetDataAccess.
SAML-based authentication integrates enterprise identity providers, so that users authenticate through existing SSO infrastructure before accessing federated data.
All Amazon Redshift connections enforce TLS encryption (ssl=true), protecting data in transit between clients and the Amazon Redshift endpoint.
Lake Formation permission vending issues short-lived, scoped credentials for each query execution rather than long-lived static credentials.
Delta Lake tables must have UniForm enabled to expose Iceberg-compatible metadata.
We recommend that source tables be well-partitioned and regularly compacted, because the federated query performance reflects how efficiently the data is organized at write time.
Clean up
To avoid ongoing charges for resources created in this walkthrough, remove them in the following order. This teardown doesn’t affect Databricks metadata or your underlying data stored in Amazon S3.
Drop the external schema in Amazon Redshift: DROP SCHEMA databricks_schema;.
Delete the resource link database in the default AWS Glue catalog (databricks_federated_db_link).
Revoke Lake Formation permissions granted to the Amazon Redshift namespace role on both the resource link database and the target tables in the federated catalog.
Delete the federated catalog in AWS Glue (databricks-federated-catalog).
Deregister the AWS Glue connection for the Databricks Unity Catalog if no longer needed.
Optionally, remove the IAM role (RedshiftServerlessNamespaceRole) if it was created solely for this walkthrough.
Conclusion
In this post, we showed how United Airlines uses AWS Glue Data Catalog federation to give Amazon Redshift Serverless analysts real-time access to double-digit terabytes of curated user interaction data on Amazon S3, without duplicating a single byte or building sync pipelines.
The architecture uses the Iceberg REST API, resource link databases, and Lake Formation credential vending to create a governed query path between Amazon Redshift and Unity Catalog. For United Airlines, this eliminated redundant ETL infrastructure costs, removed the need for catalog synchronization, and turned Amazon Redshift Serverless into a dedicated high-performance query engine for analysts and dashboards.
For questions or feedback, leave a comment on this post.
This guest post was co-written with Mukund Acharya from Sony LIV.
Real-time data analytics is transforming how streaming applications understand and serve their audiences. In this post, we share how Sony LIV used Amazon Kinesis Data Streams for sub-second processing and AWS services to build a comprehensive streaming analytics solution on AWS. In this post, we share how SonyLIV built a comprehensive streaming analytics solution on AWS using Amazon Kinesis Data Streams for sub-second event ingestion, Amazon Data Firehose for reliable delivery to storage, Amazon EMR with Apache Spark for scalable batch and micro-batch processing, and Apache Iceberg on Amazon S3 for ACID-compliant, queryable data lake tables. Together, these services enable SonyLIV to capture millions of concurrent viewer events, process them cost-efficiently at scale, and surface actionable insights from real-time engagement metrics to historical trend analysis — all within a fully managed, serverless-friendly architecture.
About Sony LIV
Sony LIV, operated by Sony Pictures Networks India, is one of India’s leading over-the-top (OTT) streaming applications offering premium content including live sports, original series, movies, and TV shows to millions of users across mobile, web, and Smart TV devices.
As part of the latest game broadcast rights for Asia Cup, Sony LIV built a streaming analytics solution using AWS services that successfully processed millions of concurrent sessions in real-time.
The challenge of batch processing
As Sony LIV’s audience grew and live sports events attracted increasingly large viewership, the team identified an opportunity to move from batch-based analytics to a real-time data application. The existing architecture processed data reliably in scheduled batches, but the growing scale of live events called for faster, more granular insights. The team set out to address three key areas:
Real-Time Visibility: Track frontend user events, journeys, and conversion funnels in near real-time, especially during high-stakes live sports streams, to see peak concurrent viewership and respond to engagement patterns as they happen.
Comprehensive Quality Monitoring: Implement end-to-end monitoring of video quality key performance indicators (KPIs) including buffering rates, playback failures, video start time, and rebuffering events to proactively enhance the viewer experience.
Unified Data Architecture: Consolidate data from multiple sources into a unified architecture to enable comprehensive customer views, supporting future personalization and machine learning (ML)-driven recommendations.
Sony LIV needed to evolve from reactive batch processing to a real-time data application to deliver actionable engagement insights at scale and establish the foundation for unified customer profiles.
Streaming analytics architecture
Figure 1: Sony LIV streaming analytics architecture on AWS
Data from mobile, web, and Smart TV applications flows through Application Load Balancers to Amazon Elastic Kubernetes Service (Amazon EKS) pods for validation and preprocessing. The architecture implements parallel processing paths to balance speed and cost efficiency.
Real Time Path: Priority events requiring immediate action such as playback failures, user interactions, and live sports engagement signals stream through Amazon Kinesis Data Streams for sub-second processing. These events flow directly to ClickHouse using the ClickHouse connector, enabling real-time analytics with minimal latency.
Batch Path: High-volume batch events are ingested via Amazon Data Firehose into a raw landing zone on Amazon Simple Storage Service (Amazon S3), partitioned by event type and time. Amazon EMR running Apache Spark then processes this raw data performing schema validation, deduplication, and transformations — and writes the curated output as Apache Iceberg tables back to S3
Unified Data Layer: AWS Glue catalogs data across streaming and batch sources, creating a unified schema that powers the customer data application. This unified data layer serves as the foundation for building comprehensive customer profiles and enabling personalized recommendations.
Analytics and Monitoring: ClickHouse serves as the low-latency analytics engine, powering real-time dashboards and video quality monitoring. Operations teams track critical KPIs including peak concurrent viewership, buffering rates, video start time, and playback failures, enabling rapid response to issues during live events.
Custom dashboards and Datadog provide visualization for business metrics and application performance monitoring, while Amazon CloudWatch tracks infrastructure health, delivering end-to-end visibility across the application.
Results and business impact
The results: product, analytics and engineering teams now operate from a single unified dashboard, making real-time decisions with data that is seconds, not hours old. Auto scaling and serverless design have optimized costs while providing reliability at peak load. At the foundation, a centralized data catalog now serves as a single source of truth for business intelligence, machine learning workloads and operational monitoring.
The transformation helped Sony LIV process millions of concurrent sessions in real time, particularly during high-profile events such as the Asia Cup 2025.
Accelerated insights: Data latency dropped from hours to seconds, enabling instant insights into viewer behavior, content performance, and streaming health.
Unified decision-making: Product, analytics, and engineering teams now use unified dashboards for real-time decision-making.
Optimized operations: The solution auto scaling and serverless design optimized costs while supporting reliability and fault tolerance.
Beyond performance gains, the new architecture established a strong foundation for personalized content recommendation and operational agility. By integrating a centralized data catalog and scalable analytics layer, the solution now provides a single source of truth for business intelligence, machine learning workloads, and monitoring.
Future innovations
Sony LIV plans to expand its analytics capabilities by integrating advanced ML models for real-time recommendations, user churn prediction, and anomaly detection. The team will also focus on building unified customer profiles to enable hyper-personalized experiences.
The AWS based architecture provides a strong foundation for future growth and innovation, enabling Sony LIV to deliver highly personalized experiences to an expanding viewer base.
Conclusion
By adopting AWS, Sony LIV transformed its analytics architecture into a real-time, scalable, and insight-driven solution. The solution reduced data latency from hours to seconds, enabled processing of millions of concurrent sessions during peak events, and positioned Sony LIV as a leader in streaming analytics innovation in India.
To learn more about how AWS can help your media organization implement real-time streaming analytics solutions, explore the following resources:
AWS for Media & Entertainment – Discover how AWS powers content delivery, streaming, and analytics for media companies worldwide.
Amazon Kinesis Data Streams – Learn about the fully managed service for real-time data streaming used in this solution.
Moovit, part of Mobileye (Nasdaq: MBLY), is a leading Mobility-as-a-Service (MaaS) solutions provider and the creator of a leading urban mobility app. Moovit’s iOS, Android, and web apps offer users a smart mobility experience to get to their destination using any mode of public and shared transportation. Transit riders can benefit from mobile ticketing to plan, pay, and ride with transit services. Introduced in 2012, Moovit now serves over 1.7 billion users in more than 3,500 cities across 112 countries, in 45 languages.
Behind these user-facing experiences is a data platform that processes large volumes of mobility, application, and operational data to support product analytics, business intelligence (BI), monitoring, and data science. As the platform grew, Moovit needed to keep analytical workloads reliable and cost-efficient without slowing down teams that depend on fresh data every day.
Over several years, Moovit’s Amazon Redshift cluster grew continuously. It started with an expanding fleet of DC2 nodes, migrated to RA3 nodes, and scaled multiple times to keep pace with growing data demands, ultimately becoming the backbone of their entire data platform.
To address this growth, Moovit transformed their data architecture by building an optimal multi-engine lakehouse architecture and assigning each workload to the most suitable option. This modernization reduced their Amazon Redshift cluster by 50 percent, while establishing a flexible, multi-engine architecture ready for future use cases.
In this post, we share how Moovit gained visibility into workload patterns, cleaned up unnecessary load, selected candidates for offloading, and ran a successful proof of concept (POC) on Amazon EMR Serverless. Moovit ultimately divided the workload between multiple engines, building a modern and cost-optimized data platform that combines provisioned Amazon Redshift, Amazon Redshift Serverless, and Amazon EMR.
The challenge: Outgrowing a single-engine data platform
The Amazon Redshift engine handled a wide variety of workloads, including:
Heavy ETL processing: Raw data ingestion from Amazon Simple Storage Service (Amazon S3) followed by complex aggregation pipelines (daily user-aggregation running once per day with a 3-day lookback, and weekly 10-day-lookback jobs).
Near-real-time operational monitoring: Queries executing every 20 minutes against raw data for system-health dashboards.
Business-intelligence reporting: Tableau extracts and live dashboards.
Data-science workloads: Exploratory analysis and model-feature engineering.
Ad-hoc analysis: Non-recurring queries done by analysts and engineers.
With business growth, storage grew by orders of magnitude over the past decade as the platform expanded. All these varied workloads competed for the same engine and pushed it to its limits. Jobs experienced increasing queue times, service level agreements (SLAs) were at risk, and adding nodes provided minimal performance gains, creating a need to isolate workloads.
Gaining visibility: Measuring workload impact
Moovit’s first modernization milestone was to create a trusted measurement foundation before changing any workloads. Instead of treating warehouse activity as a single opaque stream, the team implemented automated query attribution that continuously classified each query by workload owner and execution context. The classification combined multiple signals: who executed the query (user or service account), recognizable query-signature patterns, and metadata emitted by orchestration frameworks and scheduled processes.
This produced a historical, query-level map of platform usage that answered three critical questions: who is generating load, whatkind of workload is running, and how expensive each workload is in runtime and resource terms. With that baseline in place, the team made offload decisions from evidence rather than assumptions. This approach prioritized the largest and most stable optimization opportunities first and reduced the risk of moving business-critical workloads without visibility.
These classifications and workload metrics were reflected in a Tableau report that aggregated query activity by classification label and execution context. The view exposed operational dimensions such as classification, time granularity, service class, execution-time bucket, unload flags, and sample-query context, supporting both trend monitoring and root-cause drill-down.
The worksheet was parameterized to support multiple measurement modes over the same grouped workload population: total execution time, execution plus queue time, total CPU time, average execution time per query, and ratio-based efficiency views (execution/CPU and CPU/execution). This let the team compare “heavy by volume” workloads against “inefficient by behavior” workloads without creating separate artifacts.
For decision-making, CPU time was used as the primary impact metric because it best represented sustained compute pressure. Execution time, queue time, query-count normalization, and workload-management segmentation were treated as secondary evidence to distinguish:
compute-heavy but healthy workloads
queue-constrained workloads
high-frequency/low-cost workloads
noisy or weakly classified workloads that required attribution cleanup first
Using this framework, prioritization became systematic: first improve classification coverage, then rank workloads by CPU contribution, then validate with queue and workload management (WLM) signals, and finally choose the action path per workload (optimize SQL, reschedule, isolate, retire, or move to another engine).
The following figure shows an example of one of the dashboard widgets (CPU time by query).
Figure 1: CPU time by query, highlighting the most resource-intensive queries and their usage patterns
Cleanup: Reducing unnecessary data warehouse load
With a long-running data platform, in most cases the workloads will start accumulating, some of which become irrelevant at some point. For example, a report which was created and scheduled, yet it became irrelevant after a few years, but still running since no one disabled it. It’s important to indicate these workloads in general to reduce unnecessary load, yet even more critical before doing any significant architectural changes or migrations. Before migrating any workloads, Moovit first reduced unnecessary warehouse load.
The team:
Removed unused processes that were still consuming cluster resources.
Reduced unnecessary frequency where possible: some jobs ran more often than downstream consumers needed.
Reviewed workload-management guardrails to verify resource allocation matched actual priorities.
This cleanup phase was a prerequisite to migration. By removing waste first, the team verified that the workloads eventually selected for offloading were genuinely heavy rather than simply unoptimized or unnecessary.
The no-longer-relevant processes consumed around 7 percent of overall CPU time and were removed before the optimization work began.
Workload selection: Choosing what to offload
With a clear picture of workload patterns, Moovit faced a common decision point: continue scaling the existing Redshift cluster, or re-architect towards a multi-engine approach. The team evaluated two main paths:
Re-architect with Redshift multi-cluster and data sharing: Identify workloads that could benefit from resource isolation, then redistribute processing and queries between multiple Redshift clusters, combining both serverless and provisioned options. This would redistribute load across use-case-optimized clusters and potentially save costs through better resource use.
Re-architect with purpose-built engines: Identify workloads that could benefit from alternative processing frameworks and offload them to more suitable engines. This would reduce pressure on Amazon Redshift while building a more flexible, cost-efficient architecture.
Moovit decided to do both, because while some workloads benefited from being offloaded, others benefited from isolated Amazon Redshift compute.
The measurement data revealed a primary candidate for offloading: raw-data aggregation pipelines. This workload loaded raw data into Amazon Redshift from Amazon S3, then performed heavy sessionization and aggregation transformations. Raw tables were still used for ad-hoc and exploratory analysis, but recurring production consumers primarily depended on aggregated outputs, making these transformations strong candidates for offloading.
Proof of concept: Offloading to EMR Serverless with Spark SQL
With target workload identified, Moovit initiated a POC using Amazon EMR Serverless with Spark SQL. The choice of EMR Serverless was driven by several factors:
Spark SQL compatibility: The existing Redshift SQL logic could be ported with minimal changes to Spark SQL syntax.
Serverless simplicity: No cluster-management overhead during the evaluation phase.
Data-lake native: Processing could occur directly on data in Amazon S3.
The POC defined quantified success criteria measured over five or more consecutive runs:
Runtime reduction: Greater than or equal to 40 percent reduction for the transform portion of selected pipelines.
Amazon Redshift cost reduction: Greater than 30 percent reduction in Redshift RA3 compute with no performance degradation for remaining workloads.
Data-quality parity: Exact match between Spark and Amazon Redshift outputs on row counts, distinct users, and all published metrics over a frozen parity window.
Overcoming initial performance challenges
The first POC attempts exposed significant challenges. Early Spark jobs with 100 executors took approximately 4 hours, far exceeding the 30–40-minute baseline on Amazon Redshift. Beyond raw performance, the team encountered memory pressure, data-parity gaps between Spark and Amazon Redshift outputs, and subtle SQL behavior differences between the two engines.
The team systematically diagnosed and resolved these issues:
Execution-plan analysis: Reviewing the Spark execution plan revealed suboptimal query patterns that generated excessive data shuffles.
Query rewrites: Rewriting specific SQL constructs to align with Spark’s distributed processing model, including splitting large monolithic logic into staged transformations.
Reducing or rewriting expensive DISTINCT patterns: Identifying and eliminating unnecessary DISTINCT operations that created heavy shuffle pressure.
After applying these optimizations, execution time dropped from 4 hours to approximately 10 minutes, and the required executors dropped to fewer than 50, surpassing the original performance.
Validation: Ensuring data parity before cutover
Before transitioning any workload to production, Moovit implemented a rigorous validation process. The new Spark output was compared with the previous Amazon Redshift output using multiple dimensions:
Row counts: ensuring no data was lost or duplicated.
Metric parity: all published business metrics matched.
Daily trends: time-series patterns remained consistent.
Row-level checks: spot-checking individual records for correctness.
Only after all validation checks passed consistently over multiple consecutive runs did the team proceed with cutover for each workload.
Moving to production: Expanding workload offloading
With a successful POC demonstrating both performance gains and cost savings, Moovit progressively moved additional workloads from Amazon Redshift to EMR:
Heavy-aggregation jobs: The primary daily and weekly aggregation pipelines transitioned fully to EMR.
Data-transformation stages: Preprocessing steps that previously consumed Redshift compute moved to Spark, with only final aggregated results loaded back into Amazon Redshift for BI consumption.
Weekly batch workloads: Large batch jobs that previously created resource contention during weekend processing windows.
The transition used a measured approach: each workload was migrated individually, with data-quality validation confirming parity before decommissioning the equivalent jobs which were running on Redshift.
Additional optimizations: Redshift Serverless, workload isolation, and Amazon EMR on Amazon EC2
Beyond EMR offloading, Moovit implemented further architectural improvements to isolate workloads and optimize costs.
With heavy workloads successfully offloaded and isolated, Moovit proceeded to right-size the Redshift cluster. Rather than a single resize, the team reduced the cluster incrementally, two nodes at a time, using elastic resize. At each step, they validated that:
Existing BI workloads maintained acceptable performance.
Queue wait times remained within SLA thresholds.
No workload degradation was observed under peak loads.
This iterative approach minimized risk and allowed the team to find the optimal cluster size with confidence.
Workload isolation with Redshift Serverless
Amazon Redshift persisted as the engine of choice for serving curated BI data. However, not all Amazon Redshift workloads needed provisioned capacity:
Ad-hoc analyst queries: Moved to Redshift Serverless, isolating unpredictable workloads from the provisioned cluster through data sharing.
Data-science workloads: Transitioned to Redshift Serverless for flexible exploration without impacting production.
This workload isolation through Redshift Serverless provided resource separation without requiring additional provisioned capacity. The architecture now used data sharing to provide a unified view across provisioned and serverless clusters.
Operational isolation refinements
Moovit also refined workload isolation by rebalancing WLM priorities on the provisioned cluster. Because the ETL queue mainly handled raw data loading from Amazon S3 (which was not the bottleneck after heavy aggregations moved to Spark), its priority was reduced. At the same time, with most human users moved to Redshift Serverless, Tableau serving workloads on provisioned Redshift were prioritized higher to keep dashboard performance predictable. The final result: a 50% reduction in provisioned Redshift capacity.
Transitioning to EMR on EC2
EMR Serverless proved efficient for the POC phase: it allowed fast iteration without cluster management overhead. However, for longer-term recurring production workloads, Moovit moved to EMR on EC2 to better fit their production cost and infrastructure model, using existing compute reservations.
The transition between EMR deployment options required zero application code changes, demonstrating the flexibility of the EMR deployment options.
AI-assisted SQL translation
Additionally, Moovit used AI-assisted development tools, Claude Code and Cursor, to accelerate parts of the SQL transition process. These tools helped engineers identify Redshift SQL and Spark SQL syntax differences, suggest rewrites, and debug migration issues, while validation and production approval remained under engineer review.
Results: A modern multi-engine architecture
The architectural modernization delivered measurable outcomes:
Cluster size reduction: Redshift cluster size reduced to 50 percent of the initial capacity.
Performance improvement: Key aggregation jobs ran faster and more consistently on EMR (50 percent execution time reduction for p90).
Workload isolation: No single workload type could impact others through resource contention.
33 percent overall data pipeline cost reduction: Combined savings from cluster reduction, transition to EMR, and efficient serverless usage.
Future flexibility: The multi-engine architecture provided pathways for additional use cases without architectural changes.
The following figures compare aggregation-job performance before and after the transition.
Figure 2: Aggregation-job execution times before and after the transition
Figure 3: Wall-clock time for job executions by percentile, before and after the transition
The resulting architecture assigned each workload to the engine that fits it best:
Workload type
Engine
Rationale
Heavy ETL and aggregation
Amazon EMR (Spark SQL)
Distributed processing on Amazon S3. No data warehouse load required
Ongoing processing and BI reporting
Amazon Redshift provisioned
24/7 running processes
Ad-hoc queries
Amazon Redshift Serverless
Burst capacity with workload isolation
Data science
Amazon Redshift Serverless
Flexible exploration without impacting production
Lessons learned
The Moovit modernization journey produced several key insights applicable to similar architectural transitions:
Measure before you move: Establishing baseline metrics and automated classification was essential for identifying true offloading candidates. Without granular workload-level measurements, the team would not have identified which specific processes were exhausting the cluster.
Clean up before you migrate: Reducing unnecessary load first verified that migration efforts targeted genuinely heavy workloads rather than simply unoptimized or unused processes.
Small SQL changes, big impact: Moving from Redshift SQL to Spark SQL required relatively minor syntax adjustments. The core business logic remained intact, and most transformations translated directly with minimal refactoring.
Optimize for the engine: Porting SQL queries to Spark without optimization produced initially poor results for some workloads. Understanding Spark’s distributed execution model and optimizing for it was critical for achieving target performance.
Validate rigorously: Multi-dimensional data-parity checks (row counts, distinct users, metrics, daily trends, and row-level spot checks) gave the team confidence to cut over without data-quality regressions.
Moving between EMR options is straightforward: EMR Serverless proved very efficient for starting fast and evaluating Spark. When Moovit needed to move to EMR on EC2 to use existing reservations, the transition required no application code changes.
Iterative cluster rightsizing: Rather than a single resize, Moovit reduced the Redshift cluster incrementally (two nodes at a time) using elastic resize, validating performance at each step before proceeding further.
Conclusion
Looking ahead, as another potential optimization, Moovit will be evaluating the new Amazon Redshift RG instances for provisioned clusters, providing up to 2.2x better price performance and priced 30% lower than RA3, powered by AWS Graviton.
The broader takeaway is that AWS provides multiple purpose-built engines that can be used in a single data platform. In Moovit’s case, the biggest improvement came from assigning each workload to the engine that fit it best: Amazon Redshift for curated analytical serving, Redshift Serverless for isolated exploratory workloads, and Amazon EMR for large-scale transformations over data in Amazon S3. This architecture gives Moovit a foundation for future optimization and flexibility as data volumes grow and new analytical use cases emerge.
Razor Group is one of Europe’s leading ecommerce aggregators, operating 250+ brands across multiple global marketplaces. With a portfolio exceeding $400M in revenue, the company relies on data to power every critical business decision, from dynamic pricing and inventory optimization to advertising spend and supply chain orchestration.
At the heart of this operation sits the Razor Operating System (ROS), a proprietary platform that processes 370M+ API calls monthly through 9,300+ data pipelines, transforming marketplace signals into automated actions at scale.
In this post, we share how Razor Group optimized their data platform by implementing a lakehouse architecture on AWS. We cover the architectural decisions, the phased migration approach, and the measurable business outcomes. Whether you’re looking to optimize workload performance, reduce infrastructure costs, or unlock multi-engine flexibility for your analytics, this blueprint provides actionable insights you can adapt for your organization.
The business challenge: Scaling data infrastructure for hypergrowth
As Razor Group’s brand portfolio expanded rapidly, the demands on their data platform grew significantly. The company needed their analytics infrastructure to keep pace with the speed of ecommerce, where pricing decisions, stock replenishment, and advertising bids happen in near real time.
Their existing architecture, built on Amazon Redshift provisioned clusters, had served them well during earlier growth stages. As workloads diversified and data volumes surged, several optimization opportunities emerged:
Figure 1: The Razor Operating System data architecture before the migration
Workload contention: Over 1,000 SQL models for ETL, transformation, and analytics competed for the same compute resources, creating resource contention during peak processing windows.
Cost-to-utilization mismatch: Always-on clusters ran 24/7, but workload analysis revealed that 98% of compute demand came from batch ETL rather than interactive analytics, which resulted in significant idle capacity during off-peak hours.
Data freshness gaps: Batch-oriented pipelines delivered data with 4–6 hour latency, limiting the team’s ability to react to fast-moving marketplace dynamics.
Scaling constraints: As concurrent users and pipeline complexity grew, vertical scaling alone couldn’t address the need for workload isolation and elastic capacity.
These weren’t failures of any single service. They were signals that the architecture needed to evolve to match the scale and diversity of Razor Group’s workloads.
Why a lakehouse architecture?
Rather than replacing their existing investments, Razor Group recognized the opportunity to optimize workload placement by adopting a modern lakehouse architecture. The core principles driving this decision:
Open table formats: Apache Iceberg provides ACID transactions, time travel, and schema evolution. Data is stored once and accessed by any compatible engine without duplication.
Elastic, per-workload scaling: With data persisted on Amazon Simple Storage Service (Amazon S3), each engine independently scales compute to match its workload. Each engine spins up for peak processing and scales to zero when idle, without over-provisioning shared infrastructure.
Multi-engine flexibility: Different workloads have different requirements. Heavy ETL benefits from distributed Spark processing, ad hoc exploration from serverless queries, and business intelligence (BI) dashboards from high-performance warehouse engines, each optimized for its purpose.
This approach allowed Razor Group to right-size each workload to the best-fit engine while maintaining a single, governed copy of data accessible across the entire platform.
Solution overview
Razor Group partnered with AWS to implement a comprehensive lakehouse architecture that brings together multiple AWS services, each playing a complementary role:
Figure 2: End-to-end lakehouse architecture on AWS
Designing for scale: The lakehouse vision
The core insight driving Razor Group’s new architecture was simple: build a single, open format data lake that any engine can query. In the old model, each tool maintained its own copy of the data. In the new model, a single open-format data lake on Amazon S3 serves as the source of truth, and multiple purpose-built compute engines read from it based on the workload at hand.
This shift, commonly called a lakehouse architecture, combines the cost economics and scalability of a data lake with the query performance and governance of a data warehouse. Its open table format, Apache Iceberg, provides ACID transactions, schema evolution, time travel, and no vendor lock-in.
Storage and governance: The open data foundation
Amazon S3 Tables (a capability of Amazon S3) with Apache Iceberg — The primary storage layer, providing open-format tables with ACID transactions, partition evolution, and time travel. Data is stored once and accessible by any Iceberg-compatible engine.
AWS Glue Data Catalog — A unified metadata repository for consistent data discovery across all compute engines.
AWS Lake Formation — Fine-grained access control with column-level and row-level security so that governance scales with the platform.
Compute: Right engine for the right workload
Apache Spark on Amazon Elastic Compute Cloud (Amazon EC2) — Elastic, distributed compute for heavy ETL and transformation workloads. It uses AWS Graviton instances and Amazon EC2 Spot Instances for cost optimization.
Amazon Athena — Serverless SQL for ad hoc exploration and lightweight queries directly on Iceberg tables, with no infrastructure to manage.
Amazon Redshift Serverless — High-performance serving layer for BI dashboards, Tableau workloads, and interactive analytics. Amazon Redshift Serverless automatically scales to meet demand and pauses when idle, so it stays cost-efficient for the analytics workloads it serves best.
Orchestration and observability
Apache Airflow — Pipeline orchestration that manages 9,300+ data pipelines with dependency tracking and service level agreement (SLA) monitoring.
Comprehensive observability stack — Cost attribution, pipeline health monitoring, and data quality checks across all layers.
Note: When the architecture was originally designed, Amazon Redshift lacked Iceberg write support, making self-managed Spark the only viable ingestion path. This constraint has since been removed. Amazon Redshift now supports full Apache Iceberg DML (UPDATE, DELETE, MERGE), complementing its earlier CREATE/INSERT capabilities and AWS Glue Iceberg materialized views. This makes it a complete read/write Iceberg engine.
Migration approach
Rather than a risky big-bang cutover, Razor Group adopted a phased migration of five stages, each delivering standalone value while building the foundation for the next. Both Amazon Redshift and Spark pipelines ran in parallel during the transition, which maintained business continuity and let the team compare outputs with confidence. At no point was a production pipeline paused or a dashboard unavailable.
The migration journey: Five phases
The migration unfolded across five structured phases, each building on the previous one and delivering incremental value before the next began.
Phase 1: Establish the lakehouse foundation
Before migrating a single query, Razor Group needed to answer three questions: where does the data live, how is it managed, and how do we query it?
Why S3 Tables over self-managed Iceberg
Razor Group had already committed to Apache Iceberg as the table format: open, engine-agnostic, and equipped with ACID transactions and time travel. The question was whether to self-manage Iceberg on standard S3 buckets or use Amazon S3 Tables.
Self-managed Iceberg is powerful but operationally expensive. Someone has to run compaction jobs to prevent small-file proliferation. Someone has to expire old snapshots before metadata bloat degrades query planning. Someone has to clean up orphaned data files after interrupted writes. With 700+ models running across 40+ schemas, many of them materializing multiple times per day, that maintenance burden would scale with the platform rather than shrink.
S3 Tables eliminated this entire category of work. Compaction, snapshot management, and unreferenced file removal run continuously and automatically. The integrated Iceberg REST Catalog API means any compatible engine, such as Spark, Trino, Athena, Amazon Redshift, and Flink, can discover and query tables without maintaining a separate metastore. Discovery is unified through AWS Glue Data Catalog, which now exposes the Iceberg REST Catalog protocol as its access interface. Because tables are first-class AWS resources, access control, encryption, and lifecycle policies operate at the table level rather than through complex S3 bucket policies layered on top of file-path conventions.
For a company that didn’t want the operational burden of self-managing open table format maintenance, this was the deciding factor.
AWS Glue Data Catalog provides unified metadata discovery across all tiers. Lake Formation handles column- and table-level access control, with AWS Identity and Access Management (IAM) roles that follow least-privilege principles and AWS CloudTrail turned on for a full audit trail.
Choosing the query protocol
Prior to the rearchitecture, the Amazon Redshift cluster was 98% ETL, and only a fraction of compute hours were analyst SELECT queries. The replacement engine needed to handle both heavy batch transformations and interactive ad hoc queries.
Traditional Spark (spark-submit) handles batch ETL well, but couples clients to the cluster. Every job requires packaging driver JARs, managing classpaths, and submitting from within the cluster. For a platform running 200+ production directed acyclic graphs (DAGs) that process massive data volumes daily, this operational friction was a non-starter.
Spark Connect is the gRPC-based client-server protocol introduced in Spark 3.4, and it solved the coupling problem entirely. The cluster runs a persistent gRPC endpoint. Clients connect remotely and submit queries over the wire. Airflow operators become thin clients: they open a session, submit SQL, and get results, with success and failure mapping directly to task states. There are no driver JARs and no polling. Multiple consumers, including pipeline orchestrators, the web application, and developer notebooks, share one cluster without any of them needing Spark installed locally.
Deploying Spark Connect
Razor Group deployed a self-hosted Spark cluster on Amazon EC2: an on-demand AWS Graviton leader node, Spot workers at about 70% cost savings, and the Spark Connect endpoint exposed through an internal Network Load Balancer. Custom Amazon Machine Images (AMIs) bake in the full Spark, Iceberg, and S3 Tables stack, so private-subnet nodes have everything they need without internet access at runtime.
This phase produced no immediate business value, but it made everything that followed possible.
Phase 2: Migrate data ingestion
Razor Group’s ingestion layer pulls data from Amazon Selling Partner API, Seller Central portals, NetSuite ERP, and custom web scrapers. In the previous architecture, all of this landed in Amazon Redshift through COPY commands, which meant data freshness was dictated by batch job schedules and competed for resources on the same cluster that served analytical queries.
Razor Group migrated these pipelines to AWS Lambda functions orchestrated by Apache Airflow, writing data directly to S3 Tables in Iceberg format. The shift from schedule-driven to event-driven significantly improved freshness. Lambda functions spin up only when there’s data to process, and Airflow sensors trigger downstream transformations the moment new data lands. This replaced rigid hourly batch windows with data freshness measured in minutes.
The orchestration layer manages 200+ DAGs across 90+ flows and processes data from dozens of sources at scale. The migration required rewiring destinations from Amazon Redshift COPY to Iceberg writes, but the orchestration logic itself carried over with minimal changes.
This phase alone eliminated roughly 40% of compute costs by severing the always-on cluster dependency for ingestion.
Phase 3: Transform processing pipelines
This was the most technically demanding phase, and where Razor Group learned the most. The team migrated 1,000+ SQL models from Amazon Redshift to Apache Spark, working incrementally up the dependency chain across 40+ schemas. The models moved through a medallion structure: Bronze for raw ingested data, Silver for cleaned and conformed data, and Gold for business-ready aggregates.
Razor Group built automated conversion tooling and a validation framework that ran both Amazon Redshift and Spark outputs in parallel, comparing results row-by-row before decommissioning anything. Several categories of transformation pushed the limits of what automation could handle:
Window functions: The QUALIFY clause in Amazon Redshift has no Spark equivalent. Each instance required wrapping in a subquery with explicit row numbering, which affected dozens of models in the inventory schema alone.
JSON serialization: The most time-consuming category. Complex columns stored as JSON STRING in Amazon Redshift needed from_json() with hand-written STRUCT definitions in Spark. Every nested payload column across ads, orders, and transaction pipelines required schema introspection, with no shortcuts.
Function dialect: More than 20 function-level conversions, including NVL to COALESCE, DATEADD to interval arithmetic, and LISTAGG to ARRAY_JOIN(COLLECT_LIST()).
Snapshot elimination: The single biggest hidden cost. Full table copies that ran multiple times daily only to preserve point-in-time state consumed more than 35 hours of weekly Amazon Redshift compute. With Iceberg’s native time travel, these became zero-cost operations overnight.
When migrating 1,000+ SQL models, automated tooling handles the mechanical syntax conversions well. But roughly 30% of the models required human judgment: those with complex JSON payloads, deeply nested window functions, or cross-schema snapshot dependencies. These models consumed 70% of the migration effort.
Razor Group built a structured migration workflow that used Claude to accelerate this work: read source SQL, identify dependencies, convert syntax, resolve missing base tables, add JSON parsing, validate outputs, and write to the lakehouse. The system did more than translate SQL. It applied schema context, traced cross-model dependencies, and flagged edge cases that would have taken engineers hours to find manually. What could have been a multi-year effort became a systematic, repeatable process measured in weeks. This approach fundamentally changed the speed of migration.
Phase 4: Unify the serving layer
With data flowing through Iceberg tables, Razor Group collapsed the serving layer. End users query Gold-layer Iceberg tables through Amazon Redshift Serverless, and internal exploration and machine learning (ML) workloads read the same tables through Spark Connect. This removed the need to maintain separate data copies, materialized views, or extract jobs for different consumers.
This is the strategic payoff of an open table format. Iceberg tables on S3 are engine-agnostic: Spark for batch transforms today, Trino for interactive queries tomorrow, Flink for streaming next quarter. Any engine that speaks Iceberg can read the data without conversion or migration. Razor Group went from being locked into a single vendor’s SQL dialect to having the freedom to adopt new engines without touching the storage layer.
Phase 5: Operationalize and observe
The final phase made the lakehouse production-grade. Razor Group built a comprehensive observability stack that aggregates metrics, traces, and logs from every pipeline component into a unified view. This view supports centralized log search, anomaly detection, and automated alerting that correlates failures across the entire data platform.
This observability layer did more than provide visibility. It gave the team confidence. When you’re running thousands of pipeline executions daily, you need to know within minutes when something breaks, what caused it, and which downstream consumers are affected. That’s the difference between reactive firefighting and proactive operations.
Pipeline orchestration consolidated around three patterns: a daily pipeline (ingestion to materialization to export to AI agent analysis), an operations worker polling every 15 minutes, and weekly scraper jobs.
The cutover was zero-downtime by design: both schedulers ran in parallel for two weeks. Automated comparison checks validated that every pipeline produced identical outputs before the prior architecture system was disabled.
Results and business impact
The lakehouse architecture delivered measurable improvements across every dimension:
Metric
Before
After
Improvement
P95 query runtime
180 seconds
63 seconds
65% faster
Infrastructure cost
Always-on provisioned clusters
Elastic, workload-optimized
63% reduction
Data freshness
4–6 hour batch cycles
Event-driven pipelines
15-minute freshness
Concurrent capacity
Limited by cluster size
Elastic, independent scaling
Unlimited
Engine flexibility
Single engine
Multi-engine (Spark, Athena, Amazon Redshift)
Open format portability
The 63% reduction compares the lakehouse run-rate (January–March 2026) with the pre-rearchitecture run-rate (October–December 2025), the trailing three months before the rearchitecture. The figure is an apples-to-apples blended infrastructure number that includes compute and storage across both architectures. The before column covers Amazon Redshift cluster compute and managed storage. The after column covers Amazon EC2 (Spark workers, both on-demand and Spot), AWS Lambda, AWS Glue, Amazon Athena, Amazon Redshift Serverless, and S3 Tables storage. Data-transfer and ancillary services are excluded because they were not materially different between the two periods. Workload mix (the number of pipelines, models, and end-user query volume) was held broadly comparable across the two windows.
Lessons learned
Start with the decision loops, not the tools, and know your workload before you replace your warehouse.
The most valuable activity of the entire migration wasn’t writing a line of code. It was the Amazon Redshift workload analysis we ran before making any architectural decisions. Discovering that 98% of compute was ETL, with only a sliver going to analyst queries, validated the move to on-demand Spark. It also prevented us from over-provisioning the replacement infrastructure for interactive workloads that barely existed. Architecture decisions should always trace back to core business requirements: pricing accuracy, promotional responsiveness, intraday P&L visibility. Start there, not with the technology.
Design for multiple compute engines, and choose the right engine per workload.
One of the clearest lessons from running a single-engine architecture is what you give up. Avoid locking yourself into one compute layer for BI, ingestion, backfills, and ML alike, because they have fundamentally different cost and performance profiles. Iceberg, Spark, and S3 Tables work well together out of the box once you make the shift. The technology isn’t the hard part. The hard part is mapping 1,000+ models across 40+ schemas, tracing dependencies through 200+ DAGs, and discovering that a column is actually a JSON string silently serialized differently between two engines. Migration is as much an excavation project as an engineering one.
Automate conversion, but budget for the 30%.
Automated tooling handles mechanical syntax conversions well, and it should be the first tool you reach for. But models with complex JSON payloads, deeply nested window functions, or cross-schema snapshot dependencies require human judgment, and that work doesn’t compress. Roughly 30% of our models needed significant manual intervention, and those models consumed 70% of the total migration effort. Plan for it honestly from the start.
Observability must include cost attribution, and watch out for hidden cost bombs.
Snapshot operations were our biggest surprise. Full table copies that ran multiple times daily to preserve point-in-time state were costing more than 35 hours of weekly compute, and nobody questioned it because “that’s how snapshots work.” Iceberg’s time-travel capability eliminated their cost, and that single feature justified a meaningful portion of the migration on its own. More broadly, you cannot optimize what you cannot see, so track query-level usage and attribute it to teams and functions. Cost observability is not a nice-to-have. It’s foundational.
Governance isn’t optional. Build it into the foundation, and align stakeholders from day one.
Catalog and access control need to come first, before you scale adoption, not after. The same principle applies to people: migration is a cross-functional program, not an infrastructure project. Our two-week parallel run caught edge cases that row-level validation missed entirely: time zone differences between Amazon Redshift and Spark, partition pruning behavior under concurrent writes, and subtle ordering differences in non-deterministic window functions. That parallel run wasn’t a safety net. It was where the migration actually proved itself. None of it works without the right stakeholders involved and aligned from the very beginning.
Conclusion
Razor Group’s journey offers valuable lessons for organizations looking to optimize their data architectures:
Analyze your workload mix first. Understanding that 98% of compute was ETL rather than interactive queries guided the decision to offload heavy processing to elastic Spark, while preserving Amazon Redshift Serverless for the interactive analytics it handles best.
Design for multi-engine flexibility. Open table formats like Apache Iceberg eliminate the need to choose a single engine. Each workload runs on the engine best suited to its access pattern, cost profile, and performance requirements.
Automate migration, but budget for complexity. Automated transpilation handled 70% of SQL models, but the remaining 30% consumed 70% of engineering effort. Plan accordingly.
Observability must include cost attribution. Without per-workload cost visibility, optimization is guesswork. Razor Group discovered that Iceberg snapshot maintenance alone consumed more than 35 hours of compute weekly, a hidden cost that observability surfaced and automation resolved.
Build governance into the foundation.AWS Lake Formation and AWS Glue Data Catalog provided fine-grained access control from day one, not retrofitted after the migration.
Validate with parallel systems. A two-week parallel run between old and new architectures caught edge cases that automated testing missed, which supported a confident production cutover.
The road ahead
With the lakehouse foundation in place, Razor Group is positioned to accelerate innovation, from real-time pricing models to AI-driven inventory optimization, all powered by a unified, open, and governed data platform on AWS.
The company’s transformation demonstrates that modern data architectures aren’t about choosing between services. They’re about placing each workload where it performs best, using open formats to eliminate silos, and scaling each layer independently as the business grows.
This post is co-written with Oscar Mapfumo Sibanda from Picnic.
Picnic is an Amsterdam-based tech scale-up that reinvents how people buy food. It isn’t a supermarket with a digital layer but a tech company that happens to deliver groceries. Picnic is engineered in-house: the customer app, the fulfillment platform, the supply chain, and the routing technology that guides a fleet of thousands of electric vehicles through the Netherlands, Germany, and France. Software doesn’t merely support business. Software is a business.
At the center of this system, RabbitMQ is the core component. It’s the communication backbone connecting hundreds of microservices across the entire business lifecycle, from ordering and logistics to delivery and finance. At peak, Picnic’s platform processes close to one million messages per second. At this scale, messaging is no longer only background infrastructure. It becomes part of the company’s operational nervous system. To keep that system highly available, scalable, and resilient as Picnic grows, the company decided to use Amazon MQ as a managed service.
The next challenge was identity. Picnic’s authentication strategy clearly distinguishes between people and services. Operators sign in through Keycloak, the company’s single sign-on provider, while Picnic’s Amazon Elastic Kubernetes Service (Amazon EKS) workloads are adopting AWS Identity and Access Management (IAM) authentication to eliminate static credentials. A single broker therefore must trust two identity providers at once. The Amazon MQ documentation covers configuring OAuth 2.0 with a single provider. This post extends that guidance to a multi-provider setup on the same broker.
In this post, we show how Picnic solved that problem. You will learn how to configure an Amazon MQ for RabbitMQ broker to accept tokens from multiple OAuth 2.0 identity providers, using Keycloak and IAM as the working example. You will also see how to map each provider’s scopes to RabbitMQ permissions and how to roll the change out on a running broker without disrupting connected users.
Background and prerequisites
Amazon MQ for RabbitMQ supports OAuth 2.0 authentication and authorization, where broker users and their permissions are managed by an external identity provider. User authentication and resource permissions for vhosts, exchanges, queues, and topics are centralized through the OAuth 2.0 provider’s scope system.
RabbitMQ’s OAuth 2.0 plugin supports multiple resource servers and audiences, allowing different OAuth 2.0 providers to issue tokens that a single broker can validate. This capability is essential if you operate in multiple environments or have teams registered with separate identity providers.
Basic familiarity with RabbitMQ configuration and OAuth 2.0 concepts.
AWS Command Line Interface (AWS CLI) version 2.27 or later (required for the get-web-identity-token command used in the testing section).
Note: The information in this post reflects Amazon MQ for RabbitMQ features and behavior at the time of publication. We recommend checking the Amazon MQ documentation, release notes and best practices before implementation.
Solution architecture
The design rests on a single idea: a RabbitMQ broker can trust more than one identity provider at the same time, and it decides which one to apply per token rather than per broker. RabbitMQ does this by reading the aud (audience) claim of each incoming token and matching it against a configured resource server. Each resource server is bound to one OAuth 2.0 provider, so the audience determines both which signing keys validate the token and which permission rules apply.
Architecture diagram
The following two diagrams show how services and operators authenticate the broker through their respective identity providers.
Figure 1: Services (IAM) flow
Services authenticate through IAM. An Amazon EKS workload assumes an IAM role and generates a web identity token (1), which AWS Security Token Service (AWS STS) issues with an audience of rabbitmq-iam (2). The workload presents that token as its password when it connects to the broker over Advanced Message Queuing Protocol (AMQPS) on port 5671 (3). The broker selects the matching resource server, verifies the token’s signature against the AWS STS signing keys (4), and maps the role’s Amazon Resource Name (ARN) to the permissions the workload needs (5).
Figure 2: Operators (Keycloak) flow
Operators authenticate through Keycloak. An operator opens the RabbitMQ management console and initiates login (1), and the console redirects the browser to Keycloak (2). Keycloak authenticates the operator and issues a token whose audience targets the console resource server, rabbitmq-keycloak (3). The browser presents that token to the broker (4), which verifies the signature against Keycloak’s signing keys (5). The broker then reads the operator’s group membership and grants access (6): the Operator group receives read-only permissions, while the Administrator group receives full control.
Two constraints follow this design. First, audience verification is a single broker-wide setting that applies to every provider at once, so each provider must issue tokens carrying the exact audience its resource server expects. Second, the broker is private, deployed inside an Amazon Virtual Private Cloud (Amazon VPC) with no public exposure. Both providers’ endpoints must be resolvable, either through publicly addressable JSON Web Key Set (JWKS) endpoints or through private networking, because the broker fetches signing keys from those endpoints.
Implementation walkthrough
This walkthrough configures one Amazon MQ for RabbitMQ broker to trust two identity providers: Keycloak for operators and AWS IAM for services. The steps assume you already have a running broker, a Keycloak realm, and outbound web identity federation enabled for your AWS account. All configurations are applied through a RabbitMQ configuration revision using the AWS Command Line Interface (AWS CLI).
Enable OAuth 2.0 on the broker
The first block activates the OAuth 2.0 backend and keeps the internal backend in place. Internal authentication remains active deliberately: Amazon MQ creates an administrator user when the broker is provisioned, and that user is needed for break-glass access.
Setting verify_aud = true tells RabbitMQ to reject any token whose aud claim does not match a configured resource server. This single broker-wide setting governs every provider you add.
Add the first identity provider (Keycloak)
A resource server binds an audience value to a provider and a set of permission rules. The Keycloak resource server uses the id rabbitmq-keycloak, which is the audience the realm must place in its tokens. RabbitMQ reads the operator’s group membership from the group_membership claim and resolves it through scope aliases.
The Operator group is read-only: it can read any resource and view the management UI through the monitoring tag. The Administrator group receives full permissions plus the administrator tag. This role is least privilege by design.
The provider is configured with its issuer and JWKS endpoint:
Adding a second provider means adding a second resource server and a second entry under oauth_providers. The IAM resource server uses the id rabbitmq-iam because the audience is set when minting the token.
Note: The base64 command syntax differs between Linux and macOS. The preceding command (cat file | base64 | tr -d '\n') is portable on both operating systems. If running exclusively on Linux, you can also use base64 --wrap=0 rabbitmq.conf. On macOS, the equivalent command is base64 -i rabbitmq.conf.
Testing and validation
Validate each provider independently. For IAM, assume the role and request a token from AWS STS, then present it as the AMQP password:
TOKEN=$(aws sts get-web-identity-token \
--audience "rabbitmq-iam" \
--signing-algorithm ES384 \
--duration-seconds 300 \
--query 'WebIdentityToken' --output text)
# Username is empty (ignored by the OAuth plugin); the token is passed as the password
curl -u ":$TOKEN" https://<broker-id>.mq.<region>.on.aws/api/overview
Note: The get-web-identity-token API requires outbound web identity federation to be enabled on your AWS account and AWS CLI version 2.27 or later.
A successful response confirms the IAM resource server accepted the token. For Keycloak, open the management console, choose Keycloak SSO, and sign in as an operator.
When a login fails, decode the JSON Web Token (JWT) and check two claims. The aud claim must exactly match a resource server id. With verify_aud = true, a missing or mismatched audience is the most common cause of rejection. If the audience is correct but permissions are missing, verify the scope_prefix is set correctly.
Operational considerations
A few points deserve attention before you run this pattern in production.
Key rotation: When rotating signing keys at a provider, publish the new key in the JWKS endpoint before revoking the old one. The broker caches keys, so overlapping both during the transition window prevents authentication failures while the cache refreshes.
Audience validation: Audience remains the linchpin. With verify_aud enabled, every provider must issue tokens carrying the audience its resource server expects, so confirm this whenever you onboard a new one. Don’t disable audience validation in production. The RabbitMQ OAuth 2.0 plugin does not perform token revocation checks, which makes audience binding a critical control that prevents tokens issued for other services from granting access.
Scope prefix: scope_prefix values are optional. They’re needed only if the tokens don’t follow the default format. RabbitMQ only reads scopes carrying the expected prefix, so a token can authenticate yet grant nothing if the prefix is missing. Map each provider to the least privilege its principals need. For example, prefer narrow scopes like read:orders over blanket read:all to limit the scope of impact if a single provider’s credentials are compromised.
Token lifetime: Because the plugin does not support token revocation, token lifetime is your primary control over leaked credentials. Issue short-lived access tokens and have your client applications refresh them proactively at approximately 75 percent of the token’s lifetime to avoid connection disruptions when a token expires mid-session.
Monitoring: Authentication failures and refused tokens are recorded in the broker’s connection log group in Amazon CloudWatch, which can be reached through the Amazon CloudWatch Logs link on the broker’s page in the Amazon MQ console. Beyond logs, set up CloudWatch alarms on RabbitMQMemUsed, RabbitMQDiskFree, and ConnectionCount. An unexpected spike in failed connections is often the first sign of a token or audience misconfiguration. For unaggregated, per-node visibility, consider enabling the Prometheus metrics endpoint: metrics such as rabbitmq_auth_attempts_failed_total surface OAuth rejections faster than the CloudWatch one-minute polling interval.
Network controls: Enforce defense in depth by restricting broker access using security groups so that only authorized VPCs and IP ranges can reach the AMQPS and management endpoints. This matters especially in an OAuth setup because, once a token has been issued, the broker cannot revoke it before it expires.
Cleanup
To avoid incurring future costs, delete the resources created during this walkthrough if you no longer need them:
Delete Amazon MQ broker and configurations.
Remove test OAuth application registrations from your identity providers.
Delete any IAM roles created for testing.
Conclusion
In this post, we demonstrated how Picnic configured an Amazon MQ for RabbitMQ broker to authenticate tokens from two OAuth 2.0 identity providers: Keycloak for human operators and AWS IAM for machine-to-machine services on a single broker instance. The key mechanism is RabbitMQ’s support for multiple resource servers, where the audience claim in each token determines which provider’s signing keys and permission rules apply.
With this approach, the Picnic team was able to cleanly separate human and machine authentication without the operational overhead of running separate brokers, while retaining fine-grained access control for both token issuers.
This pattern works with any combination of OAuth 2.0 providers and is particularly valuable for organizations looking to consolidate messaging infrastructure while maintaining distinct identity boundaries.
How Gallup turned 90 years of workplace science into an AI assistant that gives leaders personalized guidance in seconds, powered by Amazon Bedrock.
Gallup delivers analytics and advice to help leaders and organizations solve their most pressing problems. With more than 90 years of experience and a global reach, Gallup has developed a uniquely deep understanding of workplace behavior and performance.
However, this knowledge wasn’t centralized or delivered in context. Leaders had to navigate multiple resources to find relevant insights and then translate them into action without guidance. The lack of real-time, personalized recommendations meant workplace challenges were often handled reactively instead of proactively.
Gallup needed to transform decades of proprietary research into real-time, personalized guidance that leaders can access instantly within their existing workflow.
In this post, we show how Gallup built Gallup AI, a generative AI assistant powered by Amazon Bedrock. It transforms decades of proprietary workplace research into real-time, personalized coaching delivered directly within the Gallup Access application.
Why Amazon Bedrock
Gallup evaluated multiple approaches to building a generative AI assistant. The team chose Amazon Bedrock for three reasons:
Access to leading foundation models like Anthropic’s Claude without managing infrastructure.
Built-in retrieval augmented generation (RAG) through Amazon Bedrock Knowledge Bases, a fully managed RAG capability, grounds responses in verified research.
Native guardrails to enforce content safety at scale.
This combination allowed Gallup to move from prototype to production in weeks rather than months, without hiring a dedicated machine learning (ML) operations team.
The approach: Building intelligence into daily workflow
Gallup built Gallup AI, a generative AI-powered assistant integrated directly into Gallup Access. The unified application lets managers review engagement results, build action plans, explore CliftonStrengths insights, and access curated content to better support their teams.
The solution uses Amazon Bedrock with Anthropic’s Claude models to deliver conversational insights grounded in Gallup’s proprietary research. Amazon Bedrock Knowledge Bases and Amazon Kendra retrieve relevant research and organizational data. This is designed to ground responses in verified workplace science. Amazon Bedrock Guardrails enforce content safety policies, while AWS Lambda with FastAPI delivers real-time streaming responses that feel natural and immediate.
The following diagram shows the Gallup Access AI application architecture.
Figure 1: Gallup Access AI application architecture
The architecture processes requests through the following stages:
Gallup’s proprietary workplace research covers decades of employee engagement studies, performance data, and organizational insights. The content is stored in Amazon S3 and ingested into Amazon Bedrock Knowledge Bases. The application also continuously crawls the Gallup website to capture the latest research publications, articles, and insights, indexing this content in Amazon Kendra for instant retrieval. This dual approach gives the AI assistant access to both historical research archives and current workplace science, delivering responses grounded in verified, up-to-date knowledge rather than generic advice.
When a leader asks Gallup AI a question, the system retrieves relevant research from both Amazon Bedrock Knowledge Bases and Amazon Kendra. The system scores documents based on confidence thresholds, filters them, and consolidates them before sending them to Claude models in Amazon Bedrock.
The conversation flows through AWS Lambda handlers that manage both real-time streaming (for web clients) and synchronous requests (for backend services). Amazon ElastiCache Serverless caches recent conversation history for instant retrieval, while Amazon RDS for MySQL serves as the durable storage layer with organized records of conversations, prompts, responses, and source citations.
Amazon Bedrock Guardrails apply content safety policies during generation, with the ability to intervene mid-stream if policy violations are detected. Interactions persist before streaming begins, preserving transactional integrity even if connections are interrupted.
AWS Systems Manager Parameter Store serves as the application’s centralized configuration hub, managing AI model settings, content safety policies, and performance thresholds. This allows the team to adjust application behavior instantly, without redeploying code or interrupting service for users.
Amazon DynamoDB provides fast, flexible storage for product-specific insights and contextual data, so the application delivers personalized experiences tailored to each user’s role and workflow.
Comprehensive metrics, including I/O tokens, cached tokens, time-to-first byte, and stop reasons, flow through Amazon Data Firehose to Amazon S3, providing visibility into cost, performance, and usage patterns across the application.
What Gallup has achieved
Gallup has transformed decades of workplace research into an intelligent assistant that delivers measurable value across thousands of organizations. Tasks that previously required navigating reports, articles, and tools now resolve through a single conversational interaction. Time to insight dropped from manual research to real-time, AI-delivered guidance within seconds. The application processes billions of tokens through production interactions, with responses grounded in verified workplace science.
Since launching in June 2024, adoption and engagement have grown rapidly:
Metric
Result
Prompts
Increased ~7x
Conversations
Increased ~4.5x
Active users
Increased ~5.5x
Engagement depth
Average prompts per conversation increased ~55%, indicating sustained, multi-turn interactions
Response latency
Sub-second time-to-first byte (TTFB) for streaming responses. Sub-millisecond session retrieval via Amazon ElastiCache Serverless
What the customer said
Gallup’s Director of Product reflects on what this shift means for how leaders access workplace science:
“Gallup AI represents a fundamental shift in how leaders access workplace science. For decades, our research helped organizations make better decisions, but it often required leaders to search, interpret, and apply those insights themselves. By building on Amazon Bedrock, we’re embedding scientifically grounded guidance directly into the flow of work, giving managers real-time support that is both personalized and actionable.”
— Andrew Bridger, Director of Product, Gallup
With this foundation in place, Gallup is focused on expanding what the application can do next.
What’s next
Gallup’s roadmap focuses on making its expertise more accessible, actionable, and embedded into everyday workflows. A key initiative is the development of an AI-curated prompt library that captures the most common questions managers and leaders ask. This library will help users quickly engage with Gallup AI through proven, high-value prompts grounded in workplace research.
In addition, Gallup is introducing guided coaching experiences built around structured conversation flows. These guided prompts walk managers through well-defined coaching scenarios, such as improving engagement, addressing team challenges, or developing employees, by sequencing prompts and responses into purposeful, outcome-driven interactions.
Gallup is building an agent-based foundation using Amazon Bedrock AgentCore. This positions Gallup AI to move beyond a user-facing assistant. By surfacing tools, workflows, and proprietary knowledge programmatically, the system can support not only end users but also other systems and integrations across the application.
Conclusion
By combining the generative AI capabilities of Amazon Bedrock with Gallup’s proprietary workplace research, leaders now have instant access to scientifically grounded guidance exactly when they need it. The serverless architecture enables the application to scale reliably while delivering low-latency streaming responses and comprehensive observability.
Accounts move between organizations in AWS Organizations whenever a business changes shape. A merger folds one estate into another. A divestiture carves one out, and some companies run more than one organization by design.
The moves take more care when AWS Resource Access Manager (AWS RAM) resource shares are involved. An organization-bound share trusts an account through its organization membership, so when the account leaves, AWS RAM removes that association. Anything in production that depends on a shared resource needs a continuity plan before the first account moves.
A leading worldwide provider of payment technology and software solutions, based out of the United States, used temporary AWS RAM resource shares to preserve AWS Lake Formation permissions during an AWS Organizations migration of 382 AWS accounts. The payment processor serves merchants and financial institutions around the world. The program separated them from their former parent company, a US-headquartered financial technology provider serving banking and capital markets clients globally, before a Transitional Service Agreement (TSA) expired in April 2026.
The migration unfolded alongside wider corporate change. In January 2026, the payment processor was acquired by a leading payments technology company headquartered in the United States. The transaction transforms the acquirer into a pure-play commerce solutions provider, serving the full spectrum of clients from small businesses to global enterprises worldwide. The migrating estate also included the payment processor’s embedded payments platform, a US-based provider of embedded payment and automated onboarding tools for software-as-a-service (SaaS) platforms.
Most workloads kept running when the original organization-bound shares broke, but the control plane lost access. They needed a migration pattern that preserved service continuity without leaving temporary permissions behind.
AWS partnered with them to design and validate that pattern in two weeks. It uses retained bridge shares for the move, then restores the original shares as the durable permission objects.
An AWS account can consume a resource to share automatically while it belongs to the same organization as the producer. When the account leaves, AWS RAM removes that organization-bound principal association. Creating a retained bridge share as an external association before the move preserves access through the organization’s change.
After the move, the company restores the migrated account to the original share, verifies access, and removes the bridge. The original share remains the AWS Lake Formation-managed source of truth. New grants and resource changes continue to attach to it, not to the point-in-time bridge copy. Keeping both would create duplicate permission state and drift.
The following diagram shows the migration wave structure.
Migration wave structure. Stage one covers fourteen non-production waves across eight months, none of which crossed an organization boundary. Stage two covers sixteen production waves: one pilot wave, twelve scheduled waves on a weekly cadence, and three contingency waves. The transitional service agreement expires inside the contingency window, leaving only the first contingency week usable.
The company’s cloud engineering team ran 14 non-production waves, a production pilot, 12 weekly production waves, and three contingency waves. The TSA expired inside the contingency window, leaving about one week of usable slack.
Challenge
The risk surfaced in a production wave in February 2026. A terraform apply against a shared AWS Transit Gateway failed with a permission error, although traffic kept flowing and no alarm fired.
This was a control plane failure. Existing attachments, DNS paths, and certificates continued to work, but engineers could not change shared resources. The dedicated AWS Transit Gateway, Amazon Route 53 Resolver, and the embedded payments platform’s AWS Glue Data Catalog waves were still ahead.
Why the issue stayed hidden
Most services retain their data plane when an AWS RAM association breaks. For example, an Amazon Elastic Compute Cloud (Amazon EC2) instance in a shared Amazon Virtual Private Cloud (Amazon VPC) keeps running, but the account cannot launch a new instance. Infrastructure-as-code exposed the problem because it needed control-plane access.
The following table summarizes the affected resource types confirmed by the customer.
Keeps the data plane, blocks new resource creation
AWS Glue Data Catalog databases and tables
AWS Glue and AWS Lake Formation
Requires bridge-share validation before migration
Some services require additional handling. Depending on its resource-cleanup configuration, an AWS Firewall Manager policy can remove AWS Network Firewall rules, which you must then redeploy, and organization-integrated AWS CloudFormation StackSets can delete stacks unless you set them to retain.
The embedded payments platform initially entered the estate through an acquisition by the former parent company. Its embedded payment and automated onboarding capabilities were subsequently integrated into the payment processor’s ecosystem to support a broader platform-focused offering for SaaS providers. The platform shared databases and tables across 10 accounts with account IDs as principals, and the workstream was paused rather than testing the migration against production.
Why the original shares broke
The company had enabled sharing with AWS Organizations in the producer account. AWS RAM therefore trusted each in-organization principal through organization membership, even when a share named an account ID. When an account left the source organization, AWS RAM removed that organization-bound association.
A share created for a principal outside the organization behaves differently. AWS RAM sends an invitation, and the accepted association is external. Because it does not depend on organization membership, it survives the account move. The bridge-share pattern uses this behavior.
Why non-production testing missed it
The company’s non-production accounts were already in a separate organization. They never crossed the boundary that caused production associations to break.
Fourteen clean waves validated the migration process but not the production-only condition. Each validation environment must cross the same trust boundary as production.
Applying the bridge-share pattern
This failure was raised with the AWS account team, which brought AWS RAM, AWS Glue, AWS Lake Formation, and AWS Organizations service teams into the response. The company first used a manual recovery path for 21 accounts while the teams automated a scalable approach.
On February 27, 2026, AWS released RetainSharingOnAccountLeaveOrganization for new AWS RAM resource shares. The setting marks principals as external after they accept the invitation. The customer confirmed that the setting does not retrofit existing shares, so those shares needed a temporary parallel share.
Retaining access during the move
AWS RAM allows a resource to belong to more than one resource share, so a parallel share can exist alongside the original. A second retained share was created alongside each original, targeting the same consumer account.
The consumer accepted the invitation before migration, creating an external association. During the move, AWS RAM removed the original organization-bound association while the bridge continued to grant access. AWS Organizations also support transferring an account directly between organizations, so the move itself does not require an intermediate standalone period.
Why restore the original share?
The bridge is a migration-only continuity copy. The original AWS Lake Formation-created share remains the durable, service-managed permission object. If a team adds a grant or changes a shared resource during the migration window, that change applies to the original share, not automatically to the bridge.
The company therefore restored migrated principals to the original share before deleting the bridge. Leaving both in place would create two permission paths that can diverge, complicate audits, and conceal which share is authoritative.
Automation deletes a bridge only after it finds a non-bridge original whose resources, principals, and permissions cover the bridge and whose associations are all ASSOCIATED. This check confirms the original share’s associations are active before removing the temporary path. Access and connectivity were validated separately, as described in the Outcome section.
Validated workflow
AWS validated the pattern across three test accounts and two organizations before it was used in production. Testing confirmed that allowExternalPrincipals alone was not enough. The bridge also required retainSharingOnAccountLeaveOrganization.
The following diagram shows the bridge before and after the account move.
Bridge share behavior before and after an account moves between AWS Organizations. Before the move, the original organization-scoped share and the accepted bridge share both grant access. After the move, the original share is revoked and the accepted bridge share continues to grant access.
Each production wave used five steps:
Inventory. Map each original share, resource, principal, permission, and Region. AWS RAM is Regional, so repeat the inventory in every in-scope Region.
Create and accept bridges. Create a retained share for the same resource and principals, then accept its invitation from each consumer account before migration.
Migrate. Move the account. AWS RAM removes the organization-bound association, while the accepted bridge keeps access active.
Restore originals. Add the migrated account IDs back to the original shares as external principals. This reactivates the durable shares and includes grants created during the migration window.
Validate and remove bridges. Confirm resources, principals, permissions, and association status, then delete only bridge shares fully covered by active originals.
This workflow ran for every remaining production wave and kept the weekly cadence.
Validating AWS Glue and Lake Formation permissions
The embedded payments platform shared AWS Glue Data Catalog databases and tables across 10 accounts, with account IDs as principals. The configuration was reproduced in disposable accounts, and the resource policy was recorded through a cross-organization move.
The validated automation records principal-to-share mappings, supports dry-run and execute modes, restores principals to the original shares, and deletes bridges only after validation. The embedded payments platform completed its production migration on July 21, 2026.
Outcome
The global payment processor migrated 378 of the 382 accounts into its landing zone. The final four awaited approvals from external stakeholders.
The TSA with the former parent company ended on schedule in April 2026. No customer-facing workload lost availability, and the company recorded no network drops across the production waves. The company and AWS moved from discovery to a validated bridge-share pattern in two weeks.
After each migration, the cloud engineering team restored the original shares, verified access and connectivity, and removed the bridges. Deleting the temporary copies confirmed that the original service-managed permission path was active and authoritative.
Production, non-production, and the embedded payments platform now run in one landing zone. They control their own guardrails, security posture, provisioning, and change process.
AWS has published the validated pattern and automation, so other organizations can start with a tested procedure.
Lessons learned
This program produced three lessons for organizations planning similar migrations.
Match validation boundaries to production
A test organization cannot expose this failure unless it crosses the same organization boundary as production. Map each production risk to an environment that can reproduce it before the first wave.
Monitor control plane changes
AWS RAM emits resource share state-change events directly to Amazon EventBridge, and AWS CloudTrail records DisassociateResourceShare API calls for audit. A weekly post-migration sweep provided a periodic reconciliation check to catch stale shares.
Inventory dependencies and destination guardrails
The Account Assessment for AWS Organizations tool inventories AWS RAM dependencies before teams set the wave plan. The company’s cloud engineering team reviewed destination guardrails at the same time. A service control policy that blocked ram:AcceptResourceShareInvitation during migration windows was temporarily adjusted.
Conclusion
The migration of this leading payment technology and software company shows how a retained bridge share can protect access while an account moves between AWS Organizations. The bridge is temporary: restoring the original share keeps AWS Lake Formation permissions aligned with future grants and avoids two sources of permission state. Inventory, dry-run-first automation, and post-move validation helped the company meet its deadline without customer disruption.
If you’re building AI agents for commerce at scale, you face two critical challenges: handling unpredictable traffic spikes and ensuring your agents can be trusted with real customer transactions.
This post shows how AgentFlo solved these challenges using Amazon Bedrock AgentCore and AWS serverless architecture. You learn the architectural patterns behind their reliability and trust frameworks, see the measurable business results (including +12% net revenue uplift based on early deployment data), and explore their roadmap for voice agents and server-side tool execution.
This is Part 2 of a two-part series. Part 1 covers velocity, standardization, and scalability.
Pillar 4: Trust: guardrails for autonomous commercial action and real-time visibility into agent operations
AgentFlo enforces trust at every layer of the stack, from pre-request filtering to post-response privacy controls, so merchants can deploy autonomous agents with confidence.
The challenge
Enterprise customers won’t deploy autonomous agents unless they can trust them. An agent in production can’t expose sensitive data, offer unauthorized discounts, or access another customer’s information. The system also prevents price hallucination, unauthorized tool calls, opt-out violations, and credential exposure.
Merchants also need fine-grained control over who can interact with their AI agents and what data each segment can access. Enterprise customers require restricted access; B2C businesses need open access for broader reach. Without identity-based controls, deploying customer-facing AI is a non-starter.
Defense in depth
In AgentFlo, trust isn’t only about safe responses. It’s about safe action. Agents can create carts, place orders, apply discounts, access customer data, and interact with backend systems, so policy enforcement must sit outside the model’s reasoning loop. The model proposes. Deterministic policy decides.
AgentFlo applies trust controls across the full agent lifecycle: before the model sees the request, during tool execution, and after the model generates a response.
Three-layer guardrails
When users deploy an agent from the AgentFlo Portal, security enforcement happens at three stages.
First, the AWS Fargate layer detects prompt injection and handles opt-outs before requests reach the agent. WhatsApp messages are authenticated using phone numbers as unique identifiers. Enterprise customers like EBM restrict access to authorized users, while restaurant deployments stay open for broader reach.
Next, the AgentCore layer verifies identity and enforces order locks during tool execution. AgentCore Gateway, a capability of Amazon Bedrock AgentCore, enforces policies that prevent sales agents from accessing customer support tools. Cedar policies enforce business rules like maximum discount percentages independently of the model’s reasoning. Cedar is an open-source policy language developed by AWS for fine-grained, verifiable authorization decisions. Additionally, Policy in Amazon Bedrock AgentCore integrates with Amazon Bedrock Guardrails, so Cedar policies can invoke configurable safeguards for prompt attack detection, content filtering, and sensitive information blocking directly at the gateway boundary.
Finally, post-turn privacy filters screen outputs to block inadvertent token disclosure and unverified price claims before customers see responses. Secrets are managed through AWS Secrets Manager with OIDC (OpenID Connect)-authenticated continuous integration and continuous delivery (CI/CD) pipelines. No credentials are stored in agent code.
Infrastructure security
Trust at the application layer requires sound underlying infrastructure. AgentFlo combines the built-in isolation of AgentCore with application-level controls:
Session isolation: Session isolation through AgentCore runtime, a capability of Amazon Bedrock AgentCore, which provides complete separation between merchants’ agent sessions through dedicated microVMs.
AgentCore Gateway policies: Fine-grained Cedar policies control which agents can access which tools and data, enforced deterministically regardless of model reasoning.
Amazon Virtual Private Cloud (Amazon VPC) integration: Agent sessions operate within AgentFlo’s VPC with domain-level network restrictions, so agents only communicate with approved endpoints.
Compliance: Data residency controls and audit trails for regulatory requirements across multiple jurisdictions.
Observability
Trust requires visibility. Merchants need to see what their agents are doing in real time, not only after something goes wrong. AgentFlo uses AgentCore Observability, a capability of Amazon Bedrock AgentCore, to provide end-to-end tracing of every agent interaction, from initial request through tool execution to final response.
AgentCore Observability captures structured traces for each agent turn, including model latency, tool invocation sequences, token usage, and error rates. These traces flow into Amazon CloudWatch, where AgentFlo builds dashboards showing active sessions, response times, and tool call patterns. Full request-to-response traces enable trace-level debugging. The system tracks P50/P95 latencies and throughput across agent types, alerting merchants when behavior deviates from baselines. Cost attribution provides per-merchant, per-agent breakdowns tied to specific conversations.
Results
Together, observability and trust give enterprises confidence to deploy autonomous agents at scale. Merchants benefit from safer execution, stronger compliance across jurisdictions, and controlled access to tools and data at the session level.
Pillar 5: Reliability: a data foundation that keeps agents grounded in reliable data
Reliable agents need reliable data. AgentFlo grounds every agent action in verified, current information through stateful sessions, merchant knowledge bases, and semantic product discovery.
The challenge
Enterprise customers won’t trust AI agents that forget context, hallucinate product details, or operate on stale data. An agent that quotes the wrong price, forgets a customer’s earlier request, or recommends discontinued products destroys confidence instantly. The system must make sure every agent action is grounded in verified, current information, from product catalogs and pricing to conversation history and business rules.
Merchants also need their agents to maintain continuity across long customer journeys, access up-to-date business-specific knowledge, and surface products through natural language, all without manual intervention or prompt engineering.
Data architecture overview
In AgentFlo, reliability isn’t only about accurate responses. It’s about accurate action grounded in verified data. Agents retrieve product information, manage carts, and complete transactions, so every data source must be authoritative and current. The model reasons. Structured data decides.
AgentFlo applies data reliability controls across three layers: stateful conversation management through Amazon DynamoDB, merchant-specific knowledge through Amazon Bedrock Knowledge Bases, and semantic product discovery through vector embeddings in Amazon S3 Vector.
State management architecture
A real sales journey can span 8 hours or 3 days. A customer might ask about a product in the morning, compare options at lunch, and complete the purchase that evening. The agent must remember context across all turns and maintain state, so customers don’t need to start over.
AgentCore runtime and Amazon DynamoDB manage this context storage. The agent replays relevant history, loads context based on intent, and continues transactions safely.
Each agent needs to be stateful (remembering earlier interactions), autonomous (deciding next steps without human intervention), and safe (operating within business rules without exposing sensitive data).
The two-table DynamoDB design covers all three requirements: session continuity through conversation replay, autonomous context loading based on detected intent, and data integrity through structured ground-truth storage that the model can’t hallucinate over.
Figure 1: Per-message conversation flow. AgentCore runtime loads the last 15 messages from the DynamoDB Session Table at the start of each turn. The Cart Table is loaded on demand only when intent detection flags the request as cart-related, preventing the model from generating incorrect prices and quantities.
Knowledge base system
Merchants upload business-specific data (restaurant menus, clinic policies, product specifications, promotion calendars) into Amazon Bedrock Knowledge Bases backed by Amazon Simple Storage Service (Amazon S3). Agents automatically retrieve and reason over this merchant-specific content. Responses stay grounded in accurate, up-to-date business information without merchants writing a single prompt.
Semantic search and vector retrieval
AgentFlo improves product discovery by giving every product in its Amazon Aurora database a lightweight vector embedding. Customers can find products by name or description.
AgentFlo also generates extra searchable tags for each product automatically, and merchants can add their own. The result: phrases like “the pink one,” “the smallest one,” “the new one,” or “the chocolate with the golden wrapper” all map to the right product. Customers ask for things naturally, and the platform finds what they’re looking for.
Observability and billing
The system captures all message interactions through Amazon Data Firehose to Amazon S3, so merchants can track cost per conversation and compare those costs against sales revenue. This pipeline shows merchants the return on investment (ROI) of their agent deployments and provides the data foundation for continuous agent improvement.
Results
The data architecture helps minimize context loss in conversations across multi-day customer journeys, with grounded responses that eliminate price and product hallucination. Merchants benefit from natural language product discovery without keyword dependency and merchant-specific knowledge retrieval without prompt engineering. Full cost visibility and ROI attribution per agent deployment give merchants clear measurement of platform value.
Business impact: measurable results across the customer lifecycle
Salesflo’s solution, powered by Strands Agents SDK and Amazon Bedrock AgentCore, delivered measurable improvements across the customer lifecycle:
Metric
Improvement
Net revenue uplift
+12%
Customer engagement
+40%
Conversion rate
+15%
Average order value
+8%
Customer reactivation
+20%
AgentFlo generated these results by comparing agent-assisted customer journeys against a control group over a 90-day early deployment period.
Note: The foundation models referenced in this post are available in select AWS Regions. For the latest information on model availability, see Supported Regions and models for Amazon Bedrock.
The compounding effect at scale
At AgentFlo’s scale, the impact is significant. With $300 billion in annual transacted value flowing through the Salesflo solution (based on platform transaction data), even single-digit percentage improvements translate to billions in incremental revenue for merchants.
Operational improvements
Beyond the metrics, merchants see operational improvements:
24/7 coverage — AI agents engage customers at any hour, in any time zone, across any channel.
Consistent quality — Every customer interaction follows best-practice selling methodologies without the variability of human agents.
Scalable personalization — Thousands of concurrent agent sessions, each maintaining unique context per customer.
Rapid merchant onboarding — New merchants go live with customized AI agents in days, not months, through the self-service configuration platform.
What’s next: voice, server-side execution, and integration expansion
AgentFlo is actively extending the platform along three directions, each at a different stage of maturity.
Real-time voice agents (in pilot)
AgentFlo already supports voice within WhatsApp. Incoming voice notes go through a two-pass transcription process: a raw initial pass, followed by domain-aware correction that fuzzy-matches text against the live product catalog. The second pass catches brand names and SKUs even when partially misheard, across more than 90 supported languages. For outbound audio, merchants pick from multiple Text-to-Speech (TTS) providers per deployment.
Because Speech-to-Text (STT), reasoning on AgentCore runtime, and TTS are fully independent components, any one can be swapped without disrupting the rest of the pipeline.
The next step: BidiAgent. The next evolution is real-time voice agents built on the Strands SDK BidiAgent and Amazon Bedrock AgentCore WebRTC support. BidiAgent supports bidirectional audio streaming, natural interruptions, and concurrent tool execution. The agent can check inventory or apply a discount while continuing to listen and respond to the customer in the same call.
The AgentCore WebRTC protocol and Amazon Kinesis Video Streams handle peer-to-peer transport for mobile and browser interactions without requiring relay infrastructure. This pushes AgentFlo beyond text messaging into proactive outbound calls and high-value B2B sales, where real-time conversation is essential for building trust.
AgentFlo is experimenting with server-side tool execution in Amazon Bedrock, which removes client-side orchestration entirely.
Traditional orchestration loops between model and tools repeatedly (model → execute → send result → repeat). With server-side execution, the agent makes a single API call to the Amazon Bedrock Responses API with an AgentCore Gateway Amazon Resource Name (ARN). The model then autonomously discovers, invokes, and processes tools through the Gateway Model Context Protocol (MCP) interface, all inside AWS infrastructure with no roundtrips back to the client.
Note: Model availability varies by AWS Region. The model and endpoint shown in this example may not be available in all Regions. See Supported Regions and models for Amazon Bedrock for current availability.
One request handles the whole loop. The Gateway ARN goes in as an MCP connector, and Bedrock takes it from there; pulling the tool list, picking the right one, invoking it, and feeding the result back to the model without anything leaving AWS. The client never sees credentials, tool schemas, or the intermediate turns. See ShopAssist: E-Commerce Agent Demo for more details.
Early results: For specialist agents with short, focused tool loops, early measurements show approximately 30% lower latency. A sales agent calling three tools in sequence (check inventory, apply discount, update cart) collapses 50+ lines of orchestration into a single API call. All credentials stay server-side, and the simplified architecture makes onboarding new agent developers faster.
Integration ecosystem expansion (ongoing)
AgentFlo adds integrations based on merchant demand. Each new platform (payment processors, shipping providers, loyalty systems, or vertical-specific ERPs) becomes another MCP server connector in AgentCore Gateway. This pattern keeps expansion modular and quick.
Key takeaways
Pick your pillars first. The right AWS stack follows. AgentFlo first defined what velocity, standardization, scalability, and trust meant for the production system. With clear requirements, the AWS stack choices became obvious: Strands for the agent layer, AgentCore runtime for stateful sessions, AgentCore Gateway for tool routing and policy, and AWS Fargate for message ingestion.
Specialized agent recipes beat multi-agent complexity. Deploying a single, well-configured agent with domain-specific tools and knowledge outperforms multi-agent orchestration for most customer interactions. Multi-agent handoffs are reserved for cross-domain transitions where context boundaries are clearly defined.
Commerce is stateful. Plan for that on day one. A real sales journey can span 8 hours or 3 days. Stateless chatbots can’t maintain context that long. AgentCore runtime stateful sessions and DynamoDB-backed context were chosen on day one for exactly this reason. Retrofitting state onto a stateless agent later is much more painful than designing for it up front.
Three-layer security builds trust. Pre-turn guards on Fargate, per-tool guards through AgentCore Gateway policies with Cedar, and post-turn output filters work together to support safe deployment of autonomous agents handling commercial transactions.
System design supports scale. By building agent customization as a software as a service (SaaS) layer on top of AgentCore infrastructure, AgentFlo serves hundreds of merchants on shared infrastructure while still delivering personalized agent behavior for each one.
Serverless + AgentCore = elastic commerce. The combination of Fargate for message handling and AgentCore for agent execution means AgentFlo scales from normal traffic to 50x flash-sale spikes without pre-provisioning or capacity planning.
The feedback loop is the product. Real customer conversations teach the agents how to close. Where customers hesitate, what language converts, when they want a human handoff — all of it feeds back into recipes, prompts, and tool definitions. A new merchant deployment benefits from every conversation that ran on the platform before it. That compounding loop is harder to copy than any single piece of the architecture.
Conclusion
Building production-scale AI agents for commerce requires scalability and trust from day one. By combining Amazon Bedrock AgentCore stateful sessions and microVM isolation with AWS serverless infrastructure, AgentFlo delivers autonomous agents that handle unpredictable traffic while maintaining the security controls enterprises require.
In this post, you learn how AgentFlo built intelligent sales agents that convert conversations into completed purchases. We show you how AgentFlo improved revenue performance in early deployments using Amazon Bedrock AgentCore and the Strands Agents SDK.
AgentFlo, the agentic commerce service by Salesflo, helps merchants deploy always-on AI sales, support, and ordering agents across channels like WhatsApp. These agents understand intent, connect to commerce systems, recommend products, create carts, and convert conversations into completed transactions. Today, AgentFlo serves eCommerce merchants managing over $300 billion in annual transacted value, according to Salesflo, across services including Shopify, WooCommerce, Magento, and SAP.
This is Part 1 of a two-part series covering the five pillars of production-grade AI agents. Part 1 covers Velocity, Standardization, and Scalability. Part 2 covers Trust, Reliability, and Business results.
The challenge: customer intent without assistance
Cart abandonment hovers around 70% industry-wide, representing trillions in unrealized revenue annually. For merchants operating on messaging platforms like WhatsApp, the gap widens further:
Cart abandonment: Customers abandon carts because a single question goes unanswered.
Generic product discovery: Ranked listings replace recommendations tailored to each customer.
Missed messaging conversations: Inbound chat volume exceeds staffing capacity across time zones and languages.
No personalized guidance: Most merchants can’t afford 1:1 assistance for every interaction.
Limited outbound engagement: Teams lack bandwidth for proactive sales motions.
Peak traffic spikes: Flash sales and seasonal campaigns can spike traffic 10–50x beyond normal capacity.
Rule-based chatbots can’t handle nuanced sales conversations. Human agents can’t scale across geographies, languages, and time zones. Merchants need specialized AI sales agents that understand customer context, run complex workflows, and operate autonomously 24/7.
What makes an agent?
At its simplest, an agent combines a model, instructions, tools, context, and memory. The model reasons over a user’s request. The instructions define the agent’s role and the limits of what it should do. Tools let the agent take action in the real world. Context grounds it in business-specific data. Memory keeps the conversation coherent across turns.
In AgentFlo, those abstract components map to concrete pieces of the platform:
Component
What it does in AgentFlo
Model
Understands user intent and decides what to do next
System prompt / persona
Defines whether the agent behaves like a sales agent, restaurant agent, support agent, or receptionist
Tools
Allow the agent to search products, check inventory, create carts, place orders, raise tickets, or trigger follow-ups
Knowledge
Grounds responses in merchant-specific data such as product catalogs, menus, policies, promotions, and FAQs
Memory / state
Maintains conversation history, cart state, customer preferences, and previous actions
Channels
Connects the agent to WhatsApp, SMS, RCS, web chat, and voice
Guardrails / Policy
Prevents unsafe, unauthorized, or incorrect actions
Observability
Tracks cost, performance, conversions, and conversation quality
Building a demo agent is straightforward. Building one that runs a business safely, repeatably, and at scale requires a different approach. AgentFlo organizes this approach around five pillars.
Figure 1: AgentFlo’s production architecture on AWS.
Customer messages arrive through WhatsApp Graph API or web/mobile channels and pass through an Application Load Balancer into the AWS Fargate messaging layer. It handles authentication, image optical character recognition (OCR), speech-to-text/text-to-speech, pre-turn guardrails, and prompt injection detection. Validated requests flow into AgentCore runtime, a capability of Amazon Bedrock AgentCore, where the Strands Agents SDK orchestrates an agent that streams model inference to an external large language model (LLM). AgentCore Gateway, a capability of Amazon Bedrock AgentCore, brokers tool calls, with IAM-based authorization, to an API layer of AWS Lambda functions (Cart, Product, and Knowledge Base). It persists state across a data layer comprising Amazon DynamoDB session and cart tables, Amazon Aurora order tables, and an Amazon Bedrock Knowledge Base backed by Amazon S3. Policy in Amazon Bedrock AgentCore enforces deterministic access control independently of model reasoning. Amazon Bedrock Guardrails can also be embedded in Policy to filter prompt attacks, harmful content, and sensitive information on both requests and responses. On the observability side, logs and traces feed into AgentCore Observability, a capability of Amazon Bedrock AgentCore, while Amazon Data Firehose captures every interaction into Amazon S3 for cost and revenue analytics.
AgentFlo evaluated several hosting options before selecting Amazon Bedrock AgentCore. Three capabilities made the difference:
Stateful sessions for long-running commerce conversations.
Agent runtime: each agent session runs in its own lightweight virtual machine, providing hardware-level security boundaries between tenants.
Native MCP integration: Model Context Protocol (MCP) is an open standard that allows AI agents to connect securely to external data sources and tools through a unified interface. AgentCore Gateway supports MCP natively for standardized tool connectivity.
Pillar 1: Velocity: from merchant idea to live agent in minutes
Speed to market determines whether merchants can capture emerging opportunities. AgentFlo addresses this with a streamlined deployment model.
The challenge
Merchants want to launch agents quickly, but each has unique workflows, tone, tools, languages, products, and business rules. Generic chatbot templates are too shallow. Custom-building each agent doesn’t scale.
Recipe-based deployment
AgentFlo uses a recipe-based agent deployment model. Merchants select from pre-configured recipes, each shipping with persona, language, tone, tool sets, prompt templates, knowledge sources, response packs, and business rules. Available recipes include:
Sales agent.
Restaurant ordering agent.
Clinic receptionist.
Support agent.
B2B reorder agent.
Cart recovery agent.
Merchants fine-tune a few choices in the AgentFlo Portal. The rest is automated.
How it works
AgentFlo chose Strands Agents SDK as its agent framework. Strands Agents SDK uses a model-driven architecture: you define tools as Python functions, write a system prompt, and let the model handle orchestration. No rigid workflow graphs or hand-coded state machines.
From the merchant’s perspective, agent creation is entirely no-code. Here’s an example of the agent customization flow:
Figure 2: Agent customization workflow
This approach makes the recipe model work. Adding a new capability (like loyalty program enrollment) means writing a new tool function and updating the system prompt. No orchestration layer rewiring is needed.
Behind the scenes, one selection triggers an automated pipeline:
The portal generates a Strands agent configuration from the recipe template.
GitHub Actions packages the agent (tools, prompts, context) into a container.
The container deploys to AgentCore runtime with appropriate Gateway policies.
The agent goes live on WhatsApp within minutes.
Figure 3: How to build a customized agent under hook workflow
Each agent is a Strands Agent instance with tool definitions mapped to AgentCore Gateway endpoints. Extending an agent’s capabilities is a code change, not an architectural one.
Results
This recipe-based approach delivers faster merchant onboarding, rapid experimentation with new agent behaviors, and quick addition of new capabilities platform-wide. It also means lower engineering effort per deployment and a tighter feedback loop between customer conversations and product iteration.
Pillar 2: Standardization: reusable recipes, tools, and commerce workflows
Consistency across deployments speeds iteration and reduces maintenance burden. AgentFlo achieves this through shared building blocks.
The challenge
As AgentFlo expanded across industries, fragmentation threatened to slow the team down. Every merchant has unique catalog structures, ERP setups, system configurations (Shopify, WooCommerce, Magento), pricing rules, languages, promotions, and support processes. Without standardization, every deployment becomes a custom project, and custom projects don’t scale to hundreds of merchants.
Repeatable building blocks
AgentFlo standardizes around several core components: agent recipes (domain-specific templates), a tool marketplace (reusable capabilities), MCP-based connectors (standardized integrations), integration contracts (consistent interfaces). Additional components include prompt and context packs (reusable templates), conversation review loops (continuous improvement), and a shared semantic layer (unified product understanding). Merchant-specific complexity is pushed to the system edges. The core remains consistent.
Single-agent architecture with domain expertise
We learned early on that a single agent with domain-specific knowledge and a curated tool set outperforms multi-agent architectures for most customer interactions. Each AgentFlo deployment configures a distinct persona, voice, language, and specialized tool set tailored to the business domain (sales agent, restaurant agent, clinic receptionist). It ships with curated contexts, prompt templates, and response packs. Merchants deploy these domain-specific agents through the self-service portal.
A single focused agent maintaining unified context converts better than multiple generalists coordinating with each other. Multi-agent capabilities remain available where valuable. For example, when conversations transition from sales to support, context hands off cleanly to the specialist agent.
Tool routing through AgentCore Gateway
Centralized tool management routes agent requests to dedicated AWS Lambda functions. When a customer asks about product availability, the agent queries the product catalog. When they’re ready to buy, it handles cart operations. For personalized recommendations, it retrieves data from Amazon Bedrock Knowledge Bases, the fully managed Retrieval Augmented Generation (RAG) capability, backed by merchant data stored in Amazon S3. Sales intelligence APIs provide additional context for each interaction.
OAuth tokens and platform credentials live in the Gateway, not in agent sessions. Policy in AgentCore sits alongside the Gateway, enforcing fine-grained, Cedar-based access control rules that operate independently of model reasoning. Cedar is an open-source policy language developed by AWS that allows fine-grained, verifiable authorization decisions.
Policies define which agent sessions can invoke which tools. For example, a sales agent can’t call customer-support-only APIs.
For standardization, this means every new tool added to the platform (payment integration, shipping provider, loyalty system) becomes available to every applicable recipe through the same mechanism. Tools aren’t re-implemented per merchant.
AgentCore Gateway as the integration backbone
AgentFlo integrates with dozens of eCommerce services: Shopify, WooCommerce, Magento, SAP, payment processors, shipping providers, and loyalty systems. Each integration is defined as an MCP server connector, with Gateway handling discovery, authentication, and routing. Furthermore, AgentFlo has many different agent recipes, each with their own specialized tool packs to provide that functionality. To make these connections modular and efficient, AgentFlo uses AgentCore Gateway.
From the agent’s perspective, the full Gateway tool surface is reachable in a few lines:
from strands import Agent
from strands.models import BedrockModel
from strands.tools.mcp.mcp_client import MCPClient
from mcp.client.streamable_http import streamablehttp_client
def create_transport():
return streamablehttp_client(
GATEWAY_URL,
headers={"Authorization": f"Bearer {access_token}"},
)
mcp_client = MCPClient(create_transport)
with mcp_client:
# Discover every tool registered on the Gateway in one call.
# cart, product catalog, knowledge base, shipping, loyalty, etc.
tools = mcp_client.list_tools_sync()
agent = Agent(
model=BedrockModel(model_id="us.anthropic.claude-sonnet-5-20260630"),
tools=tools,
system_prompt=SALES_AGENT_PROMPT,
)
response = agent("Do you have the red leather wallet in stock?")
Connecting a Strands agent to AgentCore Gateway. A single list_tools_sync() call gives the agent every integration registered on the Gateway: cart, product catalog, knowledge base, shipping, loyalty. Onboarding a new service for a merchant is a Gateway change, not an agent change.
Key capabilities:
MCP-native tool connectivity: Each platform or tool set integration is a standard MCP server connector.
OAuth and credential management: Platform API keys and OAuth tokens are managed centrally in Gateway, never exposed to individual agent sessions.
Code simplicity: The code is cleaner, shorter, and more modular, which simplifies configuration for scale. The alternative is extensive local code for each connection or tool set.
Because each new service integration and recipe-specific tool set is defined as an MCP server connector in AgentCore Gateway, expansion is modular and quick. Adding a new service or tool set requires a connector definition, not a re-architecture.
Conversation reviews as a standardization loop
Standardization also comes from learning. AgentFlo continuously reviews real conversations to understand how customers ask for products, where they drop off, which recommendations convert, when handoff is needed, and how local language affects buying behavior.
These reviews feed back into recipes, prompts, and tool definitions. Standardization is something the platform earns over time, not something declared at launch.
Results
Every deployment improves future deployments, and new integrations become reusable across the merchant base. Agent behavior stays consistent across recipes and merchants, and workflows become repeatable across industries. The service becomes harder to replicate because it learns from real commerce behavior, not generic templates.
Commerce conversations are unpredictable in volume and duration. AgentFlo’s architecture handles both dimensions without manual intervention.
The challenge
During flash sales, product launches, or restaurant rush hours, customer conversations can spike 10–50x. Human teams can’t scale that fast. AgentFlo must also support many merchants and concurrent customer sessions simultaneously. One merchant’s surge can’t affect another’s experience.
AgentFlo’s serverless architecture
AgentFlo uses a serverless architecture. Message ingestion, agent execution, tool execution, state, analytics, and billing each scale independently. Each layer absorbs its own spikes without requiring the rest of the system to over-provision.
AgentCore runtime properties for scale
Several AgentCore runtime properties specifically support scale:
Isolated microVM execution: Each agent session runs in its own environment with dedicated CPU, memory, and filesystem. The environment is sanitized on termination. One merchant’s sessions never interfere with another’s.
Stateful sessions up to eight hours: Long-running conversations don’t lose context. A customer browsing in the morning can continue the same assisted session that evening.
Framework-agnostic: AgentCore runs Strands Agents natively but also supports any containerized agent framework, giving AgentFlo flexibility to evolve the agent architecture over time.
How it works
The architecture is built end-to-end on AWS:
Messaging layer (AWS Fargate): An Application Load Balancer routes incoming WhatsApp messages to a Fargate application that handles authentication, voice message conversion (Opus OGG to MP3 transcription with fuzzy matching for product name recognition). The application also runs pre-turn security guards. AWS End User Messaging provides an alternative channel option for broader reach.
Agent orchestration (Amazon Bedrock AgentCore runtime): Each customer session spawns an isolated agent instance running the Strands Agents SDK. Sessions are stateful for up to eight hours and isolated through microVM architecture, where each session runs in its own lightweight virtual machine. They are persistent, with filesystem access for intermediate results and cached product catalogs. microVM isolation keeps merchants completely separated.
This is the entire bridge between Strands SDK agent code and a production-ready endpoint on AWS:
from strands import Agent
from bedrock_agentcore.runtime import BedrockAgentCoreApp
app = BedrockAgentCoreApp()
agent = Agent(
tools=tools, # from the Gateway, per snippet above
system_prompt=SALES_AGENT_PROMPT,
)
@app.entrypoint
def invoke(payload, context):
"""One AgentCore session per customer conversation."""
user_message = payload.get("prompt")
session_id = getattr(context, "session_id", None) # stable for up to 8 hours
result = agent(user_message)
return {"result": result.message}
if __name__ == "__main__":
app.run()
This is all the glue between a Strands agent and AgentCore Runtime. BedrockAgentCoreApp wraps the agent in the standard /invocations contract, and AgentCore handles microVM provisioning, session isolation, scaling, and stateful sessions up to eight hours. Two CLI commands take it from a local file to a live endpoint on AWS. No Dockerfile, no API routing, no web framework to maintain.
Tool execution (Amazon Bedrock AgentCore Gateway): Tool calls scale separately from agent reasoning, so a sudden burst of cart operations doesn’t slow down the agent loop itself.
Results
The architecture handles peak traffic without pre-provisioning capacity and supports long-running conversations that survive across visits. It provides strong multi-merchant isolation with lower operational overhead than traditional always-on infrastructure, resulting in a better customer experience during high-intent moments like product launches or flash sales.
What’s next
In Part 2 of this series, we explore:
Pillar 4: Trust. Guardrails for autonomous commercial action and real-time visibility into agent operations.
Pillar 5: Reliable. Data foundation that ensures agents act on reliable, up-to-date information to complete tasks with precision.
Business results: Measurable impact across the customer lifecycle.
Future roadmap: Voice agents, server-side tool execution, and integration expansion.
Summary
In this post, we explored three of the five pillars for building production-grade AI agents:
Velocity: How recipe-based deployment allows merchants to launch AI sales agents in minutes using the model-driven architecture of the Strands Agents SDK.
Standardization: How reusable building blocks and centralized tool management through Amazon Bedrock AgentCore create consistency across hundreds of deployments.
Scalability: How AgentFlo handles elastic, stateful commerce conversations at scale through Amazon Bedrock AgentCore and AWS Fargate.
Clario, part of Thermo Fisher Scientific, uses Amazon Bedrock to automate PHI (Protected Health Information) and PII (Personally Identifiable Information) detection across thousands of DICOM (Digital Imaging and Communications in Medicine) image slices in clinical trials. Each image slice may carry PII or PHI hidden in metadata tags, in custom vendor fields, or burned directly into the pixels. Across imaging sites, central labs, sponsors, and CROs (Contract Research Organizations), every one of those slices must be cleared of PII and PHI before the image moves downstream.
DICOM is the universal standard for storing, transmitting, and managing medical imaging data across healthcare systems. In clinical trials, DICOM images play a critical role by providing objective, quantifiable evidence of a patient’s medical condition throughout the study lifecycle. From baseline imaging to follow-up scans, modalities such as MRI, CT, PET, and X-ray generate DICOM files. Radiologists, clinicians, and sponsors use these files to assess treatment efficacy, monitor disease progression, and support regulatory submissions. These images serve as a core component of the clinical evidence package, making their accurate management and standardized handling essential to trial integrity.
In this post, we share how the Clario team designed an automated PHI and PII detection solution on AWS for DICOM imaging data, the key design decisions behind the architecture, and the lessons the team learned along the way.
About Clario, part of Thermo Fisher Scientific
Clario science and endpoint solutions support the clinical trials industry through the systematic collection, management, and analysis of specific, predefined outcomes (endpoints) to evaluate a treatment’s safety and effectiveness. For more than 50 years, Clario endpoint solutions have been deployed more than 30,000 times, and since 2015, they have supported more than 700 FDA and EMA new drug approvals.
Business challenge
Clearing PII and PHI from every image slice in the clinical trial is only part of the problem. The imaging workflow around this clearing process must be just as rigorous. A well-structured imaging workflow supports every DICOM file captured across globally distributed trial sites. Files are ingested automatically, consistently standardized, and rigorously validated at every step of the journey. Enforcing standardized image acquisition protocols across sites and geographies alleviates inconsistencies. These inconsistencies could otherwise impact data quality or delay regulatory submissions. A centralized imaging infrastructure that maintains complete metadata traceability, including acquisition parameters, imaging equipment details, and timestamps, supports a fully auditable workflow aligned with GCP (Good Clinical Practice) requirements. This empowers sponsors and CROs to move faster with greater confidence and significantly reduces the risk of data queries or compliance gaps.
An equally important aspect of managing DICOM imaging data in clinical trials is embedding intelligent, automated PHI and PII protection directly into the data management process. DICOM files carry more than images. They include metadata and tags, which can contain sensitive information such as patient names, dates of birth, medical record numbers, and facility identifiers. This sensitive information must be carefully managed before sponsors, CROs, or third-party stakeholders receive the data. Proactively verifying that PII and PHI are accurately identified and de-identified at the source is a critical best practice that safeguards patient privacy in compliance with HIPAA, GDPR, and ICH E6 guidelines. Automated de-identification tools that adhere to DICOM Supplement 142 and NEMA (National Electrical Manufacturers Association) standards reinforce data security and regulatory trust. They also preserve the full clinical and scientific value of imaging data, so trial teams can support confident, high-quality regulatory submissions.
To address these challenges, the Clario team built a comprehensive PHI/PII detection solution on AWS using Amazon Bedrock (Anthropic’s Claude Sonnet 4.5 on Amazon Bedrock) that combines automation, accuracy, and security throughout the clinical trial imaging workflow.
Why Amazon Bedrock and Amazon Textract
When evaluating options for building the solution, the Clario team chose to standardize on Amazon Bedrock and Amazon Textract for several key reasons:
Scalability without re-architecting: Amazon Bedrock and Amazon Textract provide scalability, reliability, and strong performance. The solution architecture can scale from a handful of documents to millions without re-architecting the solution or managing additional infrastructure. AWS manages the underlying capacity, so you can focus on building features instead of tuning servers or models.
Security and compliance: Keeping customer data secure is non-negotiable. By using Amazon Bedrock and Amazon Textract within Clario managed AWS accounts, processing remains inside a hardened AWS environment, taking advantage of AWS Identity and Access Management (IAM), Amazon Virtual Private Cloud controls, and encryption at rest and in transit. The Clario team can align closely with its organization’s security, compliance, and data residency requirements.
Managed foundation models: With Amazon Bedrock, the Clario team can access a range of high-quality foundation models as a fully managed service, without having to manage model training, hosting, or updates. This shortens the time to market, and you can iterate quickly as new models and capabilities become available in Amazon Bedrock.
Purpose-built OCR and document processing: Amazon Textract provides purpose‑built optical character recognition (OCR) and intelligent document processing, which significantly improves accuracy over traditional OCR engines. Its ability to automatically detect and extract text, tables, and key‑value pairs from complex documents and images reduces the amount of custom parsing logic that must be maintained.
End-to-end observability: Running on AWS provides end-to-end observability across the logs, metrics, and traces through services like Amazon CloudWatch and AWS CloudTrail. The Clario team can enforce governance policies, audit permissions, and track model and document processing usage centrally.
Extensible AI foundation: Because the solution builds on Amazon Bedrock and Amazon Textract, the Clario team can adopt new models and document processing features as they become available, without re-architecting.
Solution overview
The solution is built entirely on AWS, designed to bring greater efficiency, accuracy, and security to the detection of PHI and PII embedded within DICOM files. Accessible through Amazon API Gateway with TLS encryption in transit, IAM-backed authorization, and rate limiting, the detection workflow is readily consumable by multiple downstream systems with minimal integration effort.
Clinical trial sites store their DICOM images in Amazon Simple Storage Service (Amazon S3). The detection workflow retrieves each file from that bucket and processes it through the detection pipeline, so every ingestion step is logged and auditable for clinical trial security and compliance reviews. The workflow scans both standard and custom private DICOM metadata tags for PHI and PII. This covers the vendor-specific and non-standard tags where sensitive information often hides. Supporting both DICOM (.dcm) and PDF file formats, the solution is well-positioned to address PHI detection needs across the most used file types in clinical trial workflows.
The Clario AI team made a few deliberate design decisions early on. They ran the backend on Amazon Elastic Kubernetes Service (Amazon EKS) because a single DICOM series can span thousands of slices, and the detection workload is long-running and memory-intensive. They chose Amazon Relational Database Service (Amazon RDS) for PostgreSQL to persist processing metadata because the audit trail needs relational queries and strong consistency for compliance reporting. And they put the service behind Amazon API Gateway so that authentication, API-key management, and rate limiting are handled at the edge, keeping the backend focused on detection.
The following diagram and steps show how a DICOM document moves from upload through detection to structured output:
Figure 1: Solution architecture for DICOM image ingestion, detection pipeline, and data retention workflow
The following steps describe the data flow through the solution, as shown in the architecture diagram:
A consumer, such as an upstream imaging application, first uploads the DICOM image document to an Amazon S3 bucket location that is accessible to the solution.
The consumer then calls the Clario Internal API running on Amazon API Gateway, providing their consumer-specific API key and the location of the document. This call initiates the DICOM image analysis workflow.
Amazon API Gateway fronts the API and receives the incoming request. API Gateway validates the API key and, on success, forwards the request to the detection backend endpoint running on Amazon EKS to initiate processing.
The solution performs initial checks on the file location (for example, URL format, access, and basic metadata) and then begins the PII/PHI identification process. The pipeline retrieves the file from the source S3 bucket and ingests it into the detection workflow.
The file is stored in an internal Amazon S3 bucket and relevant metadata persisted in a PostgreSQL database on Amazon RDS to support downstream processing and auditability.
The workflow invokes Amazon Textract to perform OCR and intelligent document parsing. Textract extracts text, tables, and form fields from the uploaded document, returning a structured representation of the content.
The OCR output is then passed to a large language model (Anthropic’s Claude Sonnet 4.5 on Amazon Bedrock) that is configured to analyze the extracted text and identify potential PII/PHI elements. The model evaluates the content and associates each detected sensitive element with its position in the document.
Once analysis is complete, the detection workflow returns a structured response to the consumer, containing the coordinates and related metadata for each piece of sensitive data identified in the document. The consumer can take follow‑up actions, such as redaction or masking.
To minimize data exposure and support compliance requirements, the ingested files and associated records are retained only for a limited window. The document stored in Amazon S3 is automatically deleted based on an Amazon S3 lifecycle retention policy, and corresponding records in Amazon RDS are removed via a scheduled cleanup job.
Deep image analysis and detection workflow
Beyond metadata, the solution uses Anthropic’s Claude Sonnet 4.5 on Amazon Bedrock to perform a deep scan of the actual image pixel content, detecting PHI or PII that may be physically burned into the image itself. This includes patient names, dates of birth, and patient IDs across every individual slice within a DICOM series that can span thousands of images.
When PHI or PII is identified, the solution precisely captures the spatial coordinates and type of sensitive information detected, passing these bounding box details downstream to integrated systems responsible for the actual pixel-level redaction. Separating detection from masking was a deliberate design decision. It preserves flexibility, supports full auditability, and a human-in-the-loop can review the flagged findings before redaction is applied.
Figure 2: Deep image analysis and detection workflow showing the separation between AI-powered detection and human-supervised redaction
The detection solution returns structured coordinates for the identified PHI and PII, spanning burnt-in pixel text, standard DICOM header fields, and non-standard custom tags. The solution then hands these results off to two downstream processes. In the quality control (QC) flow, qualified reviewers validate the flagged findings and confirm which items require remediation. In the redaction flow, the system executes the appropriate action for each type of PHI identified: masking or overwriting burnt-in text within the image pixel data, or stripping and zeroing out sensitive DICOM metadata tags.
This separation of detection and redaction is intentional. The AI-powered detection solution focuses on comprehensive, high-recall identification across thousands of image slices and metadata fields. The redaction flow retains human oversight over the irreversible act of modifying clinical data, confirming that no PHI is left exposed and no clinically relevant information is removed.
Figure 3: Sample DICOM image with detection
With the detection pipeline in place, the next step was to measure how accurately it identifies PHI and PII across real-world clinical documents.
Evaluation methodology
The team validated the solution in three stages: building a representative test dataset, creating ground truth annotations, and running an automated evaluation pipeline.
Building a representative test dataset
The Clario generative AI team partnered with internal stakeholders to assemble a diverse dataset, including:
DICOM images with burned‑in annotations, overlays, and metadata.
PDF documents such as reports and clinical summaries.
The dataset intentionally included documents that do and do not contain sensitive PII/PHI, allowing the team to measure both the model’s ability to detect sensitive information and its ability to avoid false alarms.
Creating ground truth annotations
For each document in the dataset, ground truth labels were generated that capture:
The exact text corresponding to each PII/PHI element.
The bounding box coordinates for each element on the page where available.
These annotations form the “gold standard” that the Clario team can use to compare the output from the production pipeline.
Automated evaluation pipeline
The Clario team implemented a set of evaluation scripts that:
Run the full solution on each test DICOM or PDF, using the same workflow that powers the production system.
Collect the model predictions, including the detected PII/PHI text, associated labels (for example, name, date of birth, medical record number), and coordinates where available.
Compare predictions against ground truth using the following matching strategy:
For PDFs and DICOM images where coordinates are available, a match is performed by spatial proximity, treating a predicted bounding box as a correct match if it falls within a configurable tolerance (by default, 3 pixels for each element of the bounding box).
For DICOM metadata where coordinates are not available, a match is performed by a structured path.
Furthermore, the solution includes automated accuracy and performance (run time) checks, to improve system reliability across deployments. After validating the solution’s accuracy, the team assessed how it improves detection coverage across Clario clinical trial imaging workflows.
Results and benefits
The automated evaluation pipeline measured the solution’s detection performance against the manually annotated ground truth dataset across all three detection surfaces:
Detection surface
Detection F1
Label accuracy
PDF text
0.9775
98.12%
DICOM burned-in image text
0.9750
96.15%
DICOM metadata tags
0.9951
99.60%
Detection F1 measures how accurately the solution identifies PHI/PII instances. Label accuracy measures how correctly it classifies the type of identified PHI/PII (for example, person_name, date_of_birth, or gender).
These results demonstrate consistently high detection performance across all three data surfaces, with metadata tag detection achieving near-perfect accuracy. The solution meets the Clario generative AI team’s production-readiness bar for deployment in clinical trial workflows where compliance accuracy is non-negotiable.
Complete detection coverage
Manual QC reviewers bring deep domain expertise to PHI identification. But modern clinical trials generate an enormous volume of data: thousands of image slices per series, each with dozens of metadata tags, including non-standard vendor-specific fields. This volume makes exhaustive manual review impractical at scale. The automated solution extends that human expertise across the full dataset.
In internal testing conducted by the Clario team, the solution scanned 100% of image slices, standard DICOM header fields, and custom private tags in the test dataset. This comprehensive coverage complements the existing QC process by surfacing PHI occurrences that might otherwise require additional review passes, particularly in non-standard private tags and burned-in pixel text where sensitive data is less predictable.
Risk and compliance impact
By automating PII and PHI detection across metadata tags and image slices, the solution can strengthen an organization’s compliance posture against HIPAA, GDPR, and ICH E6 requirements.
Beyond the measurable results, the project surfaced several insights that can guide other organizations building similar solutions.
The AWS collaboration
The AWS Solutions Architecture team partnered with the Clario AI team throughout the design and optimization of the detection solution. Key areas of collaboration included:
Scaling and throughput optimization: Provided prescriptive guidance on Amazon EKS pod scaling strategy to handle DICOM series with thousands of slices per request without timeout or memory pressure and tuned concurrent invocations to Amazon Bedrock to maximize throughput within account-level quotas.
Cost-efficient inference architecture: Recommended batching strategies for Amazon Textract API calls and optimized prompt token usage for Claude Sonnet on Amazon Bedrock to reduce per-document inference cost at scale.
Data retention and security controls: Recommended auto-deletion workflow using Amazon S3 Lifecycle policies and Amazon RDS scheduled jobs to meet HIPAA and GDPR data minimization requirements.
Lessons learned and best practices
Throughout the development and deployment of this solution, several valuable insights emerged that can benefit other organizations implementing similar AI-powered PHI detection systems for clinical trial imaging data.
Evaluate models against production-representative data
The Clario AI team adopted a rigorous model evaluation process early in development. Many open-source frameworks and off-the-shelf detection models demonstrated acceptable performance on curated test samples but experienced significant accuracy degradation when exposed to the full variability of production data. This variability includes diverse imaging modalities, vendor-specific private tags, and inconsistent burned-in text formatting across globally distributed trial sites. This reinforced the importance of evaluating any AI model at realistic, production-level data volumes before adoption. The solution that proved most effective was a carefully tuned pipeline where Amazon Textract handles text extraction and Claude Sonnet on Amazon Bedrock performs PHI/PII classification, with prompt engineering optimized for the specific patterns found in clinical trial DICOM data.
Ground truth data is non-negotiable
Building a reliable, automated evaluation pipeline required the manual creation of a ground truth dataset. The team acknowledges this process is time-consuming but necessary. This highlighted a best practice that is frequently underestimated: investing in high-quality, manually validated ground truth data is a prerequisite for developing and maintaining a trustworthy automated detection system. Attempting to shortcut this step risks deploying a solution whose real-world accuracy remains unknown, an unacceptable risk in the context of clinical trial compliance.
Separating detection from masking improves flexibility and auditability
The Clario team deliberately separated the PHI/PII detection function from the actual pixel-level redaction. Rather than performing masking directly, the solution identifies the precise coordinates and type of PHI/PII detected, passing this structured output downstream to integrated systems responsible for redaction. This separation proved to be a sound best practice. It preserves workflow flexibility, a human expert can review the findings before anyone makes irreversible changes to the image data, and keeps human accountability and auditability clear at every step.
Human-in-the-loop review remains an essential safeguard
Automation accelerates the detection and flagging process, but a key lesson learned is that human oversight should remain an integral part of the workflow. Incorporating a human review step for flagged findings before masking makes sure that edge cases and model uncertainties are appropriately handled. In the context of clinical trial data, where accuracy and regulatory accountability are paramount, this human-in-the-loop approach provides an essential layer of quality assurance that purely automated systems alone cannot fully replace.
Conclusion
The Clario automated PHI/PII detection solution demonstrates how AWS services can transform clinical trial imaging workflows by combining speed, accuracy, and compliance. By replacing manual spot-checks with automated scanning of every slice and metadata tag, the solution delivers complete PHI/PII detection coverage, reducing the risk of missed detections while strengthening compliance with HIPAA, GDPR, and ICH E6 requirements.
The key architectural decisions, comprehensive coverage of custom private tags, separation of detection from redaction, and human-in-the-loop validation provide a blueprint for other organizations managing sensitive imaging data in regulated environments. These lessons learned highlight that successful automation in clinical trials requires not just advanced technology, but thoughtful design that balances efficiency with the rigorous quality standards that patient safety and regulatory compliance demand.
Next steps
Organizations looking to implement similar PHI/PII detection capabilities for clinical trial imaging can start by:
Evaluate current manual review processes: Map where reviewers spend the most time and where missed PHI poses the greatest compliance risk.
Assess custom private DICOM tags: Catalog vendor-specific and site-specific tags across your imaging network to define the full detection scope.
Build ground truth datasets: Annotate a representative sample with precise PHI labels to benchmark automated detection accuracy.
Design human-in-the-loop workflows: Define review checkpoints where qualified personnel validate flagged findings before redaction.
talabat is the leading everyday app in the Middle East and North Africa (MENA) region, offering customers a convenient and personalized way to order food, groceries, and other everyday essentials from a wide selection of restaurants and retailers. Founded in Kuwait in 2004, talabat has expanded its operations to the United Arab Emirates, Oman, Qatar, Bahrain, Jordan, Iraq, and Egypt, serving over seven million monthly active customers as of December 2025. talabat is headquartered in Dubai, United Arab Emirates, and in December 2024 successfully completed its initial public offering on the Dubai Financial Market (DFM). As a subsidiary of Delivery Hero SE, talabat uses global expertise to continuously enhance its service, expand its landscape, and drive innovation. With a strong network of partners and riders, talabat connects customers to what they need, when they need it – powering everyday convenience across the region.
In this post, we show how talabat built a hybrid, multi-cloud lakehouse that keeps a single Apache Iceberg copy of streaming data on AWS while enabling governed, near-real-time analytics from Google Cloud Platform (GCP).
Data at talabat
Data is the nervous system of talabat’s business. From the moment a customer hits “order” to the second their doorbell rings, talabat’s systems make split-second, data-driven decisions, instantaneously optimizing pricing, dispatch, routing, and order security. Over the years, talabat’s application grew into a landscape spanning two public clouds. Our transactional and operational backbone matured on AWS, where the engineering teams build and operate services. In parallel, a large population of analysts, data scientists, and analytics-engineering pipelines standardized on the Google Cloud Platform warehouse, Google BigQuery.
Both investments are deep, and both deliver value. So the strategic question wasn’t “which cloud do we consolidate on,” but rather “how do we make our data flow cleanly across the boundary between them.” That framing shaped everything that follows. The challenge isn’t only cross-cloud but cross-Region as well, with AWS services hosted in the EU region and the data in the GCP US region.
The following diagram shows how talabat’s data flows between the operational plane on AWS and the analytics plane on GCP.
Figure 1: Data flow between the operational plane on AWS and the analytics plane on Google Cloud
Historically, the data engineering team orchestrated the data movement between the two clouds, mandating a physical movement from AWS to GCP, EU to US. Moving this using conventional extract, transform, and load (ETL) tools and frameworks delayed and duplicated the data through multiple hops: Amazon Relational Database Service (Amazon RDS) to Amazon Simple Storage Service (Amazon S3) EU AWS Region, Amazon S3 EU to Amazon S3 US Region, and finally Amazon S3 US to BigQuery US.
Each hop was a copy, and every copy compounded risk: multiple failure points, compounding latency, redundant compute and storage, type fidelity, and most importantly, cross-Region and cross-cloud egress cost.
In short, the old design paid in dollars, latency, and reliability to solve a problem it had created for itself: it moved data so that BigQuery could read it. A classic data warehouse bottleneck. Could we use an open data lake instead? Yes. But the analytics usage is heavy on BigQuery, which limits access through an open source data lake layer. So the redesign started from the opposite premise: keep one copy on AWS and let BigQuery read it in place. That is what the rest of this post describes: a lakehouse for talabat.
Challenges
Operational systems emit a continuous stream of business events like order lifecycle changes, vendor, menu, logistics and rider signals, and payments information published to Apache Kafka on Amazon Managed Streaming for Apache Kafka (Amazon MSK). These events are encoded as Protocol Buffers and governed by backward-compatible schemas registered in Confluent Schema Registry, so producers and consumers can evolve safely over time.
The requirement on the analytics side is straightforward to state and hard to meet: make these events queryable, correctly typed, within minutes of being produced, and make them queryable from the tools each team already uses.
It’s tempting to view a two-cloud footprint as technical debt. For a real-time business like talabat, it’s simply the terrain, and each side plays to a genuine strength:
The event backbone lives on AWS. Our transactional and streaming systems publish to Amazon MSK. The lowest-latency, lowest-risk place to consume and process those events is next to them, in the same AWS Region.
The analytics estate lives on Google Cloud. Thousands of downstream models and dashboards, and the people who build them, assume BigQuery as the query surface.
Consolidating either side would mean a multi-year migration and a significant regression in capability for one group of users, all to remove a seam between ingestion and analytics. Data engineers decided to engineer the seam instead. The design goal became a single sentence: keep one physical copy of the data on AWS, and read it natively from both clouds. A hybrid data lakehouse makes the “which cloud” question an access-path detail rather than an architectural fork.
What we tried first: Cross-cloud writes on the hot path
Our first attempt inverted the flow we eventually shipped. Raw (also called Bronze) layer data was written from AWS directly into BigQuery-managed Iceberg tables on Google Cloud Storage. On paper, this placed the data closest to the largest consumer base. In practice, writing across clouds on an always-on streaming path introduced a class of problems we did not want to live with:
A cross-cloud dependency on the ingestion path. Every micro-batch was coupled to the availability and latency of a remote cloud’s write API.
Streaming write-API failures surfaced as ingestion incidents. The remote write became the fragile link, turning read-side concerns into write-side outages, the worst place to absorb them.
Preview-gated capabilities constrained the physical layout. Certain partitioning behaviors and features were not generally available, limiting how we could organize the data for cost and performance.
The lesson was clear: Shift left. The write path should be short, local, and straightforward. The cross-cloud concern belongs on the read path, where it can be made read-only, cached, and retried without affecting the ingestion. That reframing led directly to the architecture we run today.
Choosing how BigQuery would read AWS resident data
With the flow inverted (raw data on AWS, read from Google Cloud), we evaluated three ways for BigQuery to read tables that physically live on AWS. We assessed each against four criteria:
No data movement.
An open table format.
A governable trust model.
Minimal operational surface.
Approach
Assessment
Cross-cloud write to Google Cloud Storage
Continue writing bronze into BigQuery-managed Iceberg on Google Cloud Storage. We rejected this for the preceding reasons: it puts a cross-cloud dependency and cross-Region latency on the ingestion hot path.
Query AWS resident data through the managed cross-cloud compute of BigQuery Omni. This introduced more managed surface and more constraints than we needed for a read-only bronze layer, and we wanted to own the catalog and trust model directly.
Let BigQuery read data in Amazon S3 Tables, a capability of Amazon S3 that provides managed Apache Iceberg tables, through a federated catalog that synchronizes AWS Glue Data Catalog metadata, with access authenticated by cross-cloud IAM trust. This met all four criteria, and we chose it.
The deciding properties were that the raw data doesn’t leave AWS, the format is open Apache Iceberg (so Amazon Athena, Spark, and Iceberg-compatible engines read the same tables), and the cross-cloud relationship is expressed as identity and trust rather than as a recurring copy job.
Why Amazon S3 Tables
With the architecture settled on a single Iceberg copy living on AWS, we needed a storage layer purpose-built for Iceberg at scale. Amazon S3 Tables met the requirements without adding operational surface. Table maintenance (compaction, snapshot expiration, and unreferenced file removal) runs automatically as a service-managed policy, avoiding the need for external orchestration jobs that would otherwise grow linearly with table count. Equally important, every table is an Amazon Resource Name (ARN)-addressable resource. That means IAM policies can grant or deny access for individual tables, the same least-privilege model we apply to any other AWS resource, and AWS CloudTrail records every access decision. For a cross-cloud design where the trust boundary is expressed entirely through IAM, having tables that are first-class IAM resources isn’t a convenience but a prerequisite. S3 Tables gave us managed Iceberg housekeeping and fine-grained, auditable access control in a single construct, so the engineering team could focus on the streaming logic rather than the storage plumbing beneath it.
Solution overview
The system has two halves that meet at an open table format:
A short, local write path on AWS.
A read-only cross-cloud handshake that lets BigQuery consume the data.
The single source of truth is Apache Iceberg data in Amazon S3 Tables. Every consumer reads that one physical copy.
The following diagram shows the end-to-end architecture, from event ingestion through storage to consumption paths.
Figure 2: End-to-end architecture from event ingestion through storage to consumption paths
The write path: Short, local, and reliable
We run one Amazon EMR Serverless Spark Structured Streaming job per Kafka topic (with a prebaked Docker image, emr-7.13.0 on ARM64/Graviton) in the same AWS Region (eu-west-2) as Amazon MSK. Co-locating compute with the event backbone minimizes the data transferred per micro-batch, saving cost and latency. Each job runs the Spark foreachBatch operation with a trigger interval of roughly one to five minutes and at-least-once delivery. Every micro-batch performs five steps:
Consume from Kafka.
Decode Protocol Buffers using the registered schema.
Transform to the target Iceberg schema.
Append to the Iceberg table in Amazon S3 Tables.
Commit offsets.
The cycle repeats without interruption.
This path touches only AWS. There is no cross-cloud dependency, only one deliberate cross-Region hop: compute in the Europe (London) Region (eu-west-2), storage in the US East (N. Virginia) Region (us-east-1). This incurs standard AWS inter-Region data transfer cost, a deliberate choice so that the cross-cloud read from BigQuery stays within the same Region.
Bad records don’t block the stream. They land in a dedicated dead-letter queue (DLQ) table (<table>_dlq) in a separate S3 Tables bucket, storing the raw payload (raw_value_b64) and a skip_reason. Nothing is silently dropped. The DLQ tables are registered with the AWS Glue Data Catalog through Lakehouse, so engineers can inspect failures from Amazon Athena or BigQuery.
From this point on, Amazon S3 Tables is the source of truth.
The crux: Cross-cloud handshake
This is the heart of the design. BigQuery reads the S3 Tables Iceberg data through a Lakehouse federated Apache Iceberg REST catalog, a read-only catalog on the Google Cloud side that points at the AWS resident tables. Three mechanisms make it work.
An open catalog contract (Iceberg REST).
Amazon S3 Tables exposes an Apache Iceberg REST catalog interface, and Google Lakehouse speaks that same standard. Because both sides agree on the Iceberg on-disk format and REST catalog protocol, no translation layer or data copy is required. BigQuery reads the identical Iceberg data files that Athena and Spark read.
On the Google Cloud side this is a single Lakehouse federated catalog. A table surfaces to analysts as talabat-data.s3tables-glue.catalog.orders.
Cross-cloud identity and trust (IAM and OIDC).
The Lakehouse catalog authenticates to AWS as a Google-managed service identity (the Lakehouse REST-catalog service account) that an AWS Identity and Access Management (IAM) role trusts through OpenID Connect (OIDC) federation with accounts.google.com, using sts:AssumeRoleWithWebIdentity with the service account’s numeric ID pinned in the role’s trust policy. Requests to the S3 Tables Iceberg endpoint are SigV4-signed. It’s the same AWS request-signing scheme that any AWS SDK uses, scoped to the S3 Tables service. In other words, the handshake isn’t a proprietary connector. It’s standard AWS request signing performed by a trusted external identity.
The trust is codified as infrastructure as code (IaC) on the AWS side: granted least-privilege, and revocable at any time. The following diagram shows this authentication sequence.
Figure 3: Cross-cloud authentication sequence between the Lakehouse catalog and AWS IAM
The federated catalog periodically synchronizes table metadata from the AWS Glue Data Catalog that fronts S3 Tables. Newly created tables and new data become visible to BigQuery on a short refresh cycle (approximately 300 seconds). Reads are served against the live Iceberg data. Only the catalog pointers are synchronized.
The result is that a table written once on AWS appears in BigQuery as an ordinary catalog object and can be queried with standard SQL, while the bytes don’t leave AWS and the format stays open.
Infrastructure as code: The cross-cloud trust surface
The following section explains the authentication handshake shown in the architecture diagram. The Lakehouse catalog service account presents a Google OIDC JSON Web Token (JWT), which AWS validates through the IAM OIDC provider, returning short-lived credentials scoped to read-only S3 Tables access.
Register Google as a trusted identity provider. Scoped to our Lakehouse catalog’s service account:
Pin the trust to exactly that one identity. This is the security crux. The role can only be assumed through a Google-signed token whose subject matches our service account. A condition on the sub claim closes the door to every other principal:
data "aws_iam_policy_document" "trust" {
statement {
actions = ["sts:AssumeRoleWithWebIdentity"]
principals {
type = "Federated"
identifiers = [aws_iam_openid_connect_provider.google.arn]
}
condition {
test = "StringEquals"
variable = "accounts.google.com:sub"
values = [var.lakehouse_sa_subject_id] # nobody else can assume the role
}
}
}
resource "aws_iam_role" "lakehouse_read" {
name = "bq-lakehouse-read"
assume_role_policy = data.aws_iam_policy_document.trust.json
max_session_duration = 43200 # 12-hour sessions, then re-issued
}
Grant read-only, least privilege. The assumed role carries only enough to read the catalog metadata through AWS Glue and access the Iceberg data through S3 Tables, secured entirely by IAM policy and nothing writable:
The Google-side catalog is bound to this role. The Lakehouse federated catalog itself is created out of band (a one-time gcloud call), pointed at the preceding role so that every read presents that trusted identity. No AWS keys ever live in Google Cloud:
Together these four steps are the whole handshake: a trusted issuer, a role that only our service account can assume, a least-privilege read grant, and a catalog bound to that role.
Operational lessons: Metadata as a first-class concern
Operating an open, federated catalog across clouds taught us to treat table metadata as a first-class operational concern. In practice this means:
Snapshot retention: Keeping Iceberg snapshot retention short so that per-table metadata stays compact and synchronizes reliably.
Compaction: Standardizing table maintenance (compaction and snapshot expiry) as a uniform, service-managed policy through the S3 Tables built-in maintenance configuration.
Schema evolution: When a Protobuf schema evolves (backward-compatible additions), the Spark job appends or removes columns in the Iceberg schema in S3 Tables. The federated catalog picks up the change on its next sync cycle, and BigQuery reflects the changes without manual intervention.
These are small, well-understood settings once we know how to set them, and they are the difference between a catalog that simply works and one that drifts.
Consuming the data is a choice of engine, not a choice of copy
After a source is live, the same Iceberg table is available three ways over one physical dataset.
A BigQuery user queries it in standard SQL and joins it to the rest of the Google Cloud warehouse.
An infrastructure engineer runs the identical query in Amazon Athena for ad hoc checks and continuous integration (CI) validation.
A data scientist reads the table directly with Spark, with no BigQuery or Athena in the path.
Nobody waits for a nightly export, and nobody reconciles three divergent copies. There is only one.
Performance and cost impact
The qualitative benefits are already clear:
Minutes-fresh raw data for near-real-time analytics. The previous architecture’s latency was not a volume problem. It was a design constraint. Ingestion ran every five minutes, but a downstream hourly batch job gated end-to-end freshness to 60–90 minutes. With catalog federation, that same data is queryable within minutes of being produced: under five minutes for 95 percent of events, with the option to tune the pipeline to cover 100% of events for latency-sensitive or mission-critical workloads.
One storage copy in S3 Tables, three compute engines. BigQuery, Athena, and Spark or another Iceberg-compatible engine read a single physical Iceberg dataset in Amazon S3 Tables, avoiding duplicate storage and the reconciliation tax of keeping copies in sync.
No cross-cloud egress on the hot path. Ingestion is local to AWS. The only cross-cloud traffic is read-time metadata synchronization and query reads, not a continuous write stream. Based on an internal comparison of monthly AWS and Google Cloud data-transfer charges, orchestration overhead, multi-layered ETL workflow costs, and storage backup charges, talabat reduced data-movement costs by approximately 40 percent for comparable data volumes. The comparison spanned a two-month period before and after removing the continuous replication pipeline, and the change eliminated hundreds of terabytes of recurring cross-Region and cross-cloud data transfer per month.
Open table format, no lock-in. Because the raw bronze data layer is Apache Iceberg in Amazon S3 Tables, the data isn’t captive to any single query engine or cloud. New consumers adopt it by speaking Iceberg, not by requesting an export.
Governable cross-cloud access. The cross-cloud boundary is secured by an IAM trust relationship (least-privilege, auditable, and revocable) rather than a standing data pipeline. End-user access control within BigQuery is managed separately through the native role-based access control (RBAC) in GCP and fine-grained access controls on the federated catalog.
Future enhancements
Looking ahead, we plan to broaden source coverage by onboarding the remaining high-value event streams and batch stores onto a hybrid one-configuration pattern. We’re formalizing end-to-end freshness objectives and the observability around them: batch-level metrics, dead-letter monitoring, and catalog-synchronization health. We will continue tuning snapshot retention and compaction so the cross-cloud catalog stays fast and reliable as the number of tables grows. More broadly, we intend to make “written once, read by any engine” the default for new datasets beyond the bronze layer, leaning further into open table formats as the connective tissue between cloud service providers.
Conclusion
Being on two clouds is often framed as a problem to migrate away from. It’s simply the terrain for talabat. The event backbone is prominent on AWS, and the analytics community operates on BigQuery. By making Amazon S3 Tables with Apache Iceberg the single source of truth on AWS and letting BigQuery consume it read-only through a Lakehouse federated Iceberg REST catalog secured by cross-cloud IAM trust, we turned a two-cloud constraint into a single governed dataset that engines can read within minutes. The write path stays short, local, and reliable. The cross-cloud concern lives on the read path, where it belongs, expressed as open standards and identity, not as data movement.
That is the handshake: one copy of the data on AWS, an open catalog contract, and a signed, trusted, revocable identity reaching across the cloud boundary to read it.
This post focuses on reading AWS resident data from BigQuery. For the broader multi-cloud Lakehouse pattern, including federating catalogs from other systems into the AWS Glue Data Catalog, see Multi-cloud Lakehouse architecture on AWS for agentic AI.
Sub-second search is the starting point of every order on Zepto, a fast-growing quick-commerce platform in India, founded in 2021 with endeavor to provide delivery in minutes. Powering the search experience is Amazon OpenSearch Service, a managed retrieval engine built on OpenSearch for agentic AI, search, and analytics.
Zepto operates hundreds of delivery hubs (dark stores) across Indian cities where it provides logistics services to sellers operating on Zepto Platform. Each hub maintains its own inventory levels, pricing, and assortment spanning thousands of Stock Keeping Units (SKUs). As the company scaled to hundreds of hubs, driving linear increases in indexing volume and maintaining sub-second product search latency while controlling costs became increasingly challenging.
To address this, Zepto migrated its OpenSearch Service data nodes from memory-optimized instances to OpenSearch Optimized instances. This instance family is purpose-built for high indexing throughput and cost efficiency. It uses local storage as the primary data tier, with Apache Lucene segments copied synchronously to Amazon Simple Storage Service (Amazon S3) for durability. With this migration, Zepto now serves the same workload with two-thirds of their previous data node count, achieving over 100% higher indexing throughput and 30% cost savings.
In this post, we explore the architecture decisions along with the load testing outcomes that led Zepto to select OpenSearch Optimized instances for latency-sensitive product search. We also discuss the key lessons learned during the production migration.
Zepto’s search platform
Zepto’s search platform is built around a localized delivery hub model. Each hub maintains its own inventory, capacity, and fulfillment priority. When a customer searches for a product, the query is not resolved against a global catalog. Instead, it is resolved in the context of the specific delivery hub or hubs serving that customer’s delivery address. This distinction is critical: Every customer journey on Zepto’s application begins with product discovery through search, browse, and promotional surfaces. All these must reflect hub-specific availability in real time to fulfill orders in minutes.
An event-driven architecture powers this experience, keeping results fresh as products, prices, offers, and inventory change across hundreds of delivery hubs. The following architecture diagram illustrates Zepto’s end-to-end indexing and search pipeline, from event production through stream processing to the search indices on OpenSearch Service.
Figure 1: Zepto’s end-to-end indexing and search pipeline architecture
Event producers and consumers: Zepto’s application microservices are deployed on Amazon Elastic Kubernetes Service (Amazon EKS), a fully managed service for running Kubernetes workloads on AWS. These microservices serve as both event producers and consumers. Sellers and Zepto Admin users interact with the Zepto Partner and Admin application.
Key microservices: The Catalog Management Service emits events when product metadata changes like new product additions, attribute updates, and category reclassifications. The Inventory Management Service publishes stock-level changes across delivery hubs in real time as warehouse teams pick, pack, and replenish inventory. The Pricing Management Service generates events whenever sellers update pricing. The Offers Management Service broadcasts events when promotional offers are created, activated, modified, or expired. Together, these microservices capture every relevant update for downstream indexing, producing events into the streaming layer whenever business state changes.
Search events streaming: All domain events flow through Amazon Managed Streaming for Apache Kafka (Amazon MSK), a managed streaming data service that manages Apache Kafka infrastructure and operations.
The system organizes events into dedicated Kafka topics by business domain. These include Catalog for product metadata changes, Inventory for hub-level stock updates, Pricing for price changes across stores, and Offers for promotional offer lifecycle events and more. This topic-based partitioning provides independent scaling per domain, ensures ordered delivery within each topic and consumer isolation, so that a surge in inventory events does not disrupt catalog indexing.
Stream processing and routing: Events from MSK topics are consumed and routed into two priority-based indexing pipelines through dedicated Apache Flink OpenSearch Connector jobs deployed on an Amazon EKS cluster:
Job #1: P0 indexing events (Pipeline #1): Processes high-priority events requiring near real-time index freshness, such as inventory changes, catalog enrichment, and pricing updates.
Job #2: P1 indexing events (Pipeline #2): Handles lower-priority but higher-volume events, such as tag updates, semantic embedding generation, offer activations, and nightly revenue per impression (RPI) score recomputation. These updates improve search quality but can tolerate slightly higher latency.
With this dual-job approach, Zepto maintains sub-second freshness for critical signals like stock availability and current pricing. Compute-heavy enrichment updates are processed separately without creating backpressure on real-time updates.
Search indices: Zepto hosts the search index on OpenSearch Service, structured at the city-product level. Delivery hub-specific metadata, such as stock status and hub-level demand signals, is stored as nested documents within each record. The following example depicts a typical document in the search index.
The document structure supports store-level personalization while organizing the index by city-product pairs.
Search pipeline: Zepto’s search platform decouples the search request flow from the indexing pipeline at the application layer. When a customer initiates a search, the request passes through the Zepto application to the Search Service and Orchestration layer, which queries the OpenSearch index and assembles the response.
The Search Service and Orchestration layer handles the complete query lifecycle. This includes query understanding, candidate retrieval, machine learning (ML) ranking, ad slotting, and response assembly. For a detailed overview of Zepto’s full search architecture, refer to Building Search for a 10-Minute World on the Zepto engineering blog.
Scaling challenge
Zepto’s search platform started with a single use case, basic product search. As the business expanded, the platform introduced increasingly sophisticated experiences and each new experience added indexing signals to the pipeline like offer events, liquidation tags, pricing changes, ranking scores and more. All needed to be ingested and reflected in the index. Simultaneously, growing user traffic and the expansion of browse surfaces increased read throughput demands on the cluster.
The challenge was most acute during festive events like Diwali and New Year, when traffic surges required scaling to 1.4× the data node count. Although the cluster handled the node additions, the team needed to monitor shard relocation progress and validate that search latencies remained within service level agreements (SLAs) at each step. This operational overhead grew with each scaling event.
Adding more nodes to the cluster would address the immediate throughput constraints, but at the cost of proportionally higher infrastructure spend. To find a solution, Zepto set a clear goal: “Improve throughput without increasing the data node count.”
Solution overview
With the goal of keeping the node count intact, Zepto experimented with multiple configurations. One approach was resharding, adjusting the number of primary shards to better distribute the workload across existing nodes. However, load testing under production-representative traffic revealed that each resharding configuration degraded search latencies. The resharding operations themselves were also operationally expensive, requiring full index recreation, data migration, and extended validation windows.
The team needed a fundamentally different approach. The approach needed to improve throughput without adding nodes or resharding the index.
Evaluating OpenSearch Optimized instances
OpenSearch Optimized instances are an instance family purpose-built for workloads that require high indexing throughput with cost efficiency. They are commonly used for log analytics and time series use cases. These instances store data on Amazon Elastic Block Store (Amazon EBS) volumes for fast local access. Apache Lucene segments are synchronously replicated to Amazon S3, providing 11 nines of data durability.
Despite the common use case association with log analytics, we recommended evaluating OpenSearch Optimized instances type OR2 for Zepto’s product search workload. The team assessed two key criteria to determine viability:
Criterion 1: Does segment replication address the throughput bottleneck?
With document replication (the default on memory-optimized instances), every write is indexed on the primary shard and then re-indexed independently on each replica. This duplicates CPU work across the cluster. With segment replication on OpenSearch Optimized instances, segments are built once on the primary shard. They are then copied as complete files to replicas. This eliminates the duplicate indexing pipeline on replicas and frees their compute for serving search queries. Zepto’s workload involved continuous indexing from multiple pipelines that competed with search traffic. This separation was the key architectural advantage.
Criterion 2: Can the search platform tolerate the 10-second refresh interval?
OpenSearch Optimized instances use a 10-second segment replication refresh interval that is longer than the default one-second refresh on memory-optimized instances. This means newly indexed documents become searchable with up to 10 seconds of additional delay. The team evaluated whether this trade-off was acceptable for their search use cases.
Rahul Pradeep, Senior Architect at Zepto, explains:
“Out-of-stock or in-stock is not a primary parameter for retrieval. It is more like a tiebreaker. Relevance is our primary parameter. We retrieve hundreds of products in one query and then do a last-minute validation against our real-time inventory service. That is why we may not need one-second refresh.”
Zepto’s existing architecture where the Product Enrichment Service validates inventory after retrieval indicated that the 10-second refresh interval would not impact customer experience; see how Zepto built Product Enrichment at scale for further details. The migration was viable without any application-level changes.
Based on this evaluation, the solution involved migrating from memory-optimized Graviton-based data nodes to OpenSearch Optimized instances. This shift changed how indexing work is distributed across the cluster. Instead of a model where every node duplicates the full indexing pipeline, only the primary shard performs indexing, and replicas receive pre-built segments.
Load testing
To validate the hypothesis before committing to a migration, we designed a proof of concept, a load testing setup in their lower environment that mirrored production characteristics:
Baseline cluster with r7g.12xlarge instances and a parallel testing cluster with or2.12xlarge instances, having four nodes per cluster.
Identical shard configuration (X primary shards, Y replica, Z shard copies per node).
Simultaneous indexing and read load simulation.
Document structure improvements
In addition to validating the infrastructure change, the team identified an opportunity to optimize the document structure itself to further improve search latency. They added an active_hubs attribute to the base document, a flat array listing only the hubs where the product is currently stocked and active as shown in the following updated document structure.
The following table summarizes the key metrics from the load test comparing the r7g.12xlarge baseline cluster against the or2.12xlarge test cluster under identical conditions.
Metric
r7g.12xlarge
or2.12xlarge
Change
Peak indexing lag
~12M docs
~6M docs
2X Faster
Indexing throughput
Baseline
2× higher
100% Improvement
Search latency (p90)
187 ms
89.1 ms
52% Improvement
Search latency (p99)
244 ms
175 ms
28% Improvement
The following graph depicts the P90 search latency comparison between the two clusters.
Figure 2: P90 search latency comparison between the r7g and OR2 clusters
The following graph depicts the P99 search latency comparison between the two clusters.
Figure 3: P99 search latency comparison between the r7g and OR2 clusters
The following graphs depict the indexing latency comparison between the two clusters.
Figure 4: Indexing latency comparison between the r7g (left) and OR2 (right) clusters
Key insights
Improvement in indexing throughput: The higher indexing throughput of OR2 is most visible during nightly batch operations when events from RPI score recomputation, tag updates, and catalog enrichment flood the indexing pipeline simultaneously. On the r7g cluster, the P1 indexing lag peaked at over 12M docs. On OR2, with approximately 2× the indexing throughput, the same event volume produced a peak lag of only 6M docs. Higher throughput translates directly to lower lag and fresher search results. It is attributed to the segment replication approach of OR2 that eliminates redundant indexing work on replicas. Each document is indexed once on the primary shard rather than being replayed on each replica.
Reduction in search latency: P90 search latency dropped from 187 ms to 89.1 ms (52% improvement) and P99 from 244 ms to 175 ms (28% improvement). These gains are primarily attributable to the active_hubs document structure change rather than the instance type migration alone. By pre-computing a flat list of active hubs at index time, the query no longer needs to traverse nested hub documents to determine availability. This creates a lightweight pre-filter that eliminates unnecessary computation at search time.
Production planning and rollout
The load test results gave Zepto the confidence to make a key architectural decision: reduce the overall data node count. Higher per-node indexing throughput meant the same workload could be served with fewer nodes with OR2, and the cost savings compounded. Each eliminated node removed compute, storage, and operational overhead from the cluster. Zepto carried this forward into production, provisioning the OR2 cluster at two-thirds of the original node count. The following table summarizes the before-and-after comparison.
Metric
r7g.12xlarge
or2.12xlarge
Change
Data nodes required
3X Nodes
2X Nodes
-33.3%
Cost savings
Baseline
2/3 of Baseline
+30%
Rather than a complete cutover, Zepto adopted a phased rollout strategy using bucket-based traffic routing, completing the migration over approximately two months with zero downtime:
Provisioned a new OpenSearch Service domain on OR2 instances with segment replication turned on.
Executed parallel indexing pipelines to populate the OR2 cluster while the existing r7g cluster continued serving production traffic.
Routed internal users to the OR2 cluster first to validate search quality, relevance, and latency characteristics under real query patterns.
Gradually increased external user traffic in buckets, monitoring comparison dashboards at each increment for latency regressions or relevance drift.
Maintained parallel dashboards throughout the migration to compare the OR2 and r7g clusters in real time. Key metrics monitored included p50 and p99 search latency, indexing throughput, replica lag, Java Virtual Machine (JVM) heap utilization, circuit breaker events, I/O operations per second (IOPS) utilization, and disk throughput.
Challenges and lessons learned
During the migration, the team encountered one notable challenge: latency spikes during segment merges. This observation offers practical guidance for teams evaluating OpenSearch Optimized instances for search workloads.
Symptom: After shifting significant traffic to OR2, Zepto observed intermittent p99 latency spikes correlating with segment merge operations.
Root cause: Large segment merges consumed significant I/O bandwidth, temporarily impacting concurrent search query performance. The original 256 GB EBS volumes did not provide sufficient IOPS buffer for concurrent merge and search operations.
Resolution: Implemented the following two changes:
Increased EBS volume size to 1 TB per node. For gp3 volumes, baseline IOPS increase with volume size. This provided buffer for concurrent operations.
Tuned the segment merge policy. Reduced max_merged_segment (see OpenSearch: Force Merge API for more details) from 5 GB to 2 GB and segments_per_tier (see OpenSearch: Index Settings for more details) from 10 to 5. This produces smaller, more frequent merges that distribute I/O load more evenly rather than infrequent large merges that spike latency.
After increasing EBS volume size and tuning the segment merge policy, latency spikes decreased. Transient spikes still occurred during merges but settled quickly within acceptable bounds.
Production cutover
Finally, Zepto shifted from partial to 100% traffic over four weeks and decommissioned the previous cluster after confirming stable performance across multiple peak traffic cycles. The following table summarizes the cluster configuration before and after migration.
Parameter
Previous Cluster
Current Cluster
Instance type
r7g.12xlarge
or2.12xlarge
Data nodes
3X Nodes
2X Nodes
RAM per node
384 GiB
384 GiB
Replication strategy
Document replication
Segment replication
Default refresh interval
1 Second
10 Seconds
Durability
Cross-Availability Zone replicas
S3 synchronous replication
Conclusion
In this post, we described Zepto’s evaluation of OpenSearch Optimized instances for latency-sensitive product search and the results of their production migration. By moving from memory-optimized data nodes to OpenSearch Optimized instances with segment replication enabled, Zepto achieved over 100% higher indexing throughput and 30% cost savings while reducing their cluster to two-thirds of the previous data node count.
Zepto’s migration demonstrates that OpenSearch Optimized instances are a viable choice for latency-sensitive product search and not just log analytics. Workloads where the retrieval layer can tolerate seconds-level staleness because real-time consistency is resolved at a different layer are candidates for adopting OR2 instances. For ecommerce and quick-commerce platforms that separate candidate generation from availability validation, this pattern can deliver significant infrastructure cost reduction.
If your workload has high indexing volume, and can tolerate a 10-second refresh interval, consider evaluating OpenSearch Optimized instances for your cluster. To get started:
Assess your workload fit: review your current indexing throughput, replica count, and refresh interval requirements. Prioritize this approach if your workload has a high write-to-read ratio.
Execute a proof of concept: provision a small OpenSearch Optimized cluster in a lower environment with identical shard configuration and restore a production index snapshot. Execute simultaneous indexing and search load to validate throughput and latency.
Plan a phased rollout: use parallel indexing and bucket-based traffic routing to migrate incrementally with zero downtime, monitoring indexing lag and search latency at each step.
NaranjaX is a leading fintech platform that aims to simplify and improve the daily financial lives of millions of people in Argentina. Through its digital ecosystem, NaranjaX offers a complete suite of financial products and services, including payments, collections, financing, savings, and protection products.
NaranjaX needed to evolve from their REST-based architecture to an event-driven architecture using Amazon Managed Streaming for Apache Kafka (Amazon MSK) Serverless. In a multi-account environment, MSK Serverless clusters resolve DNS names within their hosting account. AWS published a cross-account connectivity pattern that centralizes clusters in a single account. This is an effective approach for many organizations. NaranjaX required additional flexibility to distribute clusters across accounts while avoiding centralized quota dependencies.
NaranjaX addressed this requirement by developing an approach that uses AWS Resource Access Manager (AWS RAM) and Amazon Route 53 Resolver. In this post, we show you how to expand Amazon MSK Serverless adoption across multiple accounts while maintaining scalability, availability, and reduced operational overhead.
Solution overview
NaranjaX’s solution supports cross-account MSK Serverless deployment through a centralized networking architecture that combines shared virtual private cloud (VPC) resources and DNS resolution capabilities. The solution uses a central AWS account that hosts shared private subnets and Route 53 resolver endpoints, so that MSK Serverless clusters in different accounts can communicate across account boundaries.
The architecture consists of three main components:
A central VPC with private subnets that are shared across accounts using AWS RAM.
Route 53 resolver endpoints and rules that resolve DNS across accounts for MSK Serverless clusters.
Network security configurations that control communication between components.
When an application team creates an MSK Serverless cluster in their account, they can associate it with the shared VPC subnets. The Route 53 resolver rules handle DNS resolution for the cluster’s domain names, while security groups manage access control. This design supports direct connectivity between MSK Serverless clusters and applications across different AWS accounts.
Figure 1: Cross-account architecture with a central networking account sharing subnets and Route 53 resolver endpoints
Implementation requirements and configuration
This section walks you through the steps to configure cross-account MSK Serverless connectivity using shared VPC subnets and Route 53 resolver rules. Before you begin, make sure you have the prerequisites in place.
Prerequisites
Before implementing this solution, confirm the following:
AWS RAM is enabled in your AWS Organization. For instructions, see Enabling resource sharing within AWS Organizations.
Amazon MSK supports shared subnets. When you create an MSK Serverless cluster in any account, you can associate the shared VPC as one of the up to five VPCs supported by the service.
You have a multi-account environment with at least one central networking account and one or more application accounts.
You have permissions to create VPCs, subnets, Route 53 resolver endpoints, and AWS RAM resource shares in the central account.
Step 1: Share subnets with AWS RAM in a central account
First, create a VPC with private subnets in your central networking account. These subnets are the resources you will share through AWS RAM. For details, see Creating a VPC in the Amazon VPC User Guide.
Next, create a resource share for those subnets in AWS RAM. Select the subnets you created and specify the target accounts.
Finally, specify the principals (account IDs) authorized to use the shared subnets. These are the accounts where you will create your Amazon MSK Serverless clusters.
Figure 2: Creating a resource share for the private subnets in AWS RAM
Figure 3: Specifying the target accounts for the resource share
Figure 4: Confirming the principals authorized to use the shared subnets
Step 2: Configure Amazon Route 53 Resolver rules
In your central account, create a Route 53 Resolver rule for the domain *.kafka-serverless.<Region>.amazonaws.com. Don’t associate this rule with any VPC at this point.
Figure 5: Route 53 Resolver rule for the kafka-serverless domain
Configure this as a forward rule for the kafka-serverless subdomain. Set up an outbound endpoint in the central account and point the target IP addresses to the inbound endpoint in the same account.
Figure 6: Forward rule configuration with outbound and inbound resolver endpoints
Share the resolver rule with your application accounts using AWS RAM so they can resolve the DNS names of their MSK Serverless clusters.
Make sure the central VPC has both inbound and outbound resolver endpoints configured to support cross-account DNS resolution.
Step 3: Configure network security groups
Configure security groups in each consuming account to allow inbound and outbound traffic on port 53 (DNS resolution) and port 9098 (Kafka IAM authentication). This supports both name resolution and secure connectivity to your MSK Serverless brokers across account boundaries.
Step 4: Enable and test many-to-many connectivity
With the networking infrastructure in place, you can now create MSK Serverless clusters in any of your application accounts. To do this, create an MSK Serverless cluster in your application account and associate it with the shared VPC subnets from the central account. The Route 53 resolver rules automatically handle DNS resolution for the cluster endpoints, and the security groups you configured control access. This eliminates the restriction of hosting all clusters in a single account.
You have flexibility in how you configure DNS resolution for your clients. For example, in a client account, you can associate the shared resolver rule with a VPC directly, or you can use the inbound endpoint IP addresses from the central account as custom DNS servers. Configure these either in per-connection scripts or in DHCP option sets for a separate VPC.
To verify connectivity, use the dig command from an instance in a client account VPC to test DNS resolution of MSK Serverless bootstrap strings across different accounts. The following example uses the +short flag for clarity:
Figure 7: The dig command resolving MSK Serverless bootstrap strings across accounts
The output shows that two MSK Serverless clusters (bootstrap strings starting with boot-*) in different accounts and VPCs resolve to the actual IP addresses of the three brokers listening for connections.
This confirms that the architecture supports scalable, consistent cross-account communication for event-driven workloads.
Key benefits
NaranjaX’s implementation of MSK Serverless as its integration backbone delivered measurable advantages across 15+ application teams and over 40 AWS accounts, transforming application development and operations.
Scalability with optimized cost
With MSK Serverless, teams can scale workloads automatically without managing broker capacity. Combined with AWS RAM and Route 53, the architecture supports growth across over 40 accounts while maintaining cost efficiency. By removing the need for dedicated Kafka operations staff and self-managed clusters, NaranjaX reduced infrastructure management costs by approximately 40 percent compared to their previous self-managed Kafka deployment.
Simplified governance and security
Centralized DNS management and VPC sharing keep configurations standardized across all accounts. IAM-based access control, integrated with KATHU, provides clear visibility into topic ownership and consumer access, reducing security review cycles from days to hours.
Faster developer onboarding through IDP integration
By integrating Kafka control-plane operations directly into their internal developer platform (IDP), teams can provision clusters and topics through Terraform modules or a graphical interface. This reduced onboarding time for new teams adopting event-driven architecture from weeks to less than one day.
Reduced operational overhead
Application teams can focus on delivering business features rather than managing Kafka infrastructure. Central operations handle DNS, networking, and resource sharing, while MSK Serverless abstracts broker administration. This reduced operational tickets related to Kafka by over 70 percent and freed the platform team to focus on higher-value initiatives.
Next steps
NaranjaX is evaluating extending this solution by incorporating automatic topic replication across accounts using MSK Replicator, so that certain topics can be exposed as Enterprise Topics in a central hub for global consumption. This will further simplify the architecture, improve data resiliency, and enhance visibility across event domains.
Conclusion
Through this architecture, NaranjaX successfully implemented a many-to-many connectivity model for Amazon MSK Serverless across more than 20 AWS accounts. By using AWS RAM and Amazon Route 53 Resolver, the organization achieved a scalable, secure, and centralized network topology that accelerates the adoption of event-driven architecture without operational bottlenecks. This approach complements the cross-account connectivity pattern published by Tamer Soliman, and provides additional flexibility for organizations that require distributed Kafka clusters in large-scale multi-account environments. To get started, see the Amazon MSK documentation and try this approach in your own multi-account environment.
When CI/CD deployment failures occur for GitHub hosted applications, AWS DevOps Agent reduces the hours that Development and Site Reliability Engineering (SRE) teams typically spend manually investigating across multiple AWS services, logs, and pipeline stages. This process delays critical deployments and impacts software delivery velocity. This is especially true when teams need to correlate data between GitHub commit histories, AWS CodePipeline execution logs, and Amazon CloudWatch metrics. When continuous integration and continuous delivery (CI/CD) pipelines fail, engineers often find themselves context-switching between GitHub pull requests, code build logs, deployment artifacts, and downstream service health metrics. This process of identifying root causes can extend resolution time from minutes to hours, especially in multi-service architectures.
AWS DevOps Agent reduces this manual investigation by automatically correlating pipeline failures with specific code changes. Rather than spending hours manually tracing deployment failures through multiple systems, engineers can use AWS DevOps Agent to perform this correlation. It identifies which specific code changes caused pipeline failures and provides remediation guidance. The agent analyzes pipeline failures, correlates them with specific commits and pull requests, and identifies root causes across the deployment chain.
AWS CodePipeline combined with AWS DevOps Agent helps address this challenge by creating a streamlined path from GitHub repositories to AWS deployments. This solution reduces manual handoffs, reduces configuration complexity, and provides end-to-end visibility across the entire development lifecycle.
In this post, you learn how to integrate AWS DevOps Agent with your GitHub repositories to automatically correlate deployment failures with specific commits, providing root cause analysis and remediation steps across your entire CI/CD pipeline.
Solution overview
Modern software delivery teams face a persistent challenge: when deployments fail, engineers spend valuable time manually correlating logs, tracing pipeline errors, and diagnosing root causes across disconnected tools. This reactive cycle slows recovery and increases mean time to resolution (MTTR). By integrating the AWS DevOps Agent with GitHub, AWS CodePipeline, Amazon CloudWatch, and AWS Lambda, teams can shift from manual triage to automated incident investigation, directly within their existing GitHub-based workflows.
This solution integrates AWS DevOps Agent with GitHub to automate deployment failure investigation. The following sections explain the architecture and operational benefits.
How it works
The architecture creates an automated monitoring and remediation flow that monitors your deployment pipeline and responds to issues. Your source code resides in a GitHub repository, and AWS CodePipeline orchestrates the build, test, and deployment stages. Amazon CloudWatch continuously monitors pipeline execution metrics and logs and generates alarms when it detects anomalies or failures, such as failed build stages, deployment rollbacks, or threshold breaches in downstream application of health metrics. When a failure occurs, it generates an error metric in CloudWatch. The CloudWatch Alarm detects this error and transitions to an ALARM state, which directly invokes the WebHook Executor Lambda. The WebHook Executor then sends an authenticated HTTP POST request to DevOps Agent, which receives the incident and begins an investigation.
Webhook integration acts as the bridge between the Amazon CloudWatch, the monitoring layer. Lambda parses the alarm payload and extracts contextual metadata and then invokes the DevOps Agent with a structured investigation request.
Integration with Operational Excellence
This solution directly supports the AWS Well-Architected Framework’s Operational Excellence pillar by automating the investigation process and reducing the MTTR. The investigation capability of AWS DevOps Agent aligns with AWS Incident Detection and Response (IDR) best practices, helping teams to detect, diagnose, and develop mitigation plans for pipeline failures faster while maintaining a full audit trail of agent actions and findings. This creates a delivery pipeline that accelerates resolution workflows through automated diagnostics and actionable remediation recommendations, keeping deployments moving and engineering teams focused on building rather than firefighting.
Figure 1: GitHub and DevOps Agent integration
Prerequisites
For this walkthrough, you should have access to and understanding of the following:
An AWS account with permissions to create AWS Identity and Access Management (IAM) roles: 1. Agent Space role – for basic service operations. 2. Agent Space web app role – for using the Agent Space web app functionality. 3. (Optional) Secondary source account roles if monitoring multiple AWS accounts. Refer to the DevOps Agent user guide for the details on setting up these roles.
A GitHub account: 1. You have a GitHub account with administrative permissions for your repositories, or an organization you belong to. 2. Your repositories contain code that deploys to AWS resources you want to monitor. 3. You have identified the GitHub repositories you want AWS DevOps agent to access.
Access to register DevOps Agent with your GitHub Account or Organization.
CloudWatch monitoring enabled for your application.
Create an AWS DevOps Agent Space and configure the webhook The first step is to create a dedicated Agent Space that serves as the central hub for your automated investigation workflow. The Agent Space connects your monitoring infrastructure to the DevOps Agent’s analysis capabilities. Create the DevOps Agent space by following the steps outlined in the Getting Started with AWS DevOps Agent guide Navigate to the DevOps Agent console. Create an Agent Space named after your application (for example, `myhotelapp`) 1) “Auto-create both IAM roles”. 2) “Edit the role names to be descriptive (for example, DevOpsAgentRole-AgentSpace-hotel-app and DevOpsAgentRole-WebappAdmin-hotel-app)”
Figure 2: Agent Spaces Screen
On the Capabilities tab, generate a webhook and save the credentials
Store the webhook credentials in AWS Secrets Manager:
2. Configure GitHub integration with your AgentSpace
With your Agent Space created and webhook configured, the next step is to connect your GitHub repositories. This integration allows the DevOps Agent to access commit histories, pull request data, and code changes when investigating pipeline failures.
To configure GitHub integration with your AgentSpace: 1. From the Capabilities tab within your configured AgentSpace, navigate to the GitHub Configuration section and choose “Register”
Figure 3: Capability Providers
2. Your GitHub repositories will be listed with their connection status.
3. To connect to a repository, verify that the Status shows “Ready to connect” and choose the + button in the Actions column.
4. Upon successful connection, the Status updates to ‘Connected’.
To automatically trigger AWS DevOps Agent investigations via Webhook when a CloudWatch enters the ALARM state, you can refer to sample-aws-devops-agent-cloudwatch and build based on your use case.
3. Troubleshooting application deployment 5XX errors with CloudWatch and AWS DevOps Agent
When your application encounters 5XX errors during deployment, CloudWatch alarms detect the anomaly and trigger the DevOps Agent investigation workflow. The following dashboard shows the alarm state that initiates the automated investigation process.
Figure 4: CloudWatch Dashboard
4. Resolving deployment/build errors during CI/CD deployment
The following use cases demonstrate how AWS DevOps Agent investigates and resolves common CI/CD pipeline failures. Each scenario walks through the failure trigger, the automated investigation, and the remediation guidance that the agent provides
Use case 1: Push a code change that introduces an invalid DynamoDB table name
Simulate: Push a code change that breaks the DynamoDB table name — e.g., change DYNAMODB_TABLE_NAME env var but don’t update CloudFormation to make the CodePipeline unit testing fail
A – dynamodb_table: process.env.DYNAMODB_TABLE_NAME || “Rooms”,
B + dynamodb_table: “HotelRooms”
The CodePipeline triggers 5xx alarms and the webhook triggers a DevOps Agent investigation.
DevOps Agent analyzes the 500 errors in relation to the configuration change, identifies the invalid DynamoDB endpoint, and shows the timeline: configuration update → service redeployment → requests fail with connection errors.
Figure 5: Unit test failed for the CodePipeline
Use case 2: Identifying dependency resolution failures from bad commits
1. Navigate to `package.json`
2. Change any dependency name to something invalid — for example, change `”express”` to `”expresss”` (extra ‘s’)
3. Commit the change directly to `main`
CodePipeline detects the push and starts a new execution. The CI stage runs `npm install`, which fails because the misspelled package doesn’t exist. The Amazon EventBridge rule catches the stage failure and invokes the webhook executor Lambda, which triggers a DevOps Agent investigation.
In the DevOps Agent console, select your Agent Space, then choose Operator access to open the web app. Navigate to the Incident Response tab to view the new investigation.
Figure 6: Mitigation plan step1
Figure 7: Mitigation plan steps 2-4
DevOps Agent investigates the pipeline failure, examines the CodeBuild logs showing the `npm install` error, and correlates it with the recent commit to the repository. It identifies the root cause as a dependency resolution failure introduced by the latest code change.
Clean up
This walkthrough creates AWS resources that incur charges, including AWS DevOps Agent (pay-per-use), Lambda functions, CodePipeline executions, CloudWatch alarms, and Secrets Manager secrets. Follow the cleanup steps when finished to avoid ongoing charges.
2. Delete your Agent Space from the AWS DevOps Agent console
3. Remove the GitHub pipeline connection from your settings.
4. Delete the IAM roles created for the Agent Space.
5. Delete the Lambda function, EventBridge rule, and CloudWatch alarms created for webhook integration.
6. (Optional) If you created additional source account roles, remove those as well.
Conclusion
The AWS DevOps Agent integration with GitHub fundamentally transforms how engineering teams approach CI/CD reliability by shifting from reactive troubleshooting to proactive incident prevention. By autonomously correlating CodePipeline failures with specific GitHub commits, analyzing root causes across the deployment chain, and providing intelligent remediation recommendations, this solution reduces mean time to resolution from hours to minutes while maintaining the human oversight necessary for production environments.
Organizations implementing this integration gain a resilient software delivery pipeline that combines the collaborative strengths of GitHub source control with AWS’s intelligent automation capabilities. This helps teams maintain deployment velocity, strengthen operational excellence, and focus engineering effort on innovation rather than incident response.
AWS CodePipeline, Amazon CloudWatch, AWS Lambda, and the AWS DevOps Agent integrate natively to provide end-to-end visibility and autonomous investigation capabilities. Together, they accelerate recovery workflows, reduce operational friction, and build the foundation for continuous delivery at scale.
About authors
Anjani Reddy
Anjani is a Sr. Solutions Architect at AWS. She works with Enterprise customers to provide operational guidance to innovate and build a secure, scalable cloud on the AWS platform. Outside of work, she is an Indian classical & salsa dancer, loves to travel and Volunteers for American Red Cross & Hands on Atlanta.
Jared Thompson Jared Thompson is a Senior Technical Account Manager at AWS, where he partners with strategic enterprise customers to optimize cloud operations and accelerate AI/ML workloads at scale. Jared specializes in GPU-accelerated computing, capacity planning, and cloud observability, with a passion for turning complex infrastructure challenges into automated, self-healing systems. He is a recipient of the AWS Golden Jacket award and when not at work, he can be found on a cruise ship.
Aneesh Varghese is a Senior Technical Account Manager at AWS with more than 19 years of Information Technology industry experience. Aneesh supports enterprise customers in cost optimization strategies, Cloud operations, MLOps, providing advocacy and strategic technical guidance to help plan and build solutions using AWS best practices. Outside of work, Aneesh likes to spend time with family, play Basketball and Badminton.
When Bosch Mobility Platform Solutions set out to unify vehicle tracking across India’s fragmented spot logistics market, they faced a daunting reality: dozens of telematics providers, incompatible data formats, and thousands of concurrent tracking requests — all needing real-time resolution. The result was L.OS, a serverless platform on AWS that standardizes this chaos into a single visibility layer.
In this post, we’ll show you how Bosch Mobility Platform Solutions (MPS) uses AWS services to solve these challenges through their L.OS solution. You’ll learn how Bosch built a scalable, serverless architecture that standardizes and integrates multiple tracking data sources, so you can achieve real-time visibility and data-driven decision-making across complex logistics networks.
Key challenges in logistics visibility
If you manage a modern supply chain, you face several critical challenges:
Data fragmentation and integration complexity.
Multiple tracking systems with incompatible data formats.
Different communication protocols across providers.
Lack of standardization in data exchange.
Complex and costly point-to-point integrations.
Operational inefficiencies.
Manual coordination between stakeholders.
Time-consuming reconciliation of conflicting information.
Difficulty in providing accurate ETAs.
Limited real-time visibility into shipment status.
Scale and performance issues.
High volume of concurrent tracking requests.
Variable data quality from different sources.
Performance bottlenecks during peak operations.
Cost implications of real-time tracking.
Regional complexities.
Fragmented spot logistics networks.
Multiple intermediaries in the supply chain.
Varying levels of technological adoption.
Regional compliance requirements (such as AIS140 and FASTag in India).
Introducing L.OS on AWS: A unified visibility solution
To address these challenges, Bosch’s Logistics Operating System (L.OS) on AWS provides a horizontal integration layer that connects previously siloed logistics solutions. The solution features a service catalog where solution providers and consumers can collaborate to solve complex use cases, fostering innovation in the logistics sector. Let’s explore how L.OS enhances vehicle visibility through its core workflows: discovery, tracking, and termination.
Discovery
When a client needs to track a vehicle, the service app makes a discovery call to the L.OS gateway. This call includes essential details such as the vehicle number plate or vehicle identification number (VIN). Upon receiving the request, the L.OS solution performs necessary authentication and authorization. L.OS then broadcasts the request and waits for acknowledgment from one or more connected participants. The responses contain information such as the mode, frequency, and reliability of tracking, which can be used for shortlisting and decision-making.
The following diagram illustrates the discovery workflow, showing how a client’s tracking request flows through L.OS to connected participants and back.
Figure 1 – The discovery flow: the service app sends a discovery call to the L.OS gateway with vehicle identifiers. L.OS broadcasts the request to connected participants, collects acknowledgments containing tracking mode, frequency, and reliability details, and returns them to the consumer for shortlisting.
Tracking
Once the consumer has selected a vehicle and a service provider (if there are multiple options), a request is sent to the L.OS to initiate tracking. This request is relayed to the specific service provider. The tracking mode determines who must grant consent. For SIM tracking, a consent request goes to the driver. For GPS tracking, it goes to the fleet owner. The solution waits for the tracking provider to create the trip. Upon receiving confirmation, L.OS registers the tracking request and provides a unique tracking ID to indicate that tracking has been initiated. From here, the consumer is asynchronously notified of the vehicle’s location at the specified frequency, or the maximum frequency supported by the service provider, whichever is faster. Consumers can also request the live location of the vehicle at any time between the regular reporting intervals.
The following diagram shows the tracking workflow, from initiation through consent, trip creation, and ongoing location updates.
Figure 2 – The tracking flow: the consumer sends a tracking request to L.OS, which relays it to the selected service provider. A consent request is issued (to the driver for SIM tracking, or the fleet owner for GPS tracking). Once the provider confirms trip creation, L.OS returns a unique tracking ID and begins delivering asynchronous location updates at the agreed frequency.
Termination
The tracking is automatically terminated when the vehicle enters the destination geo-fence. Alternatively, tracking can be terminated manually by sending an explicit request to L.OS, which is then relayed to the service provider.
Architecture overview
The L.OS solution built on AWS uses various services to create a scalable, secure, and maintainable system. The architecture implements serverless components (AWS Lambda adapters) where appropriate while using containers (Amazon Elastic Container Service (Amazon ECS) with AWS Fargate) for the core connector service. Let’s explore how these AWS managed services work together to create a flexible and scalable integration solution. The following diagram shows the end-to-end architecture, illustrating how requests flow from client applications through the API layer, into the core connector service, and out to individual tracking providers.
Figure 3 – L.OS architecture on AWS: Client applications connect through Amazon API Gateway to the Tracking Connector running on Amazon ECS Fargate, which handles protocol standardization, routing, and session management. Provider-specific Lambda adapters translate between the standardized connector API and each tracking provider’s API. Amazon MSK serves as the event bus for asynchronous location updates. Amazon ElastiCache provides low-latency caching for frequently accessed data, Amazon DynamoDB stores business rules and security policies, and the Marketplace Subscription Management service (also on Fargate) handles authentication, customer relationships, and provider configurations. Amazon QuickSight delivers real-time monitoring and usage analytics.
Key components
The architecture comprises five core components that work together to deliver reliable, real-time vehicle tracking at scale. Each component handles a distinct responsibility — from protocol translation to event streaming — allowing the system to scale and evolve independently.
Centralized orchestration with Amazon ECS Fargate
The Tracking Connector, running on Amazon ECS Fargate, serves as the central orchestration layer. It handles critical functions including:
Protocol standardization across multiple providers.
We use AWS Lambda to implement Tracking Adapters that handle provider-specific transformations. These adapters efficiently translate between our standardized connector API and various provider APIs, allowing for easy onboarding of new providers.
By implementing this serverless architecture on AWS, Bosch L.OS achieved significant improvements in vehicle tracking capabilities:
Operational efficiency
The combination of standardized Lambda adapters and the centralized Tracking Connector on ECS Fargate eliminates the manual coordination that previously slowed provider onboarding. Where ISVs once spent 2–4 weeks on bespoke integration work for each new customer request, the standardized connector API and adapter pattern reduces this to within 3 days. Real-time data validation at the connector layer — before events reach downstream consumers — also improves data accuracy by catching format inconsistencies at ingestion rather than during reconciliation.
Scalability and performance
Because the core connector runs on Fargate with auto-scaling task definitions, and each provider adapter is an independent Lambda function, the system scales horizontally without manual intervention. Bosch’s deployment currently handles 35,000 trips per day — each generating multiple location events — with sub-second response times for 99.9% of tracking queries. As new ISVs are onboarded, additional Lambda adapters are deployed independently, so scaling the provider network does not add load to existing integrations.
Cost optimization
Integrations in fragmented logistics markets often stall because multiple vendors must coordinate through manual processes — handoffs, SIM card provisioning, consent management, and troubleshooting. By automating these workflows within the L.OS connector layer and MSK event bus, Bosch estimates integration costs are reduced by 15–20%. The architecture also removes per-vendor overhead (SIM management, consent flows, provider-specific troubleshooting) that was previously passed on to small transporters. This potentially lowers their total tracking costs by 25–30%.
Enhanced customer experience
The unified API Gateway endpoint and MSK-powered event streaming mean consumers receive location updates from any connected provider through a single interface — regardless of the underlying tracking technology. What previously required hours of manual coordination across providers now surfaces as a consolidated event within approximately 1 minute, according to Bosch. Improved ETA accuracy is a direct result: with standardized, high-frequency location data flowing through ElastiCache, downstream planning systems can compute more reliable arrival predictions.
Compliance and security
DynamoDB-backed policy enforcement ensures that business rules, authorization configurations, and regional compliance requirements (such as India’s AIS140 and FASTag mandates) are evaluated consistently on every request. The built-in security features of AWS — IAM roles, virtual private cloud (VPC) isolation, and encryption at rest and in transit — provide the baseline. Automated audit trails captured through the event bus give organizations a verifiable record of all tracking operations.
L.OS growth
L.OS is currently operational in India with 10 integrated ISVs. The serverless adapter pattern makes geographic expansion straightforward: new region-specific adapters can be deployed as independent Lambda functions without modifying the core connector. Bosch plans to use this approach to expand into Europe for trailer monitoring use cases.
Conclusion
In this post, we showed how Bosch built L.OS, a serverless vehicle tracking platform on AWS that unifies fragmented logistics visibility into a single integration layer. By using AWS services such as Amazon ECS with Fargate for centralized orchestration and AWS Lambda for provider-specific adapters, the architecture standardizes multiple tracking providers into a unified API.
This standardization eliminates the need for maintaining multiple point-to-point integrations, freeing you to focus on core operations instead of managing repetitive integration tasks. Through strategic collaboration with key stakeholders in the visibility solutions space, L.OS is helping businesses achieve measurable outcomes: enhanced customer experience, increased operational agility, reduced operational expenses, and improved profit margins.
What started as a vehicle tracking solution is now evolving into a broader mobility services portfolio, powered by the scalable infrastructure that AWS provides. This evolution positions L.OS to address not only today’s tracking needs, but a broader range of logistics use cases as they emerge.
If you have questions or feedback about this post, leave a comment in the comments section.
To learn more about the AWS services used in this solution, explore these resources:
Real-time data streaming and event processing are critical components of modern distributed systems architectures. Apache Kafka has emerged as a leading platform for building real-time data pipelines and enabling asynchronous communication between microservices and applications. However, running and managing Kafka clusters at scale can be challenging, requiring specialized expertise and significant operational overhead.
Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that you can use to build and run production Kafka applications. With Amazon MSK, you can rely on AWS to handle the heavy lifting of provisioning and managing Kafka clusters, while you focus on building innovative applications and real-time data processing pipelines.
In this post, you learn how AppFolio adopted Amazon MSK Express brokers to replace hours-long rebalances and manual storage planning with a streaming platform that scales automatically.
About AppFolio and its data streaming platform
AppFolio is a leading Real Estate Performance Management platform, serving thousands of property management companies across the United States. AppFolio’s platform processes millions of transactions daily, from rent collection and maintenance requests to lease management and financial reporting. In this data-intensive environment, reliable streaming infrastructure isn’t only important. It’s mission-critical.
At AppFolio, real-time data is the foundation of the company’s ability to deliver powerful, intelligent solutions that power the real estate industry. To achieve this level of performance, AppFolio engineered a modern streaming data architecture built on Amazon MSK with Express brokers. This infrastructure enables high-throughput, real-time applications at scale. With Amazon MSK Express brokers, AppFolio reliably ingests massive volumes of diverse data, including Change Data Capture (CDC) and server-side events, and makes it available to downstream consumers, such as real-time fraud detection, financial reporting, and automated property management workflows, within seconds of origin.
Previous architecture and AppFolio’s evolving requirements
Until early 2025, AppFolio ran their streaming platform on a single Amazon MSK cluster with Standard brokers, supporting both customer-facing and internal workloads. The architecture served them well through earlier growth phases. As AppFolio’s data platform evolved to support increasingly complex use cases and higher throughput, two characteristics of their workload led them to look for a more elastic streaming foundation.
Figure 1: AppFolio’s previous architecture with Amazon MSK Standard brokers
First, AppFolio makes extensive use of log-compacted topics for their CDC streams. Compacted topics retain the latest value for each key indefinitely, which is exactly what they want for streams that mirror the state of operational tables. As their footprint grew, AppFolio wanted an infrastructure model that could scale storage automatically alongside data growth, without ongoing capacity planning that took multiple hours every month.
Second, AppFolio’s throughput continued to grow as they onboarded new use cases and added more event sources. They wanted the ability to scale the cluster quickly in response to traffic shifts, with minimal lead time for partition reassignments.
Third, as AppFolio’s platform matured, they needed workload isolation between customer-facing and internal data flows. Running customer-facing and internal workloads on a single cluster made it harder to size and tune each independently. As both grew, AppFolio wanted dedicated resources so each could be sized and tuned independently.
Based on these needs, AppFolio identified the following key requirements for their next-generation streaming platform:
Elastic, automatically managed storage that scales with AppFolio compaction-heavy CDC workloads, removing the need for upfront broker capacity planning.
Faster horizontal scaling and partition reassignment so AppFolio can adjust cluster shape in response to actual traffic in minutes rather than hours.
Workload isolation between customer-facing and internal data flows, so each workload can be sized and tuned for its own traffic pattern.
Why AppFolio chose Amazon MSK Express brokers
After evaluating their options, AppFolio chose Amazon MSK Express brokers as the foundation for their next-generation streaming platform. Express brokers are a broker type offered under MSK Provisioned. They include pay-as-you-go elastic storage that scales automatically, intelligent partition rebalancing, and Kafka configuration defaults tuned for production workloads. Express brokers mapped directly to the requirements AppFolio identified:
Elastic storage that scales with their data. Express brokers remove broker disk sizing and provisioning, with storage scaling automatically alongside data growth. AppFolio pays only for the storage actually used.
AWS benchmarks showed up to 20 times faster scaling. Horizontal scaling and partition reassignment that previously took hours now complete in minutes, letting AppFolio react to traffic shifts on a much shorter cycle.
Production-tuned defaults. Express brokers come pre-configured with Kafka best-practice defaults and built-in client throughput quotas, simplifying AppFolio’s operational model.
Full Kafka API compatibility. AppFolio was able to migrate without changes to its producer and consumer applications.
As part of the migration, AppFolio also took the opportunity to rethink how the cluster was being used. Rather than recreating a single shared cluster on Express brokers, they segmented their MSK clusters by workload type. This gives customer-facing and internal workloads dedicated resources, providing better isolation and more predictable performance for each workload class.
Current architecture
AppFolio’s current architecture consists of multiple Amazon MSK clusters with Express brokers, segmented by workload type. Each cluster is sized and tuned for its specific traffic pattern, providing improved isolation and more predictable performance. The following diagram shows the deployment.
Figure 2: Current architecture with workload-segmented Amazon MSK clusters using Express brokers
Benefits achieved
By migrating to Amazon MSK Express brokers and adopting a workload-segmented cluster design, AppFolio has realized several key benefits:
Elastic, hands-off storage
The pay-as-you-go storage of Express brokers scales automatically with AppFolio’s data growth. Storage capacity is no longer something the platform team plans, provisions, or monitors, and AppFolio pays only for what they use. For a workload that runs heavily on compacted topics, this is the single largest operational improvement they have seen.
Faster scaling
Partition reassignment and broker scaling that previously took hours now complete in minutes, enabling AppFolio to adjust cluster shape in response to actual traffic instead of running ahead of forecasts.
Improved workload isolation
Splitting their streaming traffic into workload-segmented clusters has given AppFolio more predictable performance. Customer-facing and internal workloads now run on dedicated infrastructure, and each cluster can be sized and tuned for its own traffic pattern.
Stable environment as data volumes grow
Since the migration, AppFolio has maintained a stable environment with no significant downtime, even as data volumes continue to grow.
Reduced operational overhead
Hands-off storage management and intelligent rebalancing have removed several recurring tasks from the AppFolio platform team’s queue, including the constant monitoring and manual intervention that storage planning required under their previous architecture.
Conclusion
By using Amazon MSK Express brokers and adopting a workload-segmented cluster design, AppFolio has built a streaming foundation that scales elastically with their data growth and adapts quickly to changes in traffic. The pay-as-you-go storage and faster scaling of Express brokers let AppFolio’s platform team focus engineering effort on building new capabilities for customers, rather than on Kafka capacity planning. As AppFolio continues to expand its platform for the real estate industry, the Amazon MSK Express brokers infrastructure provides a scalable foundation for future growth.
Adobe has used Amazon Web Services (AWS) since 2008. Adobe Firefly powers creative features across applications including Photoshop and Illustrator.
Adobe operates a GPU-based training infrastructure built on Amazon Elastic Kubernetes Service (Amazon EKS) to support Firefly. The infrastructure enables teams to run model training jobs across thousands of compute nodes and GPUs, designed to scale with growing demand.
The team initially relied on a self-hosted Prometheus infrastructure, sending data to a remote endpoint for long-term retention. As Firefly’s adoption increased and training jobs scaled, Adobe needed an observability solution that could deliver fast query performance over large metric volumes, remain highly available and scalable, and give infrastructure users self-service access to the infrastructure metrics they need to monitor and troubleshoot training jobs independently.
This post describes how Adobe evolved its observability architecture — from a self-managed Prometheus deployment for in-cluster metrics to Amazon Managed Service for Prometheus for critical metrics — and the measurable improvements in query performance, infrastructure reliability, and scale.
The challenge: GPU observability at scale
Monitoring GPU-based training infrastructure presents unique challenges that differ from traditional application monitoring. GPU training clusters generate high-cardinality telemetry across multiple dimensions like GPU health and performance metrics, compute and memory metrics and more.
Unlike CPU workloads where a single utilization metric may suffice, GPU training jobs require engineers to observe the interplay between compute, memory, and network layers to identify bottlenecks. For example, training jobs running across 2,000 nodes with 16,000 GPUs, scraped every 30 seconds, can generate over 1 billion data points in a single query window.
Self-hosted monitoring infrastructure was not meeting the performance requirements for queries at this cardinality and volume.
From self-managed Prometheus to Amazon Managed Service for Prometheus
Adobe’s observability evolution was not a single migration. It was an iterative process, with each phase addressing a specific set of limitations and informed by direct feedback from infrastructure users on what mattered most to them. As metric volumes grew, the team evaluated Amazon Managed Service for Prometheus as a fully managed alternative that could handle their horizontal scale requirements without the operational overhead of maintaining their own deployment.
Infrastructure users shaped the critical metric set iteratively through direct input on what they needed to see to run their training jobs effectively. The critical metrics were curated to support:
Job-level monitoring: GPU utilization, memory consumption, and network throughput per training job, enabling users to identify bottlenecks in distributed training.
Pod and node health: Kubernetes pod status, node readiness, and resource allocation metrics feeding into scheduler decisions.
GPU health: Metrics that determine whether a GPU is healthy or needs to be cordoned and replaced.
The team has already moved critical 2M time series metrics to Amazon Managed Service for Prometheus, targeting the specific problem of query performance at scale. Adobe used Amazon Managed Service for Prometheus collector (managed scrapers) to handle the collection of metrics from their Amazon EKS-based training clusters and forward them directly to Amazon Managed Service for Prometheus workspaces. Rather than replacing the self-managed Prometheus deployment entirely, the managed scrapers operated alongside it, taking over the scraping role for metrics destined for Amazon Managed Service for Prometheus while preserving Adobe’s existing Prometheus setup. This allowed the team to adopt Amazon Managed Service for Prometheus incrementally without disrupting their current monitoring workflows.
Why Amazon Managed Service for Prometheus
Amazon Managed Service for Prometheus provided the capabilities that addressed Adobe’s core requirements:
Query performance at scale: Purpose-built for fast queries over high-cardinality, high-volume time series data.
High availability: Built-in high availability without custom HA configurations, providing a reliable data source for downstream automated systems that depend on timely metric queries.
Migration ease: No agents required. The migration path uses remote write configuration with minimal changes to existing workflows.
Scalability: Each workspace supports up to 50 million active time series, providing headroom for growth as the infrastructure scales (up to 1 billion) [1].
AWS integration: Native integration with AWS services including Amazon EKS and Amazon Managed Grafana, simplifying metric collection and reducing configuration complexity.
Managed operations: Minimizes the operational burden of administering self-hosted monitoring infrastructure, freeing engineering resources for infrastructure development.
Note: Amazon Managed Service for Prometheus and Amazon Managed Grafana are billable services. Costs are based on metrics ingested, stored, and queried. Review the pricing pages for Amazon Managed Service for Prometheus and Amazon Managed Grafana to estimate costs for your workload before deployment.
Results
After migrating critical metrics to Amazon Managed Service for Prometheus, Adobe Firefly achieved the following measurable improvements.
Query performance: before and after
Time Range
Amazon Managed Service for Prometheus vs Self-managed
4h
3.5x faster
12h
22.6x faster
24h
28.8x faster
Figure 1: Query performance comparison for GPU utilization metrics
Conclusion
Adobe Firefly evolved its observability architecture from a self-managed Prometheus deployment to Amazon Managed Service for Prometheus, using Amazon Managed Service for Prometheus collector to handle metric collection alongside their existing Prometheus infrastructure. This approach preserves current workflows while adding managed collection.
Query performance improvement of more than 28x: Queries that previously timed out at 60 seconds or returned partial results in 2 minutes now complete in approximately 10 seconds.
Extended observability windows for training jobs: Infrastructure users now view metrics across 24-hour windows, compared to the previous practical limit of 6 hours. This is particularly impactful for large, long-running training jobs spanning 256 or more nodes, where the ability to see the full lifecycle of a job helps identify when performance degraded, correlate issues with infrastructure events, and make informed decisions.
Reduced operational overhead: Amazon Managed Service for Prometheus requires no agents and no additional Prometheus-related configuration on your end. Both data and control components are fully managed, minimizing the burden of maintaining self-hosted Prometheus infrastructure.
The performance improvements demonstrated with GPU utilization queries were consistent across other GPU metrics as well, including GPU memory usage, power consumption, and thermal monitoring. These results confirm that Amazon Managed Service for Prometheus benefits extend across the full breadth of GPU telemetry. Adobe and AWS are collaborating on the next phase of this observability architecture to extend managed Prometheus to the remaining metric tiers, enabling a multi-tenant, highly available observability stack that supports the full scale of telemetry at Adobe Firefly.
Amazon OpenSearch Service has been helpful in understanding the game ecosystem. OpenSearch Service powers two types of use cases: search workloads and analytics workloads. A team at Epic Games had a use case for storing and analyzing a sliding window of game event data. This involves supporting complex queries and multilayered aggregations that feed analytical results into other internal systems, helping them power an evolving player experience. At the scale of a game like Fortnite with a large player base, these queries run against a significant volume of incoming data.
These insights help identify emerging gameplay trends, understand how players engage with new content, and reveal more about the Fortnite ecosystem. They inform live operation decisions and help surface relevant content to players based on aggregated activity across the community.
As Epic Games’ infrastructure handles billions of telemetry events, the team identified opportunities to optimize their OpenSearch Service cluster for better performance and cost efficiency. This post details how Epic Games partnered with AWS to transform their OpenSearch Service deployment, achieving significant improvements in query latency and resource utilization while reducing operational costs.
The challenge
Epic Games runs an OpenSearch Service domain that handles continuous high-volume writes alongside CPU-intensive batch aggregation jobs. Ideally, these aggregation jobs would run more frequently to keep analytics fresh. Shorter job intervals mean fresher data for identifying gameplay trends, detecting anomalies, and informing live operations decisions. But the existing configuration couldn’t support this without scaling the domain beyond what the workload justified, driving up costs. Epic Games worked with AWS to identify where improvements could be made, focusing on areas such as hardware utilization, sharding strategy, index mappings, and query behavior.
Observations
The cluster was running on r7g memory-optimized data nodes, with 48 vCPUs and 384 GiB of memory per node. Of each node’s available memory, only a fraction (32 GiB) was allocated to Java Virtual Machine (JVM) heap, set at the maximum recommended for compressed oops. The remainder (off-heap memory) was used for the filesystem cache and the operating system. System memory was not fully utilized across the data nodes (Figure 1).
Figure 1: System memory utilization across data nodes
As shown in the preceding figure, utilization stays well below 100% throughout the observation period, confirming that much of the off-heap memory allocated to these nodes goes unused. The excess capacity could be safely exchanged for additional compute resources.
JVM memory pressure is shown in Figure 2, and the correlating garbage collection metrics (both count and time) are shown in Figure 3.
Figure 2: JVM memory pressure
Figure 3: JVM garbage collection metrics, count (top) and time (bottom)
These charts show that JVM memory pressure remains below critical thresholds, and both garbage collection count and time are low and stable, indicating healthy JVM utilization across the domain.
While cluster-level CPU metrics appeared healthy at first glance (Figure 4), zooming into node-level metrics revealed clear node hotspots. The root cause of the node hotspots was the cluster’s sharding strategy.
Figure 4: Cluster-level CPU utilization
The cluster had data nodes distributed across multiple Availability Zones. Each index used a set number of primary shards with replicas, rolling over after shards reached a certain size. At first glance, the configuration appeared well-balanced, with shard copies distributed across Availability Zones and each node holding a manageable share of the data.
However, the primary shard count was lower than the total data node count. This meant that searches targeting the latest data, which is the most common access pattern, would only execute across a subset of available nodes. As a result, some nodes developed consistent CPU-based hotspots while the rest remained underutilized (Figure 5).
Figure 5: Node-level CPU utilization showing hotspots
As shown in the preceding figure, some nodes reach as high as 90 percent CPU utilization while several others remain under 20 percent, highlighting the uneven distribution of query execution across the cluster.
Recommendations and implementation
Based on these observations, AWS worked together with Epic Games on a set of targeted optimizations spanning hardware selection, sharding strategy, index mappings, and query behavior. The following sections detail each recommendation and how it was implemented.
Right-sizing the cluster
Because aggregation queries are CPU-intensive by nature and the cluster’s JVM memory pressure was well within acceptable ranges, AWS recommended migrating from memory-optimized r7g instances to compute-optimized c7g instances. The c7g family offers a higher ratio of vCPU to RAM, which is better suited for workloads where processing power rather than memory capacity is the binding constraint.
The proposed architecture called for a larger number of c7g nodes than the existing r7g count. This migration achieved approximately 33 percent more aggregate CPU capacity across the cluster while operating with two-thirds of the original memory. The net effect was a meaningful cost reduction of approximately 10 percent, delivering more processing power at lower cost by aligning the instance profile with the actual nature of the workload (Table 1).
R7g (Before)
c7g (After)
Net Impact
Instance Family
Memory Optimized
Compute Optimized
Better CPU-to-RAM alignment for aggregation workloads
vCPUs per Node
Same
Same
Same per-node CPU. More nodes = higher aggregate CPU
Memory per Node
Higher
Lower
Reduced unused memory; JVM heap unchanged
Aggregate CPU
Baseline
+33% more total vCPUs
Distributed more evenly across higher node count
Cost
Baseline
~10% reduction
More performance per dollar spent
Table 1: Instance migration comparison, r7g compared to c7g
Sharding strategy
To support the new cluster sizing, the Epic Games team changed the sharding strategy so that the number of primary shards matches the data node count, with 1 replica. This distributes both the write-heavy load and the batch aggregation search query load evenly on all the available data nodes.
The team employed ISM (Index State Management) policies to manage shard sizing through rollover, targeting shard sizes within recommended bounds using min_primary_shard_size. This kept shard counts bounded and predictable, providing a clear scaling pattern: adjust the node count, then update the ISM policy accordingly.
After implementation, node-level CPU utilization showed a much more even distribution (Figure 6).
Figure 6: Node-level CPU utilization after sharding optimization
As shown in Figure 6, all nodes in the domain are working at similar CPU utilization levels, confirming that data and traffic are well distributed across the cluster with no node hotspots.
Mapping optimization
The index mappings had both text and keyword field types enabled on many fields, even though access patterns showed those fields were only used for aggregation, sorting, or filter context, and never for full-text match queries. Removing the redundant text field type reduced storage overhead and improved query performance by eliminating unnecessary analysis at index time.
For high-cardinality string fields, the murmur3 field type does a compute-once-and-store optimization for cardinality aggregation. Instead of hashing keyword values at query time, murmur3 computes the hash once at index time and stores it as a numeric doc_value, so the aggregation can skip the expensive string hashing step at query time (the cardinality estimate itself is still computed at query time).
The following example illustrates the mapping changes:
These mapping changes reduced overall storage, lowered shard count (which reduced CPU requirements), and reduced cluster manager node state size.
Index optimization
Additional index-level optimizations were applied to improve query performance and reduce overhead. Index sorting was configured to default to the primary date field, which improves performance for time-based access patterns by aligning the physical data layout with the most common query order. The ISM policy was updated to force merge indices down to 1 segment after rollover, reducing segment overhead on read-only indices. Finally, the refresh interval was tuned to balance indexing throughput with search freshness.
Upgrading from OpenSearch Service 2.17 to 3.1
The domain was upgraded from OpenSearch Service 2.17 to 3.1, which reduced error counts and improved throughput at the Amazon OpenSearch Ingestion pipeline level. The performance gains were notable: p99 latency on sum aggregations dropped by 40–50 percent after the upgrade alone, and large 96-hour cardinality aggregations saw p95 drop over 40 percent. General query performance improved across all query types, and thread pool pressure was reduced significantly, leading to far fewer 429 errors (Figure 7).
Figure 7: Query performance before and after the OpenSearch Service 3.1 upgrade
Upgrading from Graviton 3 to Graviton 4
The instance types were upgraded from c7g (Graviton 3) to c8g (Graviton 4). The performance gains were immediate:
p99 on all queries: 380 ms to 250 ms.
p95 on all queries: 245 ms to 230 ms.
p90 on all queries: 225 ms to 200 ms.
p50 on all queries: 100 ms to 70 ms.
Date-windowed cardinality queries saw their p99 halved from 220 ms to 98 ms, with sum-based aggregations experiencing similar gains. Overall throughput increased by 16 percent.
Tiered caching
With the upgrade to OpenSearch Service 3.1, the team enabled tiered caching. Tiered caching extends the default on-heap request cache with a disk-based tier. When items are evicted from the on-heap cache, they spill into a larger disk cache on the node’s local SSD rather than being discarded. This allows the cluster to retain cached results for a much larger set of queries without increasing JVM heap usage.
The batch aggregation jobs in Epic Games’ workload issue repeated queries over overlapping time windows. The on-heap cache alone was too small to retain results across successive job runs, so expensive aggregations were recomputed each time. With the disk tier enabled, results from longer time-window aggregations (such as the 96-hour cardinality queries) persisted between runs. This produced more consistent and faster results on some of the larger aggregation queries, particularly those spanning longer time windows.
Results summary
The following table summarizes the impact of each optimization.
Optimization Strategy
Impact
Right-sizing (r7g to c7g)
33% more CPU, 10% cost reduction
Sharding rebalance
Eliminated CPU hotspots across nodes
Mapping optimization
Reduced storage, shard count, and cluster state size
OpenSearch Service 2.17 to 3.1
p99 sum aggs reduced 40-50%, fewer 429 errors
Graviton 3 to Graviton 4
p99 380 ms to 250 ms, 16% higher throughput
Tiered caching
More consistent results on large aggregation queries
Table 2: Results summary
Conclusion
By optimizing their OpenSearch Service deployment, a team at Epic Games reduced p99 query latency from 380 ms to 250 ms, increased throughput by 16 percent, and lowered costs by 10 percent. These gains came from aligning instance types, sharding strategy, mappings, and engine versions with the workload’s actual demands.
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
