Post Syndicated from Shraddha Anil Naik original https://aws.amazon.com/blogs/big-data/building-a-scalable-personalized-recommendation-system-on-aws-from-batch-to-real-time/

Amazon.com receives millions of visits every day, and behind every product recommendation on our website is a system that needs to process customer signals, run machine learning (ML) models, and deliver results before the next visit. Doing this across global marketplaces for millions of customers at tens of thousands of requests per second, while keeping experimentation fast and infrastructure costs bounded, is an orchestration challenge as much as a machine learning one.

Our team built a system that addresses this challenge. This post shows how we did it using a batch-first architecture with AWS Lake Formation, Amazon Managed Workflows for Apache Airflow (Amazon MWAA), Amazon Athena, AWS Glue, Amazon SageMaker, and Amazon DynamoDB, and how we later extended it with Amazon MemoryDB for real-time vector similarity search when we needed to incorporate more real-time signals.

Architecture overview

Data flows from the centralized data lake (Lake Formation and Athena) through Airflow-orchestrated pipelines using Glue for processing and SageMaker for ML workloads, into DynamoDB for batch serving and MemoryDB for real-time inference

The data lake foundation: Centralized access with Lake Formation

Every recommendation pipeline starts with data. We built a centralized data lake to create a single source of truth that any pipeline or consumer can access without duplicating data or building bespoke extract, transform, and load (ETL) pipelines.

Golden datasets: Shared once, used everywhere

Before the data lake, each recommendation pipeline independently extracted and transformed its own copy of product catalog, transaction history, and embeddings. This led to subtle inconsistencies: one pipeline might use a slightly different join logic or a stale snapshot, making it difficult to compare model performance or debug discrepancies across pipelines.

Now, we publish curated, validated datasets once, and every consumer (Airflow DAGs, ML notebooks, analytics dashboards) reads from the same tables through the same governed access. This means:

• New pipelines start faster. A new recommendation model does not need its own data extraction logic. It queries the existing golden datasets from day one.
• Consistency across models. When we compare model A to model B, we know both models are trained and inferred on the same underlying data.
• Cross-team collaboration. Multiple teams share the same tables as a single source of truth.
• Two-way data flow. The data lake serves as both source and destination. Our pipelines read golden datasets as inputs and write computed outputs (model scores, feature sets, intermediate results) back to the lake, where they become inputs for other pipelines. This creates a compounding effect: each new pipeline enriches the lake for the next one.

Why Lake Formation?

Our data is stored in Amazon Simple Storage Service (Amazon S3), partitioned by marketplace. Our data consumers (Airflow pipelines, ML notebooks, analytics tools) live in separate AWS accounts from the data producers. We chose AWS Lake Formation because it adds governance on top of S3 without requiring data migration:

• Fine-grained cross-account access. Grant table-level permissions per consumer role, without managing bucket policies manually.
• Schema governance through AWS Glue Data Catalog. Scheduled Glue crawlers infer schemas from files in S3, keeping the catalog current as data evolves.
• Multi-Region consistency. We deploy identical infrastructure across multiple AWS Regions using AWS Cloud Development Kit (AWS CDK), each Region serving its local marketplaces.

Orchestration: Amazon Managed Workflows for Apache Airflow (Amazon MWAA)

We chose Amazon Managed Workflows for Apache Airflow (Amazon MWAA) as our orchestration layer. MWAA removes the operational burden of managing Airflow infrastructure: automatic scaling of workers, built-in high availability, and managed upgrades mean our team focuses on pipeline logic rather than cluster maintenance. MWAA lets us define complex multi-step workflows with rich dependencies in Python code, and its operator model lets us encapsulate team-specific conventions into reusable building blocks.

Each recommendation pipeline follows a consistent pattern:

We built a library of reusable custom Airflow operators, each encapsulating one of our core compute engines. This reduced new pipeline development from weeks to days in our team’s experience.

Athena and AWS Glue: Data access and processing

Amazon Athena is how our pipelines read from Lake Formation. Our custom operator runs SQL queries against the Glue Data Catalog and automatically runs UNLOAD to write results to S3: serverless, no infrastructure to manage, and integrated with the Lake Formation permission model.

AWS Glue handles compute-intensive data transformations through PySpark: joining datasets, filtering, deduplication, aggregation, and formatting ML outputs into final recommendation lists. We configure Glue with Auto Scaling worker pools (for example, 2–50 workers of G.8X type) so jobs scale with data volume per marketplace. All jobs run ephemerally: they read from S3, write to S3, and require no long-running clusters.

Amazon SageMaker: Training, inference, and vector search

Amazon SageMaker powers the ML-intensive stages of our pipelines across three workload types, all orchestrated as steps within our Airflow DAGs:

Training

We use SageMaker Training Jobs to train our recommendation models on GPU instances (for example, ml.g5). Training data is prepared by upstream Glue jobs and staged in S3. Our custom Airflow operator submits the training job, monitors its progress, and registers the resulting model artifact in S3. Once training completes, the artifact is immediately available for batch inference or endpoint deployment within the same DAG run. This means a single DAG can go from raw data to trained model to deployed inference without manual handoffs, and we can retrain it on fresh data every pipeline cycle with zero operator intervention.

Batch inference

SageMaker Batch Transform runs our trained models at scale, generating the outputs that feed into downstream ranking and publishing steps. Our batch inference operator handles job submission, polls for completion, and writes the output location to the DAG’s S3 convention so the next Glue step can pick it up automatically. Batch Transform lets us run inference without provisioning persistent infrastructure, and we can scale instance count and type independently for every pipeline based on data volume.

Vector search

Generating recommendations for millions of customers requires searching across hundreds of thousands of candidate products per customer. Exact search at this scale is prohibitively expensive, so we use approximate nearest neighbor (ANN) search using FAISS to find similar products efficiently, run as SageMaker Processing Jobs.

The workflow:

1. Build a FAISS index over the candidate catalog.
2. Query the index with per-customer vectors to find top-K nearest neighbors.
3. Return ranked candidate lists per customer.

We distribute query vectors across the SageMaker Processing fleet using S3-based sharding (ShardedByS3Key). Each instance receives the full candidate index but only a fraction of the query vectors. Every instance builds an identical FAISS index, searches its shard of queries, and writes results to S3. The downstream Glue step merges all shards into the final recommendation lists. This lets us scale horizontally by adding instances without changing any code.

Why SageMaker inside MWAA?

Running SageMaker jobs as MWAA tasks (rather than standalone) gives us:

• End-to-end lineage. Every model training run, inference job, and vector search is tracked as part of a DAG execution. We can trace a recommendation in DynamoDB back to the exact training run, data snapshot, and ANN search that produced it.
• Retry and failure handling. If a SageMaker job fails (spot instance preemption, transient capacity errors), Airflow retries it automatically with backoff. No manual re-runs.
• Resource sequencing. Training must finish before inference, inference before ANN search. The Airflow dependency model handles this naturally without polling scripts or step function state machines.
• Unified monitoring. One Airflow dashboard shows the health of all pipelines: Glue ETL, SageMaker training, SageMaker inference, and DynamoDB publishing. No context-switching between consoles.

Putting it together: A complete pipeline example

Our recommendation generation pipeline illustrates the full flow:

Note that the operators shown (GlueSQLOperator, VectorSearchOperator, and others) are custom internal operators built on top of the Airflow AWS provider, not open-source libraries. Here is a simplified version of what this looks like in code:

from airflow import DAG
from airflow.models.baseoperator import chain
from airflow.utils.task_group import TaskGroup

dag = DAG("recommendation_generator", schedule="0 18 * * 4")  # Weekly
task_groups = []

for marketplace in [...]:  # global marketplaces
    with TaskGroup(group_id=marketplace, dag=dag) as group:

        # Step 1: Extract data from lake
        candidates = GlueSQLOperator(
            task_id="select_candidates",
            tables=[f"{marketplace}.catalog", f"{marketplace}.products"],
            sql="select_candidates.sql", dag=dag)

        customer_history = GlueSQLOperator(
            task_id="select_customer_history",
            tables=[f"{marketplace}.transactions"],
            sql="select_customer_vectors.sql", dag=dag)

        customers = AthenaSQLOperator(
            task_id="select_customers", database=marketplace,
            query="select_customer_cohort.sql", dag=dag)

        # Step 2: Train model
        train = ModelTrainingOperator(
            task_id="train_model",
            training_input={"task_id": customer_history.task_id},
            instance_type="ml.g5", dag=dag)

        # Step 3: Batch inference
        inference = BatchInferenceOperator(
            task_id="batch_inference",
            model={"task_id": train.task_id},
            input_data={"task_id": candidates.task_id}, dag=dag)

        # Step 4: Vector search via SageMaker Processing Job
        ann_search = VectorSearchOperator(
            task_id="ann_search",
            index_input={"task_id": inference.task_id},
            query_input={"task_id": customer_history.task_id},
            k=60, instance_count=10, dag=dag)

        # Step 5: Merge and rank with Glue
        merge_and_rank = GlueSparkOperator(
            task_id="merge_and_rank",
            script="merge_results.py", dag=dag)

        # Step 6: Publish to DynamoDB
        publish = DynamoDBPublishOperator(
            task_id="publish_recommendations",
            table_name="Recommendations", dag=dag)

        # Task dependencies
        [candidates, customer_history] >> train >> inference
        [inference, customers] >> ann_search >> merge_and_rank >> publish

    task_groups.append(group)

chain(*task_groups)  # Execute marketplaces sequentially

Key patterns in this DAG

• Marketplace isolation. Each marketplace runs in its own TaskGroup. A failure in one does not block the others.
• Parallel data extraction. Independent Athena/Glue queries run concurrently before converging at the training step.
• Sequential marketplace execution. chain() runs marketplaces one at a time to avoid resource contention across large SageMaker and Glue jobs.
• Reusable operators. We built custom operators like GlueSQLOperator, VectorSearchOperator, and DynamoDBPublishOperator that encode our team’s conventions (cross-account access, S3 staging, retry logic, metrics) into shared building blocks. New pipelines are mostly configuration rather than infrastructure code, and these operators are shared across dozens of pipelines.
• Implicit data passing. Each operator writes its output to a convention-based S3 path and downstream operators automatically resolve the upstream output location. No hard-coded paths between steps.

This pattern powers hundreds of pipelines across global marketplaces, with each marketplace as an isolated TaskGroup in the DAG.

Serving layer: DynamoDB + ECS

Our Java-based Amazon Elastic Container Service (Amazon ECS) service reads pre-computed recommendations from Amazon DynamoDB at request time:

1. Receive request with customer ID, marketplace, customer context, and page context.
2. Read pre-computed recommendations from DynamoDB.
3. Apply real-time filters (availability, eligibility constraints).
4. Re-rank based on real-time context signals.
5. Return the response.

Amazon DynamoDB is designed to provide single-digit millisecond reads at our scale and time-to-live (TTL) for automatic cleanup of stale recommendations.

Extending to real-time

In recommendation system terms, our batch pipeline handles the retrieval stage offline. Although this covers the majority of our traffic, we identified scenarios where weekly freshness was not enough to capture what customers are doing right now. The real-time extension adds an online retrieval and ranking path for signals that cannot wait for the next batch cycle.

To address these, we extended our system with Amazon MemoryDB (which now supports Valkey as its open-source engine) for real-time vector similarity search and SageMaker real-time endpoints for on-demand embedding generation. The same Airflow pipelines that publish to DynamoDB also publish product vectors to MemoryDB (through our MemoryDB publish operator), and the same model in our batch pipeline is deployed to a SageMaker endpoint for single-item inference at request time.

At serving time, when we want to incorporate fresh signals, the service calls the SageMaker endpoint to generate an embedding on the fly, then queries MemoryDB for the nearest neighbors. These fresh signals include recent search queries, cart additions, and other in-session activity that changes faster than our weekly batch cycle. In our workloads, this gives us sub-millisecond vector search latency without re-running the full batch pipeline. Critically, the batch pipeline keeps the MemoryDB product index fresh. Our Airflow DAG includes a MemoryDB publish operator that refreshes the full product vector index weekly, so real-time queries always search against an up-to-date index.

Batch and real-time are not competing approaches: batch handles slow-moving signals (purchase history, catalog relationships) while real-time handles fast-moving ones (current session, new arrivals, trending items). Both paths share the same models, the same data lake, and the same serving service.

For a detailed deep-dive on this real-time architecture, see Real-time personalized recommendations with Amazon SageMaker and Amazon Managed Valkey.

Security and access control

Security is a foundational concern for a system that spans multiple AWS accounts, processes customer behavioral data, and runs across multiple AWS Regions.

Cross-account access: Lake Formation grants are issued to consumer-account AWS Identity and Access Management (IAM) roles, ensuring consumers can query tables without direct S3 bucket access. Each consumer role receives only the permissions it needs for its specific tables.

IAM execution roles: Each compute engine (MWAA, Glue, SageMaker) runs under a dedicated least-privilege IAM role.

Network isolation: MWAA environments and the ECS serving layer are deployed within virtual private clouds (VPCs), with separate VPC configurations per Region.

Reliability and failure handling

Regional isolation: Each AWS Region runs an independent copy of the system. DynamoDB tables are regional, each populated by the local batch pipeline. MemoryDB clusters are regional, with the product vector index refreshed by the local Airflow DAG. This means a regional failure or pipeline delay in one Region does not affect other Regions.

Batch pipeline failures: If a pipeline fails mid-run, the previous DynamoDB data remains live and continues serving recommendations until the next successful run. Airflow retries failed tasks automatically with configurable backoff. TTL on DynamoDB records bounds how long stale data persists. Failed pipeline runs trigger automated alerting so the on-call engineer can investigate.

Real-time path resilience: MemoryDB is deployed in a multi-AZ configuration with automatic failover. The real-time path is an extension of the batch system, and batch recommendations from DynamoDB remain available regardless of real-time path availability.

Lessons learned

1. Start batch, add real-time incrementally. Batch pipelines are easier to debug, cheaper to operate, and sufficient for most recommendation scenarios. Add real-time path only when you have clear indication that specific customer signals (for example, in-session activity, search queries) need sub-hour freshness to remain relevant.
2. Match the serving path to signal velocity. Not every signal needs real-time processing. Categorize your signals by how quickly they change, and route accordingly: batch for slow signals, real-time for fast ones, re-ranking for in-between.
3. Freshness does not always require re-computation. A batch-generated candidate set remains largely valid between runs. What changes is relative relevance. Re-ranking at serving time with recent customer activity gives the impression of real-time without the cost of real-time candidate generation.
4. A centralized data lake accelerates everything. Golden datasets eliminated weeks of per-pipeline data extraction work, made model comparisons trustworthy, and let new team members ship their first pipeline in days instead of weeks. The upfront investment in Lake Formation governance paid for itself within the first quarter.
5. Invest in reusable operators. Custom Airflow operators encapsulating Athena/Glue/SageMaker patterns let teams ship new pipelines in days. The operators encode best practices (retry logic, cross-account access, metrics) so pipeline authors can focus on business logic.
6. Separate compute from storage. S3 as the universal intermediate layer + ephemeral Glue/SageMaker jobs means you pay only for active computation. No idle clusters between weekly pipeline runs.

Conclusion

We built this system because every customer interaction is an opportunity to surface the right product at the right time. Serving tens of thousands of requests per second across millions of customers in global marketplaces, our batch-first architecture uses AWS Lake Formation for governed data access, Amazon MWAA for orchestration, Amazon Athena for data lake queries, AWS Glue for distributed processing, Amazon SageMaker for training, inference, and vector search, and Amazon DynamoDB for low-latency serving. When we needed to incorporate more real-time signals, we added Amazon MemoryDB vector search paired with SageMaker real-time endpoints for on-demand embedding generation, extending into real-time without replacing the batch foundation.

The architecture choices we have made, batch for efficiency and real-time for freshness, all serve the goal of helping customers discover what they need faster. If you are building personalized experiences at scale, we hope these patterns give you a useful starting point.

This would not have been possible without the Everyday Essentials engineering team, whose collective effort turned these ideas into a production system serving customers every day. We are also grateful for the support and guidance from Sam Heyworth, Nirav Desai, and Ankur Datta, and the broader Everyday Essentials leadership team.

About the authors