Post Syndicated from Jon Evans original https://aws.amazon.com/blogs/big-data/how-a-team-at-epic-games-tuned-amazon-opensearch-service-for-fortnite-analytics/

Since the launch of Fortnite in 2017, Epic Games has reached hundreds of millions of players worldwide. Fortnite runs on Amazon Web Services (AWS), and takes advantage of services such as Amazon OpenSearch Service to power certain internal analytics and drive decision making at scale.

Amazon OpenSearch Service has been helpful in understanding the game ecosystem. OpenSearch Service powers two types of use cases: search workloads and analytics workloads. A team at Epic Games had a use case for storing and analyzing a sliding window of game event data. This involves supporting complex queries and multilayered aggregations that feed analytical results into other internal systems, helping them power an evolving player experience. At the scale of a game like Fortnite with a large player base, these queries run against a significant volume of incoming data.

These insights help identify emerging gameplay trends, understand how players engage with new content, and reveal more about the Fortnite ecosystem. They inform live operation decisions and help surface relevant content to players based on aggregated activity across the community.

As Epic Games’ infrastructure handles billions of telemetry events, the team identified opportunities to optimize their OpenSearch Service cluster for better performance and cost efficiency. This post details how Epic Games partnered with AWS to transform their OpenSearch Service deployment, achieving significant improvements in query latency and resource utilization while reducing operational costs.

The challenge

Epic Games runs an OpenSearch Service domain that handles continuous high-volume writes alongside CPU-intensive batch aggregation jobs. Ideally, these aggregation jobs would run more frequently to keep analytics fresh. Shorter job intervals mean fresher data for identifying gameplay trends, detecting anomalies, and informing live operations decisions. But the existing configuration couldn’t support this without scaling the domain beyond what the workload justified, driving up costs. Epic Games worked with AWS to identify where improvements could be made, focusing on areas such as hardware utilization, sharding strategy, index mappings, and query behavior.

Observations

The cluster was running on r7g memory-optimized data nodes, with 48 vCPUs and 384 GiB of memory per node. Of each node’s available memory, only a fraction (32 GiB) was allocated to Java Virtual Machine (JVM) heap, set at the maximum recommended for compressed oops. The remainder (off-heap memory) was used for the filesystem cache and the operating system. System memory was not fully utilized across the data nodes (Figure 1).

Figure 1: System memory utilization across data nodes

As shown in the preceding figure, utilization stays well below 100% throughout the observation period, confirming that much of the off-heap memory allocated to these nodes goes unused. The excess capacity could be safely exchanged for additional compute resources.

JVM memory pressure is shown in Figure 2, and the correlating garbage collection metrics (both count and time) are shown in Figure 3.

Figure 2: JVM memory pressure

Figure 3: JVM garbage collection metrics, count (top) and time (bottom)

These charts show that JVM memory pressure remains below critical thresholds, and both garbage collection count and time are low and stable, indicating healthy JVM utilization across the domain.

While cluster-level CPU metrics appeared healthy at first glance (Figure 4), zooming into node-level metrics revealed clear node hotspots. The root cause of the node hotspots was the cluster’s sharding strategy.

Figure 4: Cluster-level CPU utilization

The cluster had data nodes distributed across multiple Availability Zones. Each index used a set number of primary shards with replicas, rolling over after shards reached a certain size. At first glance, the configuration appeared well-balanced, with shard copies distributed across Availability Zones and each node holding a manageable share of the data.

However, the primary shard count was lower than the total data node count. This meant that searches targeting the latest data, which is the most common access pattern, would only execute across a subset of available nodes. As a result, some nodes developed consistent CPU-based hotspots while the rest remained underutilized (Figure 5).

Figure 5: Node-level CPU utilization showing hotspots

As shown in the preceding figure, some nodes reach as high as 90 percent CPU utilization while several others remain under 20 percent, highlighting the uneven distribution of query execution across the cluster.

Recommendations and implementation

Based on these observations, AWS worked together with Epic Games on a set of targeted optimizations spanning hardware selection, sharding strategy, index mappings, and query behavior. The following sections detail each recommendation and how it was implemented.

Right-sizing the cluster

Because aggregation queries are CPU-intensive by nature and the cluster’s JVM memory pressure was well within acceptable ranges, AWS recommended migrating from memory-optimized r7g instances to compute-optimized c7g instances. The c7g family offers a higher ratio of vCPU to RAM, which is better suited for workloads where processing power rather than memory capacity is the binding constraint.

The proposed architecture called for a larger number of c7g nodes than the existing r7g count. This migration achieved approximately 33 percent more aggregate CPU capacity across the cluster while operating with two-thirds of the original memory. The net effect was a meaningful cost reduction of approximately 10 percent, delivering more processing power at lower cost by aligning the instance profile with the actual nature of the workload (Table 1).

Table 1: Instance migration comparison, r7g compared to c7g

Sharding strategy

To support the new cluster sizing, the Epic Games team changed the sharding strategy so that the number of primary shards matches the data node count, with 1 replica. This distributes both the write-heavy load and the batch aggregation search query load evenly on all the available data nodes.

The team employed ISM (Index State Management) policies to manage shard sizing through rollover, targeting shard sizes within recommended bounds using min_primary_shard_size. This kept shard counts bounded and predictable, providing a clear scaling pattern: adjust the node count, then update the ISM policy accordingly.

After implementation, node-level CPU utilization showed a much more even distribution (Figure 6).

Figure 6: Node-level CPU utilization after sharding optimization

As shown in Figure 6, all nodes in the domain are working at similar CPU utilization levels, confirming that data and traffic are well distributed across the cluster with no node hotspots.

Mapping optimization

The index mappings had both text and keyword field types enabled on many fields, even though access patterns showed those fields were only used for aggregation, sorting, or filter context, and never for full-text match queries. Removing the redundant text field type reduced storage overhead and improved query performance by eliminating unnecessary analysis at index time.

For high-cardinality string fields, the murmur3 field type does a compute-once-and-store optimization for cardinality aggregation. Instead of hashing keyword values at query time, murmur3 computes the hash once at index time and stores it as a numeric doc_value, so the aggregation can skip the expensive string hashing step at query time (the cardinality estimate itself is still computed at query time).

The following example illustrates the mapping changes:

"some_field": {
  "type": "text",
  "fields": {
    "keyword": {
      "ignore_above": 256,
      "type": "keyword"
    }
  }
},
"another_field": {
  "type": "text",
  "fields": {
    "keyword": {
      "ignore_above": 256,
      "type": "keyword"
    }
  }
},
"cardinality_field": {
  "type": "text",
  "fields": {
    "keyword": {
      "ignore_above": 256,
      "type": "keyword"
    }
  }
},

"some_field": {
  "type": "keyword"
},
"another_field": {
  "type": "keyword"
},
"cardinality_field": {
  "type": "keyword",
  "fields": {
    "hash": {
      "type": "murmur3"
    }
  }
},

These mapping changes reduced overall storage, lowered shard count (which reduced CPU requirements), and reduced cluster manager node state size.

Index optimization

Additional index-level optimizations were applied to improve query performance and reduce overhead. Index sorting was configured to default to the primary date field, which improves performance for time-based access patterns by aligning the physical data layout with the most common query order. The ISM policy was updated to force merge indices down to 1 segment after rollover, reducing segment overhead on read-only indices. Finally, the refresh interval was tuned to balance indexing throughput with search freshness.

Upgrading from OpenSearch Service 2.17 to 3.1

The domain was upgraded from OpenSearch Service 2.17 to 3.1, which reduced error counts and improved throughput at the Amazon OpenSearch Ingestion pipeline level. The performance gains were notable: p99 latency on sum aggregations dropped by 40–50 percent after the upgrade alone, and large 96-hour cardinality aggregations saw p95 drop over 40 percent. General query performance improved across all query types, and thread pool pressure was reduced significantly, leading to far fewer 429 errors (Figure 7).

Figure 7: Query performance before and after the OpenSearch Service 3.1 upgrade

Upgrading from Graviton 3 to Graviton 4

The instance types were upgraded from c7g (Graviton 3) to c8g (Graviton 4). The performance gains were immediate:

• p99 on all queries: 380 ms to 250 ms.
• p95 on all queries: 245 ms to 230 ms.
• p90 on all queries: 225 ms to 200 ms.
• p50 on all queries: 100 ms to 70 ms.

Date-windowed cardinality queries saw their p99 halved from 220 ms to 98 ms, with sum-based aggregations experiencing similar gains. Overall throughput increased by 16 percent.

Tiered caching

With the upgrade to OpenSearch Service 3.1, the team enabled tiered caching. Tiered caching extends the default on-heap request cache with a disk-based tier. When items are evicted from the on-heap cache, they spill into a larger disk cache on the node’s local SSD rather than being discarded. This allows the cluster to retain cached results for a much larger set of queries without increasing JVM heap usage.

The batch aggregation jobs in Epic Games’ workload issue repeated queries over overlapping time windows. The on-heap cache alone was too small to retain results across successive job runs, so expensive aggregations were recomputed each time. With the disk tier enabled, results from longer time-window aggregations (such as the 96-hour cardinality queries) persisted between runs. This produced more consistent and faster results on some of the larger aggregation queries, particularly those spanning longer time windows.

Results summary

The following table summarizes the impact of each optimization.

Table 2: Results summary

Conclusion

By optimizing their OpenSearch Service deployment, a team at Epic Games reduced p99 query latency from 380 ms to 250 ms, increased throughput by 16 percent, and lowered costs by 10 percent. These gains came from aligning instance types, sharding strategy, mappings, and engine versions with the workload’s actual demands.

To learn more about optimizing Amazon OpenSearch Service for your workloads, see Best practices for Amazon OpenSearch Service. For details on supported instance types, see Supported instance types in Amazon OpenSearch Service.

About the authors

Post Syndicated from Sayan Das original https://aws.amazon.com/blogs/big-data/transforming-search-at-delivery-hero-a-migration-journey-to-opensearch-service-with-radial-search/

Have you ever searched for something like “low fat yogurt” at any online grocery store and noticed how the results seem to understand what you mean? Instead of only showing items with an exact match, the top-ranked products are often semantically related. You might see items like “Greek yogurt” or “yogurt with 0.5% fat,” even when only one word matches lexically. This is the power of semantic search, and when combined with traditional lexical search, it creates a hybrid search experience that delivers both precision and recall.

At Delivery Hero, one of the world’s leading online food delivery platforms, the search team has been using semantic search for grocery verticals since 2024. What started as a proof-of-concept has evolved into a production-grade hybrid search system powered by Amazon OpenSearch Service. This system combines radial vector search with lexical retrieval to deliver highly relevant product results at scale.

In this post, we walk through how Delivery Hero migrated their semantic search infrastructure to Amazon OpenSearch Service, why they chose radial search over traditional k-nearest neighbor (k-NN) search, and the optimizations that made the system fast, cost-effective, and flexible for experimentation.

Legacy system overview

The original semantic search system was built as a standalone service using SpringBoot and Apache Lucene 9.9, deployed on Kubernetes. The retrieval flow worked as follows:

1. A user starts a search on the application.
2. The semantic search system retrieves the top 50 nearest-neighbor candidates from a static in-memory Lucene index.
3. These candidates passed through a filtering layer to remove out-of-stock items.
4. The filtered semantic results were merged with a parallel set of lexical search results.
5. A final ranking step combined both candidate sets to produce the response.

The team iterated on this system over seven versions and conducted multiple A/B tests to refine the approach. The initial system performed well, however as the business scaled, several pain points emerged:

• Scalability limitations: Running vector indices as static, in-memory structures inside Kubernetes pods meant that scaling required provisioning larger pods or adding replicas. Both options were expensive and operationally complex.
• Multi-model experimentation was difficult: Running A/B/C tests with three different product embedding model variants required fitting all models within a Kubernetes stateless workload. This created memory pressure and complicated deployment pipelines.
• Operational overhead: Managing index builds, deployments, and version rollouts for a custom Lucene-based service required significant engineering effort compared to a managed service.

Architecture modernization with OpenSearch Service

By the end of 2025, Delivery Hero had migrated their entire search infrastructure from self-managed Elasticsearch 7.x on Google Kubernetes Engine (GKE) to the fully managed Amazon OpenSearch Service 3.x. This migration created a natural opportunity to consolidate the legacy semantic search service into OpenSearch as well.

The new architecture separates concerns into two distinct pipelines: an ingestion pipeline for indexing product embeddings, and an inference pipeline for real-time hybrid retrieval.

Ingestion pipeline

For the ingestion pipeline, Delivery Hero chose Amazon OpenSearch Ingestion (OSIS) to sync product embedding data from Amazon Simple Storage Service (Amazon S3) to the OpenSearch domain.

The flow works as follows:

1. ML model
2. Airflow job: An existing Apache Airflow job periodically generates product embeddings using an external machine learning (ML) model and periodically dumps the results (product parent ID + embedding vector) to an S3 bucket.
3. OpenSearch Ingestion pipeline: An OpenSearch Ingestion pipeline is configured with a scheduled S3 scan that performs a nightly scan from S3 and updates the new k-NN index in OpenSearch Service.

version: '2'
embedding-pipeline:
  source:
    s3:
      acknowledgments: true
      scan:
        buckets:
          - bucket:
              name: my-bucket-name
              filter:
                include_prefix:
                  - vector-search/json-index/latest
        range: PT24H
        scheduling:
          interval: PT24H
      aws:
        region: eu-central-1
        sts_role_arn: arn:aws:iam::<aws-account-id>:role/osis-pipeline-role
      codec:
        ndjson: {}
      compression: none
  workers: '1'
  sink:
    - opensearch:
        hosts:
          - "https://<search-domain>.<aws-region>.es.amazonaws.com"
        aws:
          serverless: false
          region: eu-central-1
          sts_role_arn: arn:aws:iam::<aws-account-id>:role/search-xxx
        index_type: custom
        index: emb_products_v1
        template_content: ...
        template_type: index-template
        routing: '${global_entity_id}'
        document_id: '${global_entity_id}:${master_code}'
        max_retries: '3'

Because the index stores product parent IDs and embeddings are regenerated in batch, there is no need for real-time updates. This allows the team to refresh and force-merge the index once per day, resulting in highly optimized segment structures and fast retrieval speeds (p99 < 35 ms during peak hours).

Setting up the OSIS pipeline required only a few lines of Terraform, making it straightforward to provision and maintain as infrastructure-as-code.

Inference pipeline

On the retrieval side, the system runs a hybrid search strategy that combines radial vector search with lexical search in parallel:

1. Query embedding: A user’s search query first reaches the Query Understanding (QU) service, where it is encoded into an embedding using the same live ML model employed for product embeddings. To optimize performance, embeddings for top queries are cached.
2. Parallel lexical and semantic retrieval:
   • A radial k-NN search runs against the product embeddings index using min_score to retrieve all semantically similar products above a similarity threshold.
   • A lexical BM25 search runs against the product catalog index.
     

     Comparing p95 OpenSearch time for both lexical and semantic search.

3. ID resolution and inventory filter: Because the k-NN index stores product parent IDs, a resolution step maps these to individual product IDs via a secondary index that maintains near real-time inventory updates. This approach satisfies two key business requirements within a single retrieval call: product-id resolution and real-time availability filtering.
4. Merge and re-rank: A custom post-processing step combines results from both lexical and radial search, applies re-ranking logic, and returns the final result set.

Why radial search?

Traditional k-NN search in OpenSearch uses a top-k approach: you ask for the k nearest neighbors, and you get exactly k results regardless of how similar they actually are. This works well for many use cases, but it has a fundamental limitation for product search. It always returns a fixed number of results, even when some of those results are not semantically relevant.

Radial search solves this by flipping the paradigm. Instead of asking “give me the 50 closest items,” you ask “give me all items that are at least this similar.” This is done using the min_score parameter in the k-NN query:

GET product-embeddings/_search
{
  "query": {
    "knn": {
      "embedding": {
        "vector": [0.12, 0.45, 0.78, ...],
        "min_score": 0.72
      }
    }
  }
}

When using radial search with cosine similarity as the space type, OpenSearch normalizes scores using the related formula (score = (1 + cosine_similarity) / 2), as documented in the OpenSearch knn-spaces reference.

This means a min_score of 0.72 in the query example, does not directly correspond to cosine similarity. Instead, 0.72 is the normalized OpenSearch score which translates to 44% cosine similarity (that is, cosine_similarity = 2 × 0.72 – 1 = 0.44).

If you need results with at least 90% cosine similarity, apply the formula:

min_score = (1 + 0.90) / 2 = 0.95. So, you would set “min_score”: 0.95 in your query.

This approach offers several advantages for product search:

• Quality over quantity: Low-relevance results are excluded at the retrieval stage rather than relying on downstream re-ranking to filter them out.
• Variable result set size: The system naturally adapts to query specificity. Niche queries return fewer, more precise results. Broad queries return more candidates for the re-ranker to work with. For example, a highly specific query like “Oatly oat milk barista edition” might return 5 results, while a broader query like “milk” might return 200.
• Better recall-precision trade-off: By tuning the min_score threshold, the team can directly control the balance between returning too many irrelevant results and missing relevant ones.

How Delivery Hero selected the threshold for radial search

Choosing the right min_score threshold is important. Set it too high and you miss relevant products. Set it too low and you flood the re-ranker with noise.

Delivery Hero approaches threshold selection through systematic experimentation. To achieve optimal precision across diverse markets, a tailored min_score threshold is assigned to each country and query type. These thresholds are meticulously determined through rigorous offline evaluations, which use historical user interaction and manually labeled data to establish a rough estimate. This initial estimate is then further refined and validated through a series of live A/B experiments.

Evaluation of the new search system

One of the key advantages of the new architecture is how naturally it supports experimentation. At Delivery Hero, we store three variants of product embeddings within a single document:

PUT product-embeddings/_doc/1?routing=FP_DE
{
  "master_product_code": "abc123",
  "embedding_variant_1": [0.12, 0.45, 0.78, ...],
  "embedding_variant_2": [0.21, 0.4, 0.98, ...],
  "embedding_variant_3": [0.13, 0.65, 0.58, ...],
  "global_entity_id": "FP_DE"
}

In this example, embedding_variant_1, embedding_variant_2, and embedding_variant_3 are generated from three different models for A/B/C testing. After each test, the winning variant is designated as the control, while the other two are replaced with new models for further experimentation. With this approach, the team can iterate continuously while maintaining constant space complexity.

Optimizations of large scale production system

Engine upgrade: OpenSearch 2.17 to 3.3

Production metrics from one of the busiest countries.

OpenSearch 3.x introduced significant performance improvements for vector search workloads. Post-upgrade to OpenSearch 3.3, we observed a ~18% reduction in p95 latency for k-NN queries.

For Delivery Hero’s use case, the k-NN search latency was already very low on OpenSearch 2.17 (p99 of 20–30 ms), which meant the upgrade to 3.3 was not strictly necessary for all clusters. The cluster serving the control group in A/B tests still runs on OpenSearch 2.17.

Shard routing

To minimize cross-shard overhead during k-NN queries, Delivery Hero implemented custom shard routing based on geographic market. Because each market (for example, Germany, Sweden, and Finland) has its own product catalog, routing queries to market-specific shards avoids unnecessary fan-out across the entire index.

This is an example of how to configure routing at index time and search time using the _routing field:

PUT product-embeddings/_doc/1?routing=FP_DE
{
  "master_product_code": "abc123",
  "embedding_variant_1": [0.12, 0.45, 0.78, ...],
  "embedding_variant_2": [0.21, 0.4, 0.98, ...],
  "embedding_variant_3": [0.13, 0.65, 0.58, ...],
  "global_entity_id": "FP_DE"
}

And at query time:

GET product-embeddings/_search?routing=FP_DE
{
  "query": {
    "knn": {
      "embedding_variant_2": {
        "vector": [0.12, 0.45, 0.78, ...],
        "min_score": 0.72
      }
    }
  }
}

This ensures that a query for the German market only hits shards containing German products, reducing latency and compute overhead.

Refresh interval

Because the product embedding index is updated only once per day via the OSIS batch pipeline, there is no need for the default 1-second refresh interval. Delivery Hero configured the index with a longer refresh interval during ingestion and triggers a manual refresh + force merge after the nightly batch completes.

Impact on the business

The migration from self-managed Lucene on Kubernetes to Amazon OpenSearch Service achieved a ~50% reduction in p95 latency, dropping response times from a variable 200ms+ to a stable 100ms baseline. This transition significantly improved system consistency by eliminating the high variance and rhythmic latency spikes seen in the previous architecture.

End service latency after rolling out semantic search with OpenSearch for foodpanda and yemeksepeti.

Beyond raw latency, the operational benefits were significant:

• Reduced infrastructure complexity: Eliminating the standalone Lucene service removed an entire deployment pipeline, monitoring stack, and on-call rotation.
• Faster experimentation: New embedding models can be tested by creating a new index and adjusting query routing, without requiring code deployments.
• Cost efficiency: Using OpenSearch’s managed infrastructure and the batch ingestion pattern (refresh once per day) reduced compute costs compared to running always-on Kubernetes pods with in-memory indices.

Conclusion

By combining radial search with lexical retrieval, Delivery Hero’s team built a system that adapts dynamically to query intent. It returns precise results for specific queries and broader candidate sets for general ones.

The migration to Amazon OpenSearch Service demonstrates how a managed search platform can simplify the operational complexity of vector search while improving performance.

To get started with vector search on Amazon OpenSearch Service, see the AI search documentation and the OpenSearch radial search guide.

About the authors