Post Syndicated from Jon Evans original https://aws.amazon.com/blogs/big-data/how-a-team-at-epic-games-tuned-amazon-opensearch-service-for-fortnite-analytics/

Since the launch of Fortnite in 2017, Epic Games has reached hundreds of millions of players worldwide. Fortnite runs on Amazon Web Services (AWS), and takes advantage of services such as Amazon OpenSearch Service to power certain internal analytics and drive decision making at scale.

Amazon OpenSearch Service has been helpful in understanding the game ecosystem. OpenSearch Service powers two types of use cases: search workloads and analytics workloads. A team at Epic Games had a use case for storing and analyzing a sliding window of game event data. This involves supporting complex queries and multilayered aggregations that feed analytical results into other internal systems, helping them power an evolving player experience. At the scale of a game like Fortnite with a large player base, these queries run against a significant volume of incoming data.

These insights help identify emerging gameplay trends, understand how players engage with new content, and reveal more about the Fortnite ecosystem. They inform live operation decisions and help surface relevant content to players based on aggregated activity across the community.

As Epic Games’ infrastructure handles billions of telemetry events, the team identified opportunities to optimize their OpenSearch Service cluster for better performance and cost efficiency. This post details how Epic Games partnered with AWS to transform their OpenSearch Service deployment, achieving significant improvements in query latency and resource utilization while reducing operational costs.

The challenge

Epic Games runs an OpenSearch Service domain that handles continuous high-volume writes alongside CPU-intensive batch aggregation jobs. Ideally, these aggregation jobs would run more frequently to keep analytics fresh. Shorter job intervals mean fresher data for identifying gameplay trends, detecting anomalies, and informing live operations decisions. But the existing configuration couldn’t support this without scaling the domain beyond what the workload justified, driving up costs. Epic Games worked with AWS to identify where improvements could be made, focusing on areas such as hardware utilization, sharding strategy, index mappings, and query behavior.

Observations

The cluster was running on r7g memory-optimized data nodes, with 48 vCPUs and 384 GiB of memory per node. Of each node’s available memory, only a fraction (32 GiB) was allocated to Java Virtual Machine (JVM) heap, set at the maximum recommended for compressed oops. The remainder (off-heap memory) was used for the filesystem cache and the operating system. System memory was not fully utilized across the data nodes (Figure 1).

Figure 1: System memory utilization across data nodes

As shown in the preceding figure, utilization stays well below 100% throughout the observation period, confirming that much of the off-heap memory allocated to these nodes goes unused. The excess capacity could be safely exchanged for additional compute resources.

JVM memory pressure is shown in Figure 2, and the correlating garbage collection metrics (both count and time) are shown in Figure 3.

Figure 2: JVM memory pressure

Figure 3: JVM garbage collection metrics, count (top) and time (bottom)

These charts show that JVM memory pressure remains below critical thresholds, and both garbage collection count and time are low and stable, indicating healthy JVM utilization across the domain.

While cluster-level CPU metrics appeared healthy at first glance (Figure 4), zooming into node-level metrics revealed clear node hotspots. The root cause of the node hotspots was the cluster’s sharding strategy.

Figure 4: Cluster-level CPU utilization

The cluster had data nodes distributed across multiple Availability Zones. Each index used a set number of primary shards with replicas, rolling over after shards reached a certain size. At first glance, the configuration appeared well-balanced, with shard copies distributed across Availability Zones and each node holding a manageable share of the data.

However, the primary shard count was lower than the total data node count. This meant that searches targeting the latest data, which is the most common access pattern, would only execute across a subset of available nodes. As a result, some nodes developed consistent CPU-based hotspots while the rest remained underutilized (Figure 5).

Figure 5: Node-level CPU utilization showing hotspots

As shown in the preceding figure, some nodes reach as high as 90 percent CPU utilization while several others remain under 20 percent, highlighting the uneven distribution of query execution across the cluster.

Recommendations and implementation

Based on these observations, AWS worked together with Epic Games on a set of targeted optimizations spanning hardware selection, sharding strategy, index mappings, and query behavior. The following sections detail each recommendation and how it was implemented.

Right-sizing the cluster

Because aggregation queries are CPU-intensive by nature and the cluster’s JVM memory pressure was well within acceptable ranges, AWS recommended migrating from memory-optimized r7g instances to compute-optimized c7g instances. The c7g family offers a higher ratio of vCPU to RAM, which is better suited for workloads where processing power rather than memory capacity is the binding constraint.

The proposed architecture called for a larger number of c7g nodes than the existing r7g count. This migration achieved approximately 33 percent more aggregate CPU capacity across the cluster while operating with two-thirds of the original memory. The net effect was a meaningful cost reduction of approximately 10 percent, delivering more processing power at lower cost by aligning the instance profile with the actual nature of the workload (Table 1).

Table 1: Instance migration comparison, r7g compared to c7g

Sharding strategy

To support the new cluster sizing, the Epic Games team changed the sharding strategy so that the number of primary shards matches the data node count, with 1 replica. This distributes both the write-heavy load and the batch aggregation search query load evenly on all the available data nodes.

The team employed ISM (Index State Management) policies to manage shard sizing through rollover, targeting shard sizes within recommended bounds using min_primary_shard_size. This kept shard counts bounded and predictable, providing a clear scaling pattern: adjust the node count, then update the ISM policy accordingly.

After implementation, node-level CPU utilization showed a much more even distribution (Figure 6).

Figure 6: Node-level CPU utilization after sharding optimization

As shown in Figure 6, all nodes in the domain are working at similar CPU utilization levels, confirming that data and traffic are well distributed across the cluster with no node hotspots.

Mapping optimization

The index mappings had both text and keyword field types enabled on many fields, even though access patterns showed those fields were only used for aggregation, sorting, or filter context, and never for full-text match queries. Removing the redundant text field type reduced storage overhead and improved query performance by eliminating unnecessary analysis at index time.

For high-cardinality string fields, the murmur3 field type does a compute-once-and-store optimization for cardinality aggregation. Instead of hashing keyword values at query time, murmur3 computes the hash once at index time and stores it as a numeric doc_value, so the aggregation can skip the expensive string hashing step at query time (the cardinality estimate itself is still computed at query time).

The following example illustrates the mapping changes:

"some_field": {
  "type": "text",
  "fields": {
    "keyword": {
      "ignore_above": 256,
      "type": "keyword"
    }
  }
},
"another_field": {
  "type": "text",
  "fields": {
    "keyword": {
      "ignore_above": 256,
      "type": "keyword"
    }
  }
},
"cardinality_field": {
  "type": "text",
  "fields": {
    "keyword": {
      "ignore_above": 256,
      "type": "keyword"
    }
  }
},

"some_field": {
  "type": "keyword"
},
"another_field": {
  "type": "keyword"
},
"cardinality_field": {
  "type": "keyword",
  "fields": {
    "hash": {
      "type": "murmur3"
    }
  }
},

These mapping changes reduced overall storage, lowered shard count (which reduced CPU requirements), and reduced cluster manager node state size.

Index optimization

Additional index-level optimizations were applied to improve query performance and reduce overhead. Index sorting was configured to default to the primary date field, which improves performance for time-based access patterns by aligning the physical data layout with the most common query order. The ISM policy was updated to force merge indices down to 1 segment after rollover, reducing segment overhead on read-only indices. Finally, the refresh interval was tuned to balance indexing throughput with search freshness.

Upgrading from OpenSearch Service 2.17 to 3.1

The domain was upgraded from OpenSearch Service 2.17 to 3.1, which reduced error counts and improved throughput at the Amazon OpenSearch Ingestion pipeline level. The performance gains were notable: p99 latency on sum aggregations dropped by 40–50 percent after the upgrade alone, and large 96-hour cardinality aggregations saw p95 drop over 40 percent. General query performance improved across all query types, and thread pool pressure was reduced significantly, leading to far fewer 429 errors (Figure 7).

Figure 7: Query performance before and after the OpenSearch Service 3.1 upgrade

Upgrading from Graviton 3 to Graviton 4

The instance types were upgraded from c7g (Graviton 3) to c8g (Graviton 4). The performance gains were immediate:

• p99 on all queries: 380 ms to 250 ms.
• p95 on all queries: 245 ms to 230 ms.
• p90 on all queries: 225 ms to 200 ms.
• p50 on all queries: 100 ms to 70 ms.

Date-windowed cardinality queries saw their p99 halved from 220 ms to 98 ms, with sum-based aggregations experiencing similar gains. Overall throughput increased by 16 percent.

Tiered caching

With the upgrade to OpenSearch Service 3.1, the team enabled tiered caching. Tiered caching extends the default on-heap request cache with a disk-based tier. When items are evicted from the on-heap cache, they spill into a larger disk cache on the node’s local SSD rather than being discarded. This allows the cluster to retain cached results for a much larger set of queries without increasing JVM heap usage.

The batch aggregation jobs in Epic Games’ workload issue repeated queries over overlapping time windows. The on-heap cache alone was too small to retain results across successive job runs, so expensive aggregations were recomputed each time. With the disk tier enabled, results from longer time-window aggregations (such as the 96-hour cardinality queries) persisted between runs. This produced more consistent and faster results on some of the larger aggregation queries, particularly those spanning longer time windows.

Results summary

The following table summarizes the impact of each optimization.

Table 2: Results summary

Conclusion

By optimizing their OpenSearch Service deployment, a team at Epic Games reduced p99 query latency from 380 ms to 250 ms, increased throughput by 16 percent, and lowered costs by 10 percent. These gains came from aligning instance types, sharding strategy, mappings, and engine versions with the workload’s actual demands.

To learn more about optimizing Amazon OpenSearch Service for your workloads, see Best practices for Amazon OpenSearch Service. For details on supported instance types, see Supported instance types in Amazon OpenSearch Service.

About the authors