46% fewer nodes, 2× faster indexing: Confluent on Amazon OpenSearch Service

Post Syndicated from Sukanth Rajan original https://aws.amazon.com/blogs/big-data/46-fewer-nodes-2x-faster-indexing-confluent-on-amazon-opensearch-service/

Operating one of the world’s largest data-streaming platforms, Confluent, the company founded by the original creators of Apache Kafka and now part of IBM, continuously modernizes its infrastructure to stay ahead of rapid data growth. Confluent’s data-streaming products, including its Apache Kafka, Connect, and Flink fleets, are available on the AWS Cloud, and its internal logging system runs on Amazon OpenSearch Service, ingesting hundreds of terabytes of logs per day to support operations across that fleet.

At that ingest volume, indexing work dominates the hot tier. The question Confluent’s engineering team set out to answer was not “can we run on cheaper hardware,” but “how much more ingest can we absorb per node, and how much headroom does that free up?” This post shows how a measure-first migration of the OpenSearch hot tier to the purpose-built OpenSearch optimized instance type, oi2, improved indexing performance and ingestion headroom first. oi2 costs more per node, but we needed far fewer of them. The lower cost came from that efficiency, not from a lower instance price.

Why instance choice matters for ingestion

Confluent’s hot tier ran on r6gd, a previous-generation, general-purpose Graviton2 instance with local NVMe added as a bolt-on. The team migrated to the current-generation OpenSearch Optimized Instance family (oi2), purpose-built for OpenSearch. On oi2, local NVMe is the primary storage tier with no separate Amazon Elastic Block Store (Amazon EBS) volume. Data is copied synchronously to Amazon Simple Storage Service (Amazon S3) for durability, and the storage and network path is integrated specifically for indexing and merge I/O. For an ingest-bound logging workload, that storage path is the lever. It’s what lets each node sustain a higher indexing rate at lower latency.

A domain with OpenSearch Optimized Instances uses local storage as the primary tier, with data copied synchronously to Amazon S3 as it arrives. This provides increased indexing throughput with high durability (11 nines). Within the family, OR1, OR2, and OM2 use Amazon EBS volumes (gp3 or io1) locally, whereas oi2 instances use local NVMe disks. These instances benefit indexing-heavy operational analytics workloads such as log analytics, observability, and security analytics.

OpenSearch supports two replication strategies: logical (document) and physical (segment) replication. With logical replication, data is indexed on all the copies independently, leading to redundant computation on the cluster. OpenSearch Optimized Instances use the physical replication model, where data is indexed only on the primary copy. Additional copies are created by copying the segments from the primary (durably persisted to Amazon S3). Because this frees up compute resources on replicas that would otherwise redundantly index data, and network overhead on the primary, the cluster can churn more throughput. When Optimized Instances are configured with replicas, the indexing rate might appear lower than it actually is (for example, showing a rate of 1,000 when the actual indexing rate is 2,000). This is because indexing is only performed on primary shards.

Reference architecture

The following diagram shows the OpenSearch multi-domain topology, split by priority (P0/P1) and volume, behind Cross-Cluster Search.

Kubernetes clusters feed the Confluent ingestion pipeline that writes to four OpenSearch domains read through cross-cluster search

Figure 1: OpenSearch multi-domain topology, with writes going directly to the P0 and P1 domains and reads served through cross-cluster search

A measure-first approach

Each domain was modernized in two logical phases, so the efficiency of the new instance could be measured in production before capacity was removed:

  1. Phase 1, a type swap at constant node count: an in-place Amazon OpenSearch Service blue/green re-provision at the same node count, to measure real production efficiency at fixed ingest. At equal nodes, oi2 ran at roughly one-third the CPU of r6gd for the same ingest, about 2.7x the headroom per node.
  2. Phase 2, iterative rightsizing: nodes were removed in steps, holding at each step to confirm cluster health, until a guardrail became binding. That guardrail turned out to be JVM heap, not CPU, because heap pressure is driven by query patterns, and removing nodes concentrates the same shards onto fewer of them.

Spending the Phase 1 headroom this way let the fleet consolidate to roughly half the nodes. Each oi2 node carried about 1.9x the load of an r6gd node (168 ÷ 90), while indexing got faster, not slower.

Production results: Ingestion efficiency first

The following figures are production averages at the same ingest (7-day baseline on r6gd compared to a 24-hour window on oi2 at the final node counts). Because the log traffic is identical, the comparison reflects a real change per unit of work. We didn’t benchmark R8gd for this migration. Our evaluation focused on the OpenSearch optimized (oi2) family, which is built specifically for log-analytics ingestion and storage economics.

Fleet node count dropping from 168 on r6gd to 90 on oi2, a 46% reduction

Figure 2: Total hot-tier fleet size, 168 r6gd nodes compared to 90 oi2 nodes, a 46% reduction

Node count per domain falling from r6gd to oi2: Domain A 30 to 12, B 36 to 18, C 39 to 27, D 63 to 33

Figure 3: Node count per domain on r6gd compared to oi2, with reductions of 31 to 60 percent

Indexing latency per domain roughly halving from r6gd at 0.18 to 0.19 ms to oi2 at 0.08 to 0.10 ms

Figure 4: Average indexing latency per domain on r6gd compared to oi2, roughly halved across all four domains

Write queue depth per domain dropping from double digits on r6gd to near zero on oi2

Figure 5: Write queue depth in queued operations per domain on r6gd compared to oi2, cleared to near zero

Domain (anonymized) Nodes Avg CPU Indexing latency Write queue (ops)
Domain A (high-volume) 30 → 12 (-60%) 49% → 43% 0.19 → 0.08 ms 17.6 → 0.2
Domain B (streaming logs) 36 → 18 (-50%) 60% → 40% 0.18 → 0.08 ms 20.6 → 0.3
Domain C (high-volume) 39 → 27 (-31%) 58% → 39% 0.19 → 0.09 ms 37.9 → 0.2
Domain D (low-volume) 63 → 33 (-48%) 43% → 56% 0.18 → 0.10 ms 8.4 → 1.0
Fleet 168 → 90 (-46%) — ~halved <1 everywhere
  • Indexing latency roughly halved everywhere: approximately 0.18–0.19 ms on r6gd to approximately 0.08–0.10 ms on oi2, reflecting oi2’s faster storage path.
  • Write backpressure cleared. Every r6gd domain carried a double-digit write queue (8–38 queued ops, the indexing-backpressure signal). On oi2, all four dropped below approximately 1. Faster storage keeps indexing current.
  • CPU fell even on far fewer nodes for three of four domains, the clearest sign of per-node efficiency (for example, Domain B: 60% → 40% on half the nodes).
  • Heap, not CPU, is the ceiling. oi2 has the same heap per node as r6gd, so concentrating shards onto fewer nodes raised average heap on every domain. This is what set the floor on rightsizing.
  • The honest outlier. Domain D was cut the hardest relative to its load. Its CPU rose (43% → 56%) and its write queue, although far better, is the only one not near zero. It’s the domain to watch through peak traffic before calling it settled.
  • Stability: zero write/primary-write rejections in the final state. All clusters healthy throughout.

In AWS internal benchmarks, oi2 instances deliver up to 9 percent higher indexing throughput than OR2 instances and up to 33 percent more than I8g instances. Each OpenSearch Optimized instance is provisioned with compute, third-generation AWS Nitro SSDs for caching, and remote managed storage on Amazon S3. The oi2 family is available in sizes from large through 24xlarge with up to 22.5 TB of storage.

To maximize indexing throughput from OpenSearch Optimized Instances, AWS recommends:

  • Use large bulk sizes to improve buffer utilization. The recommended size is 10 MB.
  • Use multiple clients to improve parallel processing performance.
  • Consider more primary shards to raise indexing throughput, but note that over-sharding every index drives up per-node shard density and lowers it.
  • Keep the index refresh interval at 10 seconds or higher (the default for Optimized Instances is 10 seconds).

Fewer nodes, lower bill — even at a higher per-node price

oi2 carries a higher per-node price than r6gd, so a like-for-like swap alone would have raised spend. The saving came entirely from needing fewer nodes: consolidating the hot tier by approximately 46 percent more than offset the premium, for an estimated approximately 18 percent lower hot-tier cost at on-demand pricing. Committing the final footprint to 1-year reserved instances is estimated to take the hot tier to roughly 47–51 percent below the original r6gd on-demand baseline.

Cost figures are approximate, list-derived percentages at on-demand pricing (reserved-instance figures are estimates pending a quoted rate). Storage (UltraWarm and snapshots) and dedicated-master tiers were unchanged and are excluded. The takeaway for teams modernizing OpenSearch: optimize efficiency per unit of work, not instance unit price.

Conclusion

With the measure-first approach in two phases, Confluent raised ingestion efficiency and halved indexing latency on its Amazon OpenSearch Service cluster. Before removing any capacity, let the real sizing signal (heap in this case) guide the optimization work. This approach is what improved the efficiency of the system and lowered the overall cost. The same measure-first approach applies to ingest-bound OpenSearch workloads evaluating purpose-built storage-optimized instances. To learn more, see the Amazon OpenSearch Service documentation on instance types and sizing domains.

 

About the authors

Sukanth Rajan

Sukanth Rajan

Sukanth is a Technical Account Manager in Strategic Accounts at Amazon Web Services. He helps AWS’s largest customers architect, migrate, and optimize their mission-critical workloads on the cloud. He leverages his deep technical expertise, especially in databases, to help customers solve critical issues and build highly reliable, secure, and cost-effective solutions using AWS services.

Aravind Karthik

Aravind Karthik

Aravind is a software engineer on the Cloud Infrastructure & Platform team at Confluent. He helps build highly scalable observability systems that process terabytes of data every day. He is currently working on regionalizing Confluent’s metrics pipelines and optimizing the logging pipeline to be lean and efficient for the next wave of telemetry volume growth.