How Klaviyo processes 3 billion daily events at scale with Amazon MSK Express

Post Syndicated from Jonathan Li original https://aws.amazon.com/blogs/big-data/how-klaviyo-processes-3-billion-daily-events-at-scale-with-amazon-msk-express/

Klaviyo (NYSE: KVYO) is an autonomous business-to-consumer (B2C) customer relationship management (CRM) platform used by more than 200,000 brands to understand and act on customer relationships in real time. Every personalized email, SMS, and push notification that Klaviyo sends depends on a streaming pipeline that ingests roughly 3 billion events per day. These events range from email opens and website visits to purchase completions and cart abandonments.

In this post, we share how Klaviyo migrated from RabbitMQ to Amazon Managed Streaming for Apache Kafka (Amazon MSK) Express brokers. We also show how Amazon MSK Express let a small team scale its event pipeline ahead of Black Friday/Cyber Monday (BFCM) 2025. With Amazon MSK Express, Klaviyo delivered more than 22.7 billion messages, up 25 percent year over year, and scaled back down afterward over a five-day period.

The challenge

Events are the lifeblood of Klaviyo’s platform. Every customer interaction flows through the streaming pipeline. Klaviyo’s previous RabbitMQ-based event infrastructure supported the platform through years of rapid growth. As volume climbed, scaling it beyond a certain point became increasingly difficult, so the team set out to build a foundation for the next stage of scale, with three requirements:

  • Absorb extreme bursts – BFCM drives roughly 7 times normal event volume, concentrated into five days.
  • Serve two service level objective (SLO) classes on one pipeline – Real-time triggers (5-second SLO) and analytics aggregates (20-minute SLO) share infrastructure, so slower work can never hold up time-critical events.
  • Zero data loss – Every event is a customer interaction that drives personalized marketing, so nothing can be silently dropped.

The replacement had to sustain peaks of 170,000 events per second, absorb a 7 times spike, and scale back down afterward.

Solution overview

Klaviyo rebuilt its event gateway infrastructure on Apache Kafka, choosing Amazon MSK Express brokers as its fully managed streaming backbone serving as the front door for all inbound events.

Klaviyo event pipeline on Amazon MSK Express, from event sources through the event gateway and orchestrator to consumers and delivery channels

Figure 1: Klaviyo’s event gateway architecture on Amazon MSK Express, showing the gRPC publisher, p10/p20/p30 priority topics, the Kubernetes orchestrator, retry topics and dead-letter queue, Amazon S3 failure capture, and a second gateway cluster in eu-central-1

The solution includes the following key components:

  • Event gateway cluster – The entry point for all inbound events (email opens, clicks, site activity, API calls), running on express.m7g.4xlarge brokers in the US, with a second cluster in the EU.
  • Three-tier priority topics – Events are routed to priority topics (p10, p20, p30) with 360 partitions each. The partition count is set by the consumer topology rather than the broker count: the event orchestrator runs as StatefulSets with one pod per partition, which comes to 90 pods in each of four Kubernetes clusters and 360 pods in total.
  • Graduated retry architecture – Five retry delay levels with exponential backoff, plus a dead-letter queue, so no event is silently dropped.

Priority-based event processing

Rather than treating all events equally, Klaviyo routes events to priority tiers through a gRPC publisher proxy. This design degrades gracefully under extreme load. High-priority events (such as purchase confirmations that trigger transactional flows) are processed first, while lower-priority events (such as batch analytics updates) can tolerate higher latency.

Critically, Klaviyo runs one unified set of processors across all three priority levels instead of standing up separate infrastructure per level. That consolidation cut costs by roughly 30 percent through improved resource utilization. For the design in depth, see Building a Distributed Priority Queue in Kafka.

Resilient event publishing

Klaviyo designed its event publisher to survive failure at every layer. A publish that fails against one Kafka cluster has several fallback mechanisms to make sure that there will be no permanent data loss. Most failures recover automatically, and the rest are preserved for replay. Over one three-month window, roughly 20,000 events were recovered without human intervention, out of approximately 300 billion processed in that period.

Klaviyo also retains seven days of event history for replay in the case of downstream incidents.

For more on Klaviyo’s eventing design, see Rebuilding Event Infrastructure at Scale and Building a Resilient Event Publisher with Dual Failure Capture.

Scaling for Black Friday with MSK Express

The clearest demonstration of this architecture came during BFCM 2025, the largest ecommerce event of the year, when the pipeline carried the traffic behind Klaviyo customers’ most revenue-critical campaigns across five consecutive days.

The scale-out timeline

Period Brokers Context
Steady state (through September 2025) 12 Normal daily processing of ~3B events
Early October 2025 18 First capacity increase ahead of the holiday season
Late October–November 2025 (BFCM) 60 Full Black Friday / Cyber Monday capacity
December 2025 (post-BFCM) 18 Optimized steady-state fleet

The team scaled the event gateway cluster up several-fold over about seven weeks, held it through Cyber Monday, then scaled it back down in December.

Why MSK Express made this possible

Three capabilities of MSK Express were critical to making this operational with a lean team.

No storage provisioning – Express brokers remove storage sizing and provisioning entirely. You pay for the data ingested into the cluster and the data stored in it, per GB per hour, and all stored data is replicated across three Availability Zones for high availability. This pay-as-you-go model eliminates wasted capacity costs. For Klaviyo, it meant a large holiday scale-out required no storage capacity planning at all.

Intelligent Rebalancing – Adding brokers to an Apache Kafka cluster normally means redistributing partition data across the new brokers, which traditionally requires careful manual planning, monitoring, and throttle management. MSK Express Intelligent Rebalancing automates that redistribution without manual intervention or performance degradation, and AWS reports that Express brokers rebalance partitions 180 times faster than Standard brokers. For Klaviyo, this meant adding brokers at scale for BFCM without an engineer owning partition reassignment at scale.

Built-in high availability – Based on AWS internal testing, Express brokers recover from failure 90 percent faster than Standard brokers, thanks to storage improvements. A replacement node starts up with almost no disruption to the rest of the cluster, rather than triggering a significant rebalance. Express brokers are also highly available by default, with the service managing the critical cluster and topic configurations that would otherwise need hand-tuning.

Results

Since migrating to Amazon MSK Express, Klaviyo has achieved the following results:

  • Multi-fold broker scale-out in under two months to absorb BFCM peak traffic, with a clean scale-down afterward.
  • Approximately 30 percent cost reduction from running unified processors across all three priority levels instead of separate infrastructure per level.
  • 5-second SLO held for real-time event triggers at a peak of 170,000 events per second.
  • 22.7 billion messages delivered over BFCM 2025, up 25 percent year over year.
  • Multi-Region since May 2026, with a second event gateway cluster in eu-central-1 serving European customers.
  • MSK Express adopted across Klaviyo’s event gateway clusters in both US and EU AWS Regions.

As Klaviyo’s engineering team wrote in Rebuilding Event Infrastructure at Scale:

“Moving to managed Kafka infrastructure on AWS MSK was one of those decisions that keeps paying dividends.”

It freed up engineers to spend their time building features rather than operating messaging infrastructure.

Key learnings

Based on Klaviyo’s experience building and operating this architecture, here are key takeaways for teams considering a similar approach:

  • Design for extreme elasticity from day one – Klaviyo’s BFCM traffic drives a multi-fold surge in event volume into five days. Designing partition layouts, topic structures, and consumer groups to accommodate elastic scaling before you need it avoids painful re-architecture under pressure.
  • Size partitions for your consumers, not your brokers – Klaviyo’s 360 partitions per topic come from its consumer topology (one orchestrator pod per partition), which stays fixed while the broker fleet scales up and back down underneath it. Coupling partition count to broker count would have forced a partition change in the middle of the holiday ramp.
  • Scale down as deliberately as you scale up – Klaviyo returned to a right-sized fleet after BFCM rather than carrying peak capacity all year. Because Express bills storage per GB rather than per provisioned volume, scaling down left no stranded storage cost behind.
  • Run multiple SLO classes on one pipeline – A single set of processors serves all three priority tiers, which cut costs by roughly 30 percent. Priority routing still protects latency: real-time triggers held their 5-second SLO at a peak of 170,000 events per second.
  • Build fault tolerance into every layer – Cross-cluster failover, a dead-letter queue, and an Amazon Simple Storage Service (Amazon S3) fallback mean individual failures don’t cause data loss. Express broker recovery keeps a lost broker from turning into a cluster-wide disruption.

Conclusion

Klaviyo’s journey from RabbitMQ to Amazon MSK Express demonstrates how a managed streaming service helps high-growth companies operate mission-critical event infrastructure at massive scale without proportionally growing their operations team. By combining MSK Express Intelligent Rebalancing with thoughtful architecture design (priority-based topics, graduated retry queues, and layered failure capture), Klaviyo’s lean infrastructure team scaled 5 times during the largest ecommerce event of the year.

Klaviyo’s use of Amazon MSK has since grown beyond the event pipeline. Its change data capture (CDC) infrastructure now runs on Amazon MSK Express as well. The team is also evaluating migrating its remaining self-managed Apache Kafka clusters on Amazon Elastic Compute Cloud (Amazon EC2), which serve data lake workloads, to MSK.

If you’re operating Apache Kafka workloads that require elastic scaling, operational simplicity, or both, consider Amazon MSK Express brokers for your next streaming workload.


About the authors

Jonathan Li

Jonathan Li

Jonathan is a Lead Engineer at Klaviyo, where his Data Automation team owns most of the company’s Apache Kafka infrastructure and tooling, including its change data capture (CDC) platform also on MSK. He previously worked on the team that built Klaviyo’s event gateway on Amazon MSK Express, the system described in this post.

Jason Palmer

Jason Palmer

Jason is a Senior Technical Account Manager (TAM) at AWS Enterprise Support, based in Seattle, Washington. With over 6 years at AWS, Jason combines deep technical expertise with a genuine passion for people — helping enterprise customers transform complex challenges into scalable cloud solutions.

Mazrim Mehrtens

Mazrim Mehrtens

Mazrim is a Sr. Specialist Solutions Architect for messaging and streaming workloads. Mazrim works with customers to build and support systems that process and analyze terabytes of streaming data in real time, run enterprise Machine Learning pipelines, and create systems to share data across teams seamlessly with varying data toolsets and software stacks.