Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/timeseries-tiered-storage-journey-kafka-flink-streams-to-native-cassandra-cold-reads-e59d597c9d60

By Oleksii Tkachuk, Rajiv Shringi, Chris Lohfink, Abe Ratnofsky, Kartik Sathyanarayanan

Introduction

In an earlier post, we introduced the TimeSeries Abstraction, our high-throughput, low-latency platform designed to store and query temporal event data at scale. It has since become the operational backbone for Netflix’s largest datasets, which now span multiple petabytes.

However, managing such large datasets in hot storage, tuned for low-millisecond latency access, is not cheap. At Netflix, cost-efficiency is a continuous discipline rather than a temporary initiative. This post shares two solutions that significantly reduce storage costs by continuously moving older, less frequently accessed data into cheaper object storage:

• Solution 1: Uses Apache Kafka and Flink to stream data directly to S3
• Solution 2: Leverages Cassandra backups in S3 and pluggable Memtables to read data directly from S3

We’ll start with the Kafka/Flink-based design, then describe how we evolved to a native Cassandra-based cold tier.

Challenges

As Netflix’s TimeSeries datasets have expanded, we’ve encountered several compounding challenges:

• Growing retention demands: Users often push for increasing data retention to serve a broader range of use cases, which in turn increases the storage costs.
• Escalating costs: Maintaining massive, dense clusters for infrequently accessed historical data is prohibitively expensive.
• Sparse historical access: Most queries target recent events, but older data must remain readily available for occasional audits and investigations.
• Operational strain: High storage density on hot nodes complicates scaling, deployment, and cluster stability.
• Latency requirements: Even historical data sometimes requires sub-second query performance for internal tooling and analysis.

This creates a clear opportunity: keep the most recent data on fast storage and migrate older, rarely read data to cheaper storage while preserving a consistent API. Done well, this can meet user and operational needs while greatly reducing our infrastructure spend.

TimeSeries Abstraction

Before we dive into the cold storage solutions, here is a quick recap of how TimeSeries partitions its dataset into multiple tables, each covering a window of time we call Time Slices.

This enables several key optimizations, most notably allowing a table to become immutable once it crosses the write boundary, which lets us safely apply further optimizations. For more details, refer to our previous post.

This slice information is critical and is maintained in a dedicated metadata table:

Creating new Time Slices and maintaining their metadata is a primary responsibility of the TimeSeries control plane. The TimeSeries data plane uses this information to correctly route the read and write traffic to the provisioned tables.

With that background, let’s look at the different solutions we built for data tiering.

Solution 1: Streaming with Apache Kafka and Flink

Our first solution followed a familiar streaming pattern using Apache Kafka and Flink, where we streamed data to S3 in realtime while simultaneously persisting it to Cassandra.

Getting Data Into S3 Efficiently

To optimize data streaming to S3, the system hash-partitions Time Series IDs into buckets. This partitions the stream and lets dedicated Flink tasks handle each bucket, ultimately batching and reducing the write throughput to S3. Flink workers batch events into Parquet files grouped by event time within the hour. These files are great for high-throughput streaming writes, but not for low-latency reads.

Enter the Compaction job.

Compaction Job to Optimize Reads

The compaction job runs daily and merges small Parquet files into a single, optimized daily Parquet file per namespace.

The job further optimizes the Parquet files in the following ways:

• Small Row Groups: Creates small (4–8 MB) row groups to minimize lookup cost. We can then use the Parquet Java library to perform range reads of row groups from S3. This is intentionally well below Parquet’s recommended row group floor — smaller groups were better because our reads are dominated by point/range lookups rather than full scans.
• Event Sorting: Sorts events within a row group by time, eliminating brute-force scans when querying for a time range.
• Bloom Filters: Caches Bloom filters and file metadata in EVCache, allowing the TimeSeries query planner to efficiently skip unnecessary reads from S3.

Once the compaction job completes, all compacted files per partition are recorded in the TimeSeries control plane, and the hot storage slices can now be truncated!

On some of our largest datasets, this job typically takes about 12 hours, prioritizing lower compute costs over execution speed.

Streaming Cold Tier Read

Once compacted files are optimized in cold storage, they are ready to be read. The TimeSeries data plane fans out a given query across both the hot and cold tiers and merges their result. The users still use the same query interface:

The Cold Tier Data Plane uses compaction metadata for aggressive planning and pruning, which dramatically reduces the S3 query fan-out. As a result, retrieving a full day of data requires at most three sequential S3 reads, even in the uncommon case where an ID spans two row groups. This yields a p50 latency of ~150 ms and a p99 latency of ~500 ms per read.

At the time of writing, the streaming cold tier implementation was processing up to 1,250 reads per second from cold storage globally:

Why not stop here?

The streaming cold tier has delivered real savings, shrinking both Cassandra’s EBS footprint and the Cassandra backup footprint in S3. But the streaming infrastructure itself is expensive and adds moving pieces that widen the team’s on-call scope:

We wanted to improve this further and asked ourselves:

• Can we avoid streaming data into the object store and the associated data transfer costs?
• Can we reduce the compute and operational overhead of new components?

That led to a more radical question: Can we read directly from existing Cassandra backups in S3?

Solution 2: Cassandra Cold Storage for Immutable Data

At Netflix, we back up Cassandra data into S3 at regular intervals for disaster recovery. The idea for tiering is simple:

Read directly from these S3 backups instead of paying a separate data transfer cost to S3.

Once TimeSeries marks a hot Time Slice as ready for cold storage, we can customize Cassandra to read directly from its backups, allowing us to delete the on-disk data copy. This would also incur minimal changes to the TimeSeries read path!

To make this viable in production, we had to answer a series of “How…?” questions.

How Can We Read the Backups Efficiently?

We introduced a new compaction strategy in Cassandra, which we call ReadOnlyCompactionStrategy (ROCS). Here is how it works:

Once a time slice crosses the user-configured cold-tier boundary, TimeSeries stops accepting writes for that time slice making it immutable. This immutability unlocks significant optimization opportunities:

• SSTable overlap: Standard strategies (LCS, TWCS) often produce SSTables that overlap in the token space, and so a single partition key may exist across multiple files.
• Read fan‑out: Because a read must touch every SSTable covering the key’s token range, this overlap turns a single point-read into multiple lookups, increasing latency across all tiers.

ROCS re-compacts all SSTables once, aligning them precisely with token ranges:

ROCS applies more broadly to all TimeSeries use cases, driving tail latency down on all storage tiers by reducing the SSTable fan-out during read operations:

Once the immutable Time Slices have been optimized, it is time to trigger backups.

How Would We Leverage Backups?

Cassandra backs up every node in the cluster, organizing each backup on S3 as a declarative manifest, along with the SSTables backing the keyspace’s tables:

 {bucket}/
  └── {prefix}/                             ← cluster identity (env_backup/hash_app)
      └── {token}/                          ← node identity (ring token)
          ├── META_V2/                      ← manifests, one per snapshot
          │   ├── {snapshotInstantMs}.json
          │   ├── {snapshotInstantMs}.json
          │   └── ...
          └── SST_V2/                       ← actual SSTable files
              └── {backupTimestamp}/        ← snapshot timestamp
                  └── {keyspace}/
                      └── {table-with-uuid}/ 
                          └── {compression}/
                              └── {encryption}/
                                  ├── {component-file}
                                  └── ...

The backup manifest orchestrates the backup, listing the paths to all its SSTables:

manifest.json
  ├── info
  │   ├── appName              "cass-cluster"
  │   ├── region               "us-east-1"
  │   ├── snapshotInstantMs    1748908800000
  │   ├── backupPathPrefix     "prod_backup/1234_cass/token"
  └── data[]
      └── [0]
          ├── keyspaceName     "ks"
          ├── columnfamily     "tbl-a1b2..."
          └── sstables[]
              └── [0]
                  └── components[]
                      ├── na-1-big-Data.db
                      ├── na-1-big-Index.db
                      ├── na-1-big-Filter.db
                      ├── na-1-big-Statistics.db
                      ├── na-1-big-Summary.db
                      ├── na-1-big-CompressionInfo.db

Alongside the large Data.db file, backups include a compact set of index, metadata, and integrity files. This structure allows us to quickly locate, validate, and restore specific partitions or data ranges without scanning the entire dataset. Furthermore, since the manifest catalogs every S3 path, we can download most of these components directly to the Cassandra node (excluding the Data.db file) where they occupy negligible space. These local components enable efficient query planning before any data is read from S3.

How Would Cassandra Read From Backups?

Cassandra reads typically pull data from two sources: the Memtable, which holds recent unflushed writes, and flushed SSTables on durable storage. As of Cassandra 4.1, we can leverage pluggable Memtables to redirect reads from “cold” tables directly to the backup storage in S3.

-- create table for a given Time Slice
CREATE TABLE data_20260616_00_04 (
    ...
    ...,
    PRIMARY KEY ((time_series_id, time_bucket, event_bucket), event_time, event_id, key)
) WITH CLUSTERING ORDER BY (event_time DESC, event_id DESC, key DESC)
    ...
    AND compaction = ...
    AND compression = ...
    AND memtable = 'backupmemtable:timestamp=1781953200000' ← Custom memtable
    ...

The timestamp marks the point at which the backup is guaranteed to cover the full table, and the table is guaranteed to be immutable.

Opening a backup Memtable kicks off a state-machine initialization that pulls the index data from S3 onto the node once:

This process ensures all SSTable components are downloaded reliably through several resilience mechanisms:

• Retries and backoff: All S3 operations and file-system cache writes use retries with exponential backoff.
• Resource isolation: Initialization runs on a dedicated thread pool, isolated from normal Cassandra operations.
• Partial download protection: If a downloaded component’s size doesn’t match the manifest, it is deleted and re-downloaded.
• DDL change protection: A .timestamp file marks completed initialization and ties it to the backup timestamp. If the table’s backup timestamp changes, this file is removed and initialization restarts for the new timestamp.
• Read protection: Memtable initialization is asynchronous; reads to an initializing memtable block for at most half the read RPC timeout.

Once initialization has pulled every SSTable component (except Data.db) from S3 onto the local disk, the Memtable is ready to serve reads. Here is what the read process looks like:

To minimize latency, the system parallelizes S3 reads across requests, SSTables, and individual chunks. A shared cache deduplicates fetches and prefetches data, so chunks are ready before they are needed.

We further optimize cluster and table settings to reduce the number of S3 calls:

• Larger compression chunks: Reduce S3 calls per partition by reading fewer, larger chunks (accepting some over-fetch on small reads).
• Read repair disabled: Avoid expensive, unnecessary repair traffic against immutable, pre‑repaired backup data.
• Speculative retries disabled: Prevent S3’s higher latency from triggering redundant, overlapping reads.

How Would We Know When It Is Safe To Read From Backups?

However, before the Memtable can point to a backup, we must confirm two things across the whole ring, and we use Cassandra Virtual Tables to do that.

First, that ROCS re-compaction succeeded:

cqlsh> SELECT * FROM netflix_views.cluster_view                                                                                                                                                          
       WHERE keyspace_name = 'system_views' AND table_name = 'sstable_tasks';

keyspace_name | table_name    | host     | value                                                                                                                                                        
--------------+---------------+----------+------------------------------------------------
system_views  | sstable_tasks | 10.0.0.1 | [{"table_name":"<.>","kind":"compaction", ...}]
system_views  | sstable_tasks | 10.0.0.2 | [] <---- no active compactions
system_views  | sstable_tasks | 10.0.0.3 | []

And second, that backups were fully captured (denoted by uploaded=True):

cqlsh> SELECT * FROM netflix_views.complete_backups                                                                                                                                                      
       WHERE keyspace_name = 'my_ks' AND table_name = 'my_table';

keyspace_name | table_name | timestamp     | num_tokens | num_uploaded | uploaded                                                                                                                       
--------------+------------+---------------+------------+--------------+----------
my_ks         | my_table   | 1765211400000 |          3 |            3 |     True
my_ks         | my_table   | 1765213200000 |          3 |            2 |    False

How Would We Automate This Entire Flow?

The TimeSeries control plane automates the entire transition as slices cross the immutable boundary:

Each slice can be in any lifecycle stage at any time without impacting system availability, and the system remains resilient to delayed backups or lagging compactions.

How Would Cassandra Handle The Additional Load?

Serving data from S3 is much more CPU-intensive than reading from local storage. A sudden spike in read requests can overwhelm the Cassandra cluster and affect the hot path, turning a cost-saving feature into a reliability risk.

To avoid this, cold reads can optionally run on a separate Cassandra cluster that points its backup Memtables at the hot cluster’s backups:

The cold Cassandra cluster must match the original cluster in topology but need not match it in hardware! We can run the cold cluster on much smaller, cheaper, storage-light instances.

Real World Performance

Cassandra Cold Tier was rolled out to the most storage‑intensive TimeSeries Abstraction clusters. It currently stores over 15 PB of compressed data in Cassandra cold storage and has allowed us to cut attached storage significantly on all affected clusters:

At the same time, it serves thousands of reads per second from cold tier storage in any given region:

Cassandra Cold Tier improves p90 latency by ~30% over the previous cold storage implementation, thanks to more efficient S3 access patterns and smaller read sizes, while keeping p99 latency under ~500 ms. Cassandra already handles bursts of cold reads gracefully via its chunk cache, so this design preserves that same resilience for the S3-backed cold tier:

Acknowledgements: Special thanks to our stunning colleagues who contributed to the TimeSeries Abstraction’s success and collaborated with us on the cold storage solutions: Kaidan Fullerton, Matt Lehman, Sumanth Pasupuleti, Carlos Matias Herrero, Nick Pavlakis, Bryan Keller, Francesco Di Chiara

TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/behind-the-scenes-evolving-netflixs-ads-event-pipeline-for-live-part-ii-826ebf9ad9fb

By Yogesh Nagarur

Introduction

In Part 1 of this series, we shared how Netflix built the Ads Event pipeline and approached it as a system design problem: replacing fragmented pipelines with a centralized collection, enrichment, and a standard data contract. The result was Ads Event Publisher, the system that collects ad telemetry from client devices, enriches it with the context of the ad that was actually served, and publishes a single, unified stream of ad events to every downstream consumer: frequency capping, metrics, billing, measurement partners, and reporting. Any member viewing or account details that move through this pipeline end-to-end are encrypted and handled in accordance with Netflix’s strict privacy and security practices to safeguard member PII.

If ad serving is the “brain” and ad events are the “heartbeats” that feed it, then enrichment is what gives each heartbeat its meaning, turning a bare “[Somebody] watched an ad to completion” signal into “this campaign, this creative, at this price, for this break.” Part 1 described that enrichment step as a single box in the architecture. This post is about what was inside that box, and why we rebuilt it.

Part 1 ended with a look at what was on the horizon: ads on live streams, de-duplication, and richer data signals. A single project delivered all three by rebuilding the Ads Event Publisher around stateful stream processing. We replaced a database lookup with a real-time streaming join in Apache Flink, which let the system scale for Live, took a dependency off the critical path, and replaced the engine of a high-traffic pipeline without downtime.

Let’s dive in.

A quick recap: enrichment and the Metadata Registry

When an ad is served, the ad server knows everything about it: the campaign, the creative, the price of the impression, the targeting context, and the tracking URLs that measurement partners expect us to fire. The client device, by contrast, knows almost nothing: just an opaque token and when to send each event. Enrichment is the act of reuniting the two.

In the architecture from Part 1, that reunion happened through the Ads Metadata Registry, a Key-Value store:

• When the ad server made a decision, it wrote the metadata for that impression into the registry.
• It encoded only reference identifiers into the opaque token returned to the device, keeping the token small.
• Later, when the client fired an event carrying that token, the Ads Event Publisher decoded the token, looked up the metadata in the registry, and stitched the two together into an enriched ad event.

This design served us well. The registry was globally replicated, which gave us a useful property almost for free: an event arriving in any region could find its metadata regardless of where the ad was decided. It future-proofed the token against ever-growing metadata, and it carried us through the first phase of Netflix’s advertising growth.

But as we looked ahead, especially to ads on live streams, the limits of this design became clear.

Why we needed to evolve

The forcing function was Live. Netflix’s live slate has grown rapidly, from NFL games to blockbuster fight nights to the FIFA Women’s World Cup 2027, and viewership keeps growing. That traffic looks nothing like Subscription Video on Demand (SVOD): a single event can surge toward tens of millions of concurrent viewers within minutes, then drain just as fast when it ends. Building the ad pipeline for Live meant building for that trajectory.

To see why the old design could not get there, let’s put some numbers on it. When Netflix streamed the Jake Paul vs. Mike Tyson fight to more than 100 million viewers, a single ad break at that scale would generate an enormous volume of ad events. Serving those ads in the old design would have meant writing metadata for each of them to the Metadata Registry and waiting for acknowledgment before the ads could serve: millions of must-succeed database writes per second on the ad-serving critical path. That pattern created three core problems at Live scale:

1. It was a scaling bottleneck. Every ad and every ad break is a write. Multiply that by tens of millions of concurrent viewers, keep it non-lossy (a dropped write is a dropped ad, and lost revenue), add Live’s spiky on/off pattern, and the write load becomes enormous. But you cannot safely scale the registry up once the event starts. Adding nodes reshuffles data across the cluster to rebalance it, and doing that in the middle of a live event could slow or even fail the very writes you are adding capacity for. That leaves one safe option: pre-scaling for the peak in advance and hoping the forecast is accurate.
2. Fewer dependencies on the critical path. Serving an ad meant a synchronous, must-succeed write to the registry, followed by a wait for its acknowledgment. Every system on the critical path is one more thing that can stop an ad from serving, and the metadata registry had no fallback, so an unhealthy registry meant serving stopped. Taking hard dependencies off that path was the goal, and the registry was the biggest one we could remove.
3. It put event freshness at risk. If writing slowed down under load, or lookups timed out, events spilled into retries and dead-letter queues, and enriched ad events were delivered late. Some systems might be fine, but others, such as frequency capping, cannot. It needs events under a strict SLA to decide whether the next selected ad can run without breaking policies.

So we asked a simple question: the ad server already knows and logs everything about the ad it served. Does it really need to write that to a database on the hot path? Can we match that against the downstream Kafka stream of client events, which is always there, after the fact?

The write leaves the critical path entirely; no shared datastore outage can stop serving, and enrichment becomes something we scale elastically with Live rather than a cluster we babysit.

Matching two streams after the fact is a streaming join, and that is where Flink comes in.

Why Flink

We chose Apache Flink because near-real-time, stateful stream processing is exactly what it is built for, and because it has deep, production-tested platform support at Netflix. Just as important, we already knew how to run it. As Part 1 noted, our Ads Sessionizer is a large stateful Flink job, so operating one at scale, tuning state, checkpoints, and recovery under production load, was familiar ground for the team. Enrichment on Flink was built on that experience rather than starting from scratch.

The new architecture

At its core, the new Ads Event Publisher is built as an Apache Flink streaming job. Each event moves through a handful of stages:

1. Normalize. Client events arrive from several sources, including SVOD, live, playback, and embedded ad breaks, each with a slightly different shape. We first normalize them into a common form.
2. Re-route. Client events and response events may appear in different regions; we inspect and reroute them to the correct job-processing region for a successful join.
3. Join. The heart of the job: we match each client event to the ad-serving response for the same impression, enriching it with the full serving context using a KeyedCoProcessFunction
4. Deduplicate. Revenue-bearing events are de-duplicated, so nothing is counted twice.
5. Publish. We extract measurement beacons for our partners and route the enriched ad event to its downstream consumers.

The join is the interesting part. Client events and ad serving responses arrive on separate streams at different times, so the job holds recent responses in memory and matches events against them over a 60-minute window. Three things can happen:

• The response is already there when an event arrives → we enrich it on the spot.
• The event arrives first → we hold it for a configuration duration (currently set to 60 minutes) until the matching response shows up, then enrich it.
• No response arrives within the window → we mark the event as unmatched for later recovery.

We built the join as a custom Flink operator (a KeyedCoProcessFunction) rather than a built-in windowed join. We do not need a windowed join ordering and time-window semantics, since downstream consumers handle ordering; we simply match each client event to its response by key. Owning the operator at its lowest API form lets us control the matching state directly, including how long we hold it and what we do with an event that never matches. This allows us to tune retention per stream or persist unmatched events to a durable store later without rewriting the join. Our goal is always to build a strong base to reap the benefits later as we scale.

To make that concrete, picture a member watching a live game. When the ad server decides which ad to show, it logs its response, and a few seconds later, the member’s device fires a “first quartile” event carrying the opaque token. Both arrive at the job keyed to the same impression, and because the response is already waiting, the event is enriched on the spot with its campaign, creative, price, and tracking URLs, then published downstream in near real-time.

Why 60 minutes? Almost all events, roughly 99.999% of them, reach the job well before that, so extending the window further would only grow the state store and lengthen checkpoints for a vanishingly small gain. The remaining 0.001% still matters for precise billing and reporting, and we recover those events via a separate path described later on.

Cross-region: matching events where their data lives

The Metadata Registry handled cross-region enrichment for us because it was built on a globally replicated key-value store: a client event could land in any region and still find its ad’s metadata. Global replication was a feature the platform provided (see Netflix’s Key-Value Data Abstraction Layer), so serving an ad in one region and enriching its events in another was simply not something we had to think about.

A streaming join does not come with that convenience, and here the two sides of the join live in different places. The ad serving response stays in the region where the ad was served. The client event, though, is fired by the device, and the device can be anywhere: a member might start an ad on home wifi and switch to mobile data partway through, and that network change can route the event to a different region than the one that served the ad. So a client event is routed to a region that does not have the matching serving response.

The tempting fix is to stream every region’s serving responses into every other region so that each region can match any client event locally. We rejected it: that would mean every region storing every serving response worldwide, the vast majority of which will never be used in the join, which is wasteful in both cost and compute.

Instead, we treat this as a routing problem and route the client event rather than replicating the response. We encode the serving region into the ad token, so when a client event arrives in the wrong region, we forward it to the region that owns its response. For example, an ad is served in us-east-2, the member’s device switches from wifi to mobile data mid-ad, and the resulting event arrives in us-east-1. That region reads the token, sees the ad belongs to eu-west-1, and forwards the event back there, where its serving response is waiting. Only the small trickle of genuinely cross-region events pays for the extra hop.

Catching the long tail of late-arriving events

Sixty minutes covers the vast majority of events, but not all of them. A device may have been offline, an ad paused, or an engagement event delayed by hours. Rather than drop these stragglers, we set them aside and recover them through a companion path: an hourly Spark batch job, orchestrated by Maestro, Netflix’s workflow orchestrator, re-pairs each straggler with its serving response from our data warehouse (which has a far longer retention than our in-stream window) and publishes it through the same downstream sinks. This keeps the streaming job’s memory bounded while still catching the long tail.

This also lets us support event types that arrive well after the 60-minute window, including certain tracking-related events, while keeping the door open to extending the system to additional event types down the line and improving our match rate as needed. We also designed this offline job with reuse in mind: it can replay dead-letter queues and backfill historical data, giving us a single mechanism for both long-tail recovery and operational recovery.

What was hard

Beyond the architecture itself, three things took real work, and together they are what made the pipeline stable, correct, and safe to switch on.

Keeping the core lean.

Operating a stateful streaming join at live-event scale took real tuning, and most of our hard-won lessons come back to one idea: the job’s only job is matching events. Anything else we were tempted to bolt on, such as external lookups or incidental side-processing, was better moved off the hot path, or it eventually showed up as instability under load. Getting this right was the difference between a job that comfortably kept pace with live traffic and one that fell behind. It is unglamorous work, and it is most of the work.

De-duplicating Events

Duplicates are a fact of life here in an event processing pipeline. A client may retry an event, and our upstream delivery is at-least-once so that the same event can arrive more than once. For revenue-bearing events, the ones that feed billing, nothing should be counted twice.

The solution here is a simple and idempotent key. For each event, we compute a stable ID by hashing a few fields that uniquely identify it, such as the ad ID, the event type, and, for repeatable events, an offset or timestamp. The same underlying event always hashes to the same ID, so if we have seen an ID before, we drop the repeat.

What changed with Flink is where we remember those IDs. The previous design kept them in an external key-value store and did a check-and-write over the network for every event; Flink lets us keep that state local to the job instead. We key the stream by event ID, which routes every copy of a given event to the same place, and a local state store with a short time-to-live remembers the IDs we have recently seen. Keying by ID also removes the race the external store had to guard against, since a given ID is handled in only one place at a time, with no network hop on the hot path. Multiple improvements here alone.

We are deliberate about the guarantee: at-least-once delivery with best-effort de-duplication, not exactly-once. In-stream dedup removes the vast majority of duplicates within its window. However, a few can still slip past, for instance, through a cross-region reroute, a late-arriving recovery, or a restart that replays recent events. So we also carry the event ID through to the enriched ad event itself, which lets consumers who need exact counts, such as billing, run their own idempotent de-duplication over their own, longer windows.

Production Rollout

This was the most challenging part of the project. For a system that feeds billing and frequency capping, “trust me, the new one is correct” is not good enough, so we ran the Flink pipeline alongside the legacy one and made it earn its traffic.

Both pipelines continuously processed the full event stream, allowing us to compare them in real time: we published their outputs side by side. We ran automated audits that alerted us when the two drifted apart. Those audits earned their keep, surfacing subtle discrepancies we would never have caught from dashboards alone.

How do we know the enrichment logic between the two systems was equal for the audits to make sense? Early in development, we extracted that logic into a shared module that both the legacy service and the Flink job depend on, so any change to how an event is enriched is applied to both at the same time. A failure in the audits, therefore, pointed to a real difference in the pipeline mechanics.

Cutover was a separate question from comparison, and this is the part that warrants precision. We could not simply send a random percentage of events to each pipeline, because the same event landing in both would be double-counted downstream. Instead, the split was deterministic: a stable key on each event decided which pipeline was responsible for publishing it, so every event was published exactly once, even though both pipelines saw all of them. Raising the threshold moved events from the legacy pipeline to Flink a slice at a time, and we could dial it back instantly if anything looked off.

By the time we fully cut over, the two pipelines agreed on well over 99.99% of events, and we rolled it out region by region rather than all at once. Then came the satisfying part: we turned off the ad server’s writes to the Metadata Registry, and decommissioned the legacy service and the registry itself. The single point of failure on the ad-serving critical path is simply gone. One of the oldest ad serving components at Netflix was deleted.

Operating the Flink platform

Operating a large stateful streaming job in production is its own job, and we did not build that muscle alone. We leaned heavily on Netflix’s Flink platform team. Early on, the job’s checkpoints were slow and heavy. The first checkpoint in particular took a long time, and we were holding more state in memory than we needed. Working with the platform team, we trimmed the state, reduced checkpoint durations, and validated the gains in load tests. That is what keeps the job from stalling during Live bursts.

The same team also gave us operational scaffolding we did not have to invest in, including continuous delivery and canary deployments, so we could ship and roll out the Flink job with the same guardrails and confidence as the rest of our production systems. Netflix operates more than 25,000 Flink jobs (see Building a Scalable Flink Platform at Netflix), and that maturity is what we have to build on. We layered on custom canary metrics that catch issues before a release, CI/CD that lets us stagger rollouts, and a deploy flow that keeps the old job running until the new one has detected healthy checkpoints, so we never destroy a working job before the new job gives us the green light.

The results

• Enrichment is off the critical path for ad serving. The ad server no longer performs a synchronous, must-succeed database write to serve an ad. It logs and moves on, and an entire class of “the lookup store is down, so ad serving is down” failures no longer exists.
• Fewer systems, fewer schemas, fewer hops. We retired both the legacy service and the Metadata Registry.
• Built for Live, and continuously tested for it. An ad break is what this service is designed to handle, so during our journey, we also ran large-scale load tests to simulate real-life traffic. In our latest load test, a single AWS region handled one ad break for a large concurrent audience, processing hundreds of thousands of events per second, joining the large majority of them against their serving responses in real time, and outputting enriched ad events at a comparable rate, all with negligible timeouts and errors. We test this throughout development and keep doing so as live audiences grow, and we are not at the ceiling yet.
• We delivered the Part 1 roadmap. Live-stream ad events, in-stream de-duplication, and richer real-time enrichment all shipped on this foundation.

Key takeaways

• Take the database off the hot path when you can. Our biggest win wasn’t raw performance — it was blast radius. Trading a synchronous write-and-wait for a fire-and-forget log plus a downstream join removed an entire failure mode from ad serving.
• Match events where their data lives. Real-time joins beat global replication by routing events to their data rather than copying the data everywhere.
• Earn the cutover. Running both systems in parallel and diffing their output on live traffic was slower than a flag-day switch — and it’s the only reason we trust the result.

What’s next

We have an exciting list of projects on the horizon: scaling the live path for the next tier of concurrent audiences, bringing the remaining delayed engagement events fully into the streaming world, reducing the Flink state size we need to manage, working on bringing even more resilience to the eventing processing pipeline, and continuing to add new event types and richer signals for our downstream consumers.

Re-architecting the engine of a running, revenue-critical pipeline in place was a real multi-team effort. If building real-time systems at this kind of scale sounds like your thing, Ads Engineering is hiring.

Acknowledgements

This was a multi-team effort across Netflix. Thank you to everyone who designed, built, validated, and operated the new pipeline:

Ads Engineering: Yogesh Nagarur, Kinesh Satiya, Jeet Mehta, Nitin Saini, Persis Shah, Guangyu Chen, Jimmy Liu, Riley Berton, Simon Spencer, Aadesh Patel

Ads Data Engineering: Eugene Yusim, Brandon Nicklas, Indu Narayan

Ads SRE: Max Michaels, Kevin Chung, Carlos Avalos, Brian Christie

Data Movement: Mark Cho, Herbert Wang, Shraddha Ladda, Nikhil Sulegaon, James Lubin, Daniel Trager

Behind the Scenes: Evolving Netflix’s Ads Event Pipeline for Live — Part II was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Post Syndicated from The Atlantic original https://www.youtube.com/shorts/APk_2bbvYGw

Post Syndicated from Matt Granger original https://www.youtube.com/watch?v=_ImGdYyvGHY

Post Syndicated from The Atlantic original https://www.youtube.com/shorts/Q0KTKnvltcI

Post Syndicated from BeardedTinker original https://www.youtube.com/watch?v=z577_82asxU

Post Syndicated from The Atlantic original https://www.youtube.com/watch?v=er87hPH-aGI