Running Apache Spark experiments in my sleep (and on a plane)

Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/running-apache-spark-experiments-in-my-sleep-and-on-a-plane-588816b6b5f8

By Prashanth Gedde Narayanaswamy

As one of Netflix’s largest and critical data pipelines, member-sessionizer consolidates tens of billions of hourly client events, encompassing user taps, playbacks, and engagement signals, into individual viewing sessions per profile.

A quick two-week tuning effort spiraled into two months when standard approaches hit a wall. Because the failures only happened at full production scale, each test took hours, limiting me to just three or four attempts daily. The turning point was changing the goal: instead of trying to speed up a single run, I built a way to run and monitor multiple tests at once and track every result.

That mindset led to an unconventional setup: kicking off five Spark jobs at once and monitoring them over spotty inflight Wi-Fi during a day-long trip home, while cloud servers handled the actual heavy lifting. This post covers how I set up that loop, along with the Spark bottlenecks and memory quirks I uncovered along the way.

The Scale Dilemma: Small Samples Don’t Catch Scale Bugs

Standard engineering wisdom says that you should isolate bugs on small datasets to iterate quickly. That works for standard software bugs, but it fails completely for big data performance bottlenecks.

While upgrading our sessionization pipeline from a legacy logging architecture to a modern platform, the new job hit critical failures, including executor Out-Of-Memory (OOM) errors, severe shuffle spillage, and thousands of stage retries. None of these issues could be reproduced using sampled data. The failures only triggered at full production scale because of data skew: an idle Netflix tab left streaming overnight generates tens of thousands of telemetry events, while a typical session only produces a handful. A standard 1% sample misses these extreme long-tail sessions entirely.

To catch the real bugs, every test run had to execute against a full hour of unsampled production data, along with all the deployment overhead that came with it.

The Strategy: Branch-Driven Isolation

The biggest win came from a feature in Dataflow, Netflix’s command-line tool for building, testing, and deploying data pipelines. It supports branch-driven development: our workflows run on Maestro (Netflix’s workflow orchestrator), which schedules and executes the jobs, and they already use a sanitized branch name (the ${dataflow.sanitizedbranch} placeholder). So if I push a branch like pgn/cap-experiment, then the deployed workflow, its tables, and its run namespace all get tagged with that branch name. This eliminates collisions with team members or between my own concurrent runs.

With that isolation in place, the model was simple: one branch is one independent experiment. I parameterized the scheduler (cap size, shuffle partitions, heavy-device threshold, feature flags) and gave each variant its own branch and output table. Instead of queuing five sequential tests over 10 hours, I fired off five variants simultaneously against the same time window. Total execution time dropped from the sum of all runs to just the length of the single slowest job.

Managing per-branch tables added zero overhead. Dataflow provisions and updates tables automatically from declared schemas without manual DDL. Whenever I modified the event bag structure between experiments, the table schema updated seamlessly, no CREATE TABLE or ALTER TABLE required.

The only real constraint was cluster contention. Because these heavy Spark jobs shared resources with the rest of the company, peak hours sometimes limited me to two concurrent runs instead of five. But even running two or three jobs in parallel was a massive improvement over single-threaded debugging.

The key takeaway: don’t focus on making a single multi-hour run faster, build a setup where you can run many experiments simultaneously without them stepping on each other.

While the previous chart shows the time savings, the architecture that enables this isolation looks like below:

How an AI Agent Helped

Running five parallel experiments gets messy fast. Tracking instance IDs, configs, memory spills, and stage failures across variants makes it easy to lose track.

This is where an AI agent stepped in, not to write clever code, but to handle operational grunt work. I described variants in plain English, and the agent updated scheduler configs, deployed to the branch, triggered jobs, and polled metrics. Once complete, it compared spill and failure counts across variants in a summary table, allowing me to ask, “Which run is closest to OOM-free at peak load?” and get an immediate answer.

The value wasn’t deep Spark expertise, but operational memory. It turned tedious Spark UI cross-checks into a single prompt.

Eventually, I packaged this workflow into a reusable agent skill (.agents/skills/sandbox-deploy-and-run/) run via /sandbox-deploy-and-run <module> <scheduler-file> [PARAM=value]. It handles the full lifecycle—preflight, deploy, background polling, auto-diagnostics, and re-entrant resume <instance_id> support—turning multi-hour experiments into a single command.

Why tmux Mattered

All of this ran on Workbench, Netflix’s cloud dev environment, using my laptop as a thin terminal. Without session persistence, an SSH drop or sleeping laptop would kill multi-hour Spark runs mid-flight.

tmux running directly on Workbench solved this. Every experiment lived in a named cloud session: I could close my laptop, reattach hours later, and pick up where I left off. Combined with branch isolation, this made "start five and walk away" possible.

This is now my default setup: a local terminal (Ghostty), dedicated git worktrees per branch, and persistent tmux sessions on Workbench. The local laptop is disposable—closing it or losing network never interrupts the jobs.

That setup turned a long travel day into active iteration time. I kicked off batches at the gate, checked results via AI agent over spotty inflight Wi-Fi, and squeezed 4–5 full experiment cycles into a single 24-hour flight.

What All Experiments Uncovered

The root cause came down to a single operator: collect_list. Because Spark buffers each full event bag in memory, long-session OOMs were driven by individual bag size, not total dataset volume. The legacy pipeline handled identical row counts without crashing, proving that each event in the new bag had grown significantly heavier.

Two factors inflated event weight:

  • Field types and Option boxing: The session ID changed from an 8-byte Long to a UUID string. Worse, wrapping fields in Scala Option forced Spark to deserialize typed Datasets into JVM objects (Some wrappers), driving up object allocation and GC pressure across tens of thousands of events per session.
  • Bloated string attributes: Low-value fields (e.g., referrer URLs, test allocations) added several hundred bytes per event. Multiplying these strings across collect_list wasted terabytes per hour, even though downstream tasks rarely used them.

The Fixes:

  1. Unboxed primitives: Removing Option wrappers from non-nullable fields dropped peak executor memory from 112% to 58% of the slot with zero disk spill.
  2. Slimmed event bags: Dropped unused per-event attributes and hoisted session-invariant metadata (app version, country) up to the session header.

Takeaway: Reducing individual event weight delivered far better stability than throwing more cluster memory at the problem.

Two Experiments That Changed My Mind

Out of dozens of runs, two findings completely overturned my assumptions about Spark memory management:

  • Finding 1: Narrowing rows can trigger faster OOMs. Slimming our table dropped its size below Spark’s auto-broadcast threshold, prompting the query planner to switch from shuffle joins to broadcast joins. Because broadcast joins pin build-side tables directly into executor memory, narrower rows actually increased total memory pressure.
  • Finding 2: Broadcast hash joins were silently draining memory. Adaptive Query Execution (AQE) rewrote nine sort-merge joins into broadcast hash joins at runtime. While each join stayed under our 400 MB broadcast limit individually, AQE pinned ~3.1 GB of cumulative data simultaneously in a single executor, pushing containers over the edge.

The Surgical Fix: Instead of disabling broadcast joins globally, I lowered only AQE’s runtime threshold (spark.sql.adaptive.autoBroadcastJoinThreshold) to 10 MB while leaving the compile-time threshold untouched. That single change freed up ~3.1 GB of executor memory without impacting initial query planning.

Takeaway: Broadcast joins speed up queries, but pinned memory isn’t free. Watch accumulated broadcasts in your execution plan, not just dataset sizes.

Keeping a Single Experiment Log

When running dozens of parallel tests, the biggest risk isn’t job failure, it’s losing track of what you’ve already tried or confusing which JAR and config produced a result.

To solve this, I scaffolded the AI agent to append a row to a repository Markdown table after every run:

  • Run # | Jar ver | Change vs. Previous Run | Config | Key Metrics (Spill, Failed Tasks, Peak Memory, Duration)

If a run yielded a key finding, the agent added a short note explaining what the result proved or disproved.

Key Takeaways:

  • Single source of truth: It created an auditable history for the entire investigation.
  • Forced discipline: Requiring “Change vs. Previous Run” forced me to isolate variables. Whenever I bundled multiple changes, I couldn’t isolate the root cause and had to rerun the test — a mistake the log format immediately exposed.

A secondary doc (events-join-spark-tuning.md) captured the distilled findings as a quick "symptom-to-knob" lookup, so teammates didn't have to wade through the full chronological history.

A Mistake I Learned From: The Fix That Only Worked Off-Peak.

To cap session size, I added row_number().over(Window…) and filtered for the first N events. It passed off-peak tests cleanly, but crashed with OOMs at peak load.

The issue was structural: Spark’s Window operator buffers an entire session in memory before evaluating row numbers and filtering. The cap trimmed output downstream, but it didn't limit memory usage during buffering, so a single skewed session overwhelmed executor memory.

The Fix & Lesson: We moved truncation ahead of the Window stage to cap events before buffering occurs.

Takeaway: Test fixes against worst-case input skew. Off-peak runs hide structural bottlenecks that peak scale will immediately trigger.

The final fix

Pulling it together, four changes shipped, each isolated to a single experiment before it landed:

  • Unboxed the never-null fields from Option to plain primitives. Peak executor memory dropped from about 112% of the slot to about 58%, with zero spill.
  • Slimmed the event bag. Dropped per-event fields nobody used, and hoisted session-invariant fields (versions, country, connection type) to the session level so they are stored once per session, not once per event.
  • Capped session size before the Window stage. Dropped oversized sessions ahead of the buffering, and truncated the bag to the cap rather than dropping whole sessions.
  • Lowered AQE’s runtime broadcast threshold. Set spark.sql.adaptive.autoBroadcastJoinThreshold to 10 MB, leaving the compile-time threshold untouched.

The Unresolved Bottleneck

While these fixes made the pipeline run cleanly, they didn’t solve the underlying scaling issue: aggregating an entire session’s event bag into a single row has a hard ceiling. As our upstream telemetry source evolves and adds new attributes, each event in the bag becomes heavier, meaning this memory pressure will eventually return without a single line of code changing on our end.

Slimming and unboxing bought us headroom, but they didn’t change the fundamental data shape. The biggest takeaway from this effort is that tuning buys time, not infinite scale. If your underlying data model allows unbounded row growth, tuning alone won’t save you , eventually, the architecture itself has to change.

Summary

  • If the bug only appears at scale, stop trying to sample. Build a way to run many full-scale runs in parallel instead.
  • If your scheduler can use the branch name, then one branch becomes one isolated experiment, and running many at once becomes easy.
  • Use an AI agent to run the operation, not only to write code. Deploy, poll, pull metrics, compare variants: this is where most of the time goes.
  • Once a loop repeats, package it as a skill. The deploy-run-watch-diagnose cycle becomes one command.
  • Use tmux for long runs. The run must survive you closing the laptop.
  • Keep one single experiment log, and make every entry state the one change from the previous run. The format itself forces you to change one variable at a time.
  • For memory problems, look at the per-row width first, not only at the total data volume. Bigger field types, Option boxing, and fat unused strings all add up inside a collect_list bag.
  • Tuning buys headroom, not a new ceiling. If the data shape can grow without a bound, the design has to change eventually, tuning alone will not save you.

Running Apache Spark experiments in my sleep (and on a plane) was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.