Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/netflix-conductor-the-next-chapter-41ad21067649
Netflix Conductor : The Next Chapter
by Aravindan Ramkumar on behalf of the Conductor team
Conductor is the workflow orchestration engine Netflix uses to stitch micro-services into reliable, observable business processes.
If you’ve followed Conductor from the outside, the last thing you heard was probably the note on its GitHub repository: Netflix discontinued maintenance of Conductor OSS to refocus its resources on the internal fork.
That announcement marked the end of the open-source chapter, and from the outside it may have looked like the end of the story. It wasn’t. Internally, Conductor never slowed down. If anything, it grew faster, and the demands of that growth pushed us to rebuild it from the ground up. This post is the answer to the natural question that announcement leaves open: what happened to Conductor after that?
A lot, it turns out. The clearest way to put it is in terms of growth. Not long ago, Conductor crossing a billion workflow executions in a single year felt like a milestone. Today it clears that bar roughly every quarter, and the monthly volume is still climbing. Sustaining that kind of growth meant rethinking nearly every layer of the system. The data plane it runs on was modernized piece by piece. The evaluation engine was rewritten. The way it allocates work, controls concurrency, and runs compute all changes. The through-line is simple: scale forced a rethink of the evaluation engine and the layers beneath it, and everything else followed.
The rest of this post walks through that evolution: how the architecture changed, the performance and scale work at the heart of it, the leap forward in developer experience, and the newer capabilities for controlling how workflows and tasks execute.
How Netflix uses Conductor
When we last wrote about Conductor, it ran a few hundred workflow definitions for a few dozen teams. That footprint has grown by orders of magnitude. Conductor now backs roughly 200,000 workflow definitions owned by around 150 applications, spanning Content and Studio Engineering, Ads, Games, and more. The runtime side has grown to match. In a typical month it executes on the order of 420 million workflows, and on a given day it runs anywhere from 5 to 20 million, with teams adding hundreds of new workflow types every month.
Conductor and the Media Production Suite
One of the clearest ways to see what Conductor does is Netflix’s Media Production Suite (MPS). It’s a set of applications that move production data into the cloud. An average Netflix title generates around 200 terabytes of original camera files, and MPS is what gets that media ingested, validated, transcoded, and delivered to the right people around the world.
The multi-step pipelines behind MPS’s workflows (Footage Ingest, Dailies, VFX Pulls, and Conform Pulls) run as Conductor workflows. A dedicated orchestration layer called Studio Orchestrator stitches together the platform services these pipelines depend on, such as asset management, media encoding, and storage, using Conductor under the hood to ensure reliable execution and the durability to recover from the inevitable failures of long-running, distributed work.
Beyond the studio: Ads and Games
Content engineering is where Conductor started, but it’s no longer the whole story. Two of Netflix’s newer businesses lean on it just as heavily.
Netflix Ads has made Conductor a backbone of its programmatic pipelines. Creative ingestion, advertiser-certificate management, and privacy-preserving Data Clean Rooms all run as Conductor workflows.
Netflix Games uses Conductor to orchestrate the path a game takes from a partner’s submission to the app stores. When a studio uploads a build, a Conductor pipeline ingests the binary, runs the required validations (for example, iOS privacy-manifest checks), and drives App Store publishing, coordinating steps that span multiple platform services and external stores with the same reliability and visibility the studio pipelines rely on.
How the architecture evolved
Our earlier post described a Conductor built on a particular set of building blocks: DynoQueues for queueing, Dynomite for metadata, Cassandra (partially) for execution data, and Elasticsearch for indexing. That stack carried Conductor a long way, but as adoption climbed, almost every layer of it was eventually rethought. The data plane Conductor runs on today looks very little like the earlier one.
1.0: the original stack, and where it strained
In its first generation, Conductor leaned heavily on Dynomite (with DynoQueues as the queueing recipe layered on top) backed by Elasticsearch for indexing. It worked, but two classes of problems showed up as scale grew. The first was the datastore. Dynomite is an in-memory store: a clustering layer that adds sharding and replication on top of Redis. You scale it by adding memory and nodes, with each replica rack holding a full copy of the data, which suits a cache but not the durable record for millions of workflows a day. Dyno Queues, built on the same store, inherited those limits, and its zone-sharded design made task scheduling uneven. The second was the engine’s design: evaluation ran synchronously and mutated workflow state in several places, so competing updates could leave a workflow inconsistent with no automated way to repair it.
The 2.x line was a sustained campaign to address these, and two replacements did most of the heavy lifting.
2.0: Cassandra as the backbone
The most consequential change was making Cassandra the primary datastore for execution data, displacing Dynomite. Cassandra gave Conductor the elastic, horizontal scaling it needed, and removed the scaling ceiling and zone-affinity constraints that the old datastore imposed. Around the same time, large task inputs and outputs were offloaded to S3, keeping bulky data off the primary datastore and further easing the pressure on it.
Timestone: a purpose-built queueing layer
The second replacement was the queue itself. Timestone, Netflix’s high-throughput, low-latency priority queueing system, took over from DynoQueues. Beyond shedding the Dynomite dependency, it brought two capabilities that mattered for reliability: a delayed-queuing mechanism, and a background repair process that could detect and recover workflows. Timestone also turned out to be foundational for work that came later. The async evaluation queue and the queue-depth-driven dynamic thread allocation described later in this post both build on it.
Alongside these, the 2.x series also decoupled indexing from the critical path, moving to a Kafka-buffered, independently scalable indexer writing to Elasticsearch.
3.0: consolidating the modern data plane
By Conductor 3.0, these threads came together into a coherent platform. Dynomite was fully sunset. Timestone was the queue and Cassandra the datastore. A Kafka-fed indexer pushed to Elasticsearch, with Iceberg added for long-term storage. The release also shipped automated workflow repair and multi-cluster deployment, the last of which lets workflows run in fully isolated contexts (for example, a dedicated context for the Media Production Suite, kept separate from the shared default and scaled on its own).
The net effect is that by 3.0 the foundational reliability and scaling problems of the early era were largely solved, and the data plane reached roughly the shape it has today.
Rebuilding the engine: Conductor 4.0
By 3.0, everything around the evaluation engine had been modernized, but the engine and its data model were largely the original, and at this scale that became the bottleneck. The engine evaluated a workflow by loading its entire state into memory to decide what to do next, which meant heavy memory pressure and slow evaluations as workflows grew toward tens of thousands of tasks. It also stored a workflow’s metadata and its task payloads in the same Cassandra partition, producing oversized “wide rows” and leaving room for race conditions when several updates to the same task competed. Evaluation overhead, vertical-scaling pressure, and occasional inconsistent state needed to be addressed.
A partitioned task model
The old data model co-located everything about a workflow in one place. The rewrite separates workflow metadata from task and user data and gives each task its own record. The engine no longer loads the whole workflow graph to make a decision. It evaluates against a lightweight blueprint of the workflow and loads only the specific task data a decision needs. The practical result is a dramatically smaller memory footprint per evaluation and far less work to advance a workflow by one step.
The 3.0 engine topped out around 2,500 tasks per workflow. The rewritten engine scales to 30,000 tasks per workflow, more than a 10x increase. Evaluation got faster too: in production, the rewrite delivered roughly a 40% reduction in p99 workflow-evaluation latency.
Making competing updates safe using partitions
Separating the data is only half the story. The engine also has to make competing updates to the same task safe, because a task can be written from more than one place at once. Picture a long-running task whose worker sends periodic updates to keep it alive, each one marking the task in-progress, before finally reporting it COMPLETED. If a stray in-progress update arrives just after that completion, reordered by the network, it could quietly revert a COMPLETED task back to IN_PROGRESS.
The first version leaned on Cassandra lightweight transactions (LWT) to serialize these writes, but mixing LWT with ordinary writes is a known Cassandra anti-pattern. The two use different timestamping mechanisms, and combining them breaks the strict consistency Conductor needs.
So the data model evolved away from locking entirely. Task state is split across two partitions: a pending partition for tasks still in flight, and a terminal partition that a task is written to exactly once, when it reaches a terminal state (COMPLETED, FAILED, etc). When the engine evaluates a workflow, it reads both and reconciles them in the application layer with one simple rule: a terminal state always wins over a non-terminal one, and between two writes of the same kind (two in-progress, or two terminal) ordinary last-write-wins applies. So in the race above, when the task is marked COMPLETED it lands in the terminal partition, and the late in-progress update loses on read.
The net of the rewrite is reliable task lifecycle management, a lower memory footprint, faster evaluation, and far higher concurrent throughput.
Removing the last synchronous bottleneck: async workflow evaluation
A faster engine still left one structural problem. Workflow evaluation ran synchronously. When an update arrived, a new workflow or a task completing, the request itself evaluated the workflow, and to do that safely it had to take a lock on that workflow. Under load, concurrent updates to the same workflow raced for that lock. Only one won, and the rest were deferred to a background repair process that, in extreme cases, could take several minutes to catch up. The result was exactly the kind of unpredictable tail latency that hurts most under peak load.
The fix was to decouple the trigger from the evaluation. Updates no longer evaluate inline. Instead they enqueue the workflow onto a Timestone exclusive queue, and a central asynchronous evaluator processes each workflow’s pending work in order. Because the queue is exclusive, only one worker ever evaluates a given workflow at a time, so the lock contention simply goes away.
The results are most dramatic at the tail (p99 and p99.9), which is the point, since the tail is what users feel under load:
Unsuccessful lock-acquire attempts, which had spiked to around 2,700 per interval under contention, drop to essentially zero with async evaluation.
Controlling how workflows execute
Run Strategies: native control over workflow concurrency
For most of its life, Conductor would happily run as many instances of a workflow as you started, all at once. That’s the right default, but it left a gap. When several workflow instances act on the same entity, the same movie id, the same creative, the same script, there is no built-in way to serialize them or guarantee only one runs at a time. Teams that needed this had to build it themselves: custom locks, custom queues, custom coordination logic bolted onto their workflows. That’s extra complexity in every workflow that needs it, and a real source of operational risk.
Run Strategies are a way to declare, on a workflow definition, how concurrent executions should be handled. There are two strategies today. Parallel, the default, applies no restrictions and behaves exactly as workflows always have. Sequential runs at most one workflow at a time per sequence key, with subsequent requests enqueued and started automatically, in order, as each running workflow finishes. A run strategy is declared on the workflow definition, and can be set directly in the Workflow SDK as a @RunStrategy annotation argument on the @WorkflowMethod.
The key idea is the sequence key: two executions that share a sequence key run one after another, and executions with different keys still run in parallel. So a content-ingestion workflow keyed on movie id processes each title strictly serially while happily running many different titles concurrently, with no custom locking required.
Beyond polled workers: managed, serverless tasks
A task meant a polled worker: your application implements the worker, polls the Conductor server for work, runs it, and reports the result back. That model is simple and language-agnostic, and it’s still the backbone for most workloads. But it has a cost. You operate the worker fleet. You stand it up, keep it running, and scale it, even for work that’s bursty or runs rarely.
So Conductor added managed tasks: built-in task types where Conductor drives the compute directly, with no long-lived worker for you to host or scale. Two integrations carry most of this today: Titus for containers and Stratum for serverless media-processing functions.
A TITUS task launches a Titus job to run a script, whether shell, Python, or whatever your image packages, using Titus, Netflix’s container management platform, as the compute plane. You point the task at a Docker image and an entry point, and specify the shape of the run, such as cpu and memory. The lifecycle is fully managed. The task starts IN_PROGRESS, Conductor monitors the Titus job, and it transitions on the container’s exit behavior. An exit code of 0 marks the task COMPLETED, and a non-zero code, a timeout, or a missing job marks it FAILED. The upshot is that you can spin up containerized compute as a step in a workflow without running a worker that sits around polling.
A STRATUM task invokes a Stratum function and collects its output. Stratum is the serverless function layer of Netflix’s Cosmos platform, purpose-built for media-processing workloads, and a Conductor STRATUM task is the natural way to call one from a workflow. You give it a function name and an ordered list of arguments. The task stays IN_PROGRESS until the function finishes, then resolves to COMPLETED, FAILED, or TIMED_OUT. Because Stratum exposes functions to Conductor directly, teams can fold resource-intensive media operations into a workflow without owning the compute that runs them.
Polled workers, Titus jobs, and Stratum functions are three points on a spectrum, and a single workflow can mix all three: a long-lived worker for steady high-volume work, a Titus container for an occasional heavy batch step, and a Stratum function for media processing. Managed tasks shift the operational burden from the team to the platform.
Scaling task polling: dynamic thread allocation
The engine work in the previous section made Conductor evaluate workflows fast. But there’s a second half to throughput: how quickly workers pick up the tasks that evaluation schedules. That’s a polling problem, and at scale it has its own failure mode, one that showed up most sharply for Studio Orchestrator, the heavy Conductor user behind the Media Production Suite pipelines from earlier in this post.
Conductor clients poll for work using worker threads, and historically the number of threads was fixed per task type. That’s predictable, and it prevents any one task from starving the others, but it’s static. During burst traffic, when thousands of instances of a single task suddenly pile into a queue, a fixed thread count can’t ramp up to drain them. The symptom was distinctive: CPU on the instances stayed flat while queues backed up, and idle threads sat stranded on quiet queues unable to help the overloaded one.
Allocating threads by real-time queue depth
The fix is a dynamic allocation strategy that rebalances threads across task queues continuously, based on how much work each queue actually has, while still guaranteeing every queue a floor so nothing starves.
Each instance periodically polls Timestone for the current depth of every task queue it manages. It then splits its thread pool into two parts. A minimum allocation gives every queue a baseline number of threads, so no queue is ever fully starved. A floating pool holds the remaining threads, distributed across queues in proportion to each queue’s share of the total load. A queue with a deep backlog draws a large slice of the floating pool. An idle queue falls back toward its minimum, freeing threads for whoever needs them.
One guardrail keeps this from becoming its own problem: a multiplier ceiling. A queue’s thread count can grow by at most a configurable factor per cycle, so even a dramatic load spike ramps up in controlled steps rather than all at once, giving downstream and dependent services time to absorb the increase instead of being bombarded. Allocations are recomputed every polling interval, so the system tracks shifting load in near real time while always honoring the per-queue minimums.
Two things come out of this. First, because threads are allocated where they’re actually needed, each instance can safely run a much larger pool. Second, CPU utilization becomes an accurate signal of how busy an instance is, which means autoscaling on CPU makes the cluster scale out in response to real computational demand.
Developer experience: Workflow SDK
For most of Conductor’s life, authoring a workflow meant writing JSON. You declared each task, then wired one task’s output into the next with ${taskRef.output.field} string templates, and on the worker side you pulled inputs back out of a Map<String, Object> and cast them to the types you expected. It worked, but it pushed an entire class of errors to runtime, and made large workflows genuinely hard to read.
The Workflow SDK is our answer: a type-safe Java DSL for authoring Conductor workflows. You write Java code. The compiler and IDE catch mistakes before anything deploys, and a Gradle plugin translates your Java code into the JSON workflow definition the Conductor server understands.
Crucially, the SDK gives you the best of both worlds. One of Conductor’s defining strengths has always been visibility. Because a workflow is a declared graph rather than imperative code, during execution the exact control flow can be inspected, every branch, loop, and parallel fork, and light up live as an execution moves through it. Writing workflows in imperative code gives you none of that. A timeline is not a workflow diagram. The SDK keeps both halves. You write in idiomatic Java and at execution time you still get the full visual graph: the same task-by-task flow, inputs and outputs, and at-a-glance “where is this stuck and why” that Conductor has always offered.
The authoring pains it removes
No type safety. In the JSON model, a task input like “movieId”: “${awesomeTask2.output.movieId}” is just a string template. The worker reads it back with inputData.get(“movieId”) and casts. If the upstream value is actually an Integer and you cast to String, you get a ClassCastException at runtime. Fix that, and a subtle typo bites next: the JSON declares an input variable movieId but the worker reads inputData.get(movieid), which returns null and you get a NullPointerException downstream. None of this is visible until the workflow runs.
public class GetMovieWorker implements Worker {
@Override
public TaskResult execute(Task task) {
Map<String, Object> inputData = task.getInputData();
long movieId = (Integer) inputData.get("movieId"); // cast, fingers crossed
// more logic here
}
}
Verbosity. Defining a couple of tasks is already wordy. A real workflow’s JSON balloons into hundreds of lines that are difficult to review or refactor. A complex branching workflow becomes a single large JSON blob that no one wants to touch.
{
"name": "wrap_movie_workflow",
"version": 1,
"semver": "0.0.1",
"schemaVersion": 2,
"restartable": true,
"workflowStatusListenerEnabled": false,
"ownerEmail": "[email protected]",
"timeoutPolicy": "ALERT_ONLY",
"timeoutSeconds": 0,
"tasks": [
{
"name": "get_movie",
"taskReferenceName": "movie",
"type": "SIMPLE",
"inputParameters": {
"movieId": "${workflow.input.movieId}"
}
},
{
"name": "wrap_movie",
"taskReferenceName": "wrapMovie",
"type": "SIMPLE",
"inputParameters": {
"movie": "${movie.output}"
}
}
]
}
How the Java DSL works
@TaskMethod
public Movie getMovie(long movieId) { /* real worker logic */ }
@TaskMethod
public WrappedMovie wrapMovie(Movie movie) { /* real worker logic */ }
@WorkflowStub(taskNames = {"movieapp.getMovie", "movieapp.wrapMovie"})
public interface MovieWorkflows {
@WorkflowGenerated(name = "movieapp.getMovie")
Movie getMovie(int movieId);
@WorkflowGenerated(name = "movieapp.wrapMovie")
WrappedMovie wrapMovie(Movie movie);
@WorkflowMethod(name = "wrap_movie_workflow", ownerEmail = "[email protected]")
default WrappedMovie wrapMovieWorkflow(MovieInput input) {
Movie movie = getMovie(input.getMovieId()); // long in, Movie out (checked)
return wrapMovie(movie); // Movie in, WrappedMovie out (checked)
}
}
The DSL has two core building blocks. The first, @TaskMethod, is a method that is executed at runtime and can contain arbitrary code. Crucially, it takes and returns real types: a getMovie task can take a long movieId and return a Movie object directly, with no maps and no casts. The second, @WorkflowMethod, is a method that is not executed and cannot contain arbitrary code. It expresses the workflow as ordinary method calls (call getMovie(…), then pass the result to wrapMovie(…)), and a Gradle build plugin parses it into a Conductor JSON definition at build time. The server does not see the Java code. It gets the same JSON it always has.
Code generation is central to how this works. In the @WorkflowStub annotation you declare the task and workflow names you want to use, and at build time the Gradle plugin generates their input and output types along with typed stub methods that you call inside a @WorkflowMethod. Those generated stubs are what let the method body compile against real signatures with full type safety. Because the stubs are generated from names rather than shared code, teams can reuse each other’s tasks and workflows without sharing libraries.
Real control flow, expressed in Java
The examples above are linear, but most production workflows branch, loop, and fan out. The DSL expresses these as the Java constructs you’d reach for anyway, and the SDK parses them into the corresponding Conductor operators. A useful mental model: you’re describing a workflow graph in Java, not running Java, so the parser only supports the constructs that map cleanly to Conductor (if/else, switch, do-while, parallel forks, sub-workflow calls) and asks you to express a few things its way. Iteration, for instance, is always a do-while, since there are no for or while tasks in Conductor.
A common shape is to kick off work, poll until it settles, then branch on the outcome, which is a do-while loop wrapped around an if/else:
@WorkflowMethod(name = "process_asset_workflow", ownerEmail = "[email protected]")
default void processAssetWorkflow(AssetInput input) {
JobStatus status = submitJob(input);
do { // loops are do-while
status = checkStatus(input.getJobId());
} while (status.getState() == "PENDING");
if (status.getState() == "SUCCEEDED") { // branch on the result
publish(input);
} else {
flagForReview(input);
}
}
Independent work runs concurrently with ParallelExecution.fork(…) plus allOf(…) (a FORK and JOIN), and any other @WorkflowMethod can be called as a sub-workflow just by listing it in workflowNames:
@WorkflowStub(
taskNames = {"movieapp.getMovie", "movieapp.enrichMovie", "movieapp.getArtwork",
"movieapp.refreshCache", "movieapp.publishPage"},
workflowNames = {"movieapp.localize_page_workflow"})
public interface MoviePageWorkflows {
// stubs not shown for brevity
@WorkflowMethod(name = "prepare_movie_page", ownerEmail = "[email protected]")
default void prepareMoviePage(MoviePageInput input) {
// three independent branches, run in parallel
Fork<WrappedMovie> details = ParallelExecution.fork(() -> {
Movie movie = getMovie(input.getMovieId());
return enrichMovie(movie);
});
Fork<Artwork> art = ParallelExecution.fork(() -> getArtwork(input.getMovieId()));
Fork<Void> cache = ParallelExecution.fork(this::refreshCache);
ParallelExecution.allOf(details, art, cache); // FORK_JOIN and JOIN
WrappedMovie enriched = details.get();
Artwork artwork = art.get();
LocalizedPage localized = localizePageWorkflow(input); // sub-workflow call
publishPage(enriched, artwork, localized);
}
}
Sharing tasks and workflows across applications
Teams shouldn’t re-implement common tasks. Listing another team’s task or workflow in your @WorkflowStub is enough to call it like a local method, or invoke it as a sub-workflow, and the shared POJO types (a Movie, say) come along directly, with no copy-pasting between codebases.
Feature parity and rapid adoption
The SDK is gaining rapid adoption and has become the primary choice for authoring new workflows at Netflix. We continue to invest in the developer experience, with intelligent IDE plugins, better debugging, and an integrated testing framework so that testing a workflow in code feels like testing any other code.
In production: Netflix Ads
Netflix Ads runs complex, multi-step processes on Conductor with the Workflow SDK, and three examples show the range. Creative ingestion is a heavily branching pipeline where the SDK turned a sprawling JSON blob into type-safe, testable code, and a sequential run strategy eliminated a duplicate-processing race by serializing work per creative. Advertiser certificate management automates a previously manual, hard-to-scale process across several system integrations, with downstream automation when certificates expire. And Data Clean Rooms are privacy-preserving environments where Netflix and a partner run joint analysis on shared data, built on the SDK from day one. Provisioning a clean room is a multi-day sequence (create at a vendor, wait for the partner to join, activate, and eventually archive) that previously relied on hand-rolled async polling. Re-authored as Conductor workflows, each phase became a durable, observable state machine with automatic retries, state that survives restarts, and long partner waits that yield rather than hold a thread.
The common thread is that the SDK made multi-process, multi-integration workflows easy to author and maintain, and let teams lean on Conductor’s reliability and visibility instead of hand-rolling retries and distributed locking.
What’s next
Like the last time we wrote about Conductor, the road ahead is less a fixed roadmap than a wish-list, a sense of where we’re headed, shaped by what teams keep asking for.
On scale, the demands keep growing. As Netflix expands into new content types, like live, games, and podcasts, we anticipate our workflow load could grow 5x in the coming year, and the engine work described above is what gives us confidence we can absorb it without re-architecting again.
On developer experience, the Workflow SDK now covers all task types, so the focus shifts to the experience around writing workflows: a richer integrated testing story so a workflow can be validated against real worker behavior as easily as any other code, deeper agent tooling such as skills, plugins, and MCPs, and safe schema evolution: a schema registry for shared tasks and workflows, with guardrails that block non-backward-compatible changes so they can evolve without breaking the teams that depend on it.
On execution control, sequential is only the first run strategy. A family of richer policies is on the roadmap, including running N at a time per key, idempotent de-duplication, keep-first and keep-latest, and combinations of these, to cover the coordination patterns teams currently hand-build. And managed tasks will keep expanding the ways Conductor can run compute directly, beyond today’s container and serverless function integrations.
None of this is exhaustive, and priorities shift with the workloads we take on. But the direction is consistent: keep the overhead low, keep authoring safe and simple, and keep widening the range of work a single workflow can reliably coordinate.
Acknowledgments
Netflix Conductor is built and operated by the Enterprise Workflow Platform team, with contributions from many partner teams across the company.
Andrew Seier, Anoop Panicker, Christian Johnson, Christine Yu, Jasmine Lee, Jiaofen Xu, Michael Torku, Peter Lau, Prashant Rastogi, Ryan Kastilani Surafel Korse and Victor Lee have contributed to Conductor. We thank Charles Zhao, Sumukh Shivaprakash and the teams in the Content and Business Products and Ads organizations for their guidance and constructive feedback while developing and operating Conductor.
Netflix Conductor: The Next Chapter was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
