Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/netflix-stratum-media-processing-automated-container-right-sizing-ddf9aa68a801

By Violetta Pidvolotska and Naveen Mareddy

Stratum is Netflix’s internal serverless media processing platform, shaped by over a decade of experience developing large-scale distributed systems. It powers a wide range of media workloads — including encoding, packaging, inspection, and quality scoring — behind playback, advertising, studio workflows, AI embeddings, and more.

Engineers develop Stratum functions by packaging application code and OS dependencies into OCI images. Stratum then schedules these containers as batch jobs on Netflix’s Titus compute platform, with upstream services triggering function executions.

Every day, Stratum manages hundreds of thousands of containers across bursty, high-throughput, and latency-sensitive workloads. As Netflix’s core media processing platform, this load needs to scale as new content formats enter the catalog, such as podcasts, live events, sports, ads, or regional ingests like TF1 in France.

To request resources for these functions, owners specify their CPU, memory, disk, and network limits as YAML configuration, as shown below:

functionName: VideoEncoder.encodeV1
resources:
  numCpus: 8
  memoryInMB: 18000
  diskSizeInMB: 50000
  networkInMbps: 1000

Maintaining optimal values manually is challenging amidst evolving Netflix infrastructure, EC2 instance upgrades, and application-level changes. Consequently, configs are often over-provisioned upfront or slowly drift away from their ideal values over time. Mis-sized configurations are a universal pain of large-scale serverless platforms, causing additional cognitive load for function builders and operational costs for platform owners.

In this post, we share how we built the Stratum Resource Tuner (SRT) — an automated system that safely right-sizes running containers in production without human intervention. We will walk through real-world examples, our safety architecture, and what we learned when an automated change went wrong.

The Challenge: Over-Provisioning

When engineers write a new Stratum function, they typically copy a resource YAML from an existing function or size it for the absolute worst-case scenario. Once running in production, engineers rarely revisit these configurations until a failure occurs, prompting them to over-provision resources out of an abundance of caution.

Manual resource tuning does not scale for the following reasons:

• High Operational Friction: Downsizing requires days of manual metric monitoring and trial-and-error testing, diverting valuable engineering cycles away from other high-impact work. If a cut goes too far, engineers face production incidents — making manual tuning a high-risk, low-reward task.
• Continuous Infrastructure & Codec Evolution: Cloud providers regularly upgrade their hardware. A resource allocation that was optimal on an older hardware generation quietly becomes inefficient on newer hardware. At the same time, Netflix continuously advances media standards and improves encoding algorithms (such as AV1, AV2, and spatial audio). An allocation tuned for one codec might not be optimal when an improved algorithm is released.
• Dynamic Workload Evolution: As Netflix expands into new media types — such as streaming video podcasts, sports, interactive AI-adapted ad formats, and major broadcaster integrations like TF1 in France — media processing footprints can change unpredictably, rendering static container resource limits imprecise.
• Rising Industry Hardware Costs: Driven by massive global AI infrastructure demand, server hardware and DRAM prices continue to climb, making idle container memory significantly more expensive to waste.
• Compounding Platform Scale: At Netflix scale, even minor over-provisioning multiplies exponentially across hundreds of thousands of daily container executions, compounding wasted capacity across the entire Stratum platform.

To understand the scale of the challenge, we analyzed historical usage across Stratum’s workloads. The telemetry confirmed our hypothesis: over-provisioning was frequent, wasted compute was compounding, and the sustainable path forward required a fundamental re-evaluation of how resource configurations are managed.

Meet the Stratum Resource Tuner

To address the challenges listed above, we built the Stratum Resource Tuner (SRT) to automatically right-size Stratum containers. Our north star is to ensure that users never have to manually specify the resource configurations. We started with a handful of high-throughput workloads with relaxed latency requirements, which account for the majority (70%–80%) of resource usage.

As a quick note on terminology, we use “right-sizing” and “tuning” interchangeably in this post. “Downsizing” means decreasing resource limits, while “upsizing” means increasing them. Since our goal is to optimize for cost, this post focuses on downsizing.

Automatic right-sizing is not enabled by default. Function owners must manually opt in on a per resource type basis (e.g., CPU, memory, network, or disk). Once enrolled, SRT operates without a human in the loop across five steps:

1. Collect resource usage metrics across all executions of that function.
2. Generate a recommendation for each resource type (e.g., CPU or memory).
3. Verify the recommendation through experiments.
4. Apply or reject the recommendation based on the results of experiments.
5. Monitor post-apply metrics, watching for anomalies after applying the recommendation.

“Recommendation” is a key construct in SRT and serves as the primary unit of operation. As SRT executes its steps, it promotes the recommendation through several stages:

• CREATED: A scheduled workflow generated the recommendation.
• EVALUATING: Recommendation verification is running, and SRT is analyzing the results.
• APPLIED: All experiments passed, and SRT automatically rolled out the recommended configuration.
• REJECTED: At least one experiment failed, leaving the function configuration untouched.
• MONITORED: SRT monitors the recommendation during a post-apply soak period to verify stability.

The sections below examine each of these stages in detail. To illustrate the flow, we will follow the real-world lifecycle of the VideoEncoder.encodeV1 function — a high-throughput video encoding function whose over-provisioned memory and CPU footprints were successfully tuned.

Note that all operational thresholds and safety limits discussed below are fully configurable per function to accommodate varying workload characteristics and risk tolerances.

Metrics Collection

To make these recommendations possible, we need a solid metrics collection mechanism. Every Stratum container reports its function execution metadata, infrastructure dimensions, and resource utilization (CPU, memory, disk, and network) at sub-minute resolution.

These metrics flow through Keystone, Netflix’s real-time event streaming platform, and land in two separate datastores:

• Execution summaries store: Stores execution-level summary metrics indexed in Elasticsearch for fast point lookups, serving as our short-term datastore for querying and inspecting individual runs.
• High-resolution long-term store: Stores raw, high-resolution usage metrics in Iceberg tables. While slower to query, it serves as our long-term datastore, holding months of history for resource utilization trend analysis.

One record is written per execution, which is represented as a span. Here is how these records look for a single span of our VideoEncoder.encodeV1 function.

In Elasticsearch, the record captures the high-level summary of the completed span, identified by tags.stratum.spanId:

{
  "tags.stratum.spanId": "38536313f65f060a",
  "tags.functionName": "VideoEncoder.encodeV1",
  "tags.stratum.phase": "run",
  "tags.stratum.success": true,
  "tags.stratum.retriedWithMoreResources": false,
  "tags.stratum.function.execDurationSecs": 26,
  "tags.cpuCount": 8,
  "tags.memoryInMB": 18000,
  "tags.diskInMB": 50000,
  "tags.networkInMbps": 1000,
  "tags.titusTaskId": "06e1a7c0-2502-4a54-905f-b13a1e786107",
  "tags.instanceType": "r7a.24xlarge",
  "tags.nf.region": "us-east-1",
  "ts": 1780743028000
}

In Iceberg, the record contains the same metadata, but adds the sub-minute resource utilization timeseries tracked during that span:

{
  "span_id": "38536313f65f060a",
  "function_name": "VideoEncoder.encodeV1",
  "phase": "run",
  "success": true,
  "start_epoch_secs": 1780743002,
  "end_epoch_secs": 1780743028,
  "runtime_secs": 26,
  "cpu_requested": 8,
  "memory_mb_requested": 18000,
  "disk_mb_requested": 50000,
  "network_mbps_requested": 1000,
  "titusTaskId": "06e1a7c0-2502-4a54-905f-b13a1e786107",
  "instance_type": "r7a.24xlarge",
  "tags.stratum.retriedWithMoreResources": false,
  "env": "prod",
  "resource_usage": [ 
   { "cpu_usage": 1.13, "memory_mb": 3701.37, "disk_mb": 20120.4, "net_in_mb": 221.09, "net_out_mb": 0.12 },
   { "cpu_usage": 2.25, "memory_mb": 3584.09, "disk_mb": 25410.1, "net_in_mb": 0.02, "net_out_mb": 0.05 },
   { "cpu_usage": 4.16, "memory_mb": 6750.79, "disk_mb": 30400.5, "net_in_mb": 165.90, "net_out_mb": 0.38 },
   ... 
  ]
}

Recommendation Generation

A scheduled Conductor workflow runs daily, reading the past month of usage data from Iceberg and generating recommendations for every function and resource type. The workflow also generates recommendations for functions that have not opted into automatic right-sizing to preview potential savings.

Each recommendation is saved in the recommendations datastore, enabling us to query and filter recommendations by function, resource type, or status on an ad hoc basis.

For each function and resource type, the workflow applies the following generation logic:

• Filter by workload type: We only tune high-throughput workloads. This focuses our efforts on right-sizing, which yields the most meaningful fleet-wide savings.
• Aggressively tune CPU and network: We target p99 usage with no extra headroom. CPU and network are compressible resources; if a container occasionally hits these limits, the function might execute slightly slower.
• Conservatively tune memory and disk: We target p99.9 usage and include a safety buffer. Running out of physical memory or disk space causes hard failures — such as Out of Memory (OOM) or Out of Disk (OOD) crashes — making a protective margin essential.
• Apply safeguards to changes: We discard recommendations if the absolute reduction is too small to justify operational effort. For memory and disk, we clamp the maximum reduction allowed in a single cycle to 15% — a safety limit introduced after learning from early production runs.

The workflow creates new recommendations in the CREATED state. If a recommendation already exists in this state for a given function and resource type, SRT overwrites it with updated calculations. Once a recommendation advances past the CREATED state, its recommended_value and stats fields become immutable. Subsequent daily runs generate a new recommendation with a unique ID instead of modifying the existing record.

Here is how these recommendations look across resource types in their initial CREATED state for our VideoEncoder.encodeV1 function:

• CPU recommendation

{

  "function_name": "VideoEncoder.encodeV1",
  "env": "prod", "resource_type": "CPU",
  "current_value": 8, "recommended_value": 4,
  "status": "CREATED",
  "resource_cost_usd_per_hour": ***,
  "stats": {
    "usage": { "p99": 4.16, "p999": 4.57 },
    "execution": {
      "total_fn_executions": 748762182,
      "success_fn_executions_pct": 99.75,
      "total_fn_exec_runtime_hr": 88102723
    },
    "window": { "start": "2026-05-05T00:00:00Z", "end": "2026-06-05T00:00:00Z" }
  },
  "experiment_inputs": null,
  "experiment_workflow_execution_id": null,
  "outcome_message": null,
  "finalized_at": null
}

• Memory recommendation (15% reduction in one step)

{
  "id": "2edb734a-6184-4f5e-8ffa-23004bd665d7",
  "function_name": "VideoEncoder.encodeV1",
  "env": "prod", "resource_type": "MEMORY_MB",
  "current_value": 18000, "recommended_value": 15300,
  "status": "CREATED",
  "resource_cost_usd_per_hour": ***,

  "stats": {
    "usage": { "avg": 5296.43, "p99": 6750.8, "p999": 7820.3 },
    "execution": {
      "total_fn_executions": 748762182,
      "success_fn_executions_pct": 99.75,
      "total_fn_exec_runtime_hr": 88102723
    },
    "window": { "start": "2026-05-05T00:00:00Z", "end": "2026-06-05T00:00:00Z" }
  },
  "experiment_inputs": null,
  "experiment_workflow_execution_id": null,
  "outcome_message": null,
  "finalized_at": null
}

As the recommendation moves through its lifecycle, SRT populates the remaining parameters.

When proceeding to the verification stage, SRT transitions the recommendation status to EVALUATING, records the target experiment settings, and links the active workflow execution tracking the experiment.

Finally, once verification runs complete, the status changes to APPLIED or REJECTED, storing the outcome details and completion timestamp for analysis.

Retry with More Resources

We take a conservative approach to generating recommendations, prioritizing system stability over extreme efficiency. Because recommendations rely on historical usage data, they cannot always anticipate sudden spikes or novel workload profiles when functions encounter heavy, unseen requests.

To handle these unexpected spikes, we developed a fail-safe mechanism. A watcher process running in each tuned container tracks memory and disk usage during function execution. If the container exceeds the pre-set thresholds, 85% for non-cached memory usage or 95% for disk usage, the watcher flags the execution as high risk, cancels execution, and re-queues the task onto a larger container:

new_container_size = original_container_size * 1.5

This 1.5 multiplier only applies to memory and disk dimensions.

This “retry with more resources” mechanism serves as our safety net: larger container configurations are applied only when retrying tasks to self-heal from edge cases. Although these infrequent retries incur additional compute costs and latency, they allow us to keep day-to-day configurations highly optimized without having to size for rare, costly outliers up front.

For our VideoEncoder.encodeV1 function, if a heavy run causes memory usage to exceed the 85% threshold (15,300 MB) within its current 18,000 MB container, the system aborts the execution and reruns it on a 27,000 MB container.

While we prefer to trigger retries based on actual OOM or OOD events rather than pre-set thresholds, technical constraints in our current architecture make true event detection challenging. Using pre-set thresholds serves as a reliable stopgap while we develop improved detection capabilities.

SRT automatically enables the retry mechanism for any function opted into memory or disk right-sizing. Each Stratum container reports retry events, which flow into Elasticsearch and Iceberg alongside standard performance telemetry.

Recommendation Verification

Every recommendation must pass a two-stage verification process managed by a Conductor workflow before rollout. During this EVALUATING stage, SRT executes two distinct experiment types:

1. Synthetic experiment: A low-cost, low-risk test to catch obvious regressions before production rollout.
2. Production experiment: A live evaluation that validates changes across diverse production workloads to increase confidence.

A recommendation advances to production testing only after passing the synthetic experiment. When an opted-in function has multiple pending recommendations, SRT prioritizes and evaluates the candidates with the highest estimated savings first.

Synthetic experiment:

• Runs in a test environment isolated from production traffic.
• Sets up two clusters: one for the current and one for the recommended config.
• Replays identical request sets across both clusters simultaneously without splitting traffic, exposing both configurations to 100% of test requests.
• Typically runs for 1–2 hours.
• Runs to completion without early termination checks.

Production experiment:

• Runs in production on real user traffic, split three ways:
  ◦ ~90% remains on the main cluster (current production config).
  ◦ ~5% goes to a baseline cluster (current production config).
  ◦ ~5% goes to an experimental cluster (recommendation config).
• Evaluates recommendations against a similarly sized baseline cluster rather than the main cluster to preserve comparable traffic profiles.
• Typically runs from one day to one week, depending on traffic volume.
• Fails fast: cancels the experiment early if regressions are detected.

The Stratum platform executes other experiments alongside SRT experiments. To avoid over-splitting traffic, SRT experiments run at lower priority and yield resources to active, higher-priority experiments.

Traffic arrives in irregular bursts separated by multi-day or multi-week lulls. The verification pipeline accommodates these variable arrival patterns rather than expecting steady traffic.

Back to our VideoEncoder.encodeV1 memory recommendation (18,000 MB → 15,300 MB), verification is triggered with:

{
  "recommendationId": "2edb734a-6184-4f5e-8ffa-23004bd665d7",
  "syntheticExperiment": { "numberOfExecutions": 3000 },
  "productionExperiment": {
    "enabled": true,
    "maxContainers": 20, // max containers shift to baseline & experimental clusters
    "upGateTimeoutMinutes": 10080, // time to wait for traffic to start flowing
    "maxRunDurationMinutes": 1240, // max time to run the experiment once traffic starts
    "minSamples": 500 // min executions needed on both clusters before evaluating
  }
}

Both verification stages evaluate recommendations against two sequential checks, terminating immediately if either fails:

• Retries with more resources: Restricts memory and disk downsizes to a maximum 1% retry threshold over the experiment duration.
• Execution time: Compares median runtimes to prevent outlier distortion, permitting up to a 10–15% slowdown relative to baseline, depending on workload, and is configurable by function.

If a recommendation passes both verification stages, SRT updates its status to APPLIED and rolls out the configuration to the main production cluster. If a check fails at any point, SRT marks the recommendation as REJECTED and preserves the existing configuration. SRT sends notifications for both outcomes.

For VideoEncoder.encodeV1, both stages passed cleanly:

{
  "outcome": "APPLIED",
  "syntheticExperiment": {
    "result": "SUCCESS",
    "samples": { "baseline": 3000, "experimental": 3000 },
    "medianExecutionSecs": { "baseline": 59.0, "experimental": 59.0 },
    "slownessIncreasePct": 0.0,
    "retryRatePct": { "baseline": 0.0, "experimental": 0.0 }
  },
  "productionExperiment": {
    "result": "SUCCESS",
    "samples": { "baseline": 50361, "experimental": 49790 },
    "medianExecutionSecs": { "baseline": 196.0, "experimental": 198.0 },
    "slownessIncreasePct": 1.0,
    "retryRatePct": { "baseline": 0.0, "experimental": 0.0 }
  }
}

The recommendation moved to the APPLIED state, permanently setting the memory allocation for VideoEncoder.encodeV1 to 15,300 MB. We observed similarly clean results for the CPU recommendation, which reduced CPU allocation from 8 cores to 4.

Post-Apply Monitoring

Passing verification experiments does not mark the end of the recommendation lifecycle. Updated function code, modified input parameters, or underlying changes in the compute environment (Titus/EC2) can alter production behavior over time. To ensure configuration safety and foster feature adoption, we instituted continuous post-apply monitoring for applied changes.

We monitor metrics for 90 days across two rolling time windows: a 30-minute fast window and a 24-hour slow window. Each window maintains specific alert thresholds: a 3% retry rate over 30 minutes or a 2% rate over 24 hours triggers an automated notification.

At steady state, workloads exhibit baseline retry rates around 0.1%, with rates up to 2%–3% falling within acceptable operational variances. Most applied recommendations remained stable over the past three months.

VideoEncoder.encodeV1 has operated on its tuned configuration for months without breaching thresholds. However, VideoEncoder.encodeV2 presented a challenging edge case: a 50% memory reduction passed all verification checks and ran stably for two weeks before an unexpected traffic shift drove retry rates above 60%.

Because early versions lacked automated rollback capabilities, engineers manually triaged exception cases. This incident highlighted the need for automated mitigation, prompting us to construct self-healing rollback functionality.

Today, the two auto-revert windows look like this:

• Last 30 minutes: 3% retries warn us; 10% triggers an auto-revert.
• Last 24 hours: A 2% retry rate triggers both an alert and an automatic revert.

Breaching an auto-revert threshold immediately restores the function configuration to its pre-recommendation baseline.

We intentionally maintain straightforward revert logic — restoring the last known stable configuration rather than attempting dynamic resource adjustments during an incident.

The VideoEncoder.encodeV2 incident also reshaped our recommendation generation rules for memory and disk allocations. Rather than applying full target reductions immediately, SRT now applies smaller, incremental 15% step changes, advancing only after confirming stability at each step. Additionally, we introduced a global kill switch to disable automated right-sizing across all functions in case of unexpected issues or side effects.

Additionally, we enforce a mandatory cooldown period — typically one week — before initiating new experiments on a function. Pausing adjustments allows us to observe performance across diverse production traffic patterns, identifying potential risks before applying subsequent recommendations.

Where Are We Today

Oversized containers and config drift don’t have to be an accepted tax of running serverless workloads at scale. By building the Stratum Resource Tuner, we transformed manual, risk-averse resource sizing into an automated, self-healing pipeline. Through real-world workloads and hard lessons from unexpected spikes, we’ve proven that continuous right-sizing can safely deliver significant gains in infrastructure efficiency.

We initially focused on right-sizing high-throughput video encoding workloads, aiming for an initial 5% reduction in overall Stratum compute costs. For instance, VideoEncoder.encodeV1 achieved a 50% reduction in CPU usage and 40% in memory usage without compromising performance or retry rates.

However, edge cases like VideoEncoder.encodeV2 highlighted key lessons: large single-step reductions can introduce hidden risks, and manual incident response slows recovery. These insights led us to enforce stricter safety guardrails in the Stratum Resource Tuner, including capping memory and disk reductions at 15% per cycle and adding automated rollbacks when retry thresholds are breached. Guided by these safeguards, re-tuning VideoEncoder.encodeV2 yielded a stable 20% memory reduction with zero subsequent incidents.

Looking ahead, our key priorities include:

• Expand SRT beyond high-throughput encoding workloads to support a wider variety of functions and workload types.
• Further tune the resources in regions with constrained resource availability.
• Move from reactive alerts to proactive, automated upsizing for under-provisioned functions.
• Experiment with intentional upsizing for latency-sensitive workloads to evaluate performance gains and reduce function execution time.
• Upgrade our failure signals for OOM/OOD events from lagging indicators to leading indicators to improve tuning precision.

If solving problems like this sounds exciting, we are hiring in our Warsaw office. Check out open roles here.

Credits

Special thanks to Olof Johansson, Jakub Karczewski, Adam Mazur, Ethan Mathieu, Rosanna Lee, Ameya Vasani, Jean Barkhuysen, Paul Hieromnimon, and Adithya Prakash.

Netflix Stratum Media Processing: Automated Container Right-Sizing was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.