The Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scale

Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/the-lifecycle-of-llm-as-a-judge-building-aligning-and-monitoring-at-scale-c95bd8283508

By Emma Yanyang Kong, JJ Tan, Ishan Gupta, Lars Olds, Claire Campbell, David Fagnan, Ratna Kavuri, Rohan Gosain, Veli Balin, Minsu Jang, Louis Garcia

Introduction

Any team generating text with an LLM at scale hits the same problem. You cannot evaluate everything manually. You don't know which outputs are great, which are mediocre, and which are harmful, because you simply cannot read them all. The standard solution is LLM-as-a-Judge, a second model that scores the first model’s output as a proxy for human raters. It is fast, cheap, and easy to deploy. However, it introduces a new and critical question: Who evaluates the evaluator? Furthermore, if the judge drifts, quietly degrading and starting to approve bad output, nothing downstream will flag it.

This post is about how we solve that problem, by building, aligning, and continuously monitoring LLM judges so they stay trustworthy over time. Our core principle is simple: An LLM judge must never operate alone. It has to be anchored in human judgment. We keep a human-in-the-loop (HITL) at every phase of the lifecycle. Humans define what “good” means, create the labeled benchmark datasets to train and validate the judge, monitor its behavior while it runs in the system, and review and approve any updates before they’re rolled out.

We believe:

A reliable LLM judge is not a static artifact. It has a lifecycle.

It needs to be built against trusted human labels, aligned with human preference and reasoning, run under human-set policy, and maintained as the data and usage pattern shift. Each of these stages has human owners. In the sections below, we walk through these four phases, the key design decisions in each, and the lessons we’d share with a team building their own LLM judge, using one concrete task as our running example: recommendation explanations.

A recommendation explanation is a short natural-language message that accompanies a recommended item, and it shapes how users perceive and trust a recommender system. In our work, an explanation is human-readable evidence for why a title was recommended based on what the member has previously watched, shown on the title’s detail page, which is the screen a member reaches by selecting a title from the homepage. We focus on similarity-based explanations, which connect a recommended title to one reference title the member has interacted with via shared attributes such as genre and tone. For example,

A funny, heartfelt holiday romance about love and new beginnings, much like “My Secret Santa.”

Quality matters here because a misleading or poorly grounded explanation erodes the very trust the explanation is meant to build.

Everything that follows comes out of a series of controlled, user-facing online experiments we ran on the explanation feature, including a month-long experiment on the Netflix mobile app. In these tests we used an LLM to generate explanations, hundreds of thousands a week across a constantly changing catalog. Manually reviewing them at this scale isn’t feasible, so we deployed a second LLM to evaluate them.

What counts as a good explanation is specific to our task. However, the way we build, align, and monitor the LLM judge for evaluating explanations should be reusable and generalizable across different tasks.

The Lifecycle at a Glance

Figure 1 summarizes the four phases of our LLM-judge lifecycle and the data flow connecting them.

Figure 1. The four-phase lifecycle of an LLM judge, and the data that flows between the phases.

We start with Birth: Benchmark Data Creation, where human experts craft hard or near-boundary examples and human raters label them with pass/fail decisions and written rationales. We augment this with LLM-generated gray-area examples to build a richer benchmark dataset.

Next, in the second phase of Training: LLM Judge Development, we tune the LLM judge to align with human labels and rationales from the Birth phase, using self-reflection to identify mismatches between the judge and humans, and updating the rubrics based on those mismatches and anchor examples. We iterate in this phase until our metrics show that the judge is sufficiently aligned with human raters.

In the third phase, Deployment, we integrate the judge into the explanation generation system as both a gate and a critic. The LLM generator writes explanations, and the LLM judge scores them and either passes them through or blocks them and supplies a rationale that the generator can use for self-refinement and retries. Passed explanations are served to members, and low-quality explanations are gated.

Finally, in the last phase, HITL Alignment & Drift Monitoring, we keep the judge aligned with humans over time. “Human-in-the-loop” here means a person is directly involved in the decision, not just reviewing outcomes after the fact. We benchmark the judge against trained raters, and a human approves every change before it is rolled out to the system. During each test we sampled a set of explanations each week (both passed and gated, with a bias toward newer items), had human raters review them with labels and rationales, and compared the judge against these human ratings. If drift exceeded a threshold, we triggered retraining and benchmark data augmentation. This closes the loop by feeding new human-labeled data and updated signals back into the benchmark and development phases.

System components

Four automated components appear across all phases, alongside the human roles that define, validate, and supervise them.

  • Generator: produces the explanation text, optionally using prior judge rationales and traces as part of its prompt for revision.
  • Judge: evaluates an explanation and returns a pass/fail label together with a written rationale. In the deployed pipeline, it acts as both a guardrail (gating bad outputs) and a critic (providing feedback for self-refinement).
  • Reflector: revises the Judge’s rubric when the Judge makes mistakes. It updates the rubric based on mismatched examples, human rationales, and anchor examples. It tunes the rubric and does not directly evaluate explanations.
  • Meta-Judge: checks whether the Judge failed an explanation for the right reason by comparing the Judge’s rationale against a human rater’s rationale.
  • Human experts and raters: the experts who define the quality criteria; the raters whose labels and written rationales the Judge is tuned and measured against; the reviewers who monitor the Judge while it is running and explicitly approve any changes before they were rolled out.

Phase I, Birth: Establishing Ground Truth

Everything downstream depends on knowing what “good” means, so this is the most human-intensive phase. Our internal writing experts define the must-have criteria an explanation must satisfy, author detailed labeling guidelines, and hand-craft tricky adversarial examples. Those same guidelines do double duty. They’re both the rulebook for human raters and the seed rubric for the LLM judges we tune later.

Why we built our own benchmark

Off-the-shelf LLM-judge benchmarks focus on general-purpose text evaluation and miss what makes our task specific. Explanations are short, item-specific, and context-dependent (each one is tied to a target title and one or two reference titles the member has watched). We also care about alignment to our own editorial bar, not just generic language quality. As a result, we built our own custom benchmark from three complementary sources:

  • Expert-crafted adversarial cases: hand-authored pass/fail examples with known failure modes, the kind of edge cases nothing else reliably produces.
  • Boundary examples: LLM-synthesized explanations that are deliberately pushed near the pass/fail boundary, then rated by humans. These capture the ambiguous cases that sampling from live traffic rarely surfaces on its own.
  • Real sampled explanations: explanations sampled from the output of our LLM generator during test runs.

Two details that mattered

  • The benchmark is not frozen. A monitoring pipeline (Phase IV) appends a fresh batch of human-rated explanations on a weekly cadence, so over the course of a test the benchmark tracked the current catalog and new usage patterns instead of going stale.
  • Every “fail” label carries a free-text rationale explaining why the explanation failed. This looks like a small bookkeeping detail, but it is actually critical for the next phase, since those rationales are what we use to tune and audit the judge’s reasoning.

Phase II, Training: Aligning the Judge to Humans

The failure mode: right answer, wrong reason

There is a failure mode that does not show up in label-agreement metrics at all. A judge can agree with humans on the label, passing when they would pass and failing when they would fail, while reaching the right verdict for the wrong reason. On a dashboard it looks perfectly aligned. But as soon as you ask it to explain a rejection, which our pipeline does to steer self-revision, it can return misleading guidance.

We therefore care about not just label agreement but also reasoning alignment. In our setup, the Judge is always asked to output:

  • a label (e.g., PASS/FAIL), and
  • a short natural-language rationale for that label.

Here is an illustrative example. The member watched a musical, a family fantasy with a young heroine and a well-known ensemble cast. We recommend an adventure movie that is also a family fantasy with a young heroine, which is why the recommender paired them. But it shares neither the songs nor the ensemble cast.

The label (FAIL) is correct, and the stated reason is reasonable at first glance. But it turns out to be incomplete in a way that simple label agreement cannot detect, as we show below.

Reasoning-Aligned Rubric Tuning

We tune each criterion’s LLM Judge with a procedure we call Reasoning-Aligned Rubric Tuning (RART). As introduced in the system components above, the Judge, Meta-Judge, and Reflector are distinct components. The Judge scores an explanation against a rubric and gives a rationale, the Meta-Judge compares that rationale against a human-written one, and the Reflector edits the rubric text itself. RART is how the three are combined into a training loop.

Every Judge is a fixed prompt template parameterized by a rubric, so tuning it does not mean retraining the underlying model. Instead we iteratively refine the rubric text until the Judge’s behavior matches human labels and human rationales. At iteration i, the Judge is conditioned on the current rubric text and maps an explanation to a pair, a label and a rationale.

R_i is the current rubric text, y_J(x) the label, and r_J(x) the rationale.

At a high level, each RART iteration proceeds as follows.

  1. Score the training data with the current rubric.
  2. Evaluate weighted alignment metrics on a held-out validation set. If all per-criterion metrics exceed their thresholds, stop.
  3. Otherwise, construct an error-driven focus set of misaligned examples and hand it to the Reflector, which proposes a revised rubric.

The key difference from a standard loop is how we define a “mistake.” Most self-improvement loops look only at label disagreements. RART adds a second class of error. The Meta-Judge examines the cases where the Judge and a human both said fail and compares the Judge’s stated reason against the human’s rationale. When the two diverge, that is a right answer, wrong reason error, and it is also fed to the Reflector. Concretely, the focus set handed to the Reflector is the union of two error classes:

  • Label mismatches: examples where the Judge’s label differs from the human’s.
  • Right label, wrong reason: examples where the Judge and the human both said FAIL, but the Meta-Judge finds their two rationales don’t match.

So the Reflector revises the rubric from both ordinary label errors and right label, wrong reason errors. That second term is the core of RART. The Judge is pushed not only to match human decisions but to justify them in a way the Meta-Judge recognizes as aligned with human reasoning.

Returning to the earlier example, the LLM Judge said FAIL and a human said FAIL, so a label-only check would register agreement. Placing the two reasons side by side shows what that check misses.

RART routes it into the focus set, so the next rubric revision has to account for it.

Since the Meta-Judge is an LLM too, we checked it against humans first. Trained human raters labeled a sample of agreed-fail rationale pairs, and the Meta-Judge matched their verdicts on nearly all of them.

Alignment metrics

We run one Judge per criterion, each with its own rubric, tuned and scored on its own. So everything below is measured per criterion on held-out data. For each one we track three metrics.

  • Specificity (fail-recall): how often the Judge correctly rejects bad explanations. This is the one we care about most, because a bad explanation that slips through the gate is a trust hazard.
  • Recall (pass-recall): how often it correctly passes good explanations, which keeps coverage high and revision cost low.
  • Reasoning agreement: on the explanations both reject, how often they reject for the same reason.

Each criterion’s Judge gets its own three, and one can be healthy while another slips. That is also why the gate is strict. An explanation has to pass every must-have criterion to be eligible to serve.

Ablation: does the reasoning signal help?

To isolate the impact of the reasoning signal, we compare RART to an otherwise identical loop that uses only label disagreements. We call this baseline vanilla: same Reflector, same Judge, same data, but no Meta-Judge and no use of human rationales.

Figure 2. Improvement over the starting rubric, per criterion, for reasoning-aware tuning (RART) and label-only tuning (vanilla).

Figure 2 shows the result. On two of the three must-have criteria, vanilla and RART land in the same place, because the starting rubric was already near ceiling and the metrics had very little room to gain. On the third criterion, the vanilla (label-only) method degrades. The Reflector in the vanilla loop rewrites the rubric into degenerate forms, and specificity and reasoning agreement fall with every iteration, ending up worse than the rubric it started from. RART stays stable on that same criterion and lifts both, with a small recall gain as well. The reason is that a vanilla (label-only) optimizer can reach the right label through unsound reasoning, and unsound reasoning doesn’t generalize to new data. Anchoring the signal on the rationale prevents that. Reasoning alignment helps where there is headroom and doesn’t hurt where there isn’t.

Phase III, Deployment: Putting the Judge to Work

Once deployed, the judge sat inside the generation pipeline, gating and critiquing explanations as they were produced during our tests.

Two roles for one judge

Once we align the LLM Judge with humans, we reuse it in two distinct roles for every generated explanation:

  • Role 1, the gate. The Judge scores each explanation against the must-have criteria and rejects anything that fails.
  • Role 2, the critic. When an explanation fails, we don’t just throw it away. The Judge’s free-text reason gets appended to the Generator’s prompt, and the Generator tries again.

The second role turns the pipeline into a generate → judge → revise loop, with the Judge’s critique steering each revision.

Bounded retries, and why we drop rather than serve

We cap the retries. Each revision costs another round of generation and judging, and the gains flatten quickly. Figure 3 shows what the loop actually buys, on 1,000 generated explanations run through as many as 12 revisions with four generators. Every curve climbs, which is the Judge’s critique doing its job, but the steep part is early, and the gap between generators is obvious. Revision amplifies a capable generator rather than rescuing a weak one. In our test runs we used a budget of 3 retries with our strongest generator, and an explanation still failing after that was dropped rather than served.

Figure 3. Cumulative share of explanations that clear the Judge as we allow more revisions, across four generator models.

That “drop rather than serve” rule is deliberate. The two outcomes aren’t symmetric. A bad explanation is a trust hazard, while a missing explanation just forgoes an opportunity. When in doubt we’d rather show nothing, and most explanations cleared the bar within the retry budget.

Do judge-approved explanations help members?

Passing the Judge proves an explanation is good by our standards, but it doesn’t prove that judge-approved explanations are useful to members. To test that connection, we ran a one-time controlled experiment on the Netflix mobile app, explanations vs. no explanations on the title detail page, over roughly a month, with the pipeline generating and judging hundreds of thousands of explanations a week during the period.

Members who saw explanations shifted their viewing toward titles they hadn’t watched before, and more of their browse sessions ended in a meaningful play. Both effects were statistically significant, and we saw no member-initiated takedowns or escalations about explanation quality during the test. The result worth carrying over isn’t the size of that effect. It’s that offline agreement with our raters was necessary for these explanations to work but never sufficient to show they help members. Only the online A/B test could establish that judge-aligned explanations are not just quality-compliant but genuinely useful.

Phase IV, Monitoring: Keeping the Judge Aligned

A judge that’s well aligned at launch won’t stay that way. The catalog shifts, and so does what counts as high quality. The lifecycle has to close the loop.

The weekly human review

For each week a test was running, we pulled roughly 300 generated explanations for human review, stratified across the Judge’s decisions (served, revised, dropped) and weighted toward newer catalog items, where drift is most likely. The stratification stayed fixed week to week, so successive weeks were comparable. A panel of at least three raters reviewed each explanation, and the majority label was our ground truth. Those labels did two jobs. They told us whether judge-human agreement was slipping, and they grew the benchmark.

Setting the bar with human disagreement

The panel also tells us how much the raters disagreed with each other, which turns out to be the right yardstick for the Judge. Rather than hold it to a fixed threshold, we score the Judge and each rater against the same majority label and ask that the Judge fall within the raters’ own spread.

Take one criterion and one of its metrics, M. That week’s sample gives one value M(J) for the Judge and a set of per-rater values M(H) for the three or more raters on the panel. We require

so the Judge must score no worse than two standard deviations below the average rater. A harder week widens human disagreement and widens the band with it, so the Judge isn’t penalized for difficulty the humans also felt.

Falling below the band on any metric-criterion pair is a drift event, which triggers re-tuning (back to Phase II) on the augmented benchmark. A re-tuned rubric never goes live automatically. Someone on the team reads the proposed changes and signs off before it reaches the running pipeline, and we keep the previous rubric for instant rollback.

What human review catches that metrics can’t

Beyond asking whether the Judge still agrees with human raters, the weekly review answers a second question. Are the current guidelines still asking for the right things?

In the weekly review process, reviewers surfaced issues that did not show up in Judge–rater agreement. An explanation can use accurate attributes and still overstate how similar two titles really are. A single vague adjective (“exciting”, “fun”, “emotional”) can technically satisfy the rubric while saying almost nothing. Sometimes the reference itself is the problem, something the member wouldn’t think of as a comparable title at all, so the comparison lands oddly no matter how the attributes line up. Some genres show the same problem at scale. Stand-up specials can share similar metadata tags but differ completely in tone and cultural context, so a comparison may look justified on paper, not be bad enough for a rater to fail, and still land poorly with members. And some explanations meet every must-have criterion while still reading like generic “word soup” that does not help the user much.

When recurring patterns kept appearing, we updated the Phase I guidelines, which seed both the raters’ instructions and the judges’ rubrics, so one revision updates both sides. Other times we added product rules outside the Judge, for example by blocking certain title types from being used as references.

Drift detection catches a judge that has stopped agreeing with people. Only the continued human review tells us when our definition of a “good explanation” is too narrow or incomplete. We need both.

  • Drift detection keeps the Judge aligned with the current rubric.
  • Continuous human review keeps the rubric aligned with what actually works for members.

What We Learned

Five lessons we think transfer to any team shipping an LLM judge:

  1. Invest in the benchmark before the judge. A modest, rationale-annotated benchmark beats a large label-only one. Those written rationales are exactly what make reasoning-aligned tuning possible. You can’t correct a judge’s reasoning if you never wrote down the human’s.
  2. Tune on reasoning, not just labels. A judge can reach the right verdict for the wrong reason, which looks fine on a dashboard but gives misleading feedback and doesn’t hold up on new data. In our ablation, the label-only loop degraded a rubric that the reasoning-aware loop kept stable.
  3. One aligned judge can serve many roles. Reusing the same judge as both gate and critic amortizes the alignment cost and keeps production behavior consistent across roles. Alignment is the expensive part, so get one judge right and use it more than once.
  4. Plan for monitoring from day one. A recommender platform’s items and members never stop changing, so an LLM judge, like any ML model, has to be watched and re-tuned if needed, not shipped and forgotten.
  5. Keep people in the loop by design, not by exception. Automating evaluation is not the same as delegating responsibility for it. In our lifecycle, humans set the standard (Phase I), supply the rationales the judge is aligned to (Phase II), decide the conservative default of dropping rather than serving a doubtful output (Phase III), and review a standing sample of live decisions and approve every rubric change before it ships (Phase IV). These checkpoints are much harder to retrofit once a judge is already making decisions on its own. The standing review also tells us what our rubric is missing, not just when our judge has slipped.

Taken together, the judge extends the reach of our reviewers, and those reviewers are the basis for trusting the judge. That relationship is what makes it responsible to run this system at scale. An automated evaluator that no one audits provides no assurance about the decisions it is making.

Acknowledgements

This post summarizes our research paper, “The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations,” which has the full methodology, metrics, and ablations.

Our thanks to the internal human writers and review experts who defined what a good explanation is and keep those guidelines current; to our Human Data Operations team and the raters they train and calibrate, whose labels and written rationales are what the judges are aligned to; and to the engineers across the team who built and operate the pipeline; and to our UI engineering partners, who built the member-facing surfaces that bring these explanations to the homepage.

The Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scale was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.