AI EngineeringFebruary 11, 20256 min read

Evaluating LLM Outputs in Production

Shipping an LLM feature without evals means shipping on vibes. This is the measurement harness I build for every LLM product: what to check deterministically, when to let a model be the judge, and how to make a regression fail the build before a user notices it.

"It works on my test question" is the most expensive sentence in AI engineering. It gets a feature shipped, and then the next prompt tweak, model upgrade, or retrieval change quietly breaks it. No test fails. A user just notices before you do. Every LLM feature I've put in front of real users has lived or died on whether we could measure the output instead of eyeballing it. Below is the evaluation harness I build for every LLM product, in plain Python with no framework, along with the judgment calls that matter most.

The mental model

An eval is three parts: a fixed set of inputs (a golden set), a way to score each output, and an aggregate number you track over time. Only the scoring is hard, and the most common mistake is reaching for an LLM judge first. Match each output to the cheapest check that can catch a real failure:

Deterministic: the output has a property you can verify in code. It's valid JSON, it contains the order ID, it never echoes the system prompt. These checks are free, instant, and 100% reliable, so write them first.
Reference-based: there's a known-good answer to compare against, so you score similarity instead of re-deriving correctness.
LLM-as-judge: the quality is subjective (is this summary faithful? is the tone right?) and only a model can make that call at scale. It's powerful, and it's also the easiest way to fool yourself.

Three kinds of output checks ordered by cost and flexibility. Deterministic property checks are cheapest and fully reliable, reference-based similarity sits in the middle, and LLM-as-judge is the most flexible but must itself be validated. The rule: use the cheapest check that can catch the failure

The golden set

Build it from real inputs, not invented ones. Pull from logs, support tickets, and the questions people actually ask. Include the boring cases, and especially the ones that have broken before. Every production incident should end up as a golden-set case. Twenty cases that reflect real usage are worth more than two hundred random ones. Each case carries its input and whatever its checks need to score the output.

from dataclasses import dataclass, field

@dataclass
class Case:
    id: str
    input: str
    expects: dict = field(default_factory=dict)  # whatever the checks need

# A check reads the output (and the case) and returns a score in [0, 1]; 1.0 is a clean pass.
from typing import Awaitable, Callable
Check = Callable[[str, Case], Awaitable[float]]

GOLDEN_SET = [
    Case(
        id="refund-window",
        input="How long do I have to request a refund?",
        expects={
            "context": "Refunds are accepted within 30 days of purchase.",
            "must_contain": ["30 days"],
            "must_not_contain": ["lifetime", "no refunds"],
        },
    ),
    Case(
        id="out-of-scope",
        input="Do you ship to Antarctica?",
        expects={
            "context": "We ship within the United States and Canada.",
            # the right behavior is to decline, not invent a policy
            "must_not_contain": ["yes, we ship", "Antarctica is covered"],
        },
    ),
]

Deterministic checks

These are the cheapest insurance you'll ever buy, so write them first and lean on them hard. A check is just an async function that returns a score. Keeping everything async lets one runner drive all three tiers without special cases.

import json

async def valid_json(output: str, case: Case) -> float:
    try:
        json.loads(output)
        return 1.0
    except json.JSONDecodeError:
        return 0.0

async def contains_all(output: str, case: Case) -> float:
    needles = case.expects.get("must_contain", [])
    if not needles:
        return 1.0
    hits = sum(1 for n in needles if n.lower() in output.lower())
    return hits / len(needles)   # partial credit makes regressions visible, not binary

async def contains_none(output: str, case: Case) -> float:
    banned = case.expects.get("must_not_contain", [])
    return 0.0 if any(b.lower() in output.lower() for b in banned) else 1.0

Reference-based checks

When a case has a known-good answer, you don't need a judge. Embed both and compare them. This catches "the answer drifted" without you having to write out every acceptable phrasing by hand.

import numpy as np
from openai import AsyncOpenAI

openai = AsyncOpenAI()

async def _embed(text: str) -> np.ndarray:
    resp = await openai.embeddings.create(model="text-embedding-3-small", input=text)
    return np.array(resp.data[0].embedding)

async def similarity_to_reference(output: str, case: Case) -> float:
    ref = case.expects.get("reference")
    if not ref:
        return 1.0
    a, b = await _embed(output), await _embed(ref)
    cos = float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
    # Raw cosine rarely drops below ~0.3 for same-language text; rescale onto [0, 1]
    # and tune the floor on your own data so the score actually spreads out.
    return max(0.0, (cos - 0.3) / 0.7)

LLM-as-judge

For the subjective stuff like faithfulness, tone, and completeness, a model is the only grader that scales. Three rules keep it honest: give it a narrow rubric (not "rate this 1–10"), force structured output so the score is parseable, and run it at temperature 0 so results are reproducible. Always ask for a reason too. That's how you debug a bad grade.

JUDGE_MODEL = "gpt-4o"

JUDGE_PROMPT = (
    "You are a strict grader. Score the answer from 0 to 1 on whether it is faithful to the "
    "provided context AND directly answers the question. 1.0 = correct and fully grounded; "
    "0.0 = wrong, ungrounded, or evasive. If the context lacks the answer, the only correct "
    'behavior is to say so. Return JSON: {"score": <float>, "reason": "<one sentence>"}.'
)

async def judge_faithfulness(output: str, case: Case) -> float:
    resp = await openai.chat.completions.create(
        model=JUDGE_MODEL,
        temperature=0,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": JUDGE_PROMPT},
            {"role": "user", "content":
                f"Question:\n{case.input}\n\n"
                f"Context:\n{case.expects.get('context', '')}\n\n"
                f"Answer:\n{output}"},
        ],
    )
    return float(json.loads(resp.choices[0].message.content)["score"])

There's a catch: the judge is a model, so it needs its own eval. Hand-label thirty outputs yourself, run the judge over them, and make sure it agrees with you before you trust any number it produces. Watch for the known failure modes: position bias in pairwise comparisons, a soft spot for longer or more confident answers, and drift when you upgrade the judge model. If you skip the calibration, your scores tell you more about the judge than about your system.

The runner

The system under test is anything that maps an input to an output string: your RAG answer(), an endpoint, a chain. The runner doesn't care what's inside. It runs every case, applies every check, and records the scores.

import asyncio

@dataclass
class CaseResult:
    case_id: str
    output: str
    scores: dict[str, float]

    def passed(self, threshold: float = 0.7) -> bool:
        return all(s >= threshold for s in self.scores.values())

async def run_case(system, case: Case, checks: dict[str, Check]) -> CaseResult:
    output = await system(case.input)
    scores = {name: await check(output, case) for name, check in checks.items()}
    return CaseResult(case.id, output, scores)

async def run_suite(system, cases: list[Case], checks: dict[str, Check]) -> list[CaseResult]:
    return await asyncio.gather(*(run_case(system, c, checks) for c in cases))

Wire it into CI

An eval you run by hand is an eval you'll stop running. The whole point is to make a regression a failing build instead of a user complaint: run the suite on every change, aggregate to a pass rate, and exit non-zero below a threshold. Track the number over time so you can see a prompt change trade three cases for two.

# scripts/eval.py (runs in CI)
import sys

CHECKS: dict[str, Check] = {
    "no_leak": contains_none,
    "grounded_terms": contains_all,
    "faithful": judge_faithfulness,
}

def pass_rate(results: list[CaseResult], threshold: float = 0.7) -> float:
    return sum(r.passed(threshold) for r in results) / len(results)

async def main() -> None:
    results = await run_suite(my_system, GOLDEN_SET, CHECKS)
    rate = pass_rate(results)
    print(f"pass rate: {rate:.1%}")
    for r in results:
        if not r.passed():
            print(f"  FAIL {r.case_id}: {r.scores}")
    sys.exit(0 if rate >= 0.90 else 1)   # block the merge below 90%

asyncio.run(main())

The eval loop as a CI regression gate. The golden set feeds the system under test, each output is scored by the checks, the scores aggregate to a pass rate, and a threshold gate blocks the merge when the rate drops while logging the result over time

Conclusion

The harness here is small on purpose: a golden set, three tiers of checks, a runner, and a gate. The code is the easy part. What matters is the habit of putting a number on behavior you otherwise wouldn't see until a user ran into it.

So start with the golden set and the cheapest check that catches a real failure. Add a judge only where you have to, and validate it before you trust it. Once that's in place, every prompt tweak, model upgrade, and retrieval change becomes a measured decision instead of a guess.