Evals belong in the runtime, not the dashboard

You changed the prompt. Is it better?

Almost nobody can answer that. Not because they don't care — because answering it requires comparing two things that were never comparable. The old prompt ran last week, on last week's traffic, and you have a dashboard of averages. The new prompt ran on your eval set, offline, on forty cases you wrote by hand in March.

Those are different populations. The comparison is theatre.

The problem: your evals and your production runs are two different systems

Here is the shape almost everyone ends up with.

System one: the eval suite. A folder of test cases. A script that loops them through the model, scores the outputs, prints a number. It runs in CI, or more honestly, it runs when someone remembers. Its inputs are frozen. Its scores live in a CSV or a SaaS dashboard.

System two: production. Real inputs, real users, real money, running continuously. It emits traces. Somewhere there is a dashboard with latency and token counts and maybe a thumbs-up rate.

Nothing connects them. The eval suite cannot see production inputs. Production cannot run the eval suite's scorers. The score that decided you should ship lives in a different database from the runs it was supposed to predict.

So the two questions you actually have — is this better than what I had? and is this still working right now? — get answered by two systems that never speak, using two populations that don't overlap.

Agitate: what this costs

Your eval set is a museum

Every eval set drifts, because it is a snapshot of the inputs you could imagine at the time you wrote it.

Production is not a snapshot. It is the long tail arriving continuously — the ticket in a language you didn't test, the malformed payload, the customer who writes four paragraphs where everyone else writes one, the thing that only happens on the last day of the month.

Those are precisely the cases where the agent fails, and precisely the cases your eval set does not contain. So the suite goes green and the failure rate does not move, and the two facts sit next to each other without anyone reconciling them.

You could keep adding production failures to the eval set. Everyone intends to. It requires someone to notice a failure, extract the input, sanitize it, write the expected output, and commit it — for each one. In practice this happens for the first three incidents and then stops.

An average is not a comparison

The dashboard says helpfulness is 0.81 this week and was 0.78 last week.

That is not evidence the prompt is better. Different traffic mix. Different distribution of easy and hard cases. A quiet week with more simple requests moves that number more than a prompt change does. You have two averages over two populations and a difference of 0.03, and there is no honest inference available.

What you wanted was: the same input, both ways. That is the only comparison that isolates the change. And you cannot do it, because the old run is over and its inputs are in a trace viewer, not in something you can re-execute.

Scoring after the fact loses the thing you needed

The run finished an hour ago. Now you want to know whether its answer was good.

You have the output. Do you have what the retrieval step returned? The intermediate classification the agent made before it chose the tool? The arguments it passed? Whether the fallback model was the one that answered?

If your scorer runs against a log line, you get the final text and a guess. If it runs against the run, you get everything the run knew. The difference decides whether "the answer was bad" can be attributed to bad retrieval, a bad plan, or a bad model — and those have three completely different fixes.

The eval that never runs

The deepest version: teams stop measuring, and it is rational.

Standing up a second system that mirrors production, keeping it fed with representative data, and maintaining scorers in it is real work with no incident attached to skipping it. So the prompt changes ship on judgment, the agent slowly gets worse in one dimension while getting better in another, and nobody has the number that would have shown it.

The solution: score where the run already is

Put the scoring in the same place as the execution. Then a score is a property of a run, not a row in a different system, and all four problems change shape.

There are four things you need, and they are all the same primitive wearing different hats.

Score it while it runs

The cheapest case: the workflow can judge its own output as it goes.

await ctx.score({ name: "helpfulness", value: 0.9, comment: "clear and correct" });

This is durable and memoized like any other step — a replay after a crash does not record it twice. The score name is its identity, so there is no step id to invent.

Now every production run carries its own score. Not a sample. Every one. Your population is no longer forty cases from March; it is everything that actually happened.

Assert the things that are not judgment calls

Not every check needs a model. Plenty of them are deterministic: did the output match, did that step run at all, did the steps run in the right order.

Those should be assertions, not scores, because a deterministic check that fails is a bug and should read like one. Mixing "the JSON was malformed" into the same 0-to-1 average as "the tone was a bit brusque" destroys both signals.

Grade it after it finishes, with no code in the workflow

This is the one that changes the ergonomics.

When any run reaches a terminal state, Duraton emits a system event, duraton/run.finished, carrying the finished run's id, workflow, status, and output. A scorer is just a workflow subscribed to that event:

export const gradeHelpfulness = defineWorkflow<RunFinishedEvent>({
  name: "grade-helpfulness",
  triggers: [{ event: RUN_FINISHED_EVENT, if: "event.data.workflow == 'draft-reply'" }],
  handler: async (ctx) => {
    const finished = ctx.event.data;

    const judgment = await ctx.step.ai.generate<{ score: number }>("judge", {
      model: "claude-opus-4-8",
      prompt: `Rate the helpfulness 0..1 of: ${JSON.stringify(finished.output)}`,
      output: { type: "object", properties: { score: { type: "number" } }, required: ["score"] },
    });

    await ctx.score({
      runId: finished.runId,   // the FINISHED run, not this one
      name: "helpfulness",
      value: judgment.output?.score ?? 0,
      source: "llm-judge",
    });
  },
});

Look at what that is. The grader is a durable run. It retries if the judge model is flaky. It costs what it costs and that cost is attributed. It is versioned with your code. And the workflow being graded contains no evaluation code at all — you can add, change, or remove a scorer without touching the thing it measures.

One trap, and it is the obvious one in hindsight: a scorer must filter which runs it grades. The if above is not decoration. An unfiltered scorer subscribes to every finished run including its own, and grades itself, forever.

Compare one change against a real run

This is the answer to "is the new prompt better."

Take a run that actually happened. Fork it with exactly one declared change — a different model, a different prompt — on one AI step. The steps before that step replay from their recorded results. Execution resumes at the change.

const fork = await duraton.runs.fork(baseRunId, {
  step: "draft",
  model: "claude-haiku-4-5",
});

Now you have two runs, same input, same history up to the branch point, one difference. Both scored, both durable, both inspectable step by step.

That is a comparison. Not two averages over two populations — the same case, both ways, with the divergence isolated to the thing you changed.

And when you want a number over many cases rather than one, the same machinery runs a dataset through it, because a dataset eval is just a fan-out of runs that get scored like any other.

What this does not do

It does not write your scorers. An LLM judge is a model call with all a model call's problems, and the paper that popularised the technique says so itself: MT-Bench lists position, verbosity, and self-enhancement bias among the limitations of LLM-as-a-judge. The model underneath is not a fixed instrument either. Chen, Zaharia and Zou, tracking one service across two of its versions, concluded that the behavior of the "same" LLM service can change substantially in a relatively short amount of time - which makes a judge pinned to "the latest model" a ruler that changes length. And it will still confidently rate something 0.8 that a human would call wrong. Deterministic assertions are worth more than they look, and you should reach for them first.

It does not tell you what to measure. "Helpfulness 0.81" is only meaningful if that scorer means something to your product, and picking that is a product judgment no runtime can make for you.

It does not make a fork a controlled experiment. One fork is one case. It tells you what happened for that input, which is enormously more than you had, but it is not a statistically significant claim about your traffic. Run the dataset for that.

And it is not a replacement for looking at your outputs. Nothing is.

The question to ask

Not "do we have evals." Ask:

1. Is the score attached to the run, or to a row in another system?
2. Can I score every production run, or only a sample I exported?
3. Can I re-execute a real past run with one thing changed?
4. When a score is bad, can I see what the run knew at the moment it went wrong?

If scoring lives in a dashboard, the answers are no, no, no, and partly. If it lives in the runtime, they are yes by construction — because the runtime is already holding the run, the inputs, every step, and the ability to replay.

That is the whole argument. Evals are not a reporting feature. They are a runtime feature that got put in the reporting layer because that is where the traces happened to land.

We built Duraton the other way round.

This is the fourth in a series, after Your agent will die halfway, Where does it have to ask?, and Checkpointing is not completion.