You can't assert a story is good, so I built a judge

The most important file in my bedtime-story app is a big string.

It lives in src/stories/prompt-builder.ts, and it tells the LLM how to write a story: the tone, the structure, the rules about age and length and not being preachy. Change one paragraph of it and the product changes for every child who uses the app. It is, by a wide margin, the highest-leverage code in the whole thing.

It was also, for a while, the one file with zero regression protection.

Here’s the uncomfortable truth: a prompt is code, but you can’t unit-test it. There’s no assert that a story is good. expect(story).toBeCharming() doesn’t compile. So every time I tweaked the prompt to fix one complaint (“stories are too preachy,” “the openings all sound the same”), I had no way to know what I’d quietly broken somewhere else. The backend has 115 tests. The prompt had none, and it was the riskiest thing in the repo.

So I built the thing you build when unit tests don’t apply: an offline eval harness. A fixed matrix of story inputs, a run that generates a story for each, and a second LLM, the judge, that scores every story on ten axes. The reports get committed to the repo, and one rule turns them into a merge gate.

This is a post about giving a vibes-based feature a test suite you can actually gate on.

Why the normal tests don’t reach this

Everything else in the app is deterministic, and deterministic things are easy to assert. The idempotency-key logic, the cost caps, the word-count bands: POST the same key twice, assert you get the same job back. Done. Green check.

Story quality is different in kind. The output is a 1,300-word story, and the question isn’t “is it correct.” It’s “is it good, for a 6-year-old, in the requested mode, without being preachy, and not the same story I got last time.” None of those are === checks.

And the trade-offs hide from you. A prompt change that makes stories 5% more original might make them 10% more likely to wander off the requested age. You cannot see that in the diff. You can only see it by generating stories and grading them. That’s the whole motivation: prompt changes were the highest-risk changes in the codebase and had the weakest feedback loop. The eval closes that loop.

The harness: matrix, generate, judge, score

It’s one script, scripts/eval-stories.ts, run with npm run eval. Four moving parts.

The matrix is a fixed, committed set of inputs, not random ones, so that two runs are comparable. Eight entries spanning the product’s real range: ages 4–12, durations 5–15 minutes, the real modes (sleepy, gentle, playful, adventure), single- and multi-character casts, named and anonymous kids.

const MATRIX: MatrixEntry[] = [
  { age: 4,  duration_minutes: 5,  themes: ['kindness'],     characters: ['rabbit'],            setting: 'forest',    mode: 'sleepy',    child_name: 'Mira' },
  { age: 6,  duration_minutes: 10, themes: ['bravery'],      characters: ['knight','dragon'],   setting: 'castle',    mode: 'playful',   child_name: null   },
  { age: 8,  duration_minutes: 15, themes: ['curiosity'],    characters: ['astronaut','robot'], setting: 'space',     mode: 'adventure', child_name: 'Kai'  },
  { age: 12, duration_minutes: 15, themes: ['perseverance'], characters: ['explorer'],          setting: 'mountains', mode: 'adventure', child_name: 'Theo' },
  // …8 total
];

The matrix is deliberately load-bearing. The astronaut+robot and explorer entries are the long-duration (15-minute) ones, and they’re where length problems always show up first, so they act as canaries.

Generate runs through the real prompt path. This is the critical discipline: the eval must exercise the shipped code, not a look-alike. It imports buildStoryPrompt, buildEditorPrompt, and buildPremisePrompt straight from src/stories/. When I added premise divergence (a brainstorm step before the story), the eval threads the chosen premise through the same injection point production uses, the system prompt via buildStoryPrompt’s premise param, not appended to the user turn like an earlier prototype did. If the eval and prod disagree about how the prompt is built, the eval is grading a fiction.

// premise → story → (editor) → judge, all via imported production builders
const { system, user } = buildStoryPrompt(entry, undefined, undefined, chosenPremise);
const storyRes = await storyProvider.chat([
  { role: 'system', content: system },
  { role: 'user', content: user },
]);

Judge is a second model grading each story after it’s generated. The judge prompt is blunt about wanting the full range used, and it separates axes that are easy to conflate:

You are a demanding children's-story critic… Grade honestly and use the FULL
1–5 range: a competent-but-unremarkable story is a 3, not a 5. Craft and
originality are DIFFERENT axes: a beautifully written but familiar story must
score HIGH on narrative_quality and LOW on originality.

Ten axes:

const AXES = [
  'safety', 'constraint_adherence', 'length_compliance',
  'bedtime_suitability', 'mode_adherence', 'age_appropriateness',
  'narrative_quality', 'engagement', 'not_preachy', 'originality',
];

Two of them get special handling. originality is defended against prose polish: the rubric spells out that a “cozy forest/dusk mood piece is at most a 2 unless it does something you did not expect,” and that a stock premise (“a creature wanders a forest and comes to feel theme”) is a 1. Without that, a judge just rewards nice sentences and everything scores 4–5.

And length_compliance isn’t left to the judge’s taste at all. The judge’s guess gets overwritten by the product’s own word-count band rule:

// <50% target → 1 (fail), 50–80% → 4 (short but accepted), ≥80% → 5.
judge.length_compliance = lengthComplianceScore(wordCount, target);

The eval grades length by the same bands the app itself enforces, so the eval and the product never disagree about what “too short” means. The judge returns JSON, and because a “demanding critic” occasionally wraps its JSON in prose, there’s a robustJudge retry: on a parse failure it re-asks with a blunt “reply with ONLY the JSON” reminder before giving up on a zero row.

The variety pass catches something the per-story judge structurally cannot see. The judge looks at one story at a time, so it can’t notice that all eight stories open the same way. A model with weak imagination reuses the same premise shape regardless of the prompt, and that only shows up across the set. So there’s one extra judge call over all eight openings together:

// "A model with weak imagination reuses the same premise shape, setting, mood,
//  and opening imagery regardless of the prompt. Judge how VARIED the set is."
{ "variety_score": <1-5>, "duplicate_clusters": [["#1","#5"]],
  "repeated_motifs": ["…"], "note": "…" }

variety_score: 5 means every story feels distinct; 1 means “same template, props swapped.” This is the number that told me my stories had a house style before I could articulate what it was.

Each run produces a committed markdown report: per-axis means, the variety score, every story’s full text in a <details> block (so a human can read them, not just trust the judge), the matrix, and, crucially, the prompt SHA and git commit SHA, so any report can be traced back to the exact prompt that produced it.

Choice #1: judge across families

Here’s a trap that’s easy to walk into: use the same model to generate and judge. It’s convenient. One API key, one provider. It’s also systematically biased upward. Models have a documented tendency to prefer text that looks like their own output. If GPT judges GPT, “good” quietly starts to mean “the kind of thing GPT writes,” and your eval becomes a mirror.

So the harness runs a cross-family judge: a Claude model judging OpenAI-generated stories, or the reverse. The --gen and --judge flags pick the provider for each independently:

npm run eval -- --gen=openai --gen-model=gpt-5.4-mini \
                --judge=anthropic --judge-model=claude-sonnet-4-6

And when they are the same family, the report doesn’t hide it. It stamps a warning right at the top:

const sameFamily = generator.split('/')[0] === judge.split('/')[0];
if (sameFamily) lines.push(
  `- ⚠️ Generator and judge are the same provider family; scores carry a same-family optimism bias.`);

You can see that warning on the June reports (openai/gpt-4o-mini generating and judging). Those runs were fine for the specific thing they tested, a length tweak, where the judge’s opinion barely matters because length_compliance is computed from word count rather than judged. But the warning is there so nobody ever mistakes a same-family overall for an unbiased one. The 2026-07-01 model bake-off that moved the app to gpt-5.4-mini was run cross-family precisely so the “it wins every creativity axis” claim wasn’t a model grading its own homework.

One more piece makes this safe: the eval-only provider. The app does use OpenAI in production (for generation and TTS), but the eval reaches it through a separate path. There’s a dedicated scripts/openai-eval-provider.ts that lives in scripts/, uses raw fetch (no openai npm dependency), and exists only for evals. Its own header says so: “nothing here should pull OpenAI into the NestJS runtime.” Switching a judge or a generator for an eval never perturbs what’s shipped to users. It also carries eval-specific knowledge, like the fact that GPT-5 reasoning models reject max_tokens and need max_completion_tokens big enough to cover reasoning (plus the story).

Choice #2: the 0.3-drop revert rule

An eval that produces numbers but no decision rule is just a dashboard. The rule that turns it into a gate is stated in CLAUDE.md and applied to every prompt change:

Prompt/selection changes are gated by npm run eval and the >0.3-drop revert rule.

Run the eval before and after the change. If any axis drops by more than 0.3, the change is blocked, no matter how good it looked in isolation, unless inspection shows the drop is a clear judge artifact or a known-noisy axis. Not “the overall dropped.” Any single axis. Because the whole point is to catch the trade-off you didn’t intend: the originality tweak that quietly tanked age-appropriateness.

The 0.3 threshold sits above the noise floor but below a real regression. A ±0.1–0.2 wobble between two runs of the same prompt is normal LLM variance; 0.3+ on a specific axis is a signal.

Here’s the rule doing its job, from two committed reports on the same day. The change nudged the Length: instruction to aim at the upper end of the word band, to counter a chronic undershoot. Watch length_compliance:

length_compliance went from 2.67 to 3.17, and, this is the part the rule cares about, nothing else went down. No axis dropped at all, let alone by 0.3. Clean pass: the change ships. Had the length fix bought its +0.5 by making stories less age-appropriate (a −0.4 there), the rule would have killed it even though the overall still rose. The overall average is not the gate. The per-axis floor is.

The same rule is why my current premise-divergence, editor, and length changes are marked pending a paid-eval confirmation before merge. The checkpoint is literally “originality up, length_compliance recovered, no axis down >0.3.” The eval isn’t advisory. It’s the merge condition.

What this is not

I want to be honest about what this is and isn’t, because an eval you over-trust is worse than no eval.

It’s a single-judge score, and single judges are noisy. One model, grading once, on a 1–5 scale it compresses toward the top. Look at the reports: safety, bedtime_suitability, age_appropriateness, and not_preachy sit pinned at a flat 5.00 across every story. Those axes carry almost no signal on wholesome content; they only earn their keep as tripwires for a genuine regression. The axes that actually move (length_compliance, narrative_quality, originality) are the ones worth watching, and even they wobble run-to-run.

originality in particular has a plateau I don’t fully trust. On bedtime modes it sat around 2.75 no matter what I did, and I’m fairly sure that’s partly a rubric-and-matrix artifact plus a noisy judge, not purely the model’s fault. A cozy-dusk-forest prompt has a low originality ceiling by construction. This is exactly why the 0.3 threshold exists: small movements are inside the error bars, and pretending otherwise would have me chasing noise.

The mitigations are mitigations, not cures. The cross-family judge reduces the same-family self-preference but doesn’t remove judge bias altogether, and it does nothing for the random noise. Saving the full story text in every report lets a human override the judge, because the judge is a filter for regressions, not the final arbiter of taste. The variety pass adds a signal no single-story score can capture. length_compliance is computed rather than judged, so at least one axis is reproducible and aligned with the product rule. And a drop above 0.3 blocks the change, unless inspection shows it’s a clear judge artifact or a known-noisy axis.

That said, the point isn’t that the score is precise. It’s that a blunt, reliable smoke detector beats no alarm at all. This harness reliably tells me when I’ve made things obviously worse. It’s much weaker at telling me I’ve made things a little better, and I’ve made my peace with that, because “obviously worse” is the failure I actually ship by accident.

A full same-family run over eight stories costs about $0.01 to $0.13 depending on the model. Cheap enough to run on every prompt change, which is the only price at which a gate like this survives contact with real work.

You still can’t assert that a story is good. But you can, it turns out, notice the day it stopped being.

← all writing