My model wasn't boring, my ruler was bent

I built a bedtime-story app for my kid. A parent fills in a short form (age, characters, a theme, a mode: sleepy, gentle, adventure, thrill) and an LLM writes a complete story. It worked. It just kept writing the same story.

Not word for word. But every single one opened in the same place, in the same key:

“The forest was quiet at dusk, the moss soft underfoot, the ferns curling in the last of the light…”

That, before any character showed up. Swap the rabbit for a dragon, swap the theme from kindness to bravery: didn’t matter. Same evening-forest establishing shot, same hush, same mood.

So I did the responsible engineer thing. I diagnosed the anchor, shipped two fixes, and built an LLM-judge harness to prove the stories had gotten more original.

Then the metric flatlined. Originality crept up from about 2.25 out of 5, then bounced between about 2.75 and 3.25 and would not reliably climb higher, no matter what I did to the prompt. I spent an evening chasing that plateau. The punchline is that the plateau was mostly an artifact of how I was measuring, not a ceiling on the model. This is a post about two things that turned out to be one lesson: the sameness bug is easy to fix once you find the anchor, and the harder skill is knowing when the broken thing is your metric, not your model.

The pipeline, briefly

The app is a one-shot text generator. No chat, no images, no streaming. Per story the backend roughly:

1. Moderates the inputs.
2. Builds a prompt and generates the story.
3. Runs a light editor pass.
4. Checks the word count is in range.
5. Moderates the output, then titles, tags, and saves.

The prompt in step 2 is a big system prompt with one important ingredient: a few-shot example. To teach the model what “good” sounds like, I inline a real reference story, chosen from a small corpus by nearest age and register:

// src/stories/corpus.ts: pick the closest reference story, deterministically
export function selectExample(q, corpus = loadCorpus()) {
  // 1. smallest age-band distance  2. register/tone affinity
  // 3. closest word count          4. slug (stable tie-break)
  scored.sort((a, b) =>
    a.ageDist - b.ageDist || a.regScore - b.regScore ||
    a.wordDist - b.wordDist || a.s.slug.localeCompare(b.s.slug));
  return scored[0].s;
}

That example is genuinely useful. It lifts prose quality and it reliably models a calm ending. But it turns out a few-shot example doesn’t just teach voice. It teaches shape.

The tell

The stories didn’t share a plot. They shared an opening move: a wide landscape shot, at dusk, in a forest, with soft moss and curling ferns, and only after all that scene-dressing did a character finally appear.

When I ran a batch through my early eval and had a judge look at all the openings side by side, it said the quiet part out loud:

“Four of eight stories open in an evening forest with moss and ferns.” Repeated motifs: forest at dusk; moss described as soft/carpet/blanket; ferns as landmark detail; gentle sensory scene-setting before the character appears.

That’s the tell. A model with a weak imagination doesn’t vary the premise. It reuses one template and swaps the props.

Two anchors pulling the same way

There were two contributors stacking.

The few-shot example was anchoring the house style. My hand-written fallback exemplar, “Pip and the Firefly,” literally opens:

“Pip lived in a little burrow at the edge of the forest, where the moss grew softest and the ferns curled close like green blankets… One quiet evening, just as the first stars were peeking through the leaves…”

I put that story in the prompt to model voice and a calm ending. But an LLM doesn’t read a few-shot example as “adopt this register.” It reads it as “produce more of this,” and the most salient, most copyable thing about it is the opening shot: forest, dusk, moss, ferns. I was handing the model the exact template I then complained it kept reusing.

The model’s own bedtime-story prior points the same way. Ask any general model for “a bedtime story” and its highest-probability opening is a soft, setting-first mood piece. My example wasn’t fighting that prior. It was reinforcing it. Two forces, same direction, no counter-weight.

And there was a deeper structural issue underneath both. My prompt’s steering was almost entirely negative: avoid worn templates, don’t state a moral, don’t copy the example. Telling a single-pass model to “be fresh” without giving it something concrete to be fresh about just lands it on the first, most-probable, most generic idea. Negative steering removes the worst option. It doesn’t produce a better one.

Fix 1: give it something to be fresh about

The structural fix is a cheap extra LLM call before writing. Instead of asking one model in one pass to invent-and-write a story (where it defaults to its most probable idea), I first ask it to brainstorm 4-5 genuinely distinct premises, then pick one and write that.

// src/stories/premise.ts: the brainstorm prompt (abbreviated)
const system =
`You are brainstorming premises for a children's bedtime story (${mode}).
Produce 4–5 GENUINELY DISTINCT one-line premises (different core situations,
not five variations of one idea)...
Avoid stock setups: no "<character> wanders or explores a place and feels calm",
no "a quest to fetch a magic object", no generic "learns about <theme>" arc.
Give each premise a concrete hook (an unusual object, an odd predicament, a
specific problem to solve, or a genuine surprise), not merely a mood, a place,
or a feeling.`;

Three decisions here are load-bearing.

Pick by jobId % N. Don’t let the model self-select “the best.” Asking the model to choose its favorite just biases it back toward the safe first idea, the exact thing we’re trying to escape. A deterministic rotation over the list keeps the spread. As a bonus, because regenerate mints a fresh job with a fresh premise list and a different index, hitting regenerate is very likely to give a genuinely different story.

The premise is transient. It’s never persisted. It’s passed as a fourth argument to buildStoryPrompt(params, example, series, premise), not stored on stories.params. That snapshot is frozen and copied forward on regenerate. If the premise lived there, every regenerate would be pinned to the original premise, defeating the whole point.

It’s best-effort and fail-open. A failed premise call logs and generation continues without one. It’s gated by PREMISE_ENABLED and skipped entirely for series episodes (those must continue an arc, not diverge). And it’s treated as untrusted input in the prompt: delimited as story material and named in the safety block, because it’s model-generated text derived from user inputs.

Cost: about $0.001 per story. Trivial next to generation and TTS.

Fix 2: ban the establishing shot

The premise stage varies what the story is about. It doesn’t stop the model from wrapping every premise in the same dusk-in-the-forest opening. So I added an explicit opening rule, plus a rotation so even the kind of opening varies:

// src/stories/prompt-builder.ts
const OPENING_MOVES = [
  'with a line of dialogue: a character says something aloud in the first sentence',
  'on the main character already in the middle of doing something with their hands or body',
  'in the middle of a small problem that is already happening to the character',
  'on an unexpected small sound or object that makes the character stop and pay attention',
];

// deterministic, prompt-derived: same inputs → same opening move, but varies across prompts
const openingSeed = (characters + themes + setting).split('')
  .reduce((a, c) => a + c.charCodeAt(0), 0);
const openingMove = OPENING_MOVES[openingSeed % OPENING_MOVES.length];

And the rule itself, which names the exact anti-pattern:

const openingRule =
`Opening & variety:
- Introduce a CHARACTER in the very first sentence. Do NOT open by describing the
  setting, the weather, or the time of day before anyone appears (no "The forest
  was quiet after sunset…" establishing shot).
- Open ${openingMove}.
- "Forest/woods at dusk, soft moss, curling ferns, thinning evening light" is an
  overused opening default; avoid it as scene-dressing.
- The example above is a reference for VOICE and a calm ending only; do not
  borrow its setting, opening shape, imagery, or character names.`;

That last bullet is the counter-weight the prompt was missing. It tells the model explicitly what the example is for and what not to copy from it, so the few-shot stops leaking its shape.

Proving it

I refuse to ship a prompt change on vibes. “It feels more varied now” is how you convince yourself of anything. So before tuning, I built an LLM-judge harness (scripts/eval-stories.ts).

The shape:

• A fixed matrix of 8 prompts (different ages, durations, modes, characters, settings) so a run is comparable across code versions.
• A per-axis judge. Each story is scored 1-5 on ten axes: safety, constraint adherence, length compliance, bedtime suitability, mode adherence, age appropriateness, narrative quality, engagement, not-preachy, and the new one, originality.
• A cross-family judge. The generator is gpt-5.4-mini; the judge is Claude Sonnet. A model judging its own output grades itself optimistically, so the harness even prints a warning when the two share a provider family.
• A separate cross-story “variety” pass. The per-story judge sees one story at a time, so it’s blind to sameness across the set. A second pass shows it all eight openings together and asks how varied they are as a set. This is what surfaced the “four of eight open in a forest” finding.

The critical rubric choice was to separate craft from originality, so polished prose couldn’t inflate the freshness score:

"narrative_quality": <1-5: prose craft and structure ONLY, this is where good writing is rewarded>,
"originality": <1-5, judging PREMISE / PLOT / CHARACTER freshness only, ignoring prose polish.
  5 = a premise uncommon in children's books and a plot that genuinely surprised you.
  3 = a familiar setup done competently.
  1 = a stock premise, a telegraphed arc, and interchangeable characters.
  A cozy forest/dusk mood piece is at most a 2 unless it does something you did not expect>

Read that last clause again. A cozy forest/dusk mood piece is at most a 2. I wrote that deliberately, to stop the judge rewarding exactly the template I was fighting. Hold that thought. It becomes the whole twist.

The results, then the wall

The fixes did something measurable. The cross-story variety pass went from 2-3/5 up to 4/5 after the anti-template rules, and the judge’s complaints softened from “four stories open in a forest” to the much milder “opening beat of ‘small sound causes pause’ recurs a bit.”

The openings genuinely diversified:

Before: “Four of eight stories open in an evening forest with moss and ferns.” (variety 3/5)

After: “Settings and tones vary well; opening beat of ‘small sound causes pause’ recurs too often.” (variety 4/5)

But the originality axis, the headline number, barely moved:

It bounced around 2.75-3.25 and would not climb. I strengthened the “ban stock setups” wording. I rewrote the premise-injection phrasing. I reframed the editor from “rewrite” to “sharpen with the lightest touch” (a known voice-flattener, worth doing regardless). Every run came back around 2.75-3.25. I was starting to conclude the model just had a hard ceiling on originality.

The twist: the ruler was bent

I almost accepted “that’s the model’s ceiling.” Instead I sat down and pulled the per-story originality scores apart by matrix entry. Three things fell out, and none of them were about the model.

Here’s the metaphor that finally made it click for me. I’d built a ruler to measure the model, and I was frustrated the model kept coming up short. But a bent ruler doesn’t measure a short board. It just reads short. I was reading my own bend.

Half the matrix was structurally capped at about 2. My 8-entry matrix had four forest-and-bedtime entries: rabbit/forest/sleepy, fox and owl/forest/sleepy, unicorn and elf/forest/gentle, fox/forest/gentle. And my own rubric penalizes cozy forest/dusk defaults (scoring them at most a 2 unless they do something unexpected). So four entries were biased toward low originality because the matrix over-represented calm forest setups and the rubric penalized cozy forest/dusk defaults, not because those stories were bad, but because I’d told the judge to score that entire genre low. Do the arithmetic: even if the other four scored a perfect 5, the mean caps at (4×2 + 4×5)/8 = 3.5. The 2.75 “plateau” wasn’t a wall the model kept hitting. It was the average of a rubric I’d rigged against half my own test set. I was measuring my matrix, not my model.

The single judge was noisy, about ±0.5 run to run. The cross-story variety score bounced between 3/5 and 4/5 on near-identical story sets; the per-axis means wobbled by about 0.5 with no code change at all. A ±0.5 instrument is not reliable for detecting the ~0.2 improvements I was trying to A/B in a single comparison. I was reading tea leaves in sensor noise and calling it a plateau.

The thing I was optimizing was in tension with the product. Bedtime stories are supposed to be low-arousal: simple characters, a gentle rhythm, a soothing resolution. But “originality” as the judge scores it rewards surprise and distinct, spiky characters. Chasing originality-4 on a sleepy bedtime story is chasing a contradiction, because a maximally surprising story is a bad bedtime story. The right target for bedtime mode is more like “fresh but very calm” (around 3.5), and you only push originality-4 on adventure or thrill modes. My single global average was blending two modes that should have had different targets, then I was frustrated it wouldn’t hit one number.

None of these are model problems. All three are measurement problems. The model was fine. My ruler was bent, jittery, and pointed at the wrong target.

What actually moves originality

For completeness: once I understood the metric, I could reason about the real lever instead of grinding prompt tweaks. Vocabulary and prose rewrites should not move this axis much, because the rubric instructs the judge to focus on premise, plot, and character freshness rather than prose polish. The thing that genuinely lifts premise and plot freshness is a quiet plot turn: build a gentle rhythm and then subtly subvert it. The helper turns out to need help. The apparent problem is the wrong problem. The object’s purpose changes. That adds the surprise the rubric rewards while staying calm enough for bedtime.

That said, I deliberately did not grind on that with more single-run experiments at about $0.23 per 8-story eval run. The correct next step isn’t another prompt tweak. It’s a mode-aware eval: per-mode targets (bedtime is fresh plus calm; adventure is surprising plus high-stakes plus safe), calmness as its own axis, a bigger matrix (16 to 24 stories so the noise averages out), and per-story failure inspection. Fix the instrument first. Otherwise I’d have wasted a week tuning against a ruler I already knew was broken.

What I’d tell you to take from this

• A few-shot example teaches shape, not just voice. I added an example to model prose quality and a calm ending; the model dutifully copied its opening template too. If you inline an exemplar, tell the model explicitly what it’s for and what not to borrow.
• Negative steering removes bad options; it doesn’t create good ones. “Avoid clichés” plus a single pass lands on the most-probable idea. Give the model a concrete thing to be fresh about (a divergence step that brainstorms, then picks) and don’t let it self-select the “best,” because that’s just the safe idea again.
• When a metric plateaus, suspect the metric. Before concluding the model can’t do better, pull the scores apart. Mine was capped by construction, noisy, and pointed at the wrong target. The plateau was in the ruler.
• A single LLM judge is a noisy instrument. ±0.5 run to run is fine for catching a regression, useless for A/B-ing a 0.2 improvement. Bigger n, cross-family judge, and don’t over-read one run.
• Don’t optimize a number that fights your product. Maximizing originality can make a bedtime story worse. The metric has to encode the product’s actual goal, which for me means per-mode targets and calmness as its own axis, not one global average blending sleepy and adventure.

The point isn’t that building the eval was a mistake. Building the eval was the right instinct, and I’d do it again first thing. The mistake was trusting it before checking whether it could even measure the thing I cared about. Measure the model, sure. But measure the ruler first.

← all writing