My reasoning model thought itself out of an answer

I swapped the bedtime-story app over to a smarter model, gpt-5.4-mini, because it won every creativity axis in my bake-off. I was pleased with myself for about a day.

Then the stories started coming back short. A story I’d asked to be 600 words would arrive at 150. Some were worse: cut off mid-sentence, “…and then the little fox” and nothing after it.

And here’s the part that sent me hunting in the wrong place for an afternoon: no error. HTTP 200 every time. The JSON parsed fine. The usage numbers looked normal-ish. There was just less story than I’d asked for.

The symptom that lied

Downstream of generation I have a word-count guard that fails a story if it lands way under target. So the user-facing symptom was “generation failed, try again.” A truncated-but-valid response looks exactly like a lazy model, and I very nearly went off to “fix” my prompt to beg harder for length.

It wasn’t a lazy model. I’d starved it.

Read the finish_reason, then read the usage

The thing that cracked it was two fields I’d been ignoring: finish_reason and usage.completion_tokens.

const json = await resp.json();
const choice = json.choices[0];
console.log(choice.finish_reason);         // "length"   <-- not "stop"
console.log(json.usage.completion_tokens);  // ~4096, exactly my cap

finish_reason: "length" means the model didn’t choose to stop. It hit the ceiling and got cut off. And completion_tokens was pinned right at my configured cap of 4096, the old chat-model default I’d carried over without thinking.

But the visible story was only ~150 words, call it 200 tokens. So where did the other ~3,900 completion tokens go?

They went into thinking.

On a GPT-5 reasoning model, completion_tokens bills the reasoning tokens plus the visible answer, and both come out of the same budget. The model reasoned hard about my story for ~3,900 tokens, hit the 4,096 wall, and had roughly 200 tokens left to actually write with. The story I got was the sad little tail end of an exhausted budget.

The cap I set wasn’t “how long is the answer.” It was a fuel tank the model had to spend on thinking before it wrote a single word of prose. I’d sized the tank for a car that doesn’t idle, then handed it to one that warms up for a few thousand tokens first.

The two-part fix

Part one: send the right parameter. On the Chat Completions API, GPT-5 models reject max_tokens outright. They require max_completion_tokens (the Responses API uses max_output_tokens). Sending the old one gets you an API error, which at least is loud. The older models I still target (gpt-4o, gpt-4.1) accept max_completion_tokens too, so switching to it is safe everywhere: one code path, no per-model branching.

body: JSON.stringify({
  model: this.model,
  messages,
  temperature: opts?.temperature ?? 0.8,
  top_p: opts?.topP ?? 0.95,
  // GPT-5 models reject `max_tokens` and require `max_completion_tokens`;
  // gpt-4.1/4o accept it too, so it's safe for every model we target.
  max_completion_tokens: opts?.maxTokens ?? this.maxTokens,
}),

Part two: make the ceiling generous. This is the part that actually fixes the short stories. The budget has to fit reasoning plus the whole story, not just the story. I bumped LLM_MAX_OUTPUT_TOKENS from 4096 to 16384:

// src/common/config.ts
LLM_MAX_OUTPUT_TOKENS: z.string().default('16384'),
// GPT-5 max_completion_tokens covers reasoning + story; keep generous or long
// stories starve

16384 sounds wildly large for prose that’s only ~800 words (~1,100 tokens). It is, for the prose. But the reasoning tokens are the point: give the model room to think for several thousand tokens and still have plenty left to write the full story. Too low and it thinks itself into a corner and gets cut off.

While I was in there, I added a warning so a truncation can never hide again:

if (choice?.finish_reason === 'length') {
  logger.warn(
    { model: this.model, outputTokens: json.usage?.completion_tokens },
    'openai: response truncated (hit max_completion_tokens)',
  );
}

Now a truncation leaves a trail instead of masquerading as a short story.

Why it surfaced somewhere else entirely

The reason this took a detour is that the failure showed up through a component that had nothing to do with tokens. The word-count guard:

// src/stories/word-count.ts
export function classifyLength(actual: number, target: number): LengthClass {
  const ratio = actual / target;
  if (ratio < 0.25) return { kind: 'fail' };            // way too short → reject
  if (ratio < 0.8)  return { kind: 'accept', warning: 'short' };
  return { kind: 'accept' };
}

A token-starved story is, by definition, way under target. A 600-word request coming back at around 140 words is a ratio below 0.25, right into the fail band. So the job failed with “too short,” and the guard looked like the problem. It wasn’t. It was the messenger, doing exactly its job, catching a bad generation. The bad generation was upstream, caused by the token ceiling, and invisible at the HTTP layer.

That said, the guard earned its keep here: without it, I’d have shipped truncated stories silently. The point isn’t that a downstream check is noise, it’s that a silent failure in one layer resurfaces as a loud, misleading failure in the next one down. If I’d trusted the loud symptom, I’d have spent the afternoon tuning word-count bands that were never wrong.

What I’d tell you to take from this

On a reasoning model, the output budget is not “how long is the answer.” It’s reasoning plus answer, and a budget carried over from a chat model will quietly starve long generations. Size it for both, and lean generous: you pay for tokens actually produced, not for the ceiling, and a re-generated story costs more than headroom you never touch.

The parameter changed too: max_tokens became max_completion_tokens, and truncation comes back as a valid 200, never an error. The clearest runtime signal is finish_reason: "length" with completion_tokens sitting exactly at your cap; the usage object also breaks out reasoning spend in completion_tokens_details.reasoning_tokens if you want to confirm where the budget went. Log it loudly, or a cut-off answer stays indistinguishable from a lazy one.

The model wasn’t lazy. It was thinking so hard it forgot to leave room for the story.

← all writing