The queue was in the database the whole time

My bedtime-story app calls an LLM to write a full story, and that call takes 60 to 120 seconds. Far too long to hang an HTTP request on. So generation runs as a background job: POST /stories/generate returns 202 with a job_id, the client polls, and a worker does the slow work off to the side.

The textbook reflex here is “add a queue.” Redis, BullMQ, a broker, maybe a separate worker process. I didn’t add any of that.

The whole thing runs on an in-process, single-worker FIFO with a small bounded buffer, and correctness comes entirely from conditional SQL updates against the one Postgres database I already had. The trick is small and old: model the job as a state machine, and make every transition a conditional UPDATE ... WHERE status = <expected>. Postgres row locking makes “only one actor may advance this job” fall out for free.

This is a post about not reaching for infrastructure you don’t need, and about why a three-line WHERE clause is a better concurrency primitive than it gets credit for.

Why there’s a worker at all

The app is a one-shot LLM text generator. A generation is slow and expensive, so the design carries a few load-bearing constraints that matter for everything below:

• Generation is asynchronous and polled. POST /stories/generate returns 202 { job_id }, and the client polls GET /stories/jobs/:job_id. No SSE, no streaming, no long-held connection. A 90-second call must not depend on a socket staying open.
• Generation is idempotent. The request carries an Idempotency-Key, so retrying the same key returns the existing job and never re-bills the LLM. (That’s a single INSERT ... ON CONFLICT DO NOTHING RETURNING on the claim, separate from this post.)
• Every LLM call costs real money. So the one thing the system must never do is bill twice for the same job. That’s the correctness bar the whole design is measured against.

So: a request comes in, a story_jobs row is created with status = 'pending', and the worker picks it up, runs the slow pipeline, and marks it done. The job’s lifecycle is a tiny state machine:

pending ──► running ──► completed
   │           │
   └───────────┴──────► failed

Four states. And the only interesting question is: who is allowed to move a job from one state to the next, and what happens when two actors try at once?

The “do I need a queue?” question

Here’s the honest scale of this thing. It’s a personal app for one household. Concurrent generations are rare, and I process them one at a time on purpose: concurrency 1 bounds LLM spend and keeps the provider happy. The in-memory buffer is capped at 8; the 9th enqueue gets a 429.

// stories.module.ts: the entire "broker"
{ provide: JobQueue, useFactory: () => new JobQueue<number>({ max: 8 }) }

Before adding Redis, it’s worth asking what it actually buys you. Roughly three things: durability across restarts, so a queued item survives a crash; multi-process fan-out, so several workers on several boxes pull from one queue; and the fancy stuff, scheduling, retries, backoff, priorities.

Now check which of those I need. Durability? The job is already durable. It’s a row in Postgres. The in-memory buffer is just a pointer to work; if the process dies, the row is still pending (or running), and the sweeper will notice. Multi-process fan-out? I run one process and want concurrency 1 anyway. Retries and priorities? Not for a one-user bedtime app.

So a Redis queue would be a durable copy of state I already store durably, plus a second system to run, monitor, and keep consistent. When your database is already the source of truth, putting the queue somewhere else means you now have two places that can disagree.

The in-memory FIFO is genuinely tiny:

// job-queue.ts (abbreviated): a bounded FIFO with concurrency 1
export class JobQueue<T = number> {
  private readonly buf: T[] = [];
  private running = false;

  enqueue(item: T) {
    if (this.buf.length >= this.opts.max) throw new TooManyRequests(); // -> 429
    this.buf.push(item);
    if (this.handler && !this.running) void this.loop();
  }

  private async loop() {
    this.running = true;
    while (this.buf.length && this.handler) {
      const item = this.buf.shift()!;
      try { await this.handler(item); } catch { /* worker logs */ }
    }
    this.running = false;
  }
}

running is a plain boolean. There is exactly one loop, so exactly one worker draining jobs in order. No locking needed inside the process, because there’s only one consumer.

The interesting concurrency isn’t between two workers. It’s between the worker and the sweeper, and that lives in the database.

The state machine, in SQL

Every transition is a conditional update: move this job to the next state only if it’s currently in the state I expect. In Drizzle, promoting pending → running (the worker claiming the job) looks like this:

// job-worker.ts
const promoted = await db
  .update(storyJobs).set({ status: 'running' })
  .where(and(eq(storyJobs.id, jobId), eq(storyJobs.status, 'pending')))
  .returning();

if (promoted.length === 0) {
  logger.warn({ jobId }, 'worker: job not in pending, skipping');
  return; // someone/something already moved this job; don't touch it
}

The atomic safety comes from the conditional UPDATE ... WHERE status='pending', not from .returning(). Postgres takes a row lock, re-checks the WHERE under the lock, and only writes if the row still matches. Two actors racing to promote the same job cannot both write. There is no read-then-write window to slip through, because the read and the write happen in the same locked statement. .returning() just reports the outcome back to the caller: a non-empty result means I won the claim, an empty result means I lost and should bail out.

Completing running → completed has the same shape:

const completed = await db.update(storyJobs)
  .set({ status: 'completed', storyId: storyRow.id, completedAt: new Date() })
  .where(and(eq(storyJobs.id, jobId), eq(storyJobs.status, 'running')))
  .returning();

if (completed.length === 0)
  logger.warn({ jobId, storyId: storyRow.id },
    'worker: sweeper raced, story orphaned, job remains failed');

Only complete the job if it’s still running. If something else already moved it out (the only candidate is the sweeper, below), the update matches zero rows, and I know I lost the race. So I log it, rather than pretending I won.

Failing running → failed uses the same guard:

private async failJob(jobId: number, errorText: string) {
  await db.update(storyJobs)
    .set({ status: 'failed', errorText, completedAt: new Date() })
    .where(and(eq(storyJobs.id, jobId), eq(storyJobs.status, 'running')));
}

Every write to status is guarded by the status it expects to find. The row lock serializes the writers; the status predicate is the guard that turns the loser’s update into a no-op. You never hold a lock across the 90-second LLM call; you take the row lock only for the microseconds of each UPDATE.

The sweeper, and why it can’t corrupt a completing job

Processes die. If the worker crashes or is redeployed mid-generation, a job is left stuck in running forever, and the client polls something that will never finish. So a periodic sweeper fails anything that’s been in flight too long:

// sweeper.service.ts: runs on module init, then every 5 minutes
async sweepOnce() {
  const mins = loadConfig(process.env).sweeperThresholdMin; // SWEEPER_THRESHOLD_MIN, default 8
  await getDb().execute(sql`
    UPDATE story_jobs
       SET status = 'failed', error_text = 'interrupted', completed_at = NOW()
     WHERE status IN ('pending','running')
       AND created_at < NOW() - (${mins} * INTERVAL '1 minute')
  `);
}

There are two distinct situations here, and it’s worth keeping them apart. The one the sweeper exists for is crash recovery: the worker died or was redeployed, so a running row is truly orphaned, and the sweeper reclaims it by failing it. That case is uncontended, there’s no live worker to race.

The scary-sounding one is the live slow-worker case: the worker is still alive and about to complete, but the job has aged past the threshold, so the sweeper decides it’s stale at the exact moment the worker is about to complete it. Now the sweeper and the completion are genuinely competing. Doesn’t the sweeper clobber a story that was one millisecond from success?

No. The worker’s completion requires status = 'running', and the sweeper’s WHERE status IN ('pending','running') handles pending and running rows separately: for a job that’s mid-completion, the row is running, so both statements are contending for the same running row. Postgres serializes them on the row lock. Only one can win. Walk the two orderings.

Ordering A, worker completes first. The running → completed update takes the row lock, flips the row to completed, commits. The sweeper’s WHERE status IN ('pending','running') now matches zero rows for this job. Nothing happens. Clean success.

Ordering B, sweeper fires first. The sweeper flips the row to failed and commits. A moment later the worker finishes its LLM work and runs its completion update, but WHERE status = 'running' now matches zero rows, because the row is failed. So completed.length === 0, and the worker logs “sweeper raced, story orphaned, job remains failed.”

There is no interleaving where both writes land, because each one requires the row to be running, and the first writer to commit leaves it not-running. The row lock makes them take turns; the WHERE clause makes the loser a no-op.

Why the worst case is benign

In Ordering B, the worker had already generated the story and inserted the stories row before it tried to flip the job to completed. So the failed job now points at a real, finished story the user will never see. That’s the orphan.

Is that a problem? Check it against the one bar that matters, no double billing:

• The LLM was called once. The money was already spent. The sweeper racing the completion doesn’t cause a second generation; it just means the one story we paid for gets orphaned instead of surfaced.
• The client, seeing a failed job, may retry, but retry goes through the idempotency layer. A new key means a genuinely new story; the same key returns the existing (now-failed) job. Either way the orphan is never regenerated to “rescue” it. It just sits there, harmless, a few KB of wasted text in a table.

So the worst outcome of the nastiest race in the system is: we paid for one story that nobody reads. No corruption, no double charge, no inconsistent job state, no crash. Given how rare it is (you need a redeploy or crash to land in the milliseconds between story-insert and job-complete, past the 8-minute staleness threshold), that’s a trade I’ll take every day over running a broker.

And critically, the safety doesn’t depend on tuning the threshold. The sweeper could be aggressive or lazy; the conditional WHERE clauses mean it can never corrupt an in-flight completion regardless. The threshold only tunes how long a dead job lingers before the user gets an error, not whether the race is safe.

Why the row lock is enough

The objection I always hear: isn’t UPDATE ... WHERE a read-modify-write? Couldn’t two of them interleave?

No, and this is the crux. Under Postgres’s default READ COMMITTED isolation, Postgres acquires a row-level lock on each matching row and re-evaluates the WHERE predicate against the locked, current version before writing. Two concurrent UPDATE ... WHERE status='running' statements against the same row don’t both proceed: the second blocks on the first’s lock, and when it unblocks it re-checks WHERE, sees status is no longer 'running', and updates zero rows.

Think of the status column as a turnstile that only one person can push through at a time. The lock decides who reaches it first; the WHERE clause is the arm that’s already swung shut on everyone behind them. This is the same primitive as an optimistic-concurrency WHERE version = ? compare-and-swap, except the “version” is the meaningful status column itself. No extra column, no SELECT ... FOR UPDATE dance, no advisory locks. The state machine is the lock.

The honest trade-offs

I’m not claiming this scales to a job platform. It doesn’t, and here’s exactly where it stops:

• Single process means a single point of failure and no horizontal scale. One box runs the worker. If it’s down, nothing generates until it’s back. For one household that’s fine; the sweeper turns “was down for a while” into clean failed jobs the user can retry, not corruption. If I needed multiple worker boxes the in-memory FIFO wouldn’t coordinate them, but the conditional-update claim still would: I’d swap the buffer for SELECT ... FOR UPDATE SKIP LOCKED polling and keep the exact same promote/complete/fail guards. The concurrency model survives; only the “how do workers find pending rows” part changes.
• The in-memory buffer isn’t durable, deliberately. If the process dies with 3 jobs buffered but not yet running, those pointers are lost. But the jobs aren’t: they’re still pending rows, and the sweeper fails them past the threshold so the client isn’t left polling forever. Durable job, ephemeral pointer.
• The 429 at buffer-full is a real limit. Bound of 8. For this app, hitting it means something’s wrong (nobody generates 8 bedtime stories at once), so shedding load is the right call. A busier app would need a real queue depth, which is the point at which you have outgrown this.
• Sweeper granularity is coarse. 5-minute interval, 8-minute threshold, so a dead job lingers up to about 13 minutes before the user sees an error. Fine for bedtime; not fine for a checkout flow.
• The threshold has to clear the worst-case queue drain. Because the sweeper also fails stale pending rows, the threshold must be sized comfortably above the longest a job can sit waiting behind a full buffer, or the sweeper will kill jobs that were only ever queued, not stuck.

That said, I don’t want to sell this as a universal answer. For plenty of apps a broker really is the right call: high job volume, genuine multi-box fan-out, retries and priorities you don’t want to hand-roll. The point isn’t that queues are overkill; it’s that this pattern fits when your job count is low, your database is already your source of truth, and single-process operation is acceptable. That covers a huge number of apps that reflexively install Redis anyway.

If the job is a row, your database already remembers it exists, and a three-line WHERE clause already knows how to keep two actors from touching it at once. Sometimes the queue was in the database the whole time.

← all writing