Long jobs that finish exactly once

A message queue like RabbitMQ gives you at-least-once delivery. That means every message reaches a worker, and some messages reach one more than once. The queue only forgets a message when the worker acknowledges it, and a worker that goes quiet gets its messages handed to someone else.

Our first worker held each render message unacknowledged for the whole render and acknowledged it at the end. That was the simple design. It worked until a render ran longer than 30 minutes, which is RabbitMQ's consumer timeout. The queue closed the connection, put the job back, and a second worker started the same render from the start. That one also passed 30 minutes. Long jobs looped, and the queue filled up behind them.

Raising the timeout only moves the cliff. The real problem was that the queue held two jobs at once. It moved messages, and it was also the only record of who was working on what.

We split those two jobs apart. planrun-ledger is a small SQLite database on the orchestrator, the service that hands out work. It is the durable in-flight ledger, the one record of which jobs are running and on which machine. It survives restarts, and every change to a job goes through it.

A worker that receives a message first asks the ledger for a lease. A lease is a claim on one attempt at one job, and it expires unless the worker keeps renewing it. The worker renews every lease on a 30-second heartbeat. Once the lease is granted, the worker acknowledges the message within milliseconds. We call this ack-early. The queue is done with the job long before any timeout, however long the render takes.

Each grant is a compare-and-swap on the pair of job and attempt number. The database moves the job from queued to running only if it is still queued at that exact attempt. Two workers can race for the same job and exactly one of them wins.

• If a worker crashes or loses its network, its lease runs out. A sweeper on the orchestrator puts the job back in line as the next attempt number.
• Retries wait 2, then 4, then 8 seconds. When the attempts run out, the job is marked exhausted and the app that sent it gets a failure message.
• Every job ends in one of two places, finished or failed. None of them go missing.

Leases bring their own race. Picture a network blip between a worker and the orchestrator. The worker is healthy and keeps rendering. Its heartbeats stop arriving, so its lease expires and a second worker takes over as attempt two. Attempt two finishes and delivers. Then attempt one finishes too, late. Without a guard it would upload its file over the delivered one. Our graphics cards do not encode the same video to the exact same bytes twice, so nobody would ever spot the swap.

Two guards close this. The first is the settle step. Before a worker announces any result, it writes the final state to the ledger with another compare-and-swap, keyed to its own attempt number. A stale attempt loses that write and announces nothing. The second guard is the fenced upload. Right before uploading, the worker asks the ledger again whether it still holds the lease. If it lost the lease, it uploads nothing. If the orchestrator cannot be reached at that moment, the worker trusts only its own lease deadline, and that deadline moves forward only on heartbeats the queue confirmed it received.

The result is written to the ledger before it goes out on the queue. If a worker dies between those two steps, the sweeper sends the message later from the stored copy.

New kinds of work cannot skip any of this. The code for a job type receives a scratch folder, its inputs and a progress reporter. It gets no storage client, no queue connection and no lease handle. It has nothing it could upload or announce with, so the fence holds for every job type we add.

A retry only helps when the next run could turn out differently. So every error a worker can raise is sorted into one of three classes, and the class decides what happens next.

• Permanent means the same input will fail the same way every time. A plan that fails to compile, video filter settings that FFmpeg refuses, a model error that repeats on every run. The job fails once and the sender is told why. Before bad filter settings were filed here, a batch of identical broken jobs ran up to three times each.
• Transient means the machine or the network had a bad moment. A storage error, a full disk, a job that waited too long for a free slot and never started. The ledger puts the job back in line for another worker.
• Fatal means the worker itself is broken, for example a graphics card that has locked up. The worker stops taking new jobs and shuts down so it can restart clean. Its jobs return to the line when their leases run out. A single bad model error stays Permanent, because restarting the machine would only rerun it into the same failure.

Distributed systems cannot promise that a job runs exactly once. A worker cut off by the network may still be running a job that someone else has already taken over. What we can promise is an exactly-once effect. However many times a job is delivered or run, one result is stored, one message goes out, and the app that sent the job sees one answer.

Two more layers keep duplicate work cheap. Each worker keeps dedup, a short memory of the job ids it is running or has just finished, so a repeated delivery is dropped before any work starts. The ledger refuses a lease for a job that already finished, so the same check holds across restarts. Admission gates sit in front of the work. A worker runs only as many jobs as it has slots for, and it checks for enough disk to let every running job fill its output at once. A job that no machine on the fleet can run fails as soon as it arrives, with the missing capability named in the error.

renderbox, our own video and vision engine, was the first tenant. The runner now also serves AI models we host ourselves as jobs on the same fleet. everyframe-composer, our tool for making video ads in bulk, renders nothing itself. It sends every variant to the fleet as a job and relies on this runner to bring each one back once.