Long jobs that finish exactly once

Our job runner is called planrun, because it runs plans. A plan describes one job completely: the video to cut or the model to run, the files it reads and where the result goes. planrun finds a machine that can run it, runs it, and returns one result with an attestation, a receipt of exactly what ran.

Everyframe, our video and vision engine, was its first user. AI models on infrastructure we control run through it too, and Everyframe Composer sends every ad variant it makes through it. planrun knows nothing about video. Each kind of work plugs in as a job type, and this whole page holds for all of them.

Our first worker held each render message unacknowledged until the render was done. That worked until a render ran past 30 minutes, RabbitMQ's consumer timeout. The queue closed the connection and gave the job to a second worker, which also passed 30 minutes. Long jobs looped, and the queue filled up behind them.

Raising the timeout only moves the cliff. The queue was playing two roles: it moved messages, and it was the only record of who was working on what.

We split the roles. planrun-ledger is a small SQLite database on the orchestrator, the service that hands out work. It is the one record of which jobs are running and where, and it survives restarts.

A worker that receives a message first asks the ledger for a lease, a claim on one attempt at one job that expires unless renewed on a 30-second heartbeat. With the lease granted, the worker acknowledges the message within milliseconds, so the queue is done with the job long before any timeout.

Each grant is a compare-and-swap on the job and its attempt number, so of two workers racing for one job, exactly one wins. If a worker crashes or drops off the network, its lease runs out and a sweeper puts the job back in line as the next attempt. Every job ends finished or failed, and none go missing.

Leases bring their own race. A network blip stops a healthy worker's heartbeats, its lease expires, and attempt two takes over and delivers. Then attempt one finishes too, late. The graphics cards on the fleet do not encode a video to the exact same bytes twice, so its file would look like any other video.

The settle step stops the late result. Before announcing anything, a worker writes the final state to the ledger with a second compare-and-swap keyed to its own attempt. A stale attempt loses that write and announces nothing.

A check before upload narrows the window for the file. Right before uploading, the worker asks the ledger whether it still holds the lease, and uploads nothing if it lost it. A lease can still lapse during the upload itself, and a file swapped in that moment would no longer match the hash on its receipt.

Job types cannot skip any of this. Job code gets a scratch folder, its inputs and a progress reporter, and no storage client, queue connection or lease handle, so it has nothing to upload or announce with.

Attestation means a finished job can show what produced it, meaning which plan ran, on which input files, with which AI models, and which files came out. planrun builds that record for every job, of every type.

A plan's ID is a SHA-256 fingerprint of its exact bytes, computed the same way by our SDK, the orchestrator and the worker. The worker refuses any plan whose fingerprint does not match, before job code sees it, and hashes input files as they download.

When a job finishes, the orchestrator writes a receipt: the plan's fingerprint, the hash of every input and output file, a fingerprint of each AI model used, and the worker's reported environment. If the worker reports running a different plan from the one it was sent, no receipt is written.

Receipts are canonical JSON (RFC 8785), so a delivered file can be checked against its receipt later, offline. A receipt vouches for a file from the moment we hashed it on the way in, and it names no users or customers.

A retry only helps when the next run could turn out differently, so every error falls into one of three classes.

• Permanent means the same input fails the same way every time, such as a plan that fails to compile or filter settings FFmpeg refuses. The job fails once and the sender is told why.
• Transient means the machine or the network had a bad moment, such as a storage error or a full disk. The job goes back in line for another worker.
• Fatal means the worker itself is broken, for example a graphics card that has locked up. The worker stops taking jobs and restarts clean, and its jobs return to the line when their leases run out.

Distributed systems cannot promise that a job runs exactly once, because a worker cut off by the network may still be running a job someone else has taken over. What we can promise is an exactly-once effect. However many times a job is delivered or run, one result is stored, one message goes out, and the app that sent the job sees one answer.