Computer vision in the plan

A plan is a chain of typed steps

Everyframe is built on an engine called Renderbox. A plan is a chain of steps, and each step states what kind of data it takes and gives: video, audio, the faces a detector found, or a transcript. The output of a face detector is data like any other, so finding faces, following them from frame to frame and covering them fit in one plan. This one hides the faces in a street clip:

const [v, a] = g.open('street.mp4');         // video, audio
const faces = v.detect({ model: 'retinaface_mv2' })
               .track({ algorithm: 'bytetrack' });
g.write('redacted.mp4', v.redact(faces, { mode: 'black' }), a);

a.redact(faces, { mode: 'black' });
// error: 'redact' does not exist on an audio stream
v.redact(faces, { mode: 'blur' });
// error: blur needs a radius

The last two lines never reach a server. The names of every step and setting come from the engine itself, so a misspelled option or a step applied to the wrong kind of data shows up as an error in the code editor. We wrote about the same idea for web pages in Type-Safe HTML in TypeScript, and about the coding agents that rely on it in Our Claude Code Workflow.

Before a plan runs, the engine checks it. A plan is refused when a file format cannot hold its video, when a step gets the wrong kind of data, or when a setting is out of range. Video is read and written with FFmpeg, the open-source tool most video software uses.

What the engine sees in footage

The same steps run Everyframe's edits. The engine finds every face in every frame and follows it, hears every word with the second it starts, and picks out a product in a hand. An edit can then keep one face in frame, move a caption off a face, or put a label on the product.

A clip only has to be analysed once. Its faces and words are kept with it, so a library of clips can be searched by what was said in them. The same words let a cut land in a pause, and another model takes the background away while the speaker stays sharp.

The faces, words and products in these clips are what the engine's own models found in the footage, frame by frame.

Computer vision in production

These clients already use our engine in production.

University of Maribor, Faculty of Logistics

fl.um.si ↗

The faculty is studying how people ride e-scooters in city traffic. The engine hides faces and licence plates in its street footage, 480 hours of it under contract. The researchers upload files of up to 100 GB through a portal we built for them, described in What JT Digital built since March.

VARNOŠKA

varnoska.si ↗

VARNOŠKA is a workplace-health site for the Koroška region. For its 30 exercise routines the engine found the person in each clip, read the pose and counted the repetitions.

Lutkovno gledališče Maribor

www.lg-mb.si ↗

The puppet theatre had to hand over a recording in which people who took no part could be seen. The engine found their faces across the recording and burned the blur into the copy that left, so it cannot be undone. The Renderbox page covers all three of these uses.