Everyframe is the public name of our engine for making video and finding things in it. It started on 31 March inside our work for the GZS innovation awards, as a library in TypeScript, the language our web apps are written in. The library let us describe each video in code and turned that description into commands for FFmpeg, the open-source video program. Early in May we rewrote the engine's core in the programming language Rust, and that version rendered the GZS award videos.
Today the engine runs on machines we control, and programs send it work through a code library published on npm, where JavaScript developers share their code. At least seven JT Digital apps use it. It renders every video Everyframe and jt-cut make, finds the faces that huluma covers, and made the VARNOŠKA routines.
Five of the apps that send the engine work, led by Everyframe, our main product.
Video made in code
Since September the engine also does motion design. A film is written as a TypeScript program, with layers that move, blur and sit inside each other, and the machines render it. The film that looped on our stand at GZS's innovation day is now one such program, and the engine renders all 20 of its scenes within 2 pixels of the hand-built original.
The same code can find the beats in a song and place the cuts on them. In one test it found the drop (the moment the full beat comes in) 12.1 milliseconds early, under a third of a frame, and 38 of 38 cuts landed on the frames they were planned for.
One ad that holds its own square and vertical versions, in a loop, made entirely in code.
We made films of about 20 seconds for the projects in our work section the same way, from one function and one house style. The other films in this post come from that series.
VARNOŠKA
VARNOŠKA is a workplace-health site for the Koroška region. Since 27 August it has offered 15 exercise routines for the office and 15 for the production floor. Each routine is a guided video of about five minutes. About 125 exercise pages each play their own slice of those 30 videos, and none of them needed a separate clip.
The narration is a synthetic Slovenian voice from a speech service that takes plain text only, with no way to correct how a word is said. In our listening check, a reviewer plays each of the 253 narration lines and either approves it or types a respelling until the word sounds right. The final videos were rendered from exactly the audio the reviewer heard.
Showcases on everyframe.eu
Some of the engine's work is on show at everyframe.eu/showcase, including Tour de France footage with the crowd's faces pixelated. In the drone showcase the engine follows a vehicle from the air, and all the work happens on a small 17-watt computer on board. The ground gets a 68-byte record of where the vehicle is in each frame, and the video stays on the drone. For Formula 1's farewell to the Zandvoort circuit, the engine restored three newsreels of the track from 1939, 1949 and 1963.
Detection and tracking speed
We aimed to make the face detector three times faster by giving the graphics card that runs it several frames at a time, and it came out 1.42 times faster. Tracking follows each face or object from one frame to the next. It used to take 335 milliseconds on 900 frames and now takes 17.5, with the same result.
Face detection next to the three times we aimed for, and tracking time before and after.
Next · Chapter 4 of 8
Infrastructure we control
3 min read →
Stay in the loop
Don't miss the next one.
One email when we publish something worth your time.