Everyframe is the public name of our engine for making video and finding things in it. It started on 31 March inside the GZS project, as a library in TypeScript, the language our web apps are written in. The library let us describe each video in code and turned that description into commands for FFmpeg, the open-source video program. Early in May we rewrote the engine's core in the programming language Rust, and that version rendered the GZS award videos.
Today the engine runs on machines we control, and programs send it work through a code library published on npm, where JavaScript developers share their code. At least seven JT Digital apps use it. It renders every ad version Everyframe Composer makes and every video made in jt-cut, finds the faces that huluma covers, and made the VARNOŠKA routines.
Five of the apps that send the engine work, led by Everyframe Composer, our main product.
Video made in code
Since September the engine also does motion design. A film is written as a TypeScript program, with layers that move, blur and sit inside each other, and the machines render it. The loop that ran on our stand at Dan inovativnosti is one such program now, and the engine renders all 20 of its scenes within 2 pixels of the hand-built original. The same code can find the beats in a song and place the cuts on them. In one test it found the drop 12.1 milliseconds early, under a third of a frame, and 38 of 38 cuts landed on the frames they were planned for.
One ad that holds its own square and vertical cuts, in a loop. There is no footage in it, only code.
Every project page in our work section now has a film of about 20 seconds made the same way, from one function and one house style. The films in this post are those.
VARNOŠKA
VARNOŠKA is a workplace-health site for Koroška. Since 27 August it has offered 15 exercise routines for the office and 15 for the production floor. Each routine is a guided video of about five minutes. About 125 exercise pages each play their own slice of those 30 videos, and none of them needed a separate clip.
The narration is a synthetic Slovenian voice from a speech service that takes plain text only, with no way to correct how a word is said. We built a listening check for that. A reviewer plays each of the 253 narration lines and either approves it or types a respelling until the word sounds right, and the final videos were rendered from exactly the audio the reviewer approved.
Showcases on everyframe.eu
Some of the engine's work is on show at everyframe.eu/showcase, including Tour de France footage with the riders pixelated. In the drone showcase the engine follows a vehicle from the air, and all the work happens on a small computer on board that uses 17 watts of power. The ground gets a 68-byte record of where the vehicle is in each frame, and the video stays on the drone. For Formula 1's farewell to the Zandvoort circuit, the engine restored three newsreels of the track from 1939, 1949 and 1963.
Detection and tracking speed
The face detector finds the faces in each frame. We aimed to make it three times faster by giving the graphics card that runs it several frames at a time. It came out 1.42 times faster. Tracking follows each face or object from one frame to the next. It used to take 335 milliseconds on 900 frames and now takes 17.5, with the same result.
Face detection next to the three times we aimed for, and tracking time before and after.
Next · Chapter 4 of 8
Infrastructure we control
3 min read →
Stay in the loop
Don't miss the next one.
One email when we publish something worth your time.