Story of one animation: WebGL and not WebGL

My adventures of implementing one animation on a website.

The full animation on the website looks like this:

you can play it to any point

We’ve been working on this one together with the guys from MountianMaier.de. Animation you see is done in Cinema4D with a RedShift renderer. One of the main features for the website was that user could play animation to any place and it should look good on all the screens (i.e. retina )

WebGL

Of course, as a fan of WebGL and Three.js my first thought was just to export it for the web. So i opened up my Cinema4D, saved file, and, behold: the .glb file was 36Mb. That is just too much, too many details. Alright, Not an option this time.

Image Sequence

Every time WebGL is too heavy, image sequence comes to the rescue. You just save every frame of the animation as separate .PNG image:

Animation broken down to frames

To play those<canvas> element is usually used: you draw each image with drawImage function. There is one catch though, you cant play sequence in your requestAnimationFrame directly, as it runs 60 times per second, and we have 24 frames per second usually. But with some minor time fiddling you can do that:

let frameTime = 1/24;
requestAnimationFrame(){
  currentTime = Date.now();
  elapsed = currentTime - prevTime;
  if(elapsed > frameTime) {
    playTo();
    prevTime = currentTime - (prevTime % frameTime);
  }
}

And then to draw frame:

let playTo = (frameNumber) => {
    canvasContext.drawImage(frames[frameNumber],0,0);
}

Okay, playing 24 frames per second was easy. But it turned out that my 8 seconds video, got 200 frames. That prevented me from using sprites — too many images.

Not a big deal, i managed to compress it to 8–10Mb (still a lot, but a lot less than 36Mb). Also a nice trick is to over-compress in-between frames, and leave only the key frames with high quality.

All in all, i got it running good on my mac.

Working good on my macbook

But.

QA guys told me that my animation wasn’t smooth on their PCs. I went to DevTools, and here is what i found:

Frames are way over 16ms, because of `Image Decode`

Turns out, even after loading images, before drawing them, browser still need to do image decoding. So then i learned there is a Decode API (only in Chrome now):

let img = new Image();
img.src = "whatever.jpg";
img.decode().then(function() { 
    document.body.appendChild(img);  // loaded and decoded
});

But when i tried to decode my 200 images, i ran out of memory on 73'd image. So not an option again. In the end, i couldn’t do that without sacrificing quality too much. 😭

Video

And then i thought of video. Well, after all, videos were invented to save frames (so clever!). So i used ffmpeg to make a video out of all those frames. I’ve got 2Mb .mp4 file, and was very happy about that.

So instead of drawing frames to canvas, i would just set video currentTime:

let playTo = (frameNumber) => {
    video.currentTime = frameNumber/24;
}

I thought that’s the end.

Yet, that’s what i saw:

See bottom video for what i mean. It was playing like that, jaggy =(

When i saw video playing in this weird way, all my hope was lost, i had no idea why would it play so buggy via javascript. But a couple of hours later, i learned a new thing!

GOP, Intra, Keyframes

Turns out, videos have keyframes inside of them too. They have different names: GOP (group of pictures), intra frames, keyframes. Video codecs use it to compress video better.

Basic idea is: we don’t save all frames as separate images, just one full quality frame, and then difference for the next ones. We don’t really think about this parameter in most cases. But it’s always there for each video. Here is the best explanation i have found.

Frequency of keyframes is the most important part. Let’s say we have a keyframe every 5 seconds of the video. So if i want to play video to 7.5s mark, browser will need to get 5s keyframe, and all the difference frames up to 7.5s frame. Which takes some time.

From top to bottom: 1s keyframe, 5s keyframe, 15s keyframe

Experimentally i found out the best values are between 1–2s frequency for the keyframes. To change frequence, -g parameter of ffmpeg should be used:

ffmpeg −i SOURCE.mp4 −g 300 [some other stuff] OUTPUT.avi

And i finally got everything working, and the video size with H264 was just 1.7Mb!

But that wasn’t the end!

AV1 — new video codec

If you haven’t read amazing article from Evil Martians about this video codec, do that now. In short: it is the best way to compress video, better than h264 and VP8/VP9 (usually in webm). Netflix and Youtube adopted it, Chrome already supports it.

The main and only drawback of AV1 is the time of compression, it takes ages!

But, can you guess which size i got with AV1? 612 kilobytes!

So in the end, this video was compressed down to 612Kb for 70 percent of desktop users. And that is how it went online. 🎉 And everyone lived happily ever after.

In the end

So we went from 36Mb file, down to 612Kb! Hope you liked that short trip into all those cool ways we could do the same thing in a browser. Let me know what you think. And have a good day!

P.S.: I am also open to any kind of frontend development projects, so if you need something like Vue/React/WebGL/CSS just let me know =).