Jan Bot’s step by step
The filmmaking algorithm explained
Jan Bot is a computer program that works day and night producing experimental films sourced with footage from EYE’s Bits & Pieces, a collection of hand-picked found footage that curators from EYE have been gathering since 1989.
Jan Bot uses so-called Artificial Intelligence algorithms to select news items that are trending online. After studying them, it uses the news items as a reference to select images from the Bits & Pieces archive. The criteria to do so is simple: the images need to have a meaningful connection to the news items. In other words, our bot finds inspiration in trending news to choose the footage for its next film.
Every day Jan Bot produces an average of twenty films inspired by news trending online. And from these twenty films, every day Jan Bot selects the one that it considers its best to then publish through its social media channels: Instagram, Twitter and Facebook.
So, in summary, Jan Bot is mainly two things:
First, a new method to make experimental cinema that explores the aesthetics of online media and uses algorithms as editing tools.
Second, a very original way to present old audiovisual heritage by making it available through the online world.
Jan Bot’s inner workings
In order to gain a better understanding of how Jan Bot’s algorithm work, here briefly explained are the different steps required to generate a film.
Step 0: Define the source material
It all begins with sampling the source material, the aforementioned Bits & Pieces collection.
Most of the Bits & Pieces collection has been digitized. However, neither the original films nor the video files have been analyzed as individual shots, only preserved as reels. Jan Bot has processed all these reels and automatically indexed their individual shots.
To automate this process we developed an algorithm based on a program called Shot Detect. This algorithm helped us to identify cuts within the reels by showing us the difference between the pixel distribution in each and every frame.
If you want to see the archive through Jan Bot’s eyes, you have to imagine each frame as if it was an abstract painting composed of tiny black, white and gray points.
One second of video in Bits & Pieces is composed of 18 frames. If you want to see the archive through Jan Bot’s eyes, you have to imagine each frame as if it was an abstract painting composed of tiny and meaningless black, white and gray points. For example, if the frame to be examined contains the image of a ballet dancer, you must forget about the ballet dancer and focus on the rich variety of shades made of colored spots –or pixels– spread across the frame, just as you would do with an abstract painting.
After doing that mental exercise, you should repeat it with the following frame. Having these two abstract paintings in your mind is necessary for the following step: comparing them.
If their difference is significant, as if they look like two different abstract paintings, you should assume that the continuity of movement has been broken.
Jan Bot does this exercise with each of the almost eight hundred thousand frames that compose Bits & Pieces. That is how it is capable to guess when a shot ends and a new one begins.
Step 1: Generate metadata
There are two reasons why we decided to work with EYE’s Bits & Pieces collection. The first is the diverse nature of the material that it comprehends. While most of it is silent and seems to have been filmed during the first three decades of the twentieth century, the variety of events that are documented is mind-boggling. They range from mere registrations of bucolic –presumably European– landscapes, to exotic tribes in remote locations of Africa and Asia; or from rushes of old forgotten Hollywood slapsticks, to rather dusky scenes of drunk sailors beating each other almost to death.
But none of these descriptions is of particular interest for Jan Bot and its machine intelligence, for it can only see abstract compositions of pixels organized one after another within frames. And while this would also apply to all film collections in EYE, there is one more thing that Bits & Pieces lacks, thus making it very unique. Metadata.
Given that Bits & Pieces consists only of unidentified footage, there is no author, location nor precise date to be attributed to its shots. In fact, the collection has been intentionally kept that way, partly because it has been deemed as unnecessary, but also with the intention of forcing the viewer to give in to the cinematic beauty of these lost images. Such setting is very appealing for the kind of appropriation Jan Bot does, for it leaves the task of identifying images exclusively to the capacities of the algorithms that empower it. We don’t tell Jan Bot what we –its creators– see. On the contrary, we let it show us the meaning it actually finds in scanning these old monochrome images.
With the use of image recognition software developed by Clarifai, Jan Bot uses tags to annotate each of the 7000 shots found in the collection. Here are some of the clips indexed by Jan Bot under the tag ‘military uniform’.
Jan Bot’s annotations are not always accurate, at least not if our main reference is the human mind. These small errors unveil the gap that exists between human and machine intelligence. With an experimental attitude, we gladly embrace this gap and let Jan Bot surprise us with its insights.
Step 2: Select the source material
Jan Bot uses actuality as inspiration to bring these century-old images to the present. By using Google Trends it collects the current trending topics and related news items.
What you can see in the image above is data Jan Bot collected on 17 May 2018. It shows that news items containing the words “Yanny” and “Laurel” were highly popular in Great Britain, the United States, Belgium, The Netherlands, France, Germany and Denmark. Below the list you can see snippets of the news Items.
With the help of Cortical, a Natural Language Processing program whose function is to analyze the multiple meanings of a word, Jan Bot connects these texts to the tags generated for Bits & Pieces during step 1. Consequently, the best matching shots get selected to later be processed into a film.
In the image above you can see that Jan Bot associated tags such as war, military, soldier, and battle to the news about “Yanny” and “Laurel”. In this case, the selection has a very coherent set of tags –and you would understand better the connection between the tags and “Yanny and Laurel” by reading our study case– but that doesn’t mean that we don’t often run into extravagant mismatches, like, for instance, confusing a group of fashionistas wearing hats with a group of cowboys. What would a cowboy be without a hat after all? We do not correct these mismatches because, as mentioned before, it is in the spirit of this project to unveil the gap between human and machine intelligence, and let ourselves be surprised during the process.
Step 3: Compose a film
Once the film fragments are selected, the actual creation of the videos happens in a space we call the ‘black box’. Here is where algorithmic editing takes place.
The process inside the ‘black box’ is not really a single step, but rather many small steps with different variables. Here is where we experiment based on our personal speculations on how a film should be composed in order to reach a unique style and rhythm. Unlike the previous stages, which can be reasonably explained, this step is highly subjective and heavily based on trial and error. We freely take inspirations from music and scenes from films. We use them to explore concepts and sketch edits. If we like the outcome of our experiments, we take them to the following stage which consists of translating the concepts into algorithms. Once composed, these algorithms are then added to the set of algorithms that we have previously crafted, enlarging Jan Bot’s assortment of rhythms, intertitles and image loops.
Step 4: Distribute the film
At the end of each working day Jan Bot evaluates the work it has generated and selects its finest creation by using a custom ranking algorithm. This video is then uploaded to its social media profiles on Facebook, Twitter and Instagram.
What Jan Bot defines as “finest” is basically what ranked highest in the scale of trends during the day. “‘Yanny’ or ‘Laurel’” is an example of such a movie.
Final step: Enjoy the film
This final step doesn’t depend on Jan Bot’s algorithms, but on the algorithmic mindset that spectators make use of in order to appreciate Jan Bot’s films. In case we want to explore their meaning or the story these films are telling, it becomes necessary to use our imagination and try to think from unconventional perspectives. It’s good to keep in mind that these films make perfect sense to Jan Bot. From its point of view, they hold no mistake, no randomness nor choice made in the rush of impulsivity. So one way to read these films is by trying to think like Jan Bot.
That said, trying to understand Jan Bot’s genius is only one possibility. There is not only one way, there are many.
Trying to think like Jan Bot, understanding the why’s and how’s of its choices, is what we call a hermeneutic approach to reading its films. Another could be imagining a story based on the associations the films propose –a semiotic approach. A third possibility may consist in approaching Jan Bot’s films as a refreshing way to access the news, first by trying to “decipher” the “riddles” given by the film’s intertitles, and then finding the “answers” in the news that are associated to the film, which are available on the film’s page –a playful approach.
There’s certainly more ways possible, some of them more impressionistic, and others more analytical. We suggest trying them all. It is our experience that some approaches work better for some films over others.
Like for example this film about the winter solstice seems like a story in its own right:
Meanwhile, this other film, inspired by the release of a new Apple TV, seems more obscure, abstract, maybe even conceptual. Its intertitles are covered by a layer of incomprehensibility that, among the chaos, shouts a distinctive headline: “The Apple TV improved a home”.
This almost-meaningful announcement raises a kind of curiosity that the previous film didn’t convey. The former invited us to imagine a story, while the latter triggers our desire to know more about the news that inspired it.
Whatever is the approach, Jan Bot’s films will force you to think with a different mindset. So take your time, be open minded and mentally prepared. And if you don’t like a film, don’t feel gloomy. Remember the are thousands of other films already produced by Jan Bot. From those, there must be at least one you may be able to connect with. You just need to keep searching for it.
