Personalized Synthetic Advertising — the future for applied synthetic media.
It’s 2020; deepfakes are still significant media sensations and definitely unsettling news for most people. However, what’s still generally understood as deepfakes is merely just a tip of an iceberg.
This iceberg is called synthetic media. Instead of ice, it consists of not-yet materialized potential of its possible applications in a broad scope of fields: creative industries, advertising, and our daily communication. We might have happened to be on a luxury cruise ship, blinded with nostalgia for the times when things were more transparent and understandable and sailing with full speed toward the scary deepfake iceberg tip. Sorry to interrupt your fear or denial. Now it’s the time that you throw away the vintage glass of champagne and jump in the ice-cold water, for it is not the deepfakes we are going to hit, but something much more extensive and substantial. It’s still unacknowledged, hiding under the waters of shallow media coverage.
Let’s forget the celebrity and porn face-swaps for a second. I am intentionally using a considerable dose of speculation in my writing and projects, because this way it’s possible to stretch the thinking and look much further in the future, not just behind the first corner. Taking the Titanic metaphor back again, by speculating, we can see and touch the whole body of an iceberg long before we spot the small tip in a telescope.
What if I told you that your face will soon be deepfaked daily, and these deepfakes will be shown to your family, colleagues, and friends?
And bigger chance is this will not be done by any anonymous internet villain, but by one of the large companies that already have all the necessary material on you collected. Let me introduce you to the speculative scenario of personalized synthetic advertising!
In recent years, personalized marketing has become a successful marketing strategy to deliver individualized messages and product offerings to current or prospective customers based on collected data and its analysis. If you use social media, you’re most likely being targeted with personalized commercial content already. Whether it’s Zalando shoe ads on Facebook after you browsed Zalando e-shop for sneakers, or YouTube ads for feminine sanitary products after chatting in Facebook Messenger about your period, the examples range from obvious to almost creepy connections the big data has on you. Now, imagine one step further: commercials not only targeted at you but tailor-made for you. Soon you might be targeted with a synthetically made advertising that embraces the “deepfake” potential of video and sound synthesis and delivers commercial content offered to you by your friends and family.
This, and much more, can be achieved when synthetic media start being deployed for commercial purposes. Now it starts to be clear that synthetic media — a term for the more general understanding of generated AI media — go far beyond Nicholas Cage face swaps. The deep learning models able to produce synthetic media vary in complexity, size, and use cases. In essence, they can generate visuals, sound, and text. Their combination allows the existence of a completely fabricated reality, which our eyes or ears are not able to distinguish from camera-captured reality. The quality of synthetic media, in some cases, does not reach hyper-realistic results yet, but considering the exponential development in this field, it is just a matter of time until we start seeing the first professional applications of synthetic media in our daily lives.
The idea of using deepfakes in advertising is most likely to happen sooner or later. Such personalized synthetic advertising would not only increase the chance that the offered product (or services) is what you’re looking for; it might make you more likely to buy it because of the seductive power of familiarity. Seeing your mom’s face popping up on the screen of your phone (or a smart stove) with a recommendation for the right brand of butter while making her famous apple pie recipe might actually be very helpful. Seeing your friends hanging out in the park with Heineken bottles might make you crave beer, as well as it can make you organize a BBQ and socialize a little more. However disturbing such an idea is, it seems hard to evaluate whether it’s purely evil. Yes, it is probably crossing the line of privacy in terms of the non-consensual use of one’s face. On the other hand, such precedences have already happened, and society got used to losing a little bit more of private data once again.
Personalized synthetic advertising could be implemented in familiar places such as social media, but more unconventional locations could be explored in the future. Taking the inspiration in the current pandemic situation, when most of the social life moved online, we placed the synthetic advertising inside a video call, where it’s triggered by a specific phrase. In this speculative scenario, the phrase “dinner tonight” triggers an automatic full-screen ad for Uber Eats. A lot has been written about the amount of data some of the video conferencing apps collect recently. This idea speculates on the plausibility of taking the call’s video as a source for “deepfaking” the call’s participant and directly applying their face in the personalized advertising.
In the following paragraphs, I present the speculative scenario in bigger detail and explain how we made the deepfake. If you’re not interested in these details, jump straight to the takeaway part.
The scenario
VIDEO-CALL PRAGUE/UTRECHT, Sunday evening 19:00 CEST
It’s COVID-19 pandemic with widespread lock-down measures, and people resorted to connecting with each other via different video conferencing tools. Two friends, Lenka and Lisa, call each other via Room, an app that has become the most popular among all video conferencing apps. After a short, small-talk, Lisa asks Lenka: ”so…what are you having for dinner tonight?” The keywords “dinner” and “tonight” triggers a short full-screen ad for Uber Eats, similar to the ads that are popping up in the middle of YouTube videos. However strange this new feature is, there is something even more peculiar to it — the actor in the commercial has Lenka’s face. She’s dressed in a high-school cheerleader’s uniform, which does not make sense at all, but it’s definitely her who’s ordering tonight’s dinner from Uber Eats — Lisa is thinking. On the other side of the line, Lenka sees the same commercial with the only difference — deepfaked Lisa. The ad is rather short, so after a few seconds of confusion, the call continues, and they are both creeped out. That was weird..was that you? … Two friends continue talking in the Room while actually ordering some food from Uber Eats and sharing dinner together over the computer screen.
Watch the whole scenario video here:
Behind the scenes
We created this custom deepfake video for the HMM — platform for Internet culture to illustrate the speculative scenario for personalized synthetic advertising and offer a tutorial in the form of “from-behind-the-scenes” material. The deepfake was made with DeepFaceLab, currently, the most common software used for celebrity face-swaps. The goal was to swap my face with Kim Kardashian’s face from the original Uber Eats commercial. Usually, people aim to take a celebrity’s face and place it on an unknown actor (which is the case of deepfake porn as well as deepfaked politicians). We were doing the exact opposite — taking the face of an ordinary person (me) and placing it on a celebrity’s footage. For achieving this, you need some footage material for both the source and the deepfaked face.
Getting Kim’s face was easy, as there are many already pre-trained models based on celebrity datasets available, for example, at MrDeepFake Forums (warning: NSFW!). As we had my face at our disposal, we filmed a two-minute video where I keep speaking random sentences, and the camera moves around to capture my face from various angles.
For the destination, we used a cut video of Uber Eats commercial.
The first step of the production is to use DeepFaceLab to process both videos and create ‘aligned videos’, which are cropped to focus only on the face area. (Notice how it aligns the position of the eyes in every frame).
In the process, DeepFaceLab also creates a debug video where you can inspect how exactly the face is detected, and the video is cropped.
Once we have the crop-aligned videos, we can feed their individual frames into the training process of DeepFaceLab.
During this process, we can inspect the output and stop the training once the differences between iterations are barely visible. You need a decent graphics card to do this (for example 1080 Ti GPU), and it takes around 10 to 30 hours for the full process to run.
When we are satisfied with the result, we can use the “merge” operation to generate the faces and apply them to the destination video.
And that’s it! The deepfake video is ready!
As you can see, the process is not that hard to follow. If you decide to work with pre-trained models, you can achieve great results quite fast.
How speculative is this scenario really?
At the time this deepfake video was being produced (mid-March 2020), it was impossible to make real-time deepfake video out of the live video call camera footage. We obviously faked the implementation of the deepfake in a video conferencing app to create a vessel that could carry the speculative story. However, in this example, we can see how fast the synthetic media (and AI technology in general) are evolving. Two weeks after the video was finished, an open-source tool for creating basic live facial reenactment deepfakes on Zoom and Skype called Avatarify was released on GitHub. The tool uses First Order Motion Model for Image Animation, which can take a driving video and merge it with one destination image to generate new animated video. Although the logic of Avatarify does something else than what we needed for our scenario (it animates a random picture based on your live camera input instead of animating your face based on an external driving video), this could definitely take us one step closer to real-time video-call deepfakes. It’s not accessible to everyone though, as this needs a mighty graphics card to run (1080 Ti GPU can generate 33 fps, while a better-than-average MacBook would generate only ~1 fps).
The takeaway
This scenario wants to highlight a few interesting points:
- Synthetic media have an exciting future ahead, and we will be witnessing their application in all kinds of different fields, marketing/advertising being the first.
- Except using AI to generate interactive and self-evolving advertising, that’s adjusting itself based on the users’ response (or being personalized), the AI technology can be used to push the boundaries of personalized advertising even further by using deepfakes to create individualized content.
- Such personalized synthetic advertising does not have to stay in the social media realm but can follow new traces of how to find new data sources, such as video calls.
- It’s really not that hard to make a deepfake.
- It makes sense to speculate and try to envision even a bit far-stretched future scenarios. What is not possible today will be possible tomorrow. Today’s speculation is tomorrow’s reality.
While others are still chit-chatting on the ship deck about the horrible deepfakes iceberg tip, I hope you jumped with me in the waters of the unpredictable, yet fascinating evolution of synthetic media. The water is actually not ice-cold and dark and scary, as those on the deck say. And we know how to swim, for we have experiences from previous challenges and shipwrecks. More than anything else, it’s crucial to be able to have a tangible experience from the bottom of the iceberg, as well as being able to imagine the iceberg before boarding on the cruise ship.
And what do you think about personalized synthetic advertising?
Would you be comfortable with your face being deepfaked and shown to your social contacts in the form of an individualized commercial content? And how would it feel to see your friends or relatives in the ads you’re targeted with?
Curious what you think! Leave a comment here or tag me in your tweets.
Lenka Hamosova & Pavol Rusnak, 2020
Special guest appearance: Adriana Homolova
