TikTok’s game-changing algorithm for keeping users engaged
Between 2017 and 2020, TikTok (a subsidiary of China-based ByteDance) built a massively successful video-sharing social network, not only in the US but also in India and other parts of the world.
On August 6th, 2020, Donald Trump signed an executive order invoking National Security concerns and mandating ByteDance to cease operating TikTok in the US. The deadline to comply: September 15th, 2020.
TikTok users and advertisers are worried. One of the options discussed to keep TikTok running is that Microsoft acquires TikTok’s US operation.
The question that puzzles many in that saga is:
Why would Microsoft want to pay between 35 and 45Billions USD to acquire TikTok?
That’s because TikTok has built a game-changing recommendation algorithm.
Microsoft wants the code that makes TikTok… well… tick.
TikTok built a leading social network in less than 3 years:
How did they get there?
November 2017. ByteDance, one of the BAT-B (Baidu, Alibaba, Tencent, and ByteDance are the Chinese equivalent of the GAFA — Google, Amazon, Facebook, and Apple), purchased Musical.ly: a lip-syncing app popular with mostly teenage girls in the US.
The price: 1Billion USD.
August 2018. ByteDance uses the data generated from Musical.ly to feed the recommendation algorithm they built for Douyin. Douyin, launched on the Chinese market in 2016, is the Chinese version of TikTok.
July 2020. The TikTok app has more than 1 Billion downloads.
Not only that, but TikTok also became one of the largest and most influential social networks in several countries, including the US (100M+ users) and India (120M+ users before the ban).
According to Wired, TikTok’s “For You” page is now one of the most valuable digital real estate in the world.
TikTok managed to crack the code of keeping users engaged:
- 74% of TikTok users are active (source).
- A TikTok user spends, on average, 52mn per day on the app (Source).
The Exploit vs. Explore dilemma:
One of the biggest questions for every business is: How can you stay relevant and keep every customer engaged?
Any company is constantly trying to predict what customers (you) will want next.
In our age of the Internet and Cloud Computing, companies are doing this by collecting lots of data. Then, they use Machine Learning and Artificial Intelligence to try to make accurate predictions about customer needs.
At the most basic level, it works like this:
- If you look at many pictures of cute kittens, the machine will learn that you like cute kittens.
- If you click on “like” on these pictures, that’s one more piece of information: that guy really likes kittens!
- The company will try to use these data (Exploit) to keep you engaged: it’ll show you more pictures of cute kittens.
But things are a little more complicated in real life, and you’ll rapidly get bored if you only see cute kittens on your feed.
Because of that, the company will also try to show you some other things you might be interested in; pictures of cute puppies, for example.
They’ll analyze your reaction to cute puppies and learn more about what you might like next: this is the “Explore” strategy.
This “Explore” vs. “Exploit” dilemma is not limited to “which picture shall I show?”
Every day, people and companies are trying something new (Explore) with the hope that they’ll get better rewards compared to doing that same thing that they already know works (Exploit): Let’s try this new food, this new haircut, let’s read this new book, or pick a different investment strategy.
We all do this all the time.
There is a large body of theoretical work to try to find the optimal balance between “Explore” and “Exploit.” The Thompson Sampling strategy dates back to 1933; the Multi-armed bandit problem is another well known probabilistic approach to solve that. Russo and Van Roy published a paper in 2016 on how to apply Thompson sampling to online optimization problems using Information-Theoretic Analysis.
The bottom line is: finding the right equilibrium between “Explore” and “Exploit” is hard, and many people are spending enormous amounts of money trying to find the perfect balance between both options.
Some of the existing hacks to “Explore” better:
Facebook, Instagram, Snapchat, all social networks are using your Social Network Graph to try to suggest better content.
The underlying logic behind that strategy is the following:
- I know that Jack likes kittens.
- I also know that I can’t just show pictures of kittens to Jack, or else he’ll get bored and go somewhere else.
- I know that Jack is a friend of Jill.
- I know that Jill likes puppies.
- Because Jack and Jill are friends, there is a good chance that Jack will like puppies too.
- Let’s show Jack a few pictures of puppies and see if he enjoys these.
Amazon and most online stores will use your past purchase history and compare it to the purchase history of people who have bought the same kind of stuff to give you recommendations.
YouTube does it to suggest videos, Medium to suggest new articles; it’s ubiquitous.
These approaches need LOTS of data. It usually takes time and many interactions for the machine to give you relevant recommendations.
But TikTok managed to go viral fast: they seem to have a very efficient way to figure out what people will like using a (comparatively) limited set of data.
TikTok’ “Secret Sauce”:
As explained in Eugene Wei’s post: TikTok and the Sorting Hat,
After they plugged Musical.ly, now TikTok, into Bytedance’s back-end algorithm, they doubled the time spent in the app.
TikTok had become a “frighteningly addictive” app thanks to an “eerily perceptive” algorithm.
- An algorithm that does not need you to follow anyone, in an app that is easy and fun to use.
- An algorithm that adjusts to the user’s evolving tastes in near real-time.
- An algorithm that works at scale, fast, across hundreds of millions of users.
And the best part is that TikTok’s algorithm seems to be culture agnostic: it was built for Chinese customers but worked as well for US users, Indian users, people in the Middle East, and with apparently minimal to zero customization.
Borrowing from Eugene’s post again:
Now imagine that level of hyper-efficient interest matching applied to other opportunities and markets. Personalized TV of the future? Check. Education? Shopping? Job marketplace? What about personalized reading, from books to newsletters to blogs? Music? Podcasts? Yes, yes, yes, please.
Microsoft’s interest seems to make a lot more sense now.
Microsoft is saying as much:
The Microsoft Blog Post commenting on the TikTok acquisition discussion is only eight paragraphs long.
Here is the gist of what it says:
[Microsoft] is committed to acquiring TikTok subject to a complete security review.
Translation: We’ll have an in-depth and detailed look at the code that ByteDance created for TikTok.
[Microsoft] would build on the experience TikTok users currently love.
Translation: We’ll need the code to make sure that the users are still happy and have at least the same experience they have today.
Microsoft would ensure that all private data of TikTok’s American users is transferred to and remains in the United States.
Translation: Since the data will not leave the US, the algorithm will also have to run on our servers in the US.
Microsoft would ensure that this data is deleted from servers outside the country after it is transferred.
Translation: There is no way that the algorithm that creates the recommendations for users will run anywhere else but on our servers in the US (where the data is). That’s the only option.
Microsoft is cleverly using a once in a lifetime opportunity:
Microsoft is still very invested in the consumer space. They have big plans in online gaming with the xCloud initiative.
Microsoft owns LinkedIn too. How powerful would LinkedIn become if powered by the ByteDance/TikTok algorithm?
Donald Trump’s executive order has created a very particular set of circumstances that are unlikely to happen again anytime soon.
If you were Microsoft and had the opportunity to purchase the source code for Google Search Engine 25 years ago, would you have passed on that?
Microsoft’s acquisition of TikTok is not a done deal yet, but I’m betting that any agreement will include the TikTok algorithm.