Baidu is dead. Or so a veteran political journalist claimed.
On 23rd of January, Kecheng Fang published an analysis piece on Weibo titled “Search engine Baidu is dead (link in Chinese)”. In his article, he commented that Baidu has succumbed to fake news and is now largely directing people to content published under platforms owned by Baidu such as Baijiahao. Fang’s critique is important and noteworthy. Fang is a veteran Chinese political journalist and now media scholar at the University of Pennsylvania’s Annenberg School for Communication.
Within a few days, the article generated more than 830,000 real-time searches for the topic on Weibo; Microsoft’s Bing search engine, which is unblocked in China, saw a surge in user traffic; and Baidu’s stock fell by 5% after the article was published. To put things into perspective, that 5% fall in valuation for Baidu translates to about US$3 billion which was the valuation of ofo at its peak.
Battle of the platforms
Fang’s critique of Baidu and the Chinese’ discontent with it reflect a problem that has been brewing for some time. We must first go back in time to 2016 and 2017 to trace the source of the Chinese netizens’ discontent. Back then, ByteDance’s Jinri Toutiao, an artificial intelligence driven news aggregator, has started to take off and Tencent Holding’s WeChat also launched its news feed and official accounts function to give independent publishers a platform for their content. Baidu’s user base and advertising revenue were threatened as people started to use other platforms for content. Baidu responded with Baijiahao (百家号) as its answer to ByteDance’s Jinri Toutiao news aggregator and Tencent Holdings’ WeChat official accounts, giving independent publishers a platform while helping it generate more advertising revenue from online content.
Baijiahao supports many content formats including articles, albums, videos, live broadcast, augmented reality, and virtual reality. It is one of the five key products of Baidu and the number of content providers on the Baijiahao platform grew from 200,000 to 1 million in 2017. 2017 was a watershed year for Baidu as it suffered its biggest earnings drop since it went public 11 years ago. Baidu, in its urgency to increase its monetization avenues with a new content platform, started to prioritize Baijiahao in its search results. Baidu’s strategy paid off but Baijiahao’s advertising focus brought in the revenue for Baidu but it was just the start of the problems Baidu will soon face.
“You are fake news!”
Donald Trump made popular the term “fake news” to describe the media’s coverage of himself. While the term ostensibly refers to yellow journalism or propaganda which consists of disinformation of hoaxes, Donald Trump has expertly used it to describe any news which he dislikes.
Fake news and hoaxes have always been around on the internet but the 2016 United States presidential election has shown that fake news can influence elections. An analysis by BuzzFeed also found that the top 20 fake news stories during the elections received more coverage on Facebook than the top 20 real stories by the major media outlets. More than 126 million Americans were shown Russian-backed, politically-oriented fake news stories via Facebook.
Likewise, China is not isolated from the global phenomenon of fake news. In fact, Beijing has been dealing with fake news or “rumors” as the Chinese government calls it at least since the spread of internet in the late 2000s.
There have been numerous high profile cases of fake news in China. In 2008, rumors of maggots found in Sichuan-grown tangerines caused sales to decline nationwide. In 2011, rumors of the magic power of salt in protecting against radiation following an earthquake in Japan caused large crowds to empty out salt from the market. In 2013, rumors of a pro-Bo Xilai coup also spread across the internet.
Hardware is hard but fake news is harder
Fighting fake news is hard both from policy and technology angles. The problem is complex and fraught with political, economic, and societal implications and companies must work hand-in-hand with the government. From a company’s point of view, the technological challenge is huge. Both ex-Microsoft CEO Steve Ballmer and ex-Facebook’s head of security have both made remarks on its difficulty.
Here are some of the reasons why:
- The amount of information on the web far outstrips any current processing capabilities
Every day, roughly 100 million pieces of information hit the web. The most effective way of fact checking remains manual. Until AI advances to a stage where it can surpass manual fact-checking, the gap between volume and capability remains.
- Disparity in resources
Facebook is worth around US$420 billion while Baidu is worth US$52 billion. They are no small companies by any measure but the amount of resources they have pale in comparison to what state actors like China or Russia are capable of.
- As fake news detection algorithms advance, so do fake news generation algorithms
Just take a look at Buzzfeed’s video where it had Jordan Peele’s mouth swapped onto Obama to make it look as if Obama is speaking. Deep fakes make detection exponentially difficult.
However, technology companies have also stepped up in recent years to tackle fake news.
In 2017, Baidu added a ‘rumour’ tag to dispel articles which is deemed by false information using its natural language processing, big data, and artificial intelligence technology. On top of that, Baidu also launched its anti-rumour platform in September 2017 and it announced that 372 cyber police accounts from across the country had joined it.
WeChat also launched its “Anti-rumour Assistant” in June 2017 to help users check if an article is true. Likewise, Weibo also added the feature to tag misinformation and it actually did it four years before Facebook.
However, Facebook’s experience with tagging found that it was a flawed approach. Facebook implemented a Disputed Flags system which they scrapped after a year of testing. They found that the Disputed Flags buried the critical information that explained the inaccuracies and backfired by entrenching a person’s false belief.
Baidu also tried to implement an AI to filter news based on quality. However, initial reports showed that Baidu still faces an uphill task. The filter has to perform two functions. First, it has to decide if an article is well-written. Second, it has to discern if the article is truthful.
For the first challenge, Baidu can tap natural language processing to examine relationship between words and phrases to determine if an article is well-written as long as it has a large database of article as a reference. However, the AI is only capable of sorting stories into increasingly severe categories of “bad” but is only able to lump all non-bad stories into a generic “good” category. In other words, while the algorithm can tell the difference between a native speaker and a non-native speaker, it is not able to discern the Hemmingways and T.S. Eliots from any English writer. Good prose is wasted.
Baidu’s AI filter fails at the second challenge. The AI is unable to tell the difference between a grammatically perfect yet untrue article from a truthful one. In some instances, it might even penalize a truthful article with less-than-perfect grammar.
Digging your own grave
Baidu is not the only company tackling the fake news issue but it has certainly made some moves that exacerbated its own precarious position.
In Fang’s article, a search for China’s 2019 GDP showed that Baijiahao occupied the first, second, fourth, and fifth position of the results and none of them were the correct result.
Worse, Baijiahao has been unable to detect fake news and actively promoted it. Fang brought up an example of a Chinese article which has been read more than 400,000 times.
A quick search shows American parody news website The Onion as the source:
Baidu prioritized the results from Baijiahao despite its inherent inaccuracies. Baidu issued an official response that the articles from Baijiahao account for less than 10 percent of the total search results. The impenitent CEO Shen Ke also commented that the company does not regret what it has done. Yet, Baidu has not given a satisfactory answer on why most of these 10 percent appear on the first one or two pages of the search results.
Baidu faces colossal credibility problems because this is not the first time it has gotten into trouble.
Baidu ran afoul of the laws when it was implicated in the death of Wei Zexi, a 21-year old Chinese college student from Shaanxi. Wei died after an experimental treatment for a rare form of cancer at the Second Hospital of the Beijing Armed Police Corps. He learnt of the treatment from a promoted result on Baidu. While the company is free to sell top search slots to the highest bidder, the company did not ensure that the results it promote are credible. Baidu also did not mark promoted results prominently.
Two years after that incident, Baidu appears not to have learnt from its lesson. Last year, Chinese state-owned media network CCTV revealed that another user made a complaint similar to what Wei did. Ms Zhou from Ningbo wanted to seek treatment for her nose disease in an affiliated hospital of the reputable Shanghai Fudan University but Baidu’s search results directed her to a Shanghai Fuda Hospital. “Fuda” is a common shortening for Fudan University in Chinese. Zhou spent tens of thousands for treatment and an operation when she could have gotten a cure at the real Fudan University Hospital for less than RMB200.
Baidu’s spate of credibility issues persist even into 2019. Recently, it came under fire when it threatened Wang Zhi An, a prominent veteran journalist, with legal action if he didn’t remove a blog post.
In Wang’s blogpost (in Chinese), he called out Baidu for its role in facilitating the rise of Quanjian. Quanjian has been under investigation for operating a pyramid scheme and false marketing. Again, Baidu’s muddy relationship with for-profit hospitals and medical groups was brought out into the open.
In a separate incident, Baidu was also embroiled in another healthcare scandal. Baidu also has a close relationship with the Putian Medical Group which dominates most of China’s private hospitals. In 2013, Putian’s Communist Party Chief estimated that Putian may have contributed 45% of the US$4 billion in ad revenues reported by Baidu. Putian Medical Group, on the other hand, was able to further its business and gain high profits through Baidu promoting its services.
The Putian Medical Group has a checkered past. China Daily reported that Putian Medical Group was catapulted into the national spotlight when a consumer advocated claimed that practitioners from Putian sold 10-yuan bottles of STD treatment for 200 yuan. The report also questioned the hospital group’s eligibility and qualifications after healthy patients were diagnosed with STDs seemingly to make more money for the hospital.
Most patients reportedly end up in a Putian-related clinic because they saw advertisements on the internet and other forms of media. Well-known WeChat blogger Xinwenge (新闻哥) called out Baidu (link in Chinese)and the clinics for putting commercial interests above the health of their patients.
The Wild Wild East
Baidu operates in a country which is vastly different from the countries Google operates in. Baidu was able to manipulate the results to give itself an unfair advantage. When Google tried to do something similar, the European Union slapped Google with a record-breaking $2.7 billion fine for manipulating search results and giving itself an unfair advantage to its own service.
Baidu has been allowed more leeway in terms of results manipulation because it largely operates in China. Other social media or search engine companies like Facebook and Google have to operate in a society like the United States which prioritizes the freedom of speech. If Facebook or Google tries to censor too much misinformation, they might impinge on their users’ freedom of speech. Baidu does not face that problem in China.
Baidu, as a company, can build tools and teams to look inward at its products but it will never have the true geostrategic insight to assess hostile foreign threats. It will need to partner with the government or intelligence professionals to gain such insights. Furthermore, Baidu will also need to act in a quasi-governmental manner and make judgements on free speech and misinformation in tandem with the local government.
The quasi-governmental role is precisely Baidu’s biggest challenge. The conversation surrounding the United States fake news is not well understood and comprises many different actors including social media companies, hostile states, competing agenda of the United States government, political parties, and individuals. However, the issues surrounding fake news in China should be understood in the context of an older and much larger problem — government information control with Chinese characteristics.
When viewed in that lens, Baidu is really caught between the devil and the deep blue sea. As a private entity with quasi-government obligations, it has to adhere to the Communist Party’s guidance on the type of political and news content to be censored or amplified. Baidu runs the risk of a hard clampdown by the Party if it does not listen. The Party has shown that it is capable of doing so when the situation calls for it. Bytedance saw its Neihan Duanzi shut down and CEO Zhang Yiming posted an apology on WeChat for “a weak [understanding and implementation of] the “four consciousnesses” — a Xi Jinping buzzword. As David Bandurski noted, China’s leadership “has been unhappy with the idea of algorithms that wall users off from official messaging if they show no interest in such content.” If there is a lesson to be learnt from Bytedance, it is to toe the Party line.
Yet, toeing the party line makes Baidu’s position extremely tenuous. As Helen Gao of The Atlantic has pointed out, Chinese distrust the country’s traditional media which regularly covers up food scandals and human right violations leading to many of the youth turning to social media for information and news. Baidu risks alienating its users if it suppresses information too much and appears to be another extension of the state’s information control mechanism.
The reports of Baidu’s death are greatly exaggerated
Baidu’s future is not all doom and gloom. By many accounts, Baidu is a forerunner in the global AI race. Baidu is the first Chinese company to join The Partnership on AI (PAI) in October 2018. The group includes US tech giants such as Amazon and woks on ethics in AI.
In terms of research, Baidu also has numerous collaborations with prestigious Chinese and overseas institutions such as Tsinghua and Carnegie Mellon. It has also attracted top AI talents such as Andrew Ng and Qi Lu who have both since left Baidu.
In terms of product, Baidu leads the race among the Chinese AI giants — Baidu, Alibaba, Tencent, and Huawei. According to an article on Huxiu.com titled “A Map to Understand China’s AI Close-quarters Combat: Only Baidu and Huawei are Really Doing AI”, Baidu has been a complete trailblazer in the development of AI. The article divided AI into three layers — application, technology, and foundation and provided a visualization of China’s AI ecosystem. Baidu occupied the top spot with 48 positions accounting for 25% of all positions. Tencent followed in second place with 37 positions and Alibaba with 31 while Huawei accounted for 8.
(You can read Jeffrey Ding’s translation of the article here)
This is reflected in the superiority of Baidu’s AI technology. Baidu holds China’s only independent intellectual property rights for deep learning open source framework in Paddle Paddle. On top of that, Baidu also possesses China’s largest and most mature AI open neural network in Baidu brain.
Last, Baidu has stepped out of the comfort zone of its corporate genes and ventured into edge of AI development with heavy investments into autonomous vehicles. Baidu has signed a number of Chinese carmakers onto its Apollo platform which has 116 partners as of last year. Baidu aims to deliver the first batch of level four autonomous driving vehicles by 2021 and its level four minibuses are already running in ten locations throughout China.
Baidu’s long term future looks bright with its many years of investment into AI and autonomous vehicles. Yet, Baidu must survive the tightrope it is walking on before it can reap the fruits it has sown. It has survived the previous ordeals by sweeping the issues under the rug and not really addressing them and it has only been able to do so because it enjoys a virtual monopoly in the search engine market. If it fails to correct its credibility issues, the Chinese will soon turn towards social media as an alternative form of information and Baidu will soon join Renren in the graveyard of internet companies.