Fake news and disinformation are now widely recognised as weapons for fanning the flames of distrust towards governments, individuals and communities worldwide. We are inundated with disinformation every day through news reports, images, videos, memes, WhatsApp forwards, emails etc. Distorting facts for furthering an agenda, by itself, is not a new problem; politicians and advertisers are quite familiar with it. The explosive growth of social media combined with the emerging power of artificial intelligence has added new dimensions to the problem and greatly magnified it.

Artificial intelligence is getting better year after year at generating strikingly human-like content. Language models such as GPT-3 can write entire articles on their own based on only a single-line prompt given as input. Deep neural networks are being routinely used to create fake images or videos known as deepfakes. Doctoring videos used to be a tedious and expensive process requiring significant technical expertise. Open-source software such as FaceSwap and DeepFaceLab have made the technology more accessible. Today, anyone with limited expertise can easily create deepfakes using a computer or a mobile phone.

The Internet has become the most important channel for consuming information for billions of people. What we read and see on the web shapes our opinions and worldview. Access to information is vital for democracy. By incessantly hacking the truth, fake news is bleeding democracy through a thousand cuts.

While the spread of fake news is funded, aided and abetted through several vested interests and further promoted by human behaviour, the technology that helps create this can also be harnessed to fight it. Can algorithms that exacerbate the effects of fake news also be used to counter it and foster critical thinking at a mass scale?

This article explores various technological solutions to first identify and then arrest the spread of fake news.

Detecting Fake News

Can we use AI to identify fake news? To do that, we must first answer a very nuanced question: what is truth? Rather than tackling this question head-on, researchers and companies try to answer simpler variants of it.

For example, can AI determine whether a piece of content was created by a computer or a human? Yes, it can — up to a certain accuracy. Techniques analyzing linguistic cues such as word patterns, syntax construction, readability features etc. can differentiate between human and computer-produced text. Similar feature-based methods can also be used to separate synthetic images from real photographs.

However, the recent successes of generative adversarial networks suggest that algorithms will soon be able to mimic humans so well that the content they produce will be indistinguishable from that created by us.

Generative adversarial networks are deep neural networks that consist of two complementary parts. A generator algorithm takes as input a set of real samples — text or images — and tries to replicate the patterns to create synthetic samples. The discriminator algorithm, on the other hand, learns to differentiate between real and synthetic samples. The generator and the discriminator compete to outsmart each other, and in this process, both improve continuously. The process concludes with a generator that can produce synthetic samples that are impossible to tell apart from the real ones.

OpenAI’s recently launched GPT-3 can write essays, stories, emails, poems, business memos, technical manuals etc. It can answer philosophical questions, simplify legal documents and translate between languages. For short pieces of text, it is almost impossible to separate GPT-3’s output from human-written text.

A second approach to detect fake news involves determining the veracity of the facts mentioned in an article, regardless of whether a human or a computer wrote it. Services such as Snopes, Politifact, FactCheck etc. employ human editors who perform research and contact primary sources to verify a statement or an image. These services are also increasingly relying on AI to help them rummage through large volumes of data.

The way we write facts differs from the way we write lies. Researchers are using this to teach machines to discriminate between truth and fiction by training AI models using a corpus of April Fool hoax news articles written over 14 years.

An alternative approach to verification works by assigning a reputation score to each news website. A statement is considered verified only when websites with high reputation scores endorse it. The Trust Project, for instance, uses parameters such as the ethics standards, sources and methods, corrections policies etc. to assess the credibility of a news outlet.

Besides flagging potentially fake stories, social networks have also begun taking punitive action against repeat offenders. These actions include suppressing the reach or even blocking the accounts that spew fake news or hate speech. An example of this is the ‘group quality’ feature introduced by Facebook. It will help if social networks also display the trust index of news sources and red-flag stories identified as false.

Arresting Spread of Fake News

Besides aiding the generation of fake news, technology also plays a crucial role in its dissemination. Fake news, owing to its sensational nature, spreads far faster over social media than genuine news. A detailed study by researchers at MIT published in Science showed that true news took six times longer on average than false news to reach 1500 people on Twitter. When they looked at the chain of retweets, they found that true news never exceeded a chain-length of 10, but false news reached a length of 19 — and it did this almost 10 times faster than the time it took for true news to reach 10 retweets.

Why do false stories spread faster and deeper into social networks than genuine stories? Many instinctively blame bots, conjuring in our minds images of vast armies of bots employed by malicious hackers to make fake stories go viral.

A bot is a user account on a social network operated automatically by a program. Technology companies have, over the last few years, developed sophisticated mechanisms to detect bot accounts. They monitor the network to find and block accounts that indulge in malicious behaviour such as using automation to disrupt or amplify conversations; generating fake likes, shares or other forms of engagement; engaging in bulk, spam or aggressive posts; targeting, tarnishing or spamming other users etc. Sites like Twitter delete hoards of malicious bot accounts every day, and this is aided by technology. Further, machine learning applications like the Botometer use thousands of features such as the user’s profile details, connections, activity patterns, language and sentiments to predict the likelihood of an account being a bot.

But are bots solely responsible for spreading fake news? It does not seem so. Bots do amplify the spread of news, but they amplify genuine and fake news equally. In the same study mentioned above, the researchers repeated their experiments by including and excluding bot accounts from their data. In both cases, they reached the same conclusions suggesting that false news spreads faster because humans — and not bots — are more likely to share it.

Private Messaging Platforms

Unlike social media networks, messaging platforms have tighter privacy policies. For example, Whatsapp supports end-to-end encryption. The messages can be decrypted and read only by the intended recipients, not even by the Whatsapp developers. Even if government authorities realize that a fake story is circulating on Whatsapp, there’s not much they can do to stop it.

Whatsapp has made attempts in the past to mitigate the problem by rolling out measures such as limiting the number of forwards, restricting the size of groups etc. Rather than helping, this has only encouraged users to shift to other messaging platforms like Telegram and Signal that do not implement these restrictions.

Due to the nature of these platforms, it becomes almost impossible to monitor such communication to fight fake news.

Concerted Effort Required

Although high-profile incidents related to fake news hog the limelight, a larger problem is the pervasive presence of disinformation. India is witnessing a rapid rise in news stories that are high on emotion and low on facts. This has blurred the line between sensational reporting and outright fake news.

The mechanism that supports fake news is complex and involves multiple actors with diverse interests.

Individuals create fake news out of mischief, for furthering political agenda or for financial gains. Sensational content acts as clickbait and can be monetized on platforms like YouTube.

Technology companies earn advertisement revenues when stories go viral. Since they are answerable to shareholders and advertisers, they are reluctant to suppress any content — be it fake or genuine.

India, like most other countries, lacks a regulatory framework that deals specifically with disinformation on the web. We need laws that penalize malicious dissemination of false statements without affecting citizens’ constitutional right to freedom of speech — this is a challenging legal question.

Citizens should be made aware that unverified news is detrimental to their own future. Some schools in India are taking initiative by inculcating fact-checking habits in secondary school students. The students are taught critical-thinking and methods to verify what they see and hear.

Human nature and technology are both equally responsible for the problem of fake news. To solve it, we need to change human behaviour to suit our new roles as big-information consumers. We need to realize the importance of credibility and authenticity and learn to value them. This won’t happen overnight — it’s a gradual awakening that we hope will come someday. Until then, technology can lessen the impact and act as a catalyst for this change.

Viraj Kulkarni has a master’s degree in computer science from the University of California, Berkeley, and is currently pursuing a PhD in quantum computing and artificial intelligence. Sahil Deo is a co-founder of CPC Analytics, a data-driven policy consulting firm with offices in Pune and Berlin. Sanjana Krishnan is a partner at CPC Analytics.

Can Artificial Intelligence Help Us Fight Fake News? was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Imagine for a moment that you are having a conversation with a caveman. You tell him that the sun he sees in the sky every day is the same sun. A new sun does not emerge from beyond the hills every morning, and neither does it drown into the sea every evening.

“Oh, but it has to be a new sun. It comes up from here, and it goes down there!”

he retorts, his hands spread out pointing in opposite directions. You shake your head. Before explaining how the earth rotates around itself and revolves around the sun, you urge him: ‘Okay, just hear me out with an open mind…’

Here’s the thing about time: it is not real. There is nothing special about the present moment; in fact, a universal present moment does not even exist. The past and the future are equal in all respects. Our notion that time flows irreversibly from the past into the future is an illusion born out of our ignorance about the world. It exists only in our subjective perceptions and not as part of objective reality.

The distinction between past, present, and future is only a stubbornly persistent illusion.

— Albert Einstein

Modern physics makes these statements very convincingly and leaves little room to refute them. Let’s look at time and deconstruct it in this article. Let’s do it keeping an open mind.

Newton’s Absolute Time

We believe that time is universal; that it proceeds in a tick-tock fashion throughout the universe at the same speed.

We believe that time is independent and exists on its own regardless of everything else.

We believe that time is unidirectional; that things move from their past into their future but never the other way around.

Absolute, true, and mathematical time, of itself, and from its own nature flows equably without regard to anything external, and by another name is called duration.

— Isaac Newton

Newton based his theory of mechanics on this notion of absolute time. Although this has arguably been one of the most successful theories ever proposed, chinks began to appear in it at the beginning of the twentieth century when time began to lose its absoluteness. In simple, easy to understand, bite-sized steps, we will proceed to examine time and deny it these three pillars one by one.

Spacetime Diagrams

Time is not absolute; it is relative. But what do we even mean by relative?

Alice and Bob are sitting in a cafe. Bob gets a call from his mother. He needs to go home, so he starts walking in a straight line towards his house at a constant speed. Alice remains sitting at the table.

If we assume for simplicity that Alice and Bob move only on a 1-D line, and we plot their locations on the X-axis and time on the Y-axis, we get a spacetime diagram like the one below.

This is how Alice will observe Bob from her frame of reference. But how will Bob observe Alice? We will use a piece of empirical evidence here: if I see you moving away from me at a speed v, you will see me moving away from you with the same speed v in the opposite direction.

To get the spacetime diagram from Bob’s frame of reference, we slide the X-axis towards the left progressively at each time interval while keeping the Y-axis the same. This is known as a shear transformation. Now, Bob appears to be stationary while Alice appears to be moving in the opposite direction. The lines that Alice and Bob make on the graph are called their worldlines.

Simple enough. Let’s add another object there. Suppose there is a cat sitting with Alice and Bob at the table. As soon as Bob gets up, the cat also starts trotting in the direction of Bob’s house at a different speed. The two spacetime diagrams from Alice’s and Bob’s frames of reference would look like:

Note an important feature of this diagram: the cat appears to be moving at different speeds for different observers (the worldline of the cat is different in the two frames of reference). Since Alice is at the table and Bob is moving in the same direction as the cat, Alice will think the cat is moving faster than what Bob thinks. Our shear transformations work just fine in classical mechanics, but there’s a problem when we consider objects that move at speeds close to the speed of light.

Lorentz Transformations

In the late 1800s and early 1900s, physicists struggled to explain a strange phenomenon: regardless of whether you’re moving towards the source of light or away from it, the observed speed of light remains constant.

What happens then if our cat is moving at the speed of light? According to the classical spacetime diagrams we drew above, Alice should observe the cat moving at a speed faster than what Bob sees. In reality, both of them would measure the cat’s speed as 299,792,458 meters per second i.e. the speed of light.

To accommodate this thing about the speed of light, we need a different transformation in which the worldline of an object traveling at the speed of light (our cat) remains unchanged when we change the frame of reference from Alice’s to Bob’s. In other words, both Alice and Bob should see the cat moving at the same speed of light even if Alice and Bob are not at rest with respect to each other. If you have not followed this, read it again, and let it sink in — this is important.

Lorentz transformations do just this, and they are at the heart of Einstein’s special relativity. Before getting into them, eyeball the spacetime diagram below. According to special relativity, this is what Bob would see if we use Lorentz transformation to accommodate the phenomenon of constant speed of light instead of the classical shear transformation.

Note what happens to the worldline of the cat moving at the speed of light. In the classical transformation of figure 3, it changes. In the relativistic transformation of figure 5, the worldline of the cat remains unchanged. Below are the equations that govern the transformation:

If (t, x) are the time and location coordinates of an event in Alice’s frame of reference, they will become coordinates (t′, x′) in Bob’s frame of reference, where v is the speed with which Bob is moving with respect to Alice, and c is the speed of light.

Well, okay, so what’s the big deal? Here is where things get interesting. In the shear transformation, new time was the same as old time, and new location depended only on old location. Time and space were independent of each other. Movement through space and movement through time were two unrelated things.

Now, look at the equations that govern Lorentz transformation, and you will see that movement through time and movement through space influence each other. Time and space are no longer independent entities; they are relative and depend on each other!

Henceforth space by itself, and time by itself, are doomed to fade away into mere shadows, and only a union of the two will preserve an independent reality.

— Hermann Minkowski

Loss of Simultaneity

Imagine you are the chief spymaster of planet Earth, and you know that a hostile alien civilization from Andromeda is planning to attack Earth in the near future. You have invented a magical tablet, and you have given one tablet each to your two best spies, Adam and Eve. The tablet provides live coverage of whatever is happening inside the Andromedean military council building at that instant.

Your sources tell you that an important meeting is going to take place in Andromeda where the alien leadership will decide whether to attack Earth or not. You tell Adam and Eve to report to your office with their devices. You sit with Adam at the table with his tablet. Adam peers into the screen and tells you that the meeting is underway in Andromeda, and the alien military generals are arguing about the pros and cons of attacking earth. While thoughts swim in your head about what you should do, Eve bursts into the room. She checked the tablet while she was running in the corridor towards your office, and she saw that the alien spaceships have already departed for Earth!

Neither Adam nor Eve nor their tablets are lying. The present moment in Andromeda is different for Adam and Eve because they are not at rest with respect to each other. This is described as the Andromeda paradox by Roger Penrose in his fascinating book The Emperor’s New Mind.

It’s worth taking a look at the equations in figure 6 again. In our daily lives, the velocities and distances (represented by v and x) are typically small. When x<<c and v<<c, t′ is almost equal to t; therefore, if something happens in your present, it happens in my present too. But in our example above, Andromeda is far far away from earth, and our equations come into full play. What is happening in Andromeda now simultaneously with Adam’s present moment has already happened a week ago according to Eve’s. Put in some arbitrary value for v (the speed with which Eve runs in the corridor), and find out the difference in the number of days between what your two spies see. Two events that are simultaneous in one frame of reference may not be simultaneous in a different frame of reference.

Simultaneous means nothing. There is no single now.

Time is Not Universal

Let’s take down the first pillar of Newton’s absolute time. Time is not universal. What is happening in your present may lie in my past or my future. It may also lie neither in my past nor in my future but in an inaccessible region of spacetime called elsewhere — that is a story for another day.

There is no tick-tock-tick-tock that can be heard in the same rhythm throughout the universe. Something that you think will happen in the future may have already happened in my past. The order in which different observers witness two unrelated events is not fixed; Adam may say that event P happened before event Q, while Eve might argue that P happened after Q. In this scenario, a nice and clear relationship where the past causes the present, and the present causes the future breaks down. Indeed, causality itself breaks down.

Time has no Independent Existence

The year 1905 was really special for physics. Einstein showed in special relativity the dependence between space and time and also between mass and energy. In 1915, as a part of general relativity, Einstein further combined spacetime with mass-energy equivalence.

It’s not just speed that slows time. Gravity slows it too. Actually, mass slows down time, and gravity is nothing but the slowing down of time. If things fall down, it is because Earth is a massive object, and it slows down time in its vicinity. In the absence of gravity, time passes uniformly, and things just float without falling down, like they do in outer space.

Time is nothing other than the measurement of change.

— Aristotle

In his enchanting book The Order of Time, physicist Carlo Rovelli describes how Aristotle and Newton disagreed on the nature of time and space. If nothing changes, does time exist? Aristotle would say no, while Newton would say yes. In absolute emptiness where there is nothing, does space exist? Again, Aristotle would say no, while Newton would say yes. Rovelli then tells us that the synthesis between Aristotle’s and Newton’s views is the most valuable contribution Einstein has made towards physics.

Einstein says that time and space do exist even in the absence of tangible matter, but they are not absolute; they are made of the same stuff that other things like tables, chairs, protons, and electrons are made of. Spacetime is a gravitational field. Like other fields, it is neither absolute nor uniform. It influences other fields and, in turn, gets influenced by them.

Thus, we strike down the second pillar. Time does not exist independently on its own.

Past and Future

We believe there is a fundamental distinction between the past and the future. The past has already happened. We may not remember all the details exactly, but we know pretty much what happened. We cannot change the past. The milk has been spilled, and we should not cry about it.

The future, on the other hand, is yet to happen. All possibilities are open. Planning finances, talking to our spouses, negotiating with our employers, self-introspecting, exercising, learning — all these things we do to influence and shape the future into what we want.

In the fundamental laws of physics that describe reality, however, there is no distinction between the past and the future. The equations given in Newton’s laws of motion, Einstein’s relativity, Maxwell’s electromagnetism, or Schrödinger’s quantum mechanics are all reversible; they treat the past and the future as if they were equal in all respects. The equation that governs the motion of a ball rolling down a slope also governs the ball rolling up the slope — and it doesn’t care whether the ball is moving from up to down or from down to up. If there is no difference between the past and the future, why do we remember the past but not the future?

Entropy

There is only one law of physics that distinguishes the past from the future, the second law of thermodynamics, which states that the total entropy of an isolated system can only increase or remain the same, but it can never decrease. So what is entropy, and what does it have to do with time?

Entropy is a measure of disorder in the system. Consider the arrangement of gold molecules in a ring. They have more structure and order than the molecules in a chunk of gold of the same weight. Hence, we say that the ring has lower entropy than the chunk. If the configuration of a system is structured, peculiar, or special, we say that the system has more order and lower entropy. If it is random, shuffled, or disordered, it has higher entropy.

Mathematically, entropy is given by the below formula:

where k stands for the Boltzmann constant, and W is the total number of possible micro-configurations for the given macro-state.

If a system begins in a state of low entropy, it is obvious why its entropy can only increase: because any random movement of molecules will most likely take the system to a more disordered configuration. If a teacup falls down and breaks, its molecules can scatter in a near-infinite number of ways. The number of possible micro-configurations W for the macro-state of a broken cup is far larger than the number of micro-configurations for the macro-state of an intact cup. Hence, we often see a teacup shatter into innumerable pieces leading to an increase in its entropy, but we never see broken pieces of glass spontaneously rearrange themselves to form a teacup.

Let’s take another example. Gently pour some milk into a mug of black coffee. The milk forms a distinct layer at the top. Now, leave this system alone. As the molecules in the mug zip-zap around randomly, the two layers slowly mix up leading to an increase in entropy. But nothing really prevents the reverse from happening. Molecules swishing around randomly in a mug of mixed-up brown coffee can, just out of plain chance, get rearranged in neat layers of milk and black coffee. It’s just that this is highly improbable!

The Arrow of Time

A physical system always evolves from a low-entropy configuration towards a high-entropy one. This unidirectional tendency of entropy to always increase is what gives us the perception of time flowing from the past into the future. This is what introduces irreversibility in our experience of everyday lives where eggs scramble, milk mixes into coffee, glass breaks, hot water cools down, and it is easy to squeeze toothpaste out of the tube but not back into it.

Entropy is order leading to disorder, structure giving way to randomness. The universe started in a state of low entropy, and that is why life is at all possible. Entropy has since then been increasing. Our sun will eventually die. All the stars in the Milky Way will eventually die. The entire universe will slowly crumble leading to an eventual state of maximum entropy or thermal equilibrium where energy and matter will be spread out in a diffused manner throughout the universe making it a cold, dark, bleak place.

But think about this: if the universe had started in a state of high entropy, stars would never have been created, there would be no Earth, and no us. Why the universe started in a state of exceptionally low entropy is a mystery, but its inexorable evolution since then towards states of higher entropy is what creates the perception of time.

Time Arises out of Ignorance

We said that an intact teacup is more special or peculiar than a shattered teacup. But, if you decide to list down the exact location of each molecule that forms the teacup, you will find that every possible configuration is equally special and can occur with the same probability as any other configuration. The sense of order and disorder emerges because we differentiate between a teacup on one hand and all broken configurations on the other. If we were to individually differentiate between every possible configuration in which the cup can shatter, we would find each configuration to be special and peculiar and equiprobable. In such a case, entropy has no meaning.

[Trying to understand time] is like holding a snowflake in your hands: gradually, as you study it, it melts between your fingers and vanishes.

— Carlo Rovelli

Entropy is not a fundamental property. It is, like temperature and pressure, a statistical one. If you isolate a single particle, it has no entropy of its own. Entropy exists because we see reality in a blurred and approximate fashion. It exists because we cannot distinguish between the innumerable micro-configurations of the macro-states we observe. If you assimilate complete information about the exact microscopic subatomic state of the world around you, will the flow of time disappear?

Yes! With complete information about the state of the world, the difference between the past and the future vanishes. The passage of time is a distortion of reality that emerges due to our imperfect knowledge. Time thus is born out of our ignorance.

If you liked the article, check out my other pieces on Medium, follow me on LinkedIn or Twitter, view my personal webpage, or email me at viraj@berkeley.edu.

Time Is An Illusion Born Out Of Our Ignorance was originally published in Cantor’s Paradise on Medium, where people are continuing the conversation by highlighting and responding to this story.

Key challenges in productizing AI and developing solutions that are practically useful in healthcare practices

Artificial intelligence is showing amazing potential across many domains of healthcare from radiology to patient care. Applications of AI in radiology include detecting tuberculosis from chest X-rays, COVID-19 and intracranial bleeds from CT scans, cancer from mammograms, brain tumors from MRI, predicting progression of Alzheimer’s from PET scans, etc. At DeepTek, we have developed models that can diagnose 30+ conditions from chest X-rays alone. Besides radiology, AI applications have been proposed in pathology, analyzing electronic health records, contact-tracing for pandemic response, predicting re-admissions and mortality, and many more. The strong research output and the large number of companies actively working in this space are a testament to the promise AI has made to radically transform healthcare.

Be that as it may, this promise is still largely unrealized. Many exciting machine learning techniques that have shown remarkable results on benchmark datasets are lying unused in academic papers and code repositories. The valuation of AI-enabled healthcare startups is soaring, but they have no revenues, and only a handful of these have seen any form of productizing AI and commercial adoption.

As the Principal Data Scientist at DeepTek — one of the few companies that have been successful in achieving commercial adoption of their AI solutions — I would like to share the key challenges involved in developing machine learning models that actually work in practice. Here they are below in no specific order.

Lack of sufficient data

Yes, we have all heard this before, haven’t we? Data scientists perpetually whine about not having enough data, and after a point, the rest of the team becomes deaf to this complaint. It’s an inconvenient truth, but let's face the fact that there is simply no way around it. If you don’t have data, you have nothing. Machine learning projects start with data and end with data — all innovation happens in between.

Neural networks are data hungry, and they need not just volume but also variety in the data they are fed. Tonnes of medical data exists, but it is severely fragmented. It resides in silos in different hospitals, clinics, personal computers, USB disk drives, emails, and a large percentage of it is not even digitized. Even if a hospital permits you access to their data, you would find it incredibly difficult to track, compile, and organize it. Data privacy regulations such as HIPAA and GDPR, although very important in their own right, have made things even more difficult.

Noisy annotations

Supervised machine learning needs data samples along with labels. The algorithm is first trained on matching sample-label pairs where it extracts patterns from the data and distills them into a mathematical model. This model is later used to predict the label of any unseen sample. One difficulty is that these labels are often noisy.

Consider the case of radiologists annotating X-ray images as positive or negative with respect to a given pathology. As data scientists, we would like to believe that this labeling is clear, certain, and unambiguous. The reality is somewhat different. As many studies have shown [1][2], there is a high degree of inter-rater variability between radiologists when evaluating studies with kappa values ranging from 0.3 to 0.8. Therefore, if the AI models are built using radiologist annotations as the ground truth, they begin from an uncertain footing where the labels themselves may not be accurate. This problem gets somewhat compounded when the annotations come not from one radiologist but from a team of multiple radiologists — this is likely the case when working with large datasets.

The open-source research datasets (NIH, MIMIC, Padchest, etc.) come with labels that have been automatically extracted from reports using NLP algorithms. The NLP extraction process may introduce errors in the labels, but there is another subtle problem when using them. When the human radiologist evaluated the image and wrote the report, the radiologist had access to patient clinical information, and his/her diagnosis factored in that information. This additional information however is not available for the AI to train on. In the absence of this crucial information, the AI may not be able to learn the right correspondence between the samples and the labels.

Training distribution not matching that in practice

Oh, this happens all the time! Consider the following situation. We get a large collection of X-ray images from a hospital. Our radiologists annotate this data for a given pathology, say tuberculosis. The prevalence of tuberculosis in this dataset is 25% ie. one-fourth of all samples are labeled as TB. We train our TB model using this dataset. As part of a new project, we deploy this model at a healthcare center engaged in conducting population screening for TB. The prevalence of TB in this population screening program is, say, 5%. How will the model perform?

At DeepTek, these challenges are a part of our daily lives. To know how we overcome them, reach out to us, and we will be happy to discuss more.

Not very well. The model was trained and evaluated for a particular prevalence. It expects the same prevalence in the testing phase. Of course, there are many ways around this, but there are many more problems too. The X-ray equipment in the hospital may differ from that at the healthcare center. The patient position may differ. The hospital images may have tubes and chest leads showing up. Population screening images may include foreign objects like coins, pins, jewelry, and clips since these may be taken with people keeping their clothes on. Due to these factors, the data which the model was trained on may differ significantly from the data it is supposed to predict.

Distribution drift

This is related to the above problem but demands a section of its own. Processes and workflows in healthcare are in a state of constant change. Hospitals change. Doctors change. The younger generation of doctors and medical practitioners are not shy about using technology. Patients change. The net effect of all these is that data distributions change all the time. A model, once developed and deployed, is never going to work smoothly for eternity. In fact, the adoption of AI in the hospital may lead to radical changes in the hospital workflow itself, thus causing a change in data distributions.

Distribution drift (also known as concept drift or dataset shift) occurs when the data distribution begins to change over time. To make deployment of AI solutions successful in practice, they need to be equipped with techniques to automatically identify when drift occurs and retrain and update the models to incorporate this drift.

Learning irrelevant confounders instead of relevant features

Machine learning algorithms are optimized to achieve the best performance on the given dataset. Although we hope that the models learn the right set of features during training, this may not always be the case. Models are known to stealthily exploit unknown features from images that may or may not be relevant to the prediction. For instance, if the positive images in your dataset predominantly come from one hospital and the negative images from another hospital, the model may learn to differentiate the source of the image rather than the target pathology; so instead of learning to differentiate between classes based on visual symptoms that indicate the pathology, it may pick up contrast differences in the images, some text markers outside the main region of the scan, black border areas around the scan, or some other such irrelevant features (also called confounders). Such a model may even demonstrate excellent performance on a benchmark set if that too carries the same confounding features.

Inadequate understanding of how the AI solutions will be used in practice

In many organizations, both research and industry, technology developers work at an arm’s length away from medical practitioners. Due to this gap, the solution they develop solves a problem that is qualitatively different from the one that needs to be solved to make the solution practically useful to the medical folks. Identifying the right stakeholders, eliciting through dialogue what problems they face, translating these into a set of clear requirements, formulating them objectively, and then designing a solution that fulfills these requirements is not an easy task when you start counting the number of people involved in this elongated process.

Artificial intelligence can revolutionize healthcare. By removing bottlenecks in processes, automating routine tasks, improving diagnostic accuracy, improving human productivity, and ultimately reducing costs, AI can be a game-changer in making healthcare accessible to all. To realize this potential, we need to acknowledge the above challenges, give them the respect they deserve, and find ways to mitigate them. At DeepTek, these challenges are a part of our daily lives. To know how we overcome them, reach out to us, and we will be happy to discuss more.

If you liked the article, check out my other pieces on Medium, follow me on LinkedIn or Twitter, view my personal webpage, or email me at viraj@berkeley.edu.

Taking AI models from Jupyter notebooks into real hospitals was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Grover’s quantum search algorithm finds the target element from a list of unordered elements in O(√N) time.

You are given a list of numbers and a target number, and you are asked to find the index in the list at which the target number appears. If the list is sorted, you can use search algorithms such as binary search. But if the list is not sorted, there isn’t really much you can do; you simply have to traverse the whole list till you find the element. In terms of algorithmic complexity, this takes O(N) time. With a quantum computer, however, you can solve this problem in just O(√N) time. This article explains how this is achieved by the Grover’s search algorithm.

If you’re new to quantum computing, you should first read this short primer: Quantum parallelism — where quantum computers get their mojo from.

Let’s start by framing the problem. We are given:

• A set of N elements X = {x_1, …, x_i, … ,x_N} such that each x_i is an m-bit string made of 0s and 1s.
• A target element x* that is also an m-bit string made of 0s and 1s
• A function f that takes as input an m-bit string and returns 1 if the string is x* and 0 otherwise. This function can be written as:

Grover’s search works in three steps as described below.

Step 1: Setup the state

A quantum state is set up in an equal superposition of the basis states. As an example, consider N=8. We set up the state using 3 qubits as:

Step 2: Phase inversion

In the second step, we flip the amplitude of each element x if f(x)=1 and leave it unchanged if f(x)=0. This is performed using a circuit that implements the below unitary gate O:

Suppose our target element x* is present in the fourth location. Applying the gate O will give us:

Step 3: Inversion around the mean

The third step known as inversion around the mean involves flipping all the elements around their collective mean. This is performed using Grover’s diffusion operator which is given by:

Applying this operator to our quantum state gives us:

This completes one round. If we were to measure the system at this point, we would get the target element as the outcome with a probability (5/(4*√2))² which equals 78%.

The second and third steps are repeated √N times to maximize this probability. After the second iteration, we get the below state which will find the target element with a probability of 95%.

Like many quantum algorithms, Grover’s search is probabilistic. It gives you the correct result with a high probability. In order to make this probability large enough to be practically useful, you may need to run it multiple times.

The same algorithm can also be used to find k-matching entries instead of a single target element. Many variations have been proposed. One of them is the Durr and Hoyer minimization algorithm which finds the index of the smallest element from a list — this has found interesting applications in quantum machine learning. For details, please refer to: Quantum Computing Methods for Supervised Learning.

If you liked the article, check out my other pieces on Medium, follow me on LinkedIn or Twitter, view my personal webpage, or email me at viraj@berkeley.edu.

Finding the needle in a haystack on a quantum computer was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

How quantum computers harness quantum superposition to execute many computational paths simultaneously.

Quantum computers were proposed in the 1980s. Since then, physicists have been laboriously working to harness the power of nature to meet computing demands. There is no single best method of physically realising a quantum computer; the field is fragmented into several competing approaches such as ion traps, optical lattices, photon quantum bits, nuclear magnetic resonance etc. But these different approaches all work towards implementing the same model of quantum computation in hardware. As long as we work with the model correctly, we need not concern ourselves with how it is implemented in hardware. To be fair, there are more than one models of quantum computation as well, but we shall only consider the standard quantum circuit model in this article.

We break this article into four parts. In the first part, we will understand what a single qubit is. Then, we shall see how multiple qubits can be represented. In the third part, we introduce transformations that act on qubits. And finally, we put all these together to explain how quantum computers can execute multiple computational paths in parallel.

Single Qubit

Classical computers operate on bits where each bit can be in the state 0 or 1. Quantum computers operate on quantum bits or qubits. A qubit exists not as 0 or 1, but as a superposition of the two (remember Schrödinger’s cat that is both dead and alive at the same time?).

Let’s represent this mathematically. The state of a qubit ψ can be written as:

The above notation is called the braket notation, but we won’t cover it here. This qubit exists as both 0 and 1, but if it is measured, we will get either 0 or 1 but not both. The probability that we get 0 is given by |α|², and the probability we get 1 is given by |β|². Once measurement is performed, the qubit loses its superposition and continues to exist in its observed state as either 0 or 1. The braket notation is quite useful for many reasons, but the same equations can also be conveniently written using matrices and vectors as:

Disclaimer: The article uses loose notation which is not technically fully correct. For instance, the above states 0 and 1 should be written as basis vectors |0> and |1>. The reason for adopting the loose notation is the unfortunate fact that medium does not permit using mathematical symbols in text, and converting every instance into an image makes the article awkward to read.

Multiple Qubits

The state of two unentangled qubits can be represented as a tensor product of their individual states. If the first qubit has amplitudes a and b, and the second qubit has amplitudes c and d, their combined state can be written as:

Some two-qubit states cannot be decomposed into tensor products of two individual qubits. Such qubits are called entangled qubits. Entanglement plays a central role in many quantum algorithms especially in the field of quantum cryptography. There is no counterpart to quantum entanglement in classical physics. We will not be reviewing entanglement in this article.

Quantum Transformations

There’s no fun if we cannot do anything with qubits. But we can — with the help of transformations. Classical computers operate on bits with the help of logic gates such as NOT, AND, OR, NAND, NOR, XOR etc. Likewise, quantum computers operate on qubits using quantum gates. Owing to the posulates of quantum mechanics, all operations performed on qubits must be linear and reversible. Consequently, all quantum gates need to be linear and reversible.

The NOT gate is the simplest of them. It simply inverts 0 and 1.

One gate that pervades quantum computing is the Hadamard gate. It transforms a qubit existing only as 0 into an equal superposition of 0 and 1 ie. if we measure the qubit, we will get a 0 with probability 50%, and we will get 1 with probability 50%. This, as we shall see, serves as a great starting point for many quantum algorithms.

Quantum Parallelism

Now, armed with the above knowledge, let’s unravel how quantum computers do their magic. We will do this using a simple example working on two bits. Suppose you are given a classical function f that takes two bits as input and returns one bit as output.

To evaluate f on all four permutations of two bits, we will need to call f four times: f(0,0), f(0,1), f(1,0), f(1,1). Exploiting quantum parallelism allows us to evaluate all four inputs in a single call to f. Note, however, that our function f is not reversible, and all operations on qubits must be reversible. So we first define a reversible version of f as follows:

The plus symbol with the circle around it denotes the XOR operation. The quantum oracle U takes two inputs x and y and outputs two values. The first is x itself. The second is the value: y XOR f(x). It is easy to see that when y=0, the second output equals f(x).

We set up the input ϕ as an equal superposition of the four two-bit inputs 00, 01, 10, 11. To do this, we initialise two qubits as 0 and apply the Hadamard gate on both of them. This is represented as:

Next, we apply our quantum function U on this state by setting y=0. This gives us:

If we separate out the two qubits, we see that the second output qubit contains the superposition of all four results we are interested in. Thus, we evaluated four inputs using a single application of the target function.

For a more detailed but still gentle introduction to quantum computing, please see the review paper at https://arxiv.org/abs/2006.12025

Discussion

Quantum parallelism forms the heart of many quantum algorithms. There are however some important caveats we need to consider. Although the output contains a superposition of the results we are interested in, we cannot directly read them. Any direct form of measurement will give us a single result and the other three will be lost. Clever tricks therefore need to be used to read out the final answer we are interested in, and these tricks may not be computationally trivial. Secondly, for our example, preparing the input state was simple, but this may not be the case for other problems. Whether using quantum parallelism will eventually lead to speed-up over classical alternatives depends on these caveats.

Quantum computers are already here, and it’s time we start taking quantum seriously. The more people from diverse fields embrace it, the faster it will become a reality.

If you liked the article, check out my other pieces on Medium, follow me on LinkedIn or Twitter, view my personal webpage, or email me at viraj@berkeley.edu.

Quantum parallelism — where quantum computers get their mojo from was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

It is said that advice is a dangerous gift, even from the wise to the wise. And yet, everyday we are flooded with advice that reaches us through newspapers, television, magazines, Facebook feeds, blogs, even marketing pamphlets. We are not only bombarded with unsolicited advice, but we also actively seek it from the web and print media. Entrepreneurs breakfast on management blogs, new mothers are glued to the Internet for advice on how to raise children, and retired executives turn on their favorite finance news channel every morning. Whenever we read something in print (or see it on TV) we assume it must be true without giving it a second thought. Unfortunately, much of it is utter bullshit, occasionally even sheer lies, and often the author herself is a victim of misplaced reasoning. Most pieces of factual news use some sort of statistical evidence to support their claims. Although statistics is invaluable as a tool to convert raw data into information, be aware that it is also a minefield of logical fallacies that we fall prey to a bit too often. Everyone, authors and readers alike, must learn to protect themselves from getting fooled by statistics.

Journalists use statistics all the time to create drama and sensation even where none exists. What happens when a dishonest journalist desperately looking for a new story comes across the following fictitious piece of data regarding potato prices over the past one year?

We can see from the table that there is nothing to be alarmed about. The prices today, in November, are the same they were almost a year ago in January. But don’t be surprised if you read a dramatic piece of news in tomorrow’s newspaper written on the following lines:

POTATO PRICES DOUBLE CAUSING FRUSTRATION AND RAGE

Poor rainfall and short sighted decisions by policy makers have caused potato prices to double in the previous 4 months from Rs 10 per kg in July to Rs 20 per kg in November. The spike in prices has caused an increase in food costs, ahead of crucial assembly polls and general elections in 2014. The increase in the inflation rate both at the retail and wholesale level should come as a fresh policy worry for the government and the central bank which are trying desperately to tame spiralling prices.

While the article above does state objective numbers, it clearly presents an incomplete picture for the sake of telling a story which is not entirely true. Manipulation of data pervades all fields including academics, medical sciences, finance, and marketing. If you have a large enough dataset, it is almost always possible to cherry-pick data so that it leads you to the conclusion you want. Dr. Peter Wilmshurst is a vocal critic of dishonestly and fraud in the medical profession. In an exposing speech that you can read here, he talks about Amrinone, a drug intended for treatment for heart failure. His team conducted experiments on the efficacy of the drug and found it to have severe side effects. The pharmaceutical company that funded the research, Sterling-Winthrop, went to great lengths to prevent these results from going public. They resorted to data manipulation, coercion, legal threats, and even bribery. So next time you read about a study claiming health benefits of green tea, check if it was funded by a tea manufacturer!

There is another tendency, hasty rationalization, which is endemic to journalism, especially scientific and financial journalism. Journalists are trained to report facts. They are not trained to draw inferences and offer explanations. Our world is an extremely complex system where most significant events worth reporting can occur due to multiple possible causes making it difficult to single out a cause. Yet, without considering alternate causes, journalists pick up the first plausible explanation they can find and offer it in such a brash and confident manner that we are left believing that’s exactly what happened. Nassim Taleb, in his eye-opening book Black Swan, gives the following example. After Saddam Hussein was captured, Bloomberg flashed news titled US Treasuries Rise; Hussein Capture May Not Curb Terrorism. An hour later treasuries fell, and Bloomberg flashed US Treasuries Fall; Hussein Capture Boosts Allure of Risky Assets. They attributed the exact same cause — Saddam’s capture — to two opposite events, treasuries rising and then falling. In reality, the rise and fall of treasuries might have had nothing to do with Saddam’s capture. So when you read news, keep in mind that there will be several alternative explanations which the writer might not have considered.

As I was writing this post, I came across the below article freshly published by TIME magazine. The piece titled Why Owning An Inexpensive Kindle Could Cost You Hundreds? demonstrates how naive and fallacious reasoning affects even reputed sources of information like TIME. The article says:

CIRP surveyed 300 U.S.-based Amazon customers over a period of three months this fall. Based on the results, the firm estimates that Kindle owners spend about $1,233 per year on the site, compared with $790 for Amazon members who do not own one. In other words, Amazon members with Kindles spend $443 more annually.

The article correctly states the objective fact that Kindle owners spend more, but the title Why Owning An Inexpensive Kindle Could Cost You Hundreds? wrongly implies that simply owning a Kindle will make its owner spend more. This leap of logic is blasphemous, and the article should never have been published. Successful people wear Rolex doesn’t mean just wearing a Rolex will make you successful. Basketball players are taller than others doesn’t mean playing basketball will make you taller. Similarly, the fact that Kindle owners spend more than others does not mean that owning a Kindle will make you spend more. Such flawed reasoning is commonly found in market research and analytics where data is abundant and people are paid to interpret it and draw conclusions.

The saying goes correlation does not imply causation. Somebody might tell you that they have reliable statistics that, in a particular city, an increase in ice-cream sales causes an increase in drowning deaths. But now we know that the correlation between ice-cream sales and drowning deaths does not imply the causation that ice-cream causes drowning. People eat more ice-cream on warm summer days than on cold winter days. People swim more on warm summer days than on cold winter days. Consequently, both the figures, ice-cream sales and drowning deaths, are higher in summer than in winter. In the real world, the link explaining the correlation is often much more complex than this so people mistakenly attribute the observation to causation. Now, spending habits of Kindle owners may not matter to you, but when such reasoning finds its way into that magazine offering advice on feeding your 6 month old child, you should avoid trusting its claims without due diligence.

Once you start looking for it, you see data manipulation and fallacious reasoning everywhere. But there is a deeper problem called publication bias that plagues academic research and is much more dangerous and difficult to overcome. The research community has a bias towards publishing positive findings and discarding negative ones. Imagine that you are a researcher investigating whether drinking coffee increases your likelihood of catching flu. You call in a group of subjects and divide them into two groups. You serve coffee to one of the groups and just plain water to the other. A week later you find that the incidence of flu was roughly similar in both the groups so you conclude that there is no evidence linking coffee consumption to flu. You are disheartened by the results and don’t submit them for publication. Or maybe you do submit them but they get rejected because your conclusion Coffee does not cause flu is uninteresting, has no impact, and it’s just so obvious. The failure of your experiment is never published and remains inaccessible to other researchers. Now consider the fact that, like you, there might be a hundred other researchers conducting the same experiment. It may happen that, in one of those experiments, the group that consumed coffee showed higher incidence of flu out of pure chance. When that lucky researcher submits his work for publication stating Coffee increases chances of catching flu, it gets published immediately because its a novel and sensational finding even though it is incorrect when you account for the past failures! If you happen to pickup the next issue of Health Today, you might get to read the article New research links coffee consumption to an increased risk of flu. The article will also quote an authoritative doctor offering a ridiculous explanation on how coffee decreases your immunity making you more susceptible to flu. Once the observations and the conclusions are established, it is easy to find an expert to conjure up a plausible causal link between the two.

Reboxetine was a drug manufactured by Pfizer and was prescribed for treatment of depression in Europe and UK in 2001. In 2010, after 9 years of usage, it was found to be ineffective for most patients (it was effective only for some special cases of depression). Publication bias during the trials had led to thousands of patients taking an ineffective drug. Universities and research institutes have started taking initiatives to encourage researchers to publish negative findings. Many organizations now make it mandatory to register a trial before commencing it and require the results to be reported irrespective of success. Let’s hope that these initiatives prove helpful.

We are fortunate to have easy access to information. It’s difficult to imagine the time when the only source of information we had were the handful of people around us. The printing press and the Internet has changed that, and we must take advantage of it. It’s not possible for us to verify everything we read. It would also be inadvisable to distrust all information. But we can choose when to be skeptical. We can afford to be relaxed and trusting when dealing with things that aren’t critical. But when it comes to things that matter such as our health, we must be careful!

If you liked the article, check out my other pieces on Medium, follow me on LinkedIn or Twitter, view my personal webpage, or email me at viraj@berkeley.edu.

Will You Drown If You Swim After Eating Ice-Cream? was originally published in Cantor’s Paradise on Medium, where people are continuing the conversation by highlighting and responding to this story.