Data

By 2020, there will be 5,200 GB of data for every person on Earth. There are 2.5 quintillion bytes of data created every day. More data was created in 2017 than the previous 5,000 years of humanity. More data has been created in the past two years than in the entire previous history of the human race, and every 2 days we create as much information as we did up to 2003.

That’s a lot of data.

The term big data has become trendy as storage has gotten smaller, cheaper, in the cloud, and accessible with increasing speed using technologies like hadoop and MapReduce, as well as new storage paradigms like NoSQL and Graph databases.

This explosion in data storage, retrieval and analysis is going to make our current world look like the Wild West —a time and place where there was such little documentation (data) that you could murder someone and simply move to another town and live like a king. We had no documents, fingerprints, pictures, DNA. Today, we are still living in a relative wild west, just to a lesser degree. In the future, the fact that any unsolved murder could happen will likely look just as ridiculous as the wild west looks to us now.

Types of Data

The data revolution is already having a huge impact on our lives. In the book Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are, author Seth Stephens-Davidowitz analyzes several sets of data to learn what people really do compared to what we say we do. In the past, we mostly got this sort of data from surveys, where people typically lie. Now we can analyze Google searches, where no one lies.

In an interview with Vox, Stephens-Davidowitz discusses American sexuality:

Well, to learn about sex, the main approach was to ask people. But people lie on sensitive topics such as sex… I think watching a porn video is a lot more telling than answering a survey question.

Some of his findings:

It’s clear that a lot of gay men remain in the closet. In places where it’s hard to be gay, such as Mississippi, far fewer men say that they are gay than in places where it’s easy to be gay, such as New York. But gay porn searches are about the same everywhere.

The entire interview is pretty fascinating.

Here are some other areas where big data is having a huge impact:

Voting

Most polls had Hillary Clinton winning the election, likely because those polls were based off surveys — and most people didn’t want to admit they were going to vote for Trump.

Google Maps has now been used to predict voting trends using neighborhood imagery.

Trump’s data was so good his marketing strategies for winning the election were fine tuned to the level that he was able to send particular messages to particular sets of voters in certain geographic locations that may very well have won him the election.

In the book Homo Deus, Yuval Noah Harari discusses the implications of a society in which we know so much about ourselves that our concept of free will is brought completely into question.

What’s the point of having democratic elections when the algorithms know not only how each person is going to vote, but also the underlying neurological reasons why one person votes Democrat while another votes Republican? — Yuval Noah Harari, Homo Deus, 2015

Health

Soon, with sensors, we will have a complete picture of our overall health in real time. Kevin Kelly, author of The Inevitable: Understanding the 12 Technological Forces That Will Shape Our Future, describes what this will look like:

Most people are lucky to see a doctor once a year to get some aspect of their health measured. But instead of once a year, imagine that every day, all day, invisible sensors measured and recorded your heart rate, blood pressure, temperature, glucose, blood serum, sleep patterns, body fat, activity levels, mood, EKG brain functions, and so on. You would have hundreds of thousands of data points for each of these traits. — Kevin Kelly, The Inevitable, 2016

If you’re about to have a heart attack, your personal health system will know beforehand, potentially saving millions of lives. We will know the quality of sleep we get, our levels of anxiety throughout the day and even the subconscious thoughts and reactions we have to certain events.

You get your genome sequenced. You have certain traits and predispositions to disease. As companies get more and more genomes sequences along with traits and diseases, this data will be analyzed to find correlations and learn exactly which genes are responsible for certain traits. Think, as one example, about the effects this could have on diet. If we were able to provide exact information about our food intake exactly, exercise routines, and ourgenetic makeup, doctors would be able to create a near perfect diet for each individual.

Sports

Every major league baseball ballpark employs the new Statcast system to capture every movement on a baseball field during a given game:

The Statcast system, for example, uses Doppler radar and stereoscopic video from two arrays of high-resolution optical imagers to acquire seven terabytes of data during each Major League Baseball (MLB) game.

Analysts can now, in real time, quantify the speed of the ball off a batter’s bat, the angle at which the bat makes contact, and the spin of a pitcher’s curveball. This has changed how players are evaluated and led to completely new strategies for both players and teams. They are relying less on anecdotal evidence and more on real statistics. No longer do they care about what a player looks like or acts like…the bias of humans is replaced by emotionless data. And its likely that umpires will be replaced soon as well.

Crime

Better data is helping law enforcement to use data modeling and analysis to predict who is more likely to commit crimes and use programs to intervene in the lives of those with a high score in their model before they do. In Chicago, such a model is being utilized:

“The Chicago Violence Reduction Strategy,” a program that utilizes experimental computer statistics and other factors that police say help predict who is most likely to shoot a gun and or be shot. Police say the computer program sifts through arrest records, including gun arrests and a person’s history of violence both as a victim or offender to gauge a person’s chances of doing harm or having harm done to them. They also consider other factors like local gang feuds on social media when deciding who to visit.

Maybe Minority Report wasn’t as bad as I thought it was 😕.

Implications

We are killing uncertainty. Many companies know us better than we know ourselves and with growing numbers of sensors like fMRI (reading minds), we will know more about what’s actually happening in the world and for what reasons than ever before.

What gets measured, gets managed.

We can currently measure the level of income we make and that’s been the primary driver of social status for ages. But what if instead, we all had a ‘happiness score’ — would we buy cars we didn’t need to impress people we don’t know? How would it change status? Would we care instead about who has created the most joy in the world today?

Dataism

In Homo Deus, Yuval Noah Harari describes the ‘data religion’ — the idea that everything is data and information. Humans are just chips in a computing system. The economy is the interaction between those chips. Central processing is communism compared to a decentralized system like capitalism (where interesting parallels can be made with decentralization of data and the blockchain).

Why did the USA grow faster than the USSR? Because information flowed more freely in the USA.

Animals are just data processing machines, equipped with algorithms that take in data from the surroundings from their sensors, which they use to calculate probabilities to take action. Feelings are just outputs from the algorithms that make certain actions more probable.

What do we believe?

We used to worship gods, then ourselves and soon this will morph into the worship of data. When we needed guidance we looked to the stars, then to God, then to our feelings; but now, increasingly, we look to data. For instance, who should I marry? This genius passage from Homo Deus drives home how much big data will change our world:

‘Listen, Google,’ I will say, ‘both John and Paul are courting me. I like both of them, but in different ways, and it’s so hard to make up my mind. Given everything you know, what do you advise me to do?’ And Google will answer: ‘Well, I’ve known you from the day you were born. I have read all your emails, recorded all your phone calls, and know your favourite films, your DNA and the entire biometric history of your heart. I have exact data about each date you went on, and, if you want, I can show you second-by-second graphs of your heart rate, blood pressure and sugar levels whenever you went on a date with John or Paul. If necessary, I can even provide you with an accurate mathematical ranking of every sexual encounter you had with either of them. And naturally, I know them as well as I know you.

Based on all this information, on my superb algorithms, and on decades’ worth of statistics about millions of relationships — I advise you to go with John, with an 87 per cent probability that you will be more satisfied with him in the long run. ‘Indeed, I know you so well that I also know you don’t like this answer. Paul is much more handsome than John, and because you give external appearances too much weight, you secretly wanted me to say “Paul”. Looks matter, of course, but not as much as you think. Your biochemical algorithms — which evolved tens of thousands of years ago on the African savannah — give looks a weight of 35 per cent in their overall rating of potential mates. My algorithms — which are based on the most up-to-date studies and statistics — say that looks have only a 14 per cent impact on the long-term success of romantic relationships. So, even though I took Paul’s looks into account, I still tell you that you would be better off with John.’

What will this mean for how we make decisions and how we feel about free will?