Big Data…and Bigger Data

Pages from Linnaeus’s magnificent Systema Naturae, a work which launched a system to replace cumbersome species names with short, binomial names: a single genus, and a single-word epithet.

One way to think of Empire is something that happens in response to an increasing supply of — and demand for — information. As empires bring together larger and larger populations of people, more and more information is demanded; more and more information must be supplied.

I mentioned earlier that the history of human civilisation sometimes resembles a 500,000 year battle between empire-building and self-determination. But perhaps there’s a better way to consider the history of civilisation:

A 500,000 year battle between big data and bigger data.

Even before the electric telegraph could transfer information, instantaneously, from the reporters in India to audiences in England, there were some world-changing innovations involving “big data.” Perhaps the most famous was, as the communications scholar Daniel Headrick points out, the Linnaean system of classifying plants and animals. It was a major breakthrough, this binomial method of naming things. Along with the other breakthroughs — such as Lavoisier’s table of elements and the metric system of measurement — the 17th and 18th centuries would become known as the age of classification.

Classifying is different than describing. It’s purpose is not to engage with a particular thing, in a particular moment, as Keats observing his nightingale or Elizabeth Bishop her moose. Rather, classification is about creating a database, then gathering and storing as much information as possible.

This was Linnaeus’s passion — not to describe in detail a single specimen, but to know as many species as possible and to name, classify, organise everything he saw. Long before computers, Julius von Sachs, the 19th century historian, gave an apt description of Linneaus: “a classifying, co-ordinating and subordinating machine.

An interesting choice of words.

Societies that could build these artificial data structures, collect, store, display (e.g. as maps) and transport the maximum amount of information had a survival advantage over those that couldn’t. At the peak of the British Empire, it’s estimated that just 4000 civil servants administered 40% of the planet. (Compare this to, say, the 150,000 public servants administering Australia today). This highly concentrated administration of power was possible because of the British Empire’s advanced IT infrastructure. That’s right, IT is very old indeed.

Here’s a member of the British Parliament, J. Henniker Heaton, speaking to the Royal Colonial Institute around the end of the 19th century:

“Stronger than death-dealing warships, stronger than the might of devoted legions, stronger than wealth and genius of administration, stronger even than the unwavering justice of Queen Victoria’s rule, are the scraps of paper that are borne in myriad over the seas, and the two or three slender wires that connect scattered parts of her realm.”

Heaton is right about these scraps of paper, these slender wires. They carried the stuff of Empire — namely the data necessary to control larger and larger populations of people. What kind of data? A bit of everything: market data, the latest prices, banking information, commercial data, legal rulings, the movements of people and commerce, news about riots, war, social events, entertainment, politics; not to mention a durbar in Delhi to commemorate the crowing of Queen Victoria as Empress of India. Very much like the big data of today really (just less of it).

A map of the transcontinental telegraph lines, late 19th century.

When we look, however, at the classification system of the time (the data structure, if you will), as well as at the algorithms used to make decisions that would affect India in 1877, we discover some organising principles. To begin with, there were existential fears, a desire to preserve at all costs the ginormous super-organism that was the British Empire. Any data that signalled a Russian invasion from the north, for example, could completely tip the algorithmic scale. Ditto any riots, mutinies, protests, anything that might show weakness.

The same was true about any ideas that might discredit imperial visions of human progress.

One such visionary was Adam Smith, who a century earlier in the Wealth of Nations, had written that “famine has never arisen from any other cause but the violence of government attempting, by improper means, to remedy the inconvenience of dearth.” Another visionary, of course, was the early “big data” demographer, Thomas Malthus, whose ideas about overpopulation shaped much of British imperial thought at the time. The Indian population, explained Lord Lytton to the Legislative Council of 1877, “has a tendency to increase more rapidly than the food it raises from the soil.” (This claim, of course, was categorically disproven).

So while the story of 100,000 people starving to death during a week of celebration in India might sound important — a story that humanity needed to hear — the artificial systems of information decided otherwise. The most valuable information, this system would determine, involved price fluctuations. “Railroads moved grain for the purpose of trade — not for feeding,” writes Mike Davis in Late Victorian Holocausts. Rather than transmit a representation of reality, the electronic telegraph sent out a kind of Linnaean shorthand account: The devaluation of the rupee, the increase in cotton prices, a threat of war to the north, the latest imports from India.

But what was really happening?

“London,” writes Davis, “was eating India’s bread.”

And the Londoners had no idea.

…to be continued

Note: I’m writing these blog posts as notes that I may someday compile in a more formal manner. They are not meant to be interpreted as works of scholarship — not yet anyway. If I’ve made a mistake or failed to attribute a source, kindly let me know so I can fix it. Thank you.