ARPANET, Minitel, Mark I and the race to invent the internet
Some fun news first: I’ve signed a contract with No Starch Press to write my first technical book. The project can be roughly described as The Art of War for legacy modernization: when to do it, how to organize it, common problems and various tricks I’ve picked up to get stuff done. I got a wonderful opportunity this month to test out the content of early chapters on a live audience of software engineers when Øredev invited me to give a keynote in Malmö, Sweden. They asked me if I could do a talk on the intersection of technology and sociology and I was like “Oh boy… CAN I!” 😃
Anyway the book won’t be out until next year, but in researching it I’m already finding all of these fun little stories that are just slightly out-of-scope. So from time to time I’ll be writing them up into essays and posting them here instead.
Here’s the first one: France was trying to build the internet before America did. They pretty much owned us too, but things gradually fell apart when internal politics in the French bureaucracy interfered with technical choices and kneecapped the project.
I found something super interesting about this: I had heard the story before. But the protagonist in the version I heard wasn’t the French Fifth Republic, it was the Soviet Union. They also tried to invent the internet before ARPANET/NSFNET was privatized and they also failed because internal politics interfered with technical choices and crippled the project.
And that made me think: exactly how many countries tried and failed to create the internet?
As it turns out the answer is: A LOT.
Winners Write History
A term I’ve encounter over and over again in reading on the early days of computing and the internet is Whig histories. A Whig history is a highly idealized version of events in which the rise of technology is treated as inevitable and any failures, false starts or regressions simply erased. The story of the internet we all know treats it as a purely American invention that we invented by accident and that no other culture could have come up with.
In some ways there are typical American characteristics to how ARPANET became the internet. In particular, our habit of (mis)appropriating things from other places played a huge role. Most of the other network projects competing with ARPANET were nationalistic in nature. They were run by governments that thought adopting approaches developed in other places was a sign of failure. The use of local manufacturers was essential and non-negotiable. Building the internet in other nations was about the power such a network could have in advancing the nation’s economic and social dominance.
Yet despite being a military project, ARPANET managed to avoid such political trappings mainly because — unlike our rivals in other nations — we completely missed the point of a national network altogether. The nations of Europe and Asia predicted how a national network could be used as a cultural and economic force. They saw the big picture, they wanted to build to that picture. The American system, by contrast, was developed to just fix one annoying problem.
In 1957, the Soviet Union launched Sputnik I. The existential crisis and panic this triggered in the US lasted through three administrations — Eisenhower, Kennedy and Johnson. Bureaucrats and politicians were convinced that there was a critical skills gap between the US and the Soviet Union that would quickly prove a threat to national security. But no one really knew exactly how to fix that problem other than invest more in STEM somehow. So the Johnson administration did what the American system of government does really well: they lit money on fire, issuing a memo instructing government agencies to increase their grants to university research centers and not to limit the scope of inquiry.
Naturally the one thing every research facility wanted in the 1960s was a computer, but computers were expensive and too specialized at the time. The DoD’s newly created Advanced Research Projects Agency (ARPA) dumped money in research programs, those programs bought machines that fit their current needs. Often those needs changed and they found they lacked some hardware or another model was a better fit.
The DoD was dealing with seemingly endless requests for mainframes that were prohibitively expensive. ARPANET was conceived to network their research centers together so that researchers from one could use devices in another, thereby keeping the DoD from having to buy multiples of the same machine for different research sites. Although leaders at ARPA certainly understood the impact of such a network, they focused their goals on the collaborative benefit of combining the minds from a handful of the best institutions rather than broad economic and social impacts.
From the beginning the architects of ARPANET were not restricted politically in the same way their rivals were. ARPANET’s development as cost cutting and research sharing made it politically uninteresting, meanwhile in the UK and France similar projects were started up solely for political reasons.
In the UK they were worried about their computer industry being left behind and losing competitive edge to the US. France was worried about losing to the UK. National pride on the line, these project produced wins that the ARPANET eventually copied.
In France Minitel provided e-commerce services and chat rooms a full decade before such things began to appear on the internet. France integrated its network with the daily lives of millions of normal citizens by putting terminals directly in their homes for free. The Minitel experience was our future internet played out in a microcosm. Considered a threat to print media, the French government restricted access to creating Minitel services to registered newspapers. Tech entrepreneurs started registering fake newspapers in order to launch businesses to millions of potential customers. To think of the types of services that became available on the Minitel in the 1980s sounds like a science fiction novel: 25,000 online services that did everything from same-day delivery groceries to remotely controlling home appliances.
What’s even more remarkable about the Minitel story is that the same French government developed a dramatically better network than ARPANET and ultimately decided not to use it for Minitel. It was called CYCLADES.
I’ve often thought that the way to steer a project towards success in government is to make it unimportant. Once a project becomes important, political forces begin to optimize it in strange ways. One typical pattern with national networks was the assumption that it order to be successful there should only be one. Development of France’s CYCLADES was seen as undermining the development of the official network. The bureaucrats preferred the guarantees of a X.25 network, which circuit switching technology already had equivalents in telecoms. Sensing big government contracts, the telecoms obviously lobbied hard to help decision makers see it that way.
The end-to-end control of the X.25 made it seem like the smarter technical choice to government sponsors but made what was built on top of it brittle and hard to scale. Minitel was slow to innovate. The French computer scientists from the CYCLADES project shared their approach with their US colleagues and ultimately taught ARPANET how to adapt and absorb other technologies and other networks, setting the stage for the internet. But they could not convince their own government that decentralized and modifiable was a safer bet than centralized and controlled. To run a centralized computer network at scale you need to invest huge amounts of resources into the challenge of reliability. To keep people engaged in using interfaces, you need to keep them fast and modern. The investment simply wasn’t there.
Across the channel the UK Mark I was suffering from a similar problem. Although just a prototype, the UK’s Mark I was the first network to prove packet switching could be implemented. Their early success is why we call packets packets in the first place. The more common word in America for the theoretical concept before the Mark I was a block.
But strict budgets in the UK had limited its scope, and therefore its ability to demonstrate real value. On top of that their critical dependencies were public sector monopolies with no incentives to let them do anything at all. Telephone lines, for example, were controlled by the Post Office. The British government was also attempting to force British computer companies to minimize the number of machines they developed in order to create economies of scale and strengthen the market. Not only didn’t this work, it meant Mark I was originally designed for a computer which was ultimately discontinued.
The UK’s National Physics Laboratory (sponsor of the Mark I and Mark II projects) would face this problem over and over again. They developed innovative ideas but were often unable to secure funding to implement. The UK was in the middle of a decade long financial crisis — the coal market was collapsing, the pound would eventually be devalued. Computer research was only attractive if it could provide immediate economic value, products and services that British companies could sell. Theoretical packet switching networks did not fit the bill and so UK scientists, like their French cousins, ultimately shared their knowledge with ARPANET researchers where they were eventually implemented.
This general knowledge sharing between network researchers benefitted everyone. France took the ARPANET model and refined it. The US took the packet switching theories from the UK and implemented them, then took concepts from the French model and standardized them. The only group of scientists not hooked into this exchange were those behind the Iron Curtain. But perhaps that was just as well… Although the Soviet Union built lots of computer networks, three nation wide systems (OGAS, EGSVT and SOFE) were so plagued by political infighting they never got online at all. Envisioned as a way to better plan the economy the realities of data sharing and decision making across nation wide networks was enough of a paradigm shift to require organizational changes to the Soviet machine, which was not going to happen. Instead a national network for economic planning was built in a much smaller socialist country when Cybersyn went live in Chile.
There were many things that were poorly designed in the original ARPANET. Some by necessity, others a consequence of being the first to implement. Other national networks started from identifying the machine to be used and standardizing so that every node on the network was identical. ARPA was looking to network very different systems, none of them were designed to work together. Software during this era frequently had to be rewritten to work on multiple machines. The technical complexity of networking them all, including a few specialized super computers and other military projects, was beyond hopeless.
The solution was to buy identical minicomputers to act as a middleman between the research facilities and the network. These smaller, cheaper machines were called Interface Message Processors (IMP) and they were networked together to form ARPANET. Then the research computers would communicate with their designated IMP in order to talk to one another. The IMPs would break messages up into packets and route them very much the way DNS servers do.
This had two big effects. First it created the layering system we use for networking today. The research computers (the hosts) did not need to know anything about how the IMPs talked with one another. They only needed to implement a standard for talking to the IMP. Likewise the IMP didn’t need to know anything about what the host machine was or how it worked as long as the host machine honored the protocol and data contract.
Second, it meant the researchers using ARPANET needed to participate in the building of the network they were using. ARPA could not know how every machine worked or how to build out support for every operating system, but it could require that the research facilities figure it out on their own machines. This turned out to be critical. When connection speeds meant ARPANET wasn’t all that helpful with timesharing computing resources across the country, the users of the network essentially programmed in support for things that were useful, like email. Gradually ARPANET became a knowledge sharing network, rather than a resource sharing one.
TCP/IP — based on improvements developed by the French — arrived on the radar of ARPANET engineers just at the right time. As it happens the DoD hadn’t just funded one network, it had funded three different packet switching networks based on three different potential infrastructures of communication: ARPANET on phone lines, PRNET on radio, and SATNET over satellite. All three were up and running and it was not clear which would ultimately be more successful. ARPANET had a thriving community of researchers on it, but satellite and radio were more established mediums for military operations. So where a more efficient government would have probably picked one and eliminated funding for the others, the DoD instead decided to just network them together.
ARPANET’s original protocol, NCP, made that difficult. Because it had been built on telephone lines it followed the assumptions of telephone lines, meaning it was point-to-point. One of 256 logical paths between two ARPANET sites would be selected to establish a connection. Once open, no more than one message at a time could pass across the connection. Packet loss was not allowed.
PRNET, on the other hand, could not be point-to-point because radio does not send messages out across cables, but broadcasts them far and wide. ARPANET needed a new method for routing packets if the two networks were going to talk to one another.
This had a clever side effect, other government agencies who did not have access to ARPANET could now build their own networks and connect them to ARPANET, as the National Science Foundation did in the 1980s.
In America when we talk about ARPANET we talk about it as if it was the internet, but ARPANET was just a network. Computers had been networked before, ARPANET’s innovation was only that it was heterogeneous. The internet was a byproduct of nationalism of the 1950s and 1960s giving way to the globalism of the 1980s and 1990s. When ARPA developed a way to network dissimilar networks together the possibility of uniting all these national network projects into a global network became a reality.
Canada and France had connected their networks to the NSFNET by mid 1988. They were followed by Denmark, Finland, Iceland, Norway, and Sweden later in 1988; by Australia, Germany, Israel, Italy, Japan, Mexico, the Netherlands, New Zealand, Puerto Rico, and the United Kingdom in 1989; and by Argentina, Austria, Belgium, Brazil, Chile, Greece, India, Ireland, South Korea, Spain, and Switzerland in 1990 (MERIT 1995). By January of 1990 there were 250 non-US networks attached to the NSFNET, more than 20 percent of the total number of networks. By April of 1995, when the NSF ceased operating it, the Internet included 22,000 foreign networks.
What’s remarkable is that a military project was not injected with the sort of territorial nationalism that typically follows the words “national security.” Secured away under ARPA, an organization given the mandate to just dump money into research without much justification, the military remained mostly oblivious to ARPANET’s existence until 1975. And when the greater DoD did show an interest in the project, it was not grand ambitions of technical or economic dominance that made the ARPANET vision attractive but solving yet another simple, straight forward problem. It was faster to put data on a plane and physically send it to another location than to was to move it across DoD existing networks. When ARPANET seemed to be able to fix that issue the DoD took notice.
Ultimately stewardship of ARPANET itself was moved from ARPA to the National Science Foundation. Their own network by this time was outperforming ARPANET and ARPANET was eventually decommissioned in favor of NSFNET’s backbone. Eventually the network grew to the size where it could no longer be considered “experimental” and the US government privatized it.
One of my favorite lines in Benjamin Peters’s comprehensive history of Soviet national networks (How Not to Network a Nation) is in the back cover blurb:
ARPANET took shape thanks to well-managed state subsidies and collaborative research environments and the Soviet network projects stumbled because of unregulated competition among self-interested institutions, bureaucrats, and others. The capitalists behaved like socialists while the socialists behaved like capitalists.
- on ARPANET and Mark I: Inventing the Internet by Janet Abbate
- on the French Internet: Minitel: Welcome to the Internet by Julien Mailland and Kevin Driscoll
- a super interesting article using the Minitel story as an allegory for modern day net neutrality
- on the Soviet Union’s national networks: How Not to Network a Nation by Benjamin Peters