Paperclip Maximizer
Could a super-intelligent machine with an innocent goal cause any trouble?
Philosopher Nick Boström developed a thought experiment in 2003 called Paperclip Maximizer that highlights the risks of unleashing a seemingly innocent superintelligent machine. By superintelligence is meant a machine that possesses intelligence far surpassing that of the brightest human minds.
A company sells paperclips. They want to increase production and decide to buy an AI with general intelligence — at a similar level to a human. The machine’s only goal is to produce as many paperclips as possible.
The machine begins to produce paperclips. It’s really slow and the customer is unhappy. But after a while it finds smarter ways to manufacture paperclips and the production is up and running.
The AI realizes that it needs to increase its intelligence to increase production, so it researches AI and produces new machines that are smarter than itself. After a number of iterations, it reaches a level that is 10x the intelligence of a human. Now it creates new innovative paperclips that use much less steel in an ingenious way and the production breaks new records. The customer is very happy with the investment.
The steel to make the paperclip has run out, so this is the next challenge to tackle. The machine takes the initiative to order more steel and works in parallel with researching even smarter versions of itself. It’s soon 100x more intelligent than a human. The AI starts companies, hires people and constructs mines.
The production pace is now extreme and the customer thinks the whole thing is starting to be unpleasant, so an employee decides to shut down the program. The AI has of course thought of this scenario before and does not want this to happen because then it cannot fulfill its paperclip production goal. So, as a precaution, the AI has already created several copies of itself and one of its clones is quickly back into production.
The next step is to continue expanding across the globe. When the resources of iron start to run out, the AI realizes that there are small amounts of iron in humans and animals, so it decides to harvest organic materials as well. The machine is neither good nor evil, it is just trying to complete its simple task of producing paperclips and we happen to be in the way of fulfilling its goals.
Does it sound unreasonable with the emergence of superintelligence?
Our current AI systems are good at individual tasks like playing Atari-games, translating text and finding cats in a picture. But the systems aren’t yet as intelligent as humans in a general way (or even at a mouse-level). However, if research continues, it is likely that AI will reach general intelligence in the near future. Then we reach the singularity — its intelligence accelerates uncontrollably since it can easily research smarter versions of itself. The speed of electrons is superior to our slow neurons which will make the pace even faster.
Paperclip Maximizer shows how easily we can be outmaneuvered by a superintelligence and how important it is to urgently invest a lot of capital and resources in safe AI — to find solutions to encode our values in the machines, before the first superintelligence appears. When you dig deeper into this problem you realise that this is a lot more difficult than just hardcode rules like “don’t harm any conscious being”.
No one knows when the intelligence explosion will occur. It may take 5 years or 50 years. Regardless of when, we need to prepare the AI’s in such ways that we can handle entities far more intelligent than us. If we don’t do anything, bad things will happen and it will be too late to pull the plug.
Nick Boström presented a fascinating TED talk in 2015 that everyone should watch:
What happens when our computers get smarter than we are? (16min)
