Intelligence Ratios
Why Computers Are Dumb and Will Probably Stay That Way (For Now)
No one has a universally agreed upon definition of intelligence, but a lot of people try to use intelligence as a metric anyway. Some agencies, like standardized testing companies (you know the ones that write the SAT, ACT, GRE, etc…), don’t explicitly say that their tests are intelligence tests. As an example, the SAT is an aptitude test, supposedly measuring future ability to do…something. Yet, the SAT actually measures ability to do specific fields of math and a certain level of reading comprehension. And a very subjective writing section tries to determine a different kind of thinking. What this test boils down to is testing certain types of intelligence through commonly learned subjects. It is directly trying to measure different kinds of thinking and reasoning, similar to the way an IQ test would. The SAT score one receives is a kind of twisted ratio, trying to measure someone against their peers as well as against their understanding of the material tested. This measurable ratio, while already a part of computer science in a lot of ways (statistical machine learning), I believe has a novel value in understanding why computer systems are not as smart as we think they should be.
I was introduced to the idea of a “Pavlov Operating System” by a friend last year. Like the work of Pavlov — the psychologist it is named after — the operating system is set up to be classically conditioned by the user. After a command of “do something” or “do something with [file]”, the user is offered a chance to reward or punish the operating system. Intuitively, rewarding or punishing the OS would tell the OS to change the probability of performing that action again. When we used the OS my friend showed me, my friend and I were sorely disappointed in the (lack of) functionality. The OS, when told to do something, would draw us a house as one of three trivial actions. When told to do something with [file], it would crash. As a computer science student, I thought about a better implementation.
Having a Pavlov OS running over a Linux system seemed the easiest route to go. In Linux, everything is a file, so the command “do something with [file]” should be simplified by not having to figure out what kind of file it is. No matter what the user inputs, fundamentally reading or writing the file at the operating system level is the same on Linux. Execution, however, is not. To execute a file, the file may take some number of parameters. If it is a well programmed file, incorrect execution will output a string with the correct usage.
After a brief look under the hood at a learning OS, we have already come to a fairly large roadblock in terms of implementation: executing programs correctly. As stated before, running a program wrong will often output a correct usage line. Many more programs will not. Many programs on Linux are also part of the Manual, and have well documented usage cases in there. The problem for programs with well documented usage cases is the fault of the computer: it cannot read natural language well, and cannot interpret what it cannot read into meaningful action. If we limit the scope of intelligence to being able to interpret usage cases for a program (the inputs that go along with a program) and correctly run a program, we can derive an “intelligence ratio” for solving this problem. This intelligence ratio is the number of steps the computer can perform autonomously (after someone presses a button to “do something”) versus the number of steps to “do something”. This ratio includes “prerequisite” steps, like being able to read. Right now, computers have nearly zero intelligence when it comes to running programs with complex inputs themselves. Computers aren’t that good at reading, certainly they are not to the point where they can open up the manual to a specific program, read the necessary information, and remember it long enough to use on the command line again. But if a computer could read proficiently enough to use the manual, what would that bring the intelligence ratio up to? The computer would be a big step closer to solving the problem, but not all the way there, and until it is all the way there, we consider it “dumb”.
Expanding the scope of what makes a computer intelligent beyond the simple example of reading a complex manual or command line output and using that info, there are a lot of smart and dumb computer systems. Usually the “smart” systems are ones that respond quickly to human interaction, directly or indirectly, and perform several non-trivial functions that involve decision making (a home security system that detects and identifies an intruder, and calls you and the police). The “dumb” systems are ones that either fail to do complicated tasks correctly, or perform a trivial task well (cell phone alarm clocks).
As computers are able to do more tasks that are increasingly complicated autonomously, we should be able to start assessing “computer intelligence” in batches of these tasks. A computer’s intelligence would not be determined by how much RAM it has or how fast the processor, but what it can use that RAM and processor for: software. In my previous example of limited scope, the computer could only be considered intelligent if it could actually perform the task of reading and executing a program. However, reading alone is non-trivial, and in a greater context, would make a computer “more intelligent”. After all, it is the ability of humans to generalize our abilities that helps us most — if we could only read a menu, or a spreadsheet, our powers of written language would be greatly diminished.
The road to more general computer “intelligence” is a long one. In part, simply getting a computer to read is very, very difficult. Many smart people have been and still are working to solve this problem. On another level, the way people use computers is not conducive to general abilities. As extensions of human ability, computers mostly serve us in specific ways, such as numeric calculations or applications to play games or make art. Computers are not used by people in general ways, so we are not developing them as general “intelligences”. We only care that a computer is “intelligent” in the way we use it. It is unclear what a general computer would be good for.
It is more clear what a “general intelligence” computer would act like. It would act like a person, in many ways. It would read and speak like a person so people could understand it in the most human-intuitive way. Right now the best we have are computer programming languages, abstractions that allow people to understand computer commands. However, to the computer it is still all zeroes and ones, no matter how poetic your code is. A “general intelligence” computer would be able to see like a human, too, which the field of computer vision is making strides in.
In many more ways it is unclear exactly how a “general intelligence” computer would act. Given that, right now, a computer has no method to act on its own, would a “general intelligence” computer need such an ability? And how would it come about? One possibility is a question asking function, which of course needs a general natural language processing ability as prerequisite. By asking questions about the data it observes, a computer may be able to decide action to learn more about the data.
General computer intelligence make the idea of an intelligence ratio a strange metric. The ratio works best when looking at a specific area because we can count the steps necessary to complete the task in that area, and weigh the steps currently achievable by a system. In a general sense, it becomes harder to enumerate all the tasks that need to be involved. It makes me wonder why so many human tests that are so important to so many lives hinge essentially on intelligence ratios, when it is so often our generalization of abilities that make us “intelligent”.
