Data, Information, Knowledge, and GPT-3
An ex-student of mine (now a colleague) reminded me the other day of something I once did in an introductory course that he believes got him ‘passionately’ interested in computer science. I thought the story is relevant to this blog that is concerned with matters AI…
The Tale
In what seems like a century ago, back when I was briefly in academia, I used to teach an introductory course in computer science that I designed to introduce, albeit at a basic level, and using a highly declarative programming language (Haskell) most of the important concepts in computing. In the very first lecture, and after the complimentary comments, I drew on the white board what looked like a strange symbol. I then told everyone in the large lecture room: I claim that all the knowledge in the world, everything that the human mind has ever created, is in that symbol. How many of you agree with me, or believe me?
As expected, a shrug and/or a sarcastic smile was the unanimous reaction. Not even a single student raise their hands. Murmuring of “How could that strange symbol represent all the knowledge of the world?” can be heard throughout the large room.
I then said: hey guys, believe me, I am not trying to open the semester by proving to you I can be funny. I am serious. OK. Let me start simplifying this for you. You all have a laptop or a mobile phone or both. You know you can get there any text, video, animations, music, photos, etc. You can read any book that was ever written, look at the best paintings, listen to symphonies, watch movies and plays, anything the human mind has ever created can be digitized. Right? I stopped for a few seconds to let that sink in. And then asked: before I go further, is there anyone now that would like to flip their answer? Is there anyone now that believes the symbol above might after all represent all the knowledge in the world? At this point I got a few hands raised, though very hesitantly (half raised, so to speak).
It is bits all the way down
Well, although I got a few onboard, I knew they raised their hands for the wrong reasons. So I proceeded to ‘convince’ them that the symbol above could be (or could ‘contain’) all the knowledge in the world. So I then asked: how are the paintings, photos, melodies, documents, movies, etc. stored in your machine? This was an easy question, and almost all replied: they are bits (binary digits)— sequences of 0 and 1 (actually, it is a certain state in a flip flop, but essentially a 0 or 1, for our purposes). Alright, I said. So we have the picture below:
In other words, all that the human mind has ever created is now a large sequence of binary digits. But, any binary sequence is, after all, just a number. Right? They all agreed, because they all know that the binary sequence 1101 is the decimal number 13, for example. So, I said: all the knowledge and all creations of the human mind are nothing but a very very very large binary sequence. In other words, all the knowledge and all creations of the human mind are, in the end, just a very large number. It is such a large number that, for now, I will refer to it by this symbol:
Now, I said, how many of you believe that the symbol above contains all the knowledge in the world?
Where is the interpreting and understanding mind?
Amazingly enough, when I asked the question after I proved we can digitize all what the human mind has ever created, thus all that knowledge is represented by just one very very large number, more than half said ‘yes’ — the symbol above does represent all the knowledge in the world. But I was looking for those extra brilliant students. And I found a few. The ones that agreed with me half-way, now were very hesitant and did not raise their hand. I asked one of them: initially and before we went through the digitization process and obtaining a large number (which is the symbol above), you raised your hand in agreement that all the knowledge in the world could be in that symbol. Why did you change your mind? He replied like the brilliant AI scientist he is now: I changed my mind because now I realize that that symbol above might have all the data in the world, but not all the knowledge in the world. I was mesmerized. A star is born! Bingo.
Yes, I said. You are absolutely correct. That symbol might contain all the data in the world. But that data needs a mind to interpret it and understand it. We can even structure that data into information, and the machine could have not just all the data in the world, but all the information in the world. But still, to get at all the knowledge buried in there, the machine needs an interpreter, a mind. Data alone is … just data.
You are correct my friend — all that is in that symbol is just data. Now let’s start the course to see if we can one day build that mind-interpreter.
PS: I did not mention GPT-3 although it is in the title, because the entire article is about GPT-3 — that model that can (potentially) store all the data in the world — and perhaps some of the information — with a lot of rich correlations between all that data. But like that symbol above, GTP-3 needs the real AI, the real mind-interpreter to get at the knowledge and the understanding. And that’s where we should spend our time and our resources.
___
ONTOLOGIK — Medium
