Losing Serendipity in the Quest for Knowledge

But I’ve been in the wrong place
But it must have been the right time
I been in the right place
But it must have been the wrong song

Dr. John (1973)

I hope that all of you reading this have been in the right place at the right time at least once. It could be anything from buying that lottery ticket to bumping into that old friend sitting at the next table in the café. Or it could have been, “Of all the gin joints in all the towns in all the world, she walks into mine.”

Serendipity is more than coincidence, less than fate. It is the collision of innumerable micro choices that leads to something good. You chose that train, you read that book, you smiled. Of course, calamities and tragedies are also the consequence of a constellation of choices. But serendipity is a word reserved for the confluence of events that leads to a happy ending, not a tragedy.

How many things do you think you know? A thousand? A hundred thousand? A million? Chances are that some of what you know was the product of serendipity. That class you took because the one you wanted was filled. That TV program about penguins you half-watched after scrolling through 100 channels. That random conversation while waiting at the checkout.

And, what about scientific discoveries? How many of them could have just not happened if one chance occurrence hadn’t led to the next and the next and the next? This is how the American Chemical Society (ACS) describes Alexander Fleming’s discovery of penicillin.

Returning from holiday on September 3, 1928, Fleming began to sort through petri dishes containing colonies of Staphylococcus, bacteria that cause boils, sore throats and abscesses. He noticed something unusual on one dish. It was dotted with colonies, save for one area where a blob of mold was growing. The zone immediately around the mold — later identified as a rare strain of Penicillium notatum — was clear, as if the mold had secreted something that inhibited bacterial growth. (https://www.acs.org/education/whatischemistry/landmarks/flemingpenicillin.html)

Fleming was not looking for penicillin, but that happy accident led to saving millions of lives. Which leads me to my main point. We don’t always know what we’re looking for.

When search engines first appeared in the 1990’s and early 2000’s, we were happy to have any way at all to look something up without dragging out a heavy book, running to the library, or “phoning a friend.” We were OK that you had to dig through a bunch of inappropriate results to find something close to what you were looking for. When Google’s search engine hit its stride in the early 2000’s, we became huge fans because its algorithms had matured to the point of giving us “in the ballpark” results. We still had to scroll through the HTML links and browse a few sites (sometimes going down the rabbit hole), but we more often than not found what we were looking for.

Interestingly enough, Google’s search engine algorithms owe something to serendipity. By using page ranking of what humans chose most among the links, Google was learning what we were most likely looking for based on search terms.

We also found and read a lot of information that was not what we were looking for.

Sometimes what you were not looking for was pretty interesting. Ideas can be sparked by following the breadcrumbs from one thing to the next to the next. Or by following that tangent that takes you down another path entirely. The current state of search has been junked up by ads, but it still functions as a source to browse by.

If we stay on the Google track, we can compare its search engine results to the results we get when working with Google Bard. If I ask Bard a question, I get back the “definitive” answer, definitive meaning the best answer that Bard can give at the time. I don’t get to make a serendipitous choice among results. I have no choice but to ask it another question to get it to refine the answer. And sometimes I don’t know the next question to ask. I have lost the opportunity for serendipity in the quest for knowledge.

We still don’t know how AI and LLMs are going to change the world. But we do know if we start relying solely on LLMs to give us answers, we will gradually lose the opportunity for serendipity. Currently, we don’t trust the models enough to let them be the only source of answers and knowledge. But as the models increase in reliability, we will probably reach that trust point where all questions will be fed into the machine and all answers will be spoon-fed to the questioner.

I used to admire a world like the one depicted in Star Trek: TNG. The computer never gives the wrong answer unless some alien force has meddled with it.

“Computer, what is the vector for…”

“Computer, give me the analysis of…”

But now the idea of that world makes me sad. I want us to cherish serendipity because we really don’t know what we don’t know. Serendipitously, I just looked up the movie from 2001 with John Cusack and find that it is starting in 15 minutes on HBO. Long live serendipity!