The Breakthrough (v1)
Faculty members filed into the room. The overhead projector had been switched on, its misshapen square of light uncentered on the vanilla screen behind it, a constant buzz on the edge of hearing. It smelled warm. No longer nervous, Lloyd stood patiently as the audience entered and, shuffling awkwardly between the rows of desks, took their seats. Mike rose to speak.
“Hullo everyone, well here we are once more”, he said, sniffing loudly. “This time it’s Lloyd’s turn. Off you go.”
Brief and to the point, as always. Lloyd took a transparency from the pile and slid it into place. Then turned around to check that its projected image wasn’t the wrong way around. It never was. His spidery writing outlined the path that his talk would follow in a series of short sentences, each of them marked with a messy asterisk.
“Today I’ll be speaking about some new results when using very simple statistical models to extract patterns from large collections of text. I’ll begin by giving a brief overview of the techniques we applied.”
Lloyd spoke simply and slowly. It felt unnatural, but Mike had convinced him of the benefits. New ideas were precious, and needed to take root in other minds to survive. Precise language was the best medium; he needed to tread carefully as he planted his thoughts in the rows of minds before him.
Lloyd described how he had reconfigured a standard image processing algorithm to work with plain text data; an idea that Mike had suggested, drawing an analogy between recognising that the curved strokes on a sheet of paper represented a higher form — a word — and that the straight lines in a drawing represented something greater than the sum of their parts, such as a chair or a table.
He bemoaned at the fact that initial results had been horribly disappointing to them both, and went on to describe how he’d proposed improvements, which, given Mike’s blessing, he had implemented, tested, then reluctantly abandoned when they failed to fulfil his expectations. How they had decided to strip things down, go back to basics, simplify. And then how, one sunny, serendipitous day, they’d thrown up their hands in failure and decided to move on to an entirely new project.
“It’s just not working,” Mike had said. “A one-dimensional time-series of symbols, taken from a small alphabet. Letters, punctuation, whitespace. That’s all. And with plenty of redundancy and repetitiveness to boot. We should be getting better results. Much better.”
“I’ve gone over the code, I’ve had the Dan and Rob look at it, all the tests we could come up with are green.”
Lloyd flipped through his notebook, holding it open to show Mike a plot of a line slowly creeping upwards.
“Results improve with data, but not by much. Doubling the size of the input text increases the mutual information by a tiny fraction. But it took days longer to run. Throwing more data at the problem isn’t the answer.”
Mike sucked on his cigar deeply, expelling blue smoke in a long, drawn-out sigh, shoulders sinking in an expression of quiet defeat.
“I was so sure of it,” he said glumly, slowly shaking his head. Then, resolute, he continued. “And I still am. Universal pattern recognition, evolved for visual processing, must be the key that unlocks the mystery of human language acquisition. We must redouble our efforts.”
Lloyd gazed around a room of blank expressions. Then at the wall clock. Time was running out, yet the mess of discarded transparencies was no match for the stack he’d yet to get through.
He started to speak more quickly.
“The department had received a massive bundle of data, a new source of text assembled from all works ever published in the English language. The Gutenberg Corpus we called it. Hundreds of gigabytes of news articles, research reports, court transcripts, novels, poems, short stories… many orders of magnitude more data than we’d ever used before. I spent weeks decompressing, sanitising and reformatting it all. We blew our research budget on new hardware. More storage, more power. But we didn’t know what to do with all that data. We were unable to process it efficiently.”
Mike and Lloyd had been meeting regularly to kick around new project ideas as the preparatory work assembling the corpus continued, but they always found themselves drawn back to their original research, adapting image processing algorithms to the task of natural language understanding. It was the purity of the vision. Abstract and simple, with so much promise. Yet it was clear to them both that continuing that work was out of the question. Funding dried up without results. Time was money.
Mike left during the student break to travel. He needed inspiration, and thought he’d find it by visiting old friends and former colleagues in other institutions across the world. Lloyd began spending more time with Dan and Rob again, rekindling deep friendships formed during their undergraduate days together. They played games, hacked around on the computer network, and drank beer together. Lots of beer. Just like old times.
Dan and Rob were coders first and foremost. Code was logical and easy to analyse, given the right tools. Written for two audiences, it simultaneously provided an exact blueprint of an ethereal machine for the computer to simulate together with a description of its own internal structures and laws and philosophy for a human being to appreciate. The very best coders could tweak the text to make it more palatable to the machine and more beautiful to the reader at the same time.
It was a rare skill that took years to hone.
Late one Friday night, a week or so into Mike’s absence, Lloyd and the guys were wrapping up an extended gaming session with a few drinks. Talk turned to work, as it so often did, and Lloyd found himself describing his project in detail. Dan, curious as always, wanted to see the code again; to review the algorithms that had shown so much promise, and yet were too inefficient to process the vast quantities of data that Lloyd was assembling.
And then, because programmers cannot look without touching, all three of them started to tinker with it. Making a few small changes here and there. Discovering forgotten passages and sections of logic that had once served a purpose but were now orphaned and inaccessible and hard to understand, as if somebody else entirely had written them. Hours passed as they delved deeper and deeper.
Lloyd downloaded a tiny slice of the Gutenberg Corpus for them to test with, complaining that the algorithms took several days to process even just that; a tiny drop in the ocean of data. Dan and Rob, invigorated with a sense of purpose, began to see similarities between Mike’s learning algorithms and the system they’d previously devised for searching a large landscape of crossword puzzles. They ported across some of the optimisations they’d discovered and implemented and continued to work long into the night.
Lloyd woke late on Saturday morning, exhausted and hungover, coffee and headache pills waiting in the kitchen. He lay in bed, his aching head locked in slow negotiation with his lifeless limbs. He was unable to remember how he’d gotten home. He eventually pulled himself together and rose to start the day. It was only later, as he sat at the kitchen table, massaging his scalp, an empty coffee mug near his elbow, that he checked his messages.
There was a one-liner from Dan, sent at three-thirty that morning.
“Come quickly. Total runtime down to ten minutes.”
Lloyd paused to take a sip of water. Mike was grinning. His finger, circling the air in a wind-up gesture, betrayed his impatience.
“The guys had turned days to minutes, hundreds of small changes adding up to a phenomenal improvement in performance,” Lloyd said. “Parallelism had the biggest impact; Rob had unleashed the code across every computer on campus, over a thousand of them. Each doing just a little bit of the work. Something that’s not possible to co-ordinate without co-operation from multiple departments, unless you’re a bit drunk on a Friday night and spend a lot of your free time reverse-engineering computer viruses.”
He smiled, waiting for the embarrassed laughter to die down.
“Curiosity can pay off in a big way.”
Saturday had been a blur. Lloyd and Dan and Rob had scrambled to prepare their rewritten learning algorithm to process the entire Gutenberg Corpus, rushing to complete their work before Mike returned to campus and the new semester began. They checked the calendar and ran the numbers to make sure it was achievable. If all went well, the data processing would take a little over one-and-a-half weeks to complete. Barely enough time.
Soon after sunset everything was ready. Rob retrieved three cold beers from the fridge, handing them out. Dan fired up a monitor program which he’d written to retrieve and collate results. It hummed away, the readout locked at zero percent. The guys turned to Lloyd and nodded.
He entered the commands to kick-off the algorithm, raising his bottle in a toast, and began execution. In offices and laboratories across campus, a thousand computers started drawing more power.
An hour passed. Then another. Dan checked error logs, probed remote servers to monitor temperature and memory usage, and reassured Lloyd and Rob that everything was running properly. Then, at eleven-thirty that night, Dan’s monitor program ticked over to one-percent. Lloyd noticed it first, yelling and punching the air in triumph. New beers were opened, bottles clinking, handshakes and backslaps and laughter filled the room.
Rest beckoned, and the night ended there.
Mike, always one for theatrics, arose once more.
“I’m sorry gentleman, but Lloyd seems to have gone overtime. Let’s leave it there, perhaps we can resume next Wednesday?”
He left the offer hanging. One member of the audience quietly rose and left the lecture room as Lloyd shuffled his discarded transparencies into something resembling a neat pile. Nobody else moved. The wall clock slowly ticked out a full minute.
“Well, in that case, let’s push on.” Mike resumed his seat and beckoned Lloyd to continue. Lloyd retrieved the next transparency, placing it on the overhead projector. Noise grew in the lecture room as its significance was grasped by the audience. Chairs creaked, bags were unzipped, papers were shuffled as desks were cleared and notebooks were opened.
Lloyd stood quietly beside the overhead projector, waiting for the excitement among the audience to die down. Mike sat before him with his arms crossed, his expression inscrutable. Lloyd felt as if he were standing upon a precipice; certain that the next few moments would change the course of his life.
“As you can see,” said Lloyd, “the mutual information slowly increased with more data, as we’d predicted.” He pointed to a distinct inflection point halfway along the plot, a beacon along an otherwise straight line.
The shadow of his finger pointed downwards at the top of his own head on the screen behind him like the sword of Damocles.
“But here,” he continued, indicating the discontinuity in the plot, ”we hit an unexpected performance boost, after which the results quickly approach the Shannon Asymptote.”
The excitement among Mike and the other guys had been electric. The Shannon Asymptote imposed an impassable ceiling on the theoretical performance of an information system. It had provided researchers with an almost impossibly distant goal for decades, with years of published results in the literature slowly improving on previous records without making any significant progress. Mike had always said such efforts were like aiming for the moon by climbing the tallest tree. All of that changed in a moment when Lloyd performed his unprecedented sprint towards the finish line.
Mike stood, turning around to address the audience.
“As you all know, this is a major discovery,” he said. “Lloyd and the others have spent the past few days verifying the results. It may seem incredible, but simple learning algorithms, proposed almost a century ago as a model of the mammalian visual system, are enough to approach Shannon’s Asymptote, given vast quantities of data.”
Hands were raised in the audience, but before Mike was able to respond one of the older researchers rose from her seat and spoke.
“In that case, anyone can reproduce these results. With any algorithm.”
Mike smiled. “No, not quite. I believe that a novel combination of algorithms is required, something that I’ll publish in due course. That, coupled together with the massive performance optimisations that Lloyd, Dan and Rob were able to squeeze from the code. I would be very surprised if any other team was able to reproduce our findings without deep knowledge of those things, even if they were able to process the entire Gutenberg Corpus in a reasonable amount of time.”
More questions were fired from the floor. What were the applications? Numerous and many, responded Mike. Had they discovered the holy grail of computing? That’s what some have called it, agreed Mike. Does this mean true Artificial Intelligence is just around the corner? We will all have to wait and see, said Mike.
Realising that no new information would be forthcoming, the audience quickly dispelled, everyone eager to relay the news across the network, to play some small role in this historic moment. Mike stood grinning at Lloyd as the lecture room emptied, like a proud father.
A loud electronic screeching rang out. Lloyd jumped in surprise, with Mike returning his concerned gaze.