On Gen AI Fatigue
This is my first Medium post — words I shudder at even writing. I don’t want to be a thought leader…please. I just like words, and find writing clarifies my thinking and is generally therapeutic. I will share this on LinkedIn though.
I am, at the best of times, fairly allergic to bullshit. This makes starting a company or doing AI research in the year of our lord 2023, an interesting experience. In general, I view this as a positive, though there’s a fine line between being that and being a cynic which I generally find exhausting. I wouldn’t have spent my career working in AI if I didn’t believe in it; but when the crypto boys turn up, words need to be had.
This writing is generally aimed at the non-AI/non-technical side of the audience; or at least someone who hasn’t been on AI Twitter constantly for a decade. If that’s not you, I hope you enjoy this, but YMMV. Also, goes without saying, but this is just like my opinion, man.
I’ll start by saying that if you’ve followed the news, or god forbid, use Twitter, you likely can’t avoid Generative AI (“GenAI”). So I’ll skip the preamble. I will say that like “big data” or the “cloud” or “blockchain” — GenAI today means whatever you want it to mean. Customer service software? GenAI. Emails? GenAI. Food delivery chatbot? That’s GenAI. Everything is GenAI. We’ve moved away from pedestrian terms like “statistics” or even “Machine Learning”; anything can and will be GenAI if you try to fundraise hard enough.
Amid all the blabber and fluff, there is real substance here of course. Some of the smartest researchers in the world have spent decades thinking long and hard about “intelligence” — what it means, where it comes from, how we can replicate it. Neuroscientists, philosophers, computer scientists, mathematicians and many more have each approached this from their own perspectives. This is real stuff — actual hard work, graft (in the British sense of the word, though now leaning to the American sense) and expertise applied to tough questions even when the funding was well into winter and nothing about it was exciting. The stuff you win Turing Prizes for.
Since, let’s call it 2010, we’ve been in a revolution of sorts. We found GPUs were useful and your pension fund started buying NVIDIA stock. People in disparate fields started talking to each other, nowhere more so than in DeepMind which changed the game. Big Tech resources combined with the who’s who of academia teamed up in a collaboration never before seen (till it was surpassed by The Expendables 2).
Then in a flash we had AlphaGo, AlphaFold; chess, proteins, starcraft, science — one by one knocking off targets in the race to AGI — artificial general intelligence. And then “GenAI”.
Generative Modelling is and was a thing. If you roughly start with the belief that the best way to learn is to predict, you end up with some notion similar enough to the current crop of methods. If you can predict what a missing word in a sentence is, you’ve probably had to grasp some basic concepts about words, objects and the world. Put another way, optimising a model to fill in the blanks, forces it to try and make sense of things. The best way to know that 1+1 is 2 and 48 + 65 is 113 is to learn the algorithm for adding; not just memorize all possible combinations.
In the mid 2010s, Generate Modelling (now rebranded “GenAI”, then part of “unsupervised learning”) was the sort of middle child in the AI family. Reinforcement Learning (teach things by giving them rewards) and its cousin, Deep RL, had been pushed to the top of the pile by DeepMind and AlphaGo; quickly followed up by work done at OpenAI (then, but a humble open source outfit, pushing the boundaries of human knowledge). “Supervised learning” was your bread and butter of ML — is this photo a cat or dog? Is this email spam or not? Classification, sorting, and assigning objects to neat piles. Unsupervised learning was the ignored child that plodded along well and good — but nobody really knew what its game was. Like where does this go? For those who had “AI” on their CV before ChatGPT — we’ve been here before. We generated beautiful photos in 2018 — no one knew what to do with them. We made tennis balls that looked like dogs [I will cite this at some point].
ChatGPT wasn’t even the first chatbot. If “chatbot” can even vaguely capture the magic of ChatGPT. So how does something essentially trained to predict words, start doing such cool things? And why does it capture the public consciousness like it does?
In short, in the intervening years two things changed: data and computational power. We understood the value of good, high quality data; and we learnt how to make these models massive on consumer hardware. Given enough capacity and enough data (say, the internet), these things developed emergent capabilities. The combination of maturity of research, market forces, competition and talent all came together in one sweeping crescendo. Your child’s year 12 band suddenly makes it to the big leagues. This is also why it captured the public’s imagination. If you train on the internet, you become good at important things that people want to play with: haikus, memes, inside jokes, banter. And thus started the GenAI revolution: based on a bedrock of cutting edge research, actual applications, memes, Twitter and a lot of fluff.
Today everyone and their siblings wants to do GenAI. Every software engineer is secretly an AI researcher. Every VC fund is an AI fund. Everyone has the next best product. I’ve been in this field for a bit and I am tired of the noise. And you might be too.
If there’s one thing you take from this, it’s the following: this thing is real. And like all real things, it attracts a lot of bullshit. It’s here to stay. Have we solved AI? Absolutely not. In the long list of options, generative modelling is just the first paradigm that broke through the barriers of big tech into the world of TikTok. Who knows what’s next, but there will be something. Maybe robots will get things done, finally. I’ve given up on self-driving cars. AI will be transformative like nothing we’ve seen, but it will also get commoditised. Every incumbent technology will morph to incorporate AI. Some startups will win. Bread and butter SaaS businesses will use AI under the hood somewhere, the way even your grocery business needs to use a database, eventually. Eventually everything will be impacted by AI; but not every AI application will be a billion dollar business. In the meantime, they’ll all try to raise at 400x ARR multiples.
