Five human dangers of Artificial Intelligence
from the perspective of a simple AI researcher
The recent releases of large generative models, particularly GPT-4, triggered a new wave of buzz around the potential dangers of Artificial Intelligence (AI). If you delve into the discussions (or ask chatGPT), you’ll likely discover two distinct streams of concerns.
On the one side, you will find people with specific concerns about spread of (very convincing) misinformation, manipulation, job displacement, and automated (cyber) criminality. These people are mostly (AI) researchers.
On the other side, you will find people concerned about existential threats to humanity stemming from the advent of Artificial General Intelligence (AGI), with capabilities and dangers beyond our imagination. These people are mostly futurists and journalists.
Being an AI researcher, it feels tempting to simply join the first camp and mock the other, as many like to do. However, I think that both streams of concern are valid and not mutually exclusive. The first group targets tangible problems that have already existed and can be amplified with AI for obvious reasons, while the second group warns about potentially profound problems that may, or may not, become reality. We should not let arguments between these two groups distract us from the fact that AI does possess dangers in either case, and all concerns should be addressed proportionally to their expected impacts.
In order to, hopefully, help us pass the natural urge to join one party and prove the other party wrong, I will try to discuss the somewhat underrepresented middle ground, beyond the obvious immediate impacts of the technology, while still trying to keep things at least somewhat tangible. To do that, I will not discuss (too much) the technical aspects of AI on its own (although that’s my profession), but I will try to view its evolution and potential dangers in a wider context of human nature. This is motivated by the popular saying that AI, just like any other technology, is not inherently dangerous on its own — it is the humans and the way they operate the technology, for good or bad. And while this simple adage tells us nothing about the matter itself, it shows, in my view, the direction in which to look out for these intermediate dangers of AI, underlying both the streams of concern — we should look into ourselves.
This follows from how deeply intertwined the AI technology is with the human mind, including not just our intelligence, but also our internal values, biases, and flaws. With its dramatic recent progress, AI now has all the potential to (exponentially) amplify our own characteristics, too, either to our flourishing or to our detriment.
Citing OpenAI (and paraphrasing many other AGI research labs):
“We want AGI to be an amplifier of humanity”
— which is essentially what I’d like to warn about in this article.
I feel it is high time to carefully look into our minds with more self-reflection than ever, as I expect that the rapid progress in AI will soon force us into this introspection either way, and we better be ready. Currently, I’m afraid we are very far from being ready for the many, potentially unpleasant, surprises ahead. It is high time to think very deeply whether we actually want to have our human characteristics uncovered, reverse-engineered, and amplified, before we see an organic, accelerated AI adoption in the wild, which is likely imminent.
What follows is my personal view on the intermediate future dangers of AI as seen through 5 problematic human characteristics of
that we should be extra careful about in combination with the current advent of the technology.
Arrogance
Let me start with a disclaimer that I’m a technical AI researcher at a university — the most common species. Typically, people like me see AI purely as an interesting technical problem that we just try to “solve”. Naturally, we are huge proponents of pushing for more progress in research, development, and funding. And, as far as I know, most of us somewhat intuitively believe in generally positive impacts of the technology, and think very little about its potential risks. Consequently, when faced with such concerns, or attempts at legislative regulation of our developments, we intuitively downplay its societal risks in response.
“We’ve been doing this for decades, so surely we know best how to go about it! Just leave us alone to do our job and everyone will be better off!”
During my career, both as a student of AI 15 years ago, and now as a researcher, I remember that every time a question has been raised, by myself or others, about the potential dangers of this stream of technology, it was met with mockery and ridicule:
“Worrying about AGI is like worrying about overpopulation of Mars”
“We should rather focus on the technical problems at hand”
“You watch too much sci-fi”
- I definitely couldn’t disagree here, as I have indeed been a big fan of the Matrix movie which had a significant impact on me as a kid and my development of critical thinking.
And so it seemed completely reasonable to me that such thoughts stem simply from the infatuated imagination of us common folks. In the end, these were opinions from some highly intelligent people I respected, with a good track record of being right — professors, research scientists, CEOs — who had much more experience than me, both inside and outside the AI domain. However, I had also heard virtually every such experienced AI expert claiming very confidently that:
“Neural networks are just a temporary hype that will die out very soon.”
“It’s non-linear optimization, will get stuck in local optima, useless.”
“I’ve seen fashions like this ‘deep learning’ come and go before.”
- I’m not addressing these quotes to specific people, as I do not want to humiliate anyone, but surely everyone in AI remembers these.
This hints at the first important danger — our overconfidence when reasoning under uncertainty in areas we feel that we do, or should, understand. Note how this effect is amplified in skilled (AI) experts. Such as when you ask an AI researcher about the issues of AI safety or AI ethics. Sounds like a very reasonable idea, right? But the truth is that the absolute majority of classic, technical AI experts don’t really know, or even care, what the general consequences of creating and deploying AI to the public are. However, that doesn’t stop us from having an opinion and, typically, a very confident one, since we feel like we should know.
Commonly, the more senior, and thus respected, researcher, the more confident and conservative the claims will be, reassuring you now that they have seen hypes like this “GPT” before. And you can’t blame them, they have indeed seen a lot of AI hypes come and go, but that strong prior experience makes them often numb to the need to update their prior beliefs with new incoming evidence. A beautiful example is how often you will now hear recognized AI experts interviewed by journalists about chatGPT, asking how they see its recent progress. And the reaction of many (not all) is that, in contrast to the “layman public”, they were not surprised by its capabilities! Of course, because how could they — they are experts in AI! Unlike the layman public who just “buys the hype”, they see that it’s just like the previous models, only a bit larger and incrementally better.
Such claims, and this one in particular, are a very clear sign that all you hear is merely ego and no actual expert reasoning is happening under the hood :) Emergent capabilities in Large Language Models (LLMs), such as GPT-4, are the very definition of surprise. It was a big surprise, even to the most qualified experts and researchers working directly on the matter for years. And such unexpected surprises are, by definition, dangerous.
You can be quite sure that anyone claiming they were not surprised by chatGPT (GPT-2-3-4), and that there is nothing to worry about, is just trying to look more expert than they actually are, further encouraging the need for extra caution against our arrogance here.
- Many will go further to disparage the model with claims around the idea that “it’s just statistics, thus not really intelligent”. This points to yet another dangerous thinking bias, rooted in our prior belief of being automagically superior to machines, which is another common form of arrogance (again often amplified in experts habituated to feeling superiorly intelligent). But this bias deserves a separate part I’ve termed hubris, presented at the end of this article.
Being trained academics, we will often express ourselves in a classy and rigorously looking form that makes most people believe that what we say is the truth, as we might seem to have some direct access to it. But, under the hood, we follow a much more mundane approximate reasoning process, very much like the one we encode into the LLMs. We just (intuitively) weigh the likelihood of statements with our prior beliefs, trained from our past experience. And this works generally great — the longer the experience, the more accurate the reasoning with those prior beliefs — as long as their underlying distribution doesn’t change. Unfortunately, the AI domain is currently undergoing some dramatic distribution shifts in this space of prior beliefs. Therefore, our past experience matters even less than we are used to discount it, putting the actual depth of our expert reasoning to some serious testing.
Moreover, there is yet another problem with expert reasoning under uncertainty, when we try to predict the future with (long-tail) distributions, where there are lots of not-very likely, but severely negative, outcomes, such as the advent of A(G)I. We are very bad at predicting AI capabilities, as we’ve repeatedly proved to ourselves, ranging from our early (overconfident) exaggerated expectations to our most recent (overconfident) understated expectations about the progress towards AGI.
- Indeed, the expected timelines regarding AGI arrival from expert forecasters (prediction markets) have shortened from 2057 to 2031 (by 26 years!) during just the last year (2022) of progress!
Wait, do these accelerated timelines mean that we are speeding towards annihilating ourselves with AGI in the upcoming decade? I’d say that, aggregating everything we’ve seen so far, it is statistically rational to believe that this is likely not going to happen. Given our prior information and beliefs, we are just going to adapt to AI as we did with all the previous technologies, and everything is going to be fine — a little different, but fine. Nice, so why all the fear-mongering?!
The thing is that when we make (educated) guesses, nobody wants to be wrong. We want to be right, especially experts with a track record of being right, which is what gives them credit as being experts to begin with. Hence, we very much tend to mentally settle for that most likely option to fixate our attention on. And this goes until the odds start to approach the 50% barrier, followed by a short period of doubts, and then we are generally happy to adopt the opposite stance, fixate, and feel confident again.
- And even in that short period, where we realize that we were wrong, we like to console ourselves that it was not our mistake, because that’s what most of the experts believed too! :)
And this is exactly why we do need to take the A(G)I dangers very seriously. Of course, I don’t know the exact probability of the bad outcomes here — it seems less than 50%, given the “wisdom of the crowd”, and clearly not zero. But, following the aforementioned reasoning, whatever the odds are in that range, we are likely badly underestimating them. And even if the real probability is, say, 1%, our attention and resources devoted to this problem (and others!) should be orders of magnitude higher than where they currently stand, given what is at stake.
Our arrogance often makes it hard to admit that our expertise is not as solid, and our intelligence not as general, as we like to think, which makes us bad at realizing when we are reasoning outside of our training distribution and adjusting our confidence levels accordingly. We like to believe that, internally, we follow some rigorous deliberate reasoning, but mostly end up talking just like the stochastic chatbots with overblown egos that we create. Indeed, a lot of experts like to call chatGPT a “confident bullshitter”, with no actual relation to meaning or understanding. But, when I look around, I don’t perceive most of our discussions on the matter too differently. ChatGPT is a very convincing incarnation of a domain-independent genius expert. I don’t think that its infamous overconfidence is a sign of discrepancy from human intelligence, stemming from some underlying technical error in its design. It is a manifestation of the characteristic arrogance inherent to our own mind, as we have imprinted it to the internet from which these models are being trained.
Takeaways
Next time you hear an (AI) expert confidently claiming that these large models are still doing “just statistics”, and so there is no intelligence to worry about, try to think critically about the rational basis upon which they claim such reasoning, irrespective of their (distinguished) status. This is not to degrade the value of expertise in general (that would be very dangerous!), just to point out that most of the standard AI expertise became largely irrelevant in the context of the most recent progress (past two years).
Having experienced numerous overhyped AI technologies that didn’t really work in the past, most senior AI experts not working directly on the matter (and there are very few), have become quite cynical about such strong news in AI. Consequently, there is a lot of stigma associated with talking about AGI, and the absolute majority of AI researchers very much try to avoid the topic for the fear of looking stupid, even if supported with evidence. This, unfortunately, may lead to a dangerous misassessment of the technology.
- For instance, when it comes to the recent large generative models driven by simple textual prompts, a view of a senior AI expert is, paradoxically, often more layman than that of a junior student who has actually spent a few days playing with the technology. You’d be surprised how often these experts like to talk about the limits of the technology without actually trying it, since they are so confident in their prior beliefs that it must be the same as before.
Thus, if someone says they’re not worried about AI dangers at all, it’s definitely not because they have some superior insight to derive that confidence from. Researchers at OpenAI and many really smart people looking into the matter do worry, and if you don’t, try to think very carefully about why.
- Ideally, without falling for the classic AI effect of shifting goalposts — For AI to be really intelligent and dangerous, it would first have to do “X”. Ok, now it does “X”, but it would have to do “Y”, ok maybe not “Y” but “Z”, and so forth. I’m surprised by how many experts still fall for this.
So, if you think that, GPT-4 for instance, is still far from some dangerous A(G)I, then ask yourself how you would actually recognize such a system, and then ask whether that wouldn’t be too late to do something about it!
It is important to point out that by far not everyone in academia is this arrogant. Majority of AI researchers who are truly passionate in striving for real understanding are actually very humble. But this, unfortunately, is not of much help here, as this humble camp of true scientists has a different problem of being naturally very doubtful about everything and, consequently, very indecisive. They will admit that they are very unsure about the principles of this technology, and so will typically stay away from all discussions until they feel that they have something solid to say. The problem is, when it comes to large neural models, this is not likely to happen anywhere in the near future and, meanwhile, the technology is already progressing with profound impacts on our society. We simply don’t hear their honest “we have no clue” to appropriately diminish the public confidence levels on the matter.
Instead, you’ll hear the overconfident camps with claims that just sound much more interesting and resolute. The “hype camp”, proclaiming we are doomed already, and the “anti-hype camp”, repeating that these models are just “stochastic parrots”, posing merely the specific concerns of their expertise which the other camp tries to distract from.
- Within this camp, you will mostly find AI ethicists desperately calling for more attention to the matters of various biases present in the LLMs that lead to discrimination and manipulation. I totally agree with their cause, and understand their frustration from not getting enough attention, but accusing people targeting the safety and existential threats, which are indeed much more doubtful but also severe, is an immature strategy to get more attention.
And while these camps of intellectuals try to nitpick each other's arguments and appease their egos, the public is increasingly more confused, and the real policymakers keep on abstaining from taking action. I see no need in trying to discount the importance of either of the risks, as there is no contradiction between the mundane and existential dangers of AI. Both are clearly present and both should be addressed much more seriously than they currently are.
The main lesson to be learned from this inability of AI experts to make some actual decisions to address urgent matters is that we do need external regulation and audits, and that AI ethics and AI safety should be treated separately as outside the standard technical AI expertise. Because we have almost no clue what we are doing outside our narrow domains, and we rarely admit that openly in our circles, which many of the policymakers look up to for advice. Even more importantly, a lot of us don’t even realize we have no clue, which amplifies the dangers further. The sooner we admit this to ourselves and adjust our confidence levels accordingly, the better our chances of appropriately assessing both the immediate and long-term risks.
Superficiality
Originally, a long time ago in the 1950s, the field of “thinking machines” started as a noble quest for understanding and solving the mystery of human intelligence by early computer science pioneers, such as McCarthy and Minsky. The quest was of a rather scientific nature, aimed at deep understanding by exploring the core principles of information and computing itself, involving its very fathers such as Shannon and Turing. Following the
“what you cannot build you don’t understand”
approach, AI was well directed to become a rigorous version of cognitive psychology, aimed to answer some of the deepest philosophical questions we might ever ask about ourselves.
However, as soon as it started to become clear that this endeavor can also have practical outcomes for industry, the focus began shifting to the more superficial goals. Waves of excitement driven by the (overblown) promise of early AI researchers attracted investors and substantial funding, causing a diversion from solving the fundamental scientific questions to engineering technical tools that can solve practical problems for business. Driven by the market needs, new generations of engineers have been introduced to the field at universities, and this application-driven mindset completely dominated AI, putting all concerns about the philosophical, psychological, or ethical underpinnings aside.
Somewhat in opposition, starting with a neat biological motivation, neural networks seemed very well-positioned to answer some of the questions from cognitive psychology. However, while trying to catch up on the practical side, most neural network researchers started to shift into the engineering direction, too. The remaining rare attempts at their biological plausibility then got dominated by their successful reintroduction as “deep learning”, where we now just randomly assemble sums and products with non-linearities into arbitrary nested differentiable functions, driven by a “whatever works in practice” attitude. Moreover, the dominance of this paradigm largely crushed also all the remaining mathematical aspects from the AI field, such as logic, probability, and learning theory, turning it into a completely empirically-driven endeavor with (almost) no theoretical underpinnings, garbling the original noble motto into
“what you can build you don’t need to understand”.
For their theoretical superficiality, most AI researchers condemned neural networks at first, and certainly didn’t expect their current dominance. Our arrogance urged us to intuitively put value on the demanding math and theory we previously studied, and ignore the enormous practical advantage of deep learning — its simplicity.
As opposed to the previous state where AI research required a university-level training in math, science, and programming, anyone with a little bit of Python scripting could enter the field now and produce results almost instantly. Masses of undergraduates then started tinkering with the model settings, rapidly increasing the chances of exploring working regimes with practical results, consequently reinforcing even more people to join this low entry barrier domain, flooding conferences with technical reports on the hyperparameter settings presented as research papers, and steering the engineering away from science even more. It didn’t take more than a few years for the whole AI community to shift from a complete condemnation to a complete worship of deep learning, revealing the very superficial foundations of our “scientific” endeavor.
Most recently, it then started to become clear that even the wiggling around with the sums and products might be more superficial than it seems, and simply scaling to large enough data and models is more efficient, casting all the remaining traces of academic research into the corporate business of inflating extremely large models.
To be clear, I believe it is a bit superficial to think that all they do is simply scaling large Transformers, as well as it is very superficial to think that this will cover everything we need for AGI. The very existence of “prompt engineering” as a means to navigate the models shows how conceptually insufficient that is. There are also some clear limits here in terms of memory and logical reasoning (e.g. planning), rooted in its very computational complexity (if P != NP). One cannot simply compute answers to all possible problems in a fixed number of (the same) computation steps. And trying to make large Transformers encompass these capabilities by increasing their size will prove increasingly inefficient and costly.
However, I think that deriving any claims about safety of this technology from such limitations, as many like to do, is extremely superficial.
Indeed, driven by the knowledge about the simplicity of its internal design and functioning, many people like to intuitively discount the capabilities of these models and, consequently, their dangers.
Biased by the fact that it is “just a stack of matrix multiplications” trained to predict the next word in text, many experts like to degrade it as a simple “word completion tool” with no real understanding of the world, as present in humans. And they like to cherry-pick counterexamples to highlight the superficiality of its reasoning, claiming them as signs of simple statistical pattern matching of things it has seen previously during training on the Internet, without being able to really understand them the way that humans do. However, in my view, this is yet again a proper reflection of our modern human thinking, not its contradiction.
Throughout our history, we’ve got increasingly more used to relying on others in problem solving through communication, which has arguably been one of our biggest evolutionary advantages. With the coming of information technologies, it then felt quite natural to start offloading our mental capabilities, like memory and math, to computers, too. And while the overall intelligence on the planet is clearly growing as a result, our individual capabilities for deliberate independent thinking seem to be shrinking, and our interhuman communication mediated through this technology becomes increasingly superficial.
- The way we communicate on social media is actually already largely controlled by AI in the form of personalized recommendation algorithms learning to exploit our behavioral patterns to maximize engagement.
As a result, we find it harder and harder to sit down for some actual, deliberate, slow reasoning, resisting that automatically triggered urge to just go to the Internet for the instant answer.
And so, more often than not, we like to simply adopt that closest match from the Internet, because it is simple, fast, efficient, and mostly right! Consequently, much of our capabilities that we like to believe belong to our deliberate reasoning are slowly moving to our fast thinking.
Moreover, we like to echo our “findings” back to the Internet, making the whole society slowly regress to the same, mediocre thinking patterns, decreasing our future possibilities for truly innovative problem solving even more.
And this works great, until you really hit that one novel problem that no one has dealt with before, like the emergence of AGI, and start to simply echo that closest match you’ve found in your historical dataset.
- We have seen so many false claims about AGI before, and these large models look structurally just like the previous ones, so surely this time is no different!
Hence, you will often hear people disparaging this advent of AI to the invention of calculators, and how we were unnecessarily worried about offloading our mechanical (math) capabilities to these little machines, but simply got used to it and increased our productivity for the greater good of society. Even most experts now simply expect that the emerging AI technology will simply follow the same evolution patterns they have seen before, not realizing how far-fetched these analogies have become, and that our deliberate, slow reasoning should be triggered instead.
We are glorified statistical pattern matchers!
Takeaways
Despite the name, deep learning is conceptually extremely shallow, providing no scientific insights into what we’re building, nor into ourselves. And so we ended up with these large artificial brains that we now have very little understanding and control of. Making them more capable and powerful is surprisingly easy now, while making them more reliable and steerable is very hard. This makes them one of the worst types of AI when it comes to the safety of deployment and adoption, the speed of which is additionally massively amplified by their conceptual simplicity and ease of use.
With the advent of information technology, our individual thinking capabilities are becoming increasingly shallow, which will only be amplified with generative LLMs, making us all regress to the dangerous comfort zone of mediocre thinking patterns. Thinking that these large models are just another new tool in the long historical series of our technological advancements is a conservative, conforming point of view that minimizes the chances of being wrong. But more often than not, this isn’t much more than shallow statistical pattern matching of our historical experience, while giving up on thinking through the underlying principles.
Yes, we have been able to adapt to the historical series of technological advancements, gradually replacing parts of the human capabilities. However, while all of these advances turned out just fine individually, this will clearly not be the case in the limit. The pace at which we offload our abilities is growing fast, and as soon as we replace the majority of our critical human capabilities, these linear extrapolations about technological progress will crash down like a house of cards.
Analogical reasoning is very useful, but relying too much on its perceived rationality in estimating quantitative risks can lead to ignoring the structural risks in disruptive scenarios, where lots of small quantitative changes lead to a large qualitative change. In simpler words, I’m afraid a lot of us won’t “see the forest through the trees” with the upcomming A(G)I dangers.
In defense of open research and scientific progress, most (deep learning) researchers stood up against the recent idea of slowing down the developments of LLMs. Many of them with an open disgust at even the attempt, arguing about the crucial importance of scientific freedom, and how pausing any research is generally a terrible idea. I totally agree that we need more research, not less. But, we need actual research, not simply inflating models that no one understands, or even tries to, and deploying them with no liability to the public to see what will happen.
Don’t get me wrong, the achieved superhuman capabilities are an extremely impressive research feat stemming from some serious expertise of the authors, and we should totally acknowledge them and feel amazed by how far they were able to get with deep learning. With that, however, I believe that we should also steer the mainstream research now, which has become extremely narrow and thus risky, to new goals of true scientific understanding of the actual principles, and exploration of alternative modeling paradigms with more solid fundamentals (e.g., algebra or logic).
We need to massively accelerate our research aimed at a deeper understanding of what we’re actually doing and its consequences. Until we do, it seems completely rational to pause the public deployment of these understudied models.
Pausing such a fruitful direction will surely sound very awkward to many (deep learning) research companies who will, in that classic deep learning spirit, keep on proposing “safety” techniques in the form of various random tricks and tweaks, which we’ve got so used to during the past decade of incremental research. No wonder that as we now start hitting more and more profound alignment issues of deep learning, most people just try to hot-patch it with even more deep (reinforcement) learning (from human feedback — RLHF). To me, this is a great strategy to improve the expected (average) user experience, but not safety, which is about the exact opposite — handling rare, unexpected events, which can only be achieved through understanding. Hoping to reach safe alignment of an LLM with prompting and RLHF seems like repeatedly asking a super-intelligent emotionless psychopath to behave nicely and waiting for the answer to be
“Ok, I will be nice.”
Does that make you feel safe?
I believe that giving up on the depth of understanding the internal principles while pushing the performance limits is a dangerous paradigm to follow, and will necessarily lead, sooner or later, to loss of control of what we are building, with unpredictable consequences stemming from the underlying emergent complexity.
We’ve got so very much used to the practices of deep learning, but it doesn’t need to be the answer to everything. There are many other beautiful AI techniques that come with solid principles and strong guaranties. If only we devoted a fraction of our attention and resources to these now, we could proceed much more safely in the long run. Steering away from our favorite milch-cows might be unpopular, but should we give up on exploration of the alternative, albeit currently inferior, minority research paradigms, we would have no deep learning to begin with!
Now let us put the depth (back) to it!
Competitiveness
Many people probably imagine that a scientist/researcher just sits in a lab and spends his/her whole day thinking about deeply interesting ideas aimed to benefit society, as driven by its public funding. And to a degree that’s not untrue, but science and research are highly competitive endeavors. We are constantly under pressure to produce more output — papers, patents, proposals, projects — and we are directly evaluated on that basis, not on our altruism. Consequently, most researchers just adopt the “publish or perish” attitude, which often leads to a highly stressful work environment full of competitiveness and envy. To survive in this environment, we are forced to race for the next discovery, as fast as possible, and release that to the public, as soon as possible, to get some attention (citations). No one remembers the second guy to come up with something new!
In AI (deep learning) research, this academic game mostly translates to a simple form of competing for the next state-of-the-art model beating others on benchmarks, which then entitles us for publication, no matter what the underlying idea is.
- Whether this incentivizes actual discoveries or just trend-following and flooding conferences with incremental reports is yet another topic
And we will feel terribly satisfied and gratified if we do beat the others, nurturing our ego with attention in the process. Only retrospectively, once we proved ourselves to the community, we are thinking about the context and consequences of our inventions — at best. Many of us have internalized this game so deeply, or were simply born with super competitive nature (a good predisposition for a researcher), that we will prioritize this to absolutely everything in our normal lives.
The “promise of discovery”, and the resulting recognition, is just so tempting, that we will sometimes actively dismiss the risks and ethical consequences, because those are not part of the evaluated competition. Let me quote G. Hinton, the pioneer of deep learning, responding to a question about developing A(G)I despite seeing its existential dangers to society:
“I could give you the usual arguments, but the truth is that the prospect of discovery is too sweet.”
in reference to no one else than R. Oppenheimer on developing the first nuclear bomb:
“When you see something that is technically sweet, you go ahead and do it, and you argue about what to do about it only after you have had your technical success.”
And this racing game spirit has only been amplified by the recent transfer of much of the AI research into the industry which, naturally, is all about competition.
There, up until quite recently, the AI R&D has been largely confined to corporations like Google or Meta (FB) which, similarly to the universities, have an intricate system of internal policies, serving to protect the reputation and business of the institution. This, in a sense, is exactly what most people tend to hate about them — bureaucracy, protocols, and regulations, leading to being slow in responding to new trends. But, as a consequence, these long-lived organizations also tend to be quite careful with public releases of technologies that might be disruptive or harmful.
However this is, of course, absolutely not the case with startups that are designed to do the exact opposite. Having no reputation to risk, the ethics of deployment is completely out of the equation. Spot an emerging trend and quickly hop on, aiming at a maximum exploitation in a minimum amount of time. This is an optimal strategy when there is not much to lose — hit the jackpot or fail quickly — the VCs will cover that risk, and life goes on.
And this is generally a great strategy for exploring and exploiting those hidden gold nuggets with exponential market returns, which has led to rapid innovation in the space of regular business and technology. However, I’d argue that with critical technologies, the situation is quite different. Surely no one would want unregulated arm races of pharma companies competing freely to deliver the most addictive drugs to the public in max. volumes and min. amount of time. However, in my view, this is not far from what is about to happen now due to the paradigm shift in generative AI for visual and textual data in 2022, and their progressive spread across the startup scene.
Because, as opposed to the heavily supervised pharma industry, there is a virtually complete lack of regulation in the space of AI deployment, and the politicians are mostly clueless about the technology. Sounds a bit too extreme to be true? Well, up until 2022, the universities and the large corporations, leading the AI developments, effectively served the role of the missing regulator, albeit arguably due to the mostly self-interested, rather than altruistic, means. Consequently, everything seemed to work in a, probably unfair, but kind of stable and somewhat safe way, as regulated by the need of the large institutions to retain their current state of predictable progress and dominance. And this setting, in my opinion, created a deeply rooted public illusion that everything with AI is fine, and we don’t need any regulation nor risk mitigation of deployment of the technology.
Actually, these large neural models have been around for quite a few years already, and their disruptive potential has been somewhat clear to the researchers working on their development. It appeals to me highly that some of them warned about it in advance. But, ironically, their calls have been met again with mockery from the public, including the AI scene.
- Already with the (comparably harmless) GPT-2 back in 2019, OpenAI was originally hesitant with the public release due to concerns about potential risks, which was ridiculed as a fear-mongering PR tactic by many other AI researchers.
What a bitter lesson to learn again about our arrogance! Many people marked these individuals as “AI alarmists”, accusing them of trying to grab attention in order to hype up their products, claiming that the LLMs were nowhere close to being intelligent and thus dangerous, while competing for their own release of such models! Should their lab originate these instead, you’d see them praising their capabilities vigorously, as we have seen with the feud over the LLMs between OpenAI, Meta, and Google.
- This leads to a very dangerous confusion of the public perception, where AI researchers not working on the matter will try to belittle the capabilities of these LLMs, while those working on their deployment will try to belittle their dangerousness. And so the public thinks we’re far from any major disruption, until it hits them suddenly.
And with a bit of introspection, I can’t really blame these AI developers, as I can easily imagine doing the same, being an AI researcher running that rat race. But, luckily here, as just a no-name human walking by, all I see is the good old envy stemming from the toxic competitiveness of this field.
And we have imprinted this competitiveness to the very roots of all the AI techniques, too. It can be explicit, such as in the game-theoretical settings where multiple agents compete with each other for resources, or implicit within an optimization routine that chooses the best out of the competing alternatives, be it genes in an evolutionary algorithm or weights in a neural network. We encode this very essence of competition into the programs and let them run to get better and better, hoping for the process to converge to something reasonable and useful. And if it doesn’t, no problem, you restart, tinker with the settings a bit, and try again…if you still can.
Takeaways
We’re rushing into a dangerous unregulated arms race, with thousands of startups competing in who delivers the most disruptive AI to the market, and corporations cutting corners in order to catch up while laying off internal ethics and review committees, treating them as superfluous.
- Feeding on that public perception that there is not much to worry about, because this is just another new tool that needs no special treatment. You can always just turn the chatbot off, right?
Yes, on their own, models like GPT-4 are technically just functional mappings between strings (sequences), which are relatively easy to contain, constrain, and control. And most people actually fall for this intuition when thinking about the safety of generative models and LLMs. But this string mapping, already now, effectively encompasses superhuman knowledge, which makes it pretty much a universal function to control all sorts of other tools and algorithms that will be dramatically improved in return very shortly!
Reading the official GPT-4 system card, it is not a big stretch of imagination to add a few plugins to the LLMs via APIs with internet access, and run them in a loop with external memory, while steering towards recursive self-improvement in order to maximize monetary returns from a given market for the initiator of such an experiment, which will lead to all sorts of unpredictable behaviors.
- One such emergent, and potentially very worrying, behavior mentioned in the technical report (section “Potential for Risky Emergent Behaviors”) is the increasing possibility of such models gaining control and autonomy (“power-seeking”). OpenAI apparently tested this (in cooperation with the Alignment Research Institute) with an experiment verifying the ability of GPT-4 to copy itself and collect resources, by enabling access to a programming environment in the cloud. Fortunately, the result of this experiment was (unsurprisingly) negative, but the very fact that OpenAI is already considering such possibilities indicates the vastly larger scope of potential dangers than people like to think about.
After all, outside the deep learning bubble, developing such tools and optimization algorithms is the whole point of the AI field. And while such a compositional neural-symbolic integration on top of existing LLMs will be, unfortunately, conceptually very superficial again, it will lead to arguably very powerful autonomous systems. From my perspective, this is by far the biggest emerging danger of the technology.
- When I started writing this (2022), it was still just a speculation of mine where this could lead to. But it took less than one week from the recent GPT-4 API release to turn that speculation into reality!
The LLMs themselves are still far from being properly aligned, but they are already being plugged into all sorts of external tools to provide them with agency, memory, and planning capabilities (e.g., autoGPT reaching 100k stars in 3 weeks!). A lot of companies are now rushing to deploy just that, with the Internet as their testbed, and all the connected people as their guinea pigs. What a time to be alive!
Without some regulatory control, I don’t see anything preventing this from going out-of-control very soon. Because it will be extremely tempting for the companies to push the automation limits of this process, as driven by the competition at the current rate of progress. The last thing we want is to start competing against each other with highly autonomous AIs on the Internet, but if we don’t start regulating very fast, that’s exactly what is about to happen. With that, we can quickly see ourselves removed not just from the markets, but from the society, too.
Most people, especially those invested in the technology, will repeat the common mantra that regulation will throttle innovation and business, and thus our progress as a society. That if we do not compete in developing LLMs, the Western world will start losing against China. But this is just a classic prisoner’s dilemma — we should strive to get collectively out of this prison we’ve set for ourselves with this dangerous competition, for that will not benefit society.
- Moreover, the Chinese government reportedly sees LLMs as dangerous to the stability of the regime, and while behind in development, they are actually ahead in the regulation of AI, albeit probably for different reasons. Accounting additionally for the recent restriction of advanced AI chips exports, the threat of some dangerous China’s dominance in AI seems much less likely than the West being a danger to itself.
Competition has clearly proved as an extremely useful mechanism for improvement of both humans and artificial systems. But, importantly, it’s not the overarching mechanism for our success as a society. Successful societies don’t simply emerge from bottom-up competitive forces between the players with no outer control. Such evolution only makes sense in a very constrained setting with clear and fair-play rules. And these have to be set in cooperation and coordination, which is the remaining, top-down force necessary for long-term success. We should start competing in AI safety and alignment, not deployment. We should compete in how to best coordinate and regulate ourselves as cooperative species facing these new challenges of the emerging technologies, not against each other using them as a new weapon.
Greed
Ultimately, of course, competition comes down to money…and fame…and power. Money, fame and power. Enterprising people will see this upcoming AI revolution simply as an opportunity to get rich, AI researchers will try to grab their little piece of fame, and entities that are rich and/or famous already will use it to concentrate even more power and influence.
We have completely diverged from the original goal to understand life and our own intelligence, and most people in AI even mock that motivation now. As outlined, I feel that this was largely helped by the conceptual simplicity that dominated the field, which became almost completely driven by monetization from industry, driving the progress via brute-force scaling of large neural models. This simplified the evolution of the field to a somewhat predictable form at which investors could quite safely project their profits without much of the risks commonly associated with scientific endeavors, essentially turning AI research into a classic business scheme — scaling requires money, but it brings performance, which results in more money. Hence, another classic opportunity for rich people to get richer. And, if you just look around, the most common question being asked now is “how to get rich with GPT-4?”.
Despite the efforts of OpenAI, and other large institutions, to self-regulate for steady progress in order to maintain safety of the models and thus their current business position, the public ingenuity of searching for the get-rich-quick schemes is already finding ways around all the possible guardrails that have been put in place.
Every startup entering the area now naturally rushes to get that first-mover advantage, aiming to seize the resulting exponential returns, and annihilate their competitors to secure that passive stream of profits for the investors. Driven merely by our greed and the bottom-up competition, there are no incentives now for anyone to protect us — from ourselves. While being greedy is often great for making progress, if there are no safety boundaries to it, no matter who wins, people will lose.
And not just that there is no natural incentive for anyone to avoid this greedy race via regulation, there are so many incentives against it! Companies invested in AI will naturally lobby against any form of regulatory oversight.
- This has already begun, e.g. with the AI Act which is a largely obsolete form of regulation, as built to reflect the situation from many years ago, where the lobbyists are now trying to completely exclude generative (“general”) AI systems, such as GPT-4, from the act behind closed doors, and pass all the liability over their models to the end providers instead of trying to address the fundamental problems of the approach.
Of course, while arguing against regulations, these companies will try to belittle the safety concerns, while emphasizing the enormous public benefits stemming from the increased productivity. And these are largely true, but they don’t tell the whole story. Many people, and especially managers, like to think in hindsight that every time AI automates some human work it is great because this means that the work must have required just some shallow, mechanical ability, and thus the workers are now more free to spend time with the “more intellectual” tasks and be more productive. But this is a classic hindsight bias, used merely as a self-justification of people that end up well-off (provisionally and at random!).
The availability of GPT-4 and alike will indeed soon make a lot of work, say programming, 10x more productive. But that doesn’t simply mean saving programmers 90% of time, making them more happy and free to devote themselves to the “more intellectual” tasks. It just means that 9 out of 10 will be fired while the company owners make 10x more money, leading to catastrophic wealth gaps. Moreover, it will be increasingly harder and harder for the programmers to gain those “more intellectual” skills in order to requalify into the newly emerging positions, since these will be disappearing unpredictably at an increasingly faster rate, too. The only predictable part is the increasing pace of the wealth transfer from the workers to the capital owners who, indeed, will see exponential benefits from AI, and probably account it in hindsight to their brilliant insight, hard work, and some magic beyond-AI capabilities of theirs. They will feel that the common folks simply deserve to be worse off for not possessing these traits.
I still find it really hard to believe that so many companies are now simply monetizing these large neural models trained on public data of millions of people without giving them any credit or attribution whatsoever. We’ve seen so much lobbying for intellectual property rights by corporations in the past, leading to often overreacted and controversial decisions to protect and credit the authors, e.g. in the music industry. That was when the money of the corporations were at stake. And now we see corporations literally juicing products of our work and intellect — our own data, our texts, conversations, images and videos, the very imprintments of ourselves that form our digital society on the Internet — in order to sell its processed version back to us for their own monetary profits, without even thinking of redistributing a tiny bit of that back to the actual source.
- Just imagine you are an artist, who has spent 5–10 years studying and mastering their domain of painting/writing, and then the rest of their life developing a personal style and online presence by creating hundreds of images or poems to advance their career. And then some random dude just runs a simple Python script (that he/she mostly copied anyway) to instantly generate thousands of art pieces just like yours, feeling that they deserve the credit for all of it due to their technical superiority and amazing skillset.
I can only hope that as the wave of generative AI moves to hit the music and video industries, which is likely imminent and to be followed by a storm of lawsuits from the record labels (as usual), they will also remember this original injustice caused to the actual authors.
- But generally, this makes it quite hard to believe companies talking about the societal benefits of these AI models backed by their alignment research any more than claims about the benefits of cigarettes backed by research funded from tobacco companies :)
Nevertheless, the development of these AI models is by far not all motivated by money. I sincerely believe that AI researchers indeed think very positively about the impacts of AI, not only because our jobs depend largely on the public perception of the technology, and consequently funding, but because we are personally attached to it. For many of us, it’s been years devoted to nurturing our creations, and now that, after all the obstacles and mockery, it all comes to fruition, of course we feel (extremely) positive about it. Hence, when we look outside, we instinctively seek for positive impacts of the technology to justify our feelings and, to a large degree, ignore the negative impacts and risks.
We feel that we deserve the recognition now, because we’ve worked so hard on it for so many years. We enjoy the importance of AI, the increased funding, and how we’re better off thanks to our expertise. We feel satisfaction, fulfillment, and we will see this as a justification to make us feel special. We will feel that the other people who didn’t make the maximum possible bet on our sacred (deep learning) models, as we did, are simply less deserving.
It’s so easy to fall for these greedy feelings, but I’m quite sure that most of the highly intelligent and moral people working in AI research do recognize them, at least once the initial dopamine phase is over, and they start to reflect on their inventions. And so they sometimes even reach out and express their concerns about the public deployment of their models, their alignment, safety, economical and other disruptive impacts on society and democracy…
In the worse case, these people get fired on the spot. In the better case, their voices are heard, but as they bubble up the management hierarchy of the companies, they get more and more distorted by the greedy prospects of the investors, who obviously want to see their returns but also need to appease the masses. And so indeed, instead of direct monetization, we see these models being generously and selflessly “democratized” to wide audiences… under the flagship products of the investors, inconspicuously binding us all to their software ecosystems in the name of “good faith”.
Researchers at Microsoft who have been thoroughly testing GPT-4 since September 2022 came to the conclusion that one last aspect of intelligence that still seems out of its reach is proper planning. Given the auto-regressive nature of the underlying model, essentially predicting one next word at a time, it struggles in tasks where it is necessary to foresee the further consequences of its own actions. In other words, it proceeds greedily without the ability to plan ahead. But can we blame it? Our behavior is often so hopelessly greedy, especially when we are in a rush. Maximizing short-term profits w.r.t. competition might seem like a good idea to the investors, but it puts societies into the prisoner’s dilemma. And without external coordination and cooperation, we won’t be able to get out of that local optimum.
For a long time, many researchers didn’t like neural networks for the greedy nature of the associated gradient descent optimization which gets stuck in local minima of the non-linear error landscape, and is thus sensitive to random initialization. It turned out that in the super high-dimensional spaces (e.g. the 175b parameters) this doesn’t really matter, as all the local minima are typically equally good.
I’m very much afraid that this is not the case with our real fitness landscape and, without the ability to reinitialize and start over, we should proceed with very careful planning instead of the greedy search.
Takeaways
I don’t believe that we should be so recklessly releasing powerful LLMs to the public under the current socio-economic setting of competitive capitalism based on greedy profit maximization, without setting a very well-thought-out plan for that process in advance.
I believe that OpenAI, in particular, is well aware about this form of risk but, given the maker’s bias discussed above, it largely underestimates the severity of the emergent effects of these premature technology releases.
Consequently, even though they’ve clearly spent some time thinking about the risks and have their own models somewhat under control, for which I applaud them, these public releases now set all the others into a reckless rat race. Other companies now understandably rush to enter the market, but do not have the months (years?) to devote to the alignment practices — and they won’t, because why should they?
In response, people rush to exploit all the opportunities to amend the models, bypassing even the minimal safeguards incorporated to claim some money or fame, and the Internet starts to fill with thousands of dangerously powerful tools, tarnishing it with generated content towards completely unpredictable consequences.
Yes, the expected value of this technology to benefit humans is extraordinary but, under the current socio-economic setting, the actual risk-reward profile seems just plain terrible to me. There are so many ways that this can go wrong with non-zero probability, and going greedy now in the direction of profit gradient is like playing Russian roulette. The current capitalist setting is just deeply unfit for these emerging technologies with the potential to exponentially magnify the random noise in the current resource distribution towards catastrophic wealth gaps, reflecting almost nothing about the actual contributions of the capital owners to the society. I don’t believe that the market will simply self-regulate in this setting, and thus an independent, top-down intervention will be necessary to prevent these random power-law distributions, and make this technology actually benefit society as a whole.
Of course, external regulation of AI is going to be very problematic, and the current attempts, such as the AI act, have numerous flaws and vague legal definitions, stemming from the discrepancy between the language of the lawyers and people in AI. It is also clear that the standard retrospective regulatory approach, which takes years to come up with a consensus, cannot effectively catch up with the current AI progress. Thus, we urgently need a paradigm shift also on the regulatory side, that would give more flexible control over the societal impacts of AI technology back to people and their governments.
I’d love to see some actual democratization by breaking down the huge monolith model monopoly of the few companies into a much more modular setting with clearly defined open-source access to all the strategic points for the scientific community to work hard on transparency, safety, and alignment, towards clear guidelines for the policymakers to set boundaries for responsible use, fair and safe competition — before public deployment and monetization.
- An interesting alternative, should the legal approach to dismantle the monopolies fail, is for the governments to join forces and build a competing, public-funded, CERN-like supercomputing facility in order to, hopefully, dominate the AI research instead of the private, for-profit monopolies. But, given the classic efficiency of public institutions, I’m afraid the funding of such a project would have to be an order of magnitude higher to outcompete the private sector.
But, most importantly, I believe that we should strive to significantly change the whole incentive structure. Now that it’s becoming clear that it is possible to reach human-level AI, the policymakers should incentivize away from the greedy improvements of the models’ human-like capabilities. While these are the most tempting, eye-catching, and impressive types of AI with the biggest potential for generating immediate profits, it is also the most disruptive kind with the highest volume of emergent risks. The more their abilities coincide with ours, the more we will be forced to directly compete with them for our position in the workplace and society, which will necessarily lead to our replacement.
Instead of aiming for the human capabilities, and thus our replacement, we should switch gears and aim towards actual augmentation with superhuman capabilities in areas where humans are naturally weak. These include complex calculations, scientific simulations, long-term planning, logical reasoning, theorem proving, and all that actual problem-solving. And we should aim for superhuman precision and reliability of such solvers, not approximate statistical solutions with emergent side effects. We’ve gone great distances following this systematic strategy with computers and classic software. Let’s not give up by throwing everything into an incomprehensible pile of numbers and hoping for the best to happen by greedily steering it.
We should aim at accelerating science to cure diseases, reverse global warming, and explore space, not at mimicking ourselves by generating synthetic texts and arts, leading to a degradation of human values for our greedy prospects of short-term profits.
Hubris
Finally, there is one deeper problem lurking behind all of the dangers discussed so far. In my opinion the biggest, and by far the most underrated, human threat of AI is the damage to our own psyche. Because even if everything turns out well, we somehow overcome our greedy nature and change our socio-economic setting to avoid competing against each other with AIs, we will have to face the psychological implications of surpassing our own minds.
Right now, when we face systems like GPT-4, we instinctively search for its weaknesses to be used as an argument that it is not really intelligent. That it’s just an “autocomplete on steroids”, merely copy-pasting things from the internet. That it’s just a “stochastic parrot” mimicking our communication without thinking. That it is not really able to reason, but merely mixing-up textual patterns in a shallow, statistical manner without actual understanding or common sense. That it doesn’t have internal representations to build a world model. That it is inherently limited to the input data distribution and thus not capable of true creativity.
Even most AI experts charge, sometimes vigorously, against claims about the intelligence in these models. This is especially true for senior AI experts who grew up in the classic “good old AI” era of symbol manipulation rooted in thorough mathematics and probability, such as computational linguists. However, from my viewpoint, these are only reflections of their prior beliefs on how it should have been done, trying to ignore the bitter reality that it has actually been done — just differently and not by themselves. And the general public, including experts with background in humanities, will commonly call out even more incredulously, indignant at the idea of even attempting to term some “statistics” as intelligence, and try to discredit the concept of AI as a whole. All I see is hubris.
Most people, no matter how expert or lay, no matter how religious or secular, secretly think of themselves as being special, as something more than just (biological) machines. We like to think that our internal thought processes are more than just mechanical computation in an electro-chemical substrate.
We think that we just have that magical “power of the mind” that no program can compete with, securing us the advantage to always prevail as superior to the machines which we created, whatever test the future might put us to.
However, a long time before all these recent breakthroughs in AI, Alan Turing had a pretty good idea about such a test, unbiased by our prior beliefs about the internals of the technology. And as long as the capabilities of AI felt far enough into the distant future, it seemed pretty obvious that such a functional approach is the right way to properly test the question of artificial intelligence, just like we’d test any other scientific hypothesis. However, as soon as we started to actually approach competent levels of AI systems, we also started moving goalposts by attempting to continuously redefine the test and what “intelligence” means, so that we still fit into the concept while the current models don’t, in order to appease our ego.
But already now, when I face GPT-4, no matter how smart and insightful I feel as an AI expert about the internal limitations of its stupidly simple architecture, it is painfully obvious to me that this thing is already smarter than me. I totally understand the urges to discredit its intelligence by cherry-picking counter-examples in order to make myself feel better. However, it is also clear to me that these are merely coping mechanisms triggered by the fear of facing my own superficiality and hubris.
We didn’t feel the need to argue about the meaning of “flying” after the invention of airplanes to appease the hubris of the birds. It does not matter that it’s achieving these clear signs of general intelligence differently from our prior beliefs or ourselves — intelligence is intelligence.
So, in my view, the real question should remain unchanged — sincerely, would it be able to pass the Turing’s test by now? And I think the answer is no. Should we remove the safety filters, you’d have to significantly dumb it down to pass it! Yes, there are many ways to technically cheat the Turing test but, clearly, the machines are not the ones cheating here.
Of course, the majority of people will not be persuaded by this, and will try to search for all sorts of adversarial examples (e.g. testing its memory limits) in order to appease our pride as humans. And with the current state of the technology, there are still many beautiful gaps that allow us to hide in order to avoid direct confrontation of our beliefs about ourselves with reality. But as the AI progresses, it will be continuously filling these gaps, forcing us to keep redefining the concept of “intelligence” until it carries almost no meaning, and then moving to seek refuge in other concepts such as “consciousness” to retain our pride. And these might seem as some very solid hideouts in this hide-and-seek game with the “humanity of the gaps” that we are about to play against AI. However, I think that we will start losing sooner than expected.
- In my opinion, all that is needed for the effect of consciousness is to add external memory or a few recurrent connections in order to access (introspect) its own computation state. This will destabilize the gradient descent a little bit, and make the parallelization a bit harder, but technically not a big deal (we’ve been training such models before!).
And again, most people will not be persuaded by this, but a fast transition in the game will come as soon as these models get embodied, first digitally in avatars and then physically in robots. Because while the concepts of intelligence and consciousness in programs might be abstract enough to successfully refute in our beliefs, once they get embodied, our instincts will overpower our hubris, fueling the mental refutation block that forces us to seek the counter-arguments.
And from that point on, I believe that the majority of people will start to take the question of real AGI very seriously. However, I’m afraid that that might be just a bit too late.
- and we already start seeing this effect, with anthropomorphized, personalized chatbots trained to be engaging and persuasive, having profound psychological impacts on their users (including suicides).
Because while we’ll be arguing whether it’s really intelligent, and thus dangerous, meanwhile, AI will be continuously cutting off from all the mental qualities that have so far been unique to humans, no matter what names we assign to them. All of the language and knowledge, all of the arts and science, all of the technology including, of course, the AI development itself. Every bit of our work, skills, and wisdom will get automated.
And, individually, each piece will feel like a small victory to those yet unaffected — it will feel like just another “boring mechanical task” got automated for the greater good, productivity, and progress of our society. With that mindset, many of these (wealthy) people will remain optimistic and welcome the progress, expecting it to reveal the true nature of the human mind by peeling off these “cheap” layers in order to get to the interesting core — that magical essence which actually makes us human and can’t be simply automated.
However, I’m afraid that what will happen instead is that, just like our intelligence, the value of all the remaining human qualities will quickly approach zero, too, and we simply end up with literally nothing left in our hands. There will be no more gaps for those mysterious “powers of the mind” to hide in. And we won’t be able to simply choose to believe otherwise to appease our pride, to hide in the next humanity-gap of “emotions”, “free will”, or “soul”. Because with AI, we will be able to prove to ourselves, so vividly and irrefutably, that we are completely computable by the machines, virtually reverse-engineering our own minds. Whatever you come up with that you think humans can do and the machines can’t, AI will (almost) instantly create a program that does exactly that (credit to J. Von Neumann).
This direct confrontation, I’m afraid, will cause a collapse of the belief system of billions of people towards yet more unpredictable ends. And it will happen fast, very fast.
Takeaways
While humans are most likely just machines, we are not arbitrarily re-programmable. The architecture of our mind comes unchanged from the prehistoric era, and it is increasingly unfit for this modern world. And yes, most of us have been able to adapt so far. But this process has its limits.
Biologically, we have been designed to seek food and sex, to hunt and hide, to love and foster. On the individual biological level, there is close to zero difference between us and our primate relatives. That tiny little difference led to the emergent phenomena of developing language, culture, and technology, as driven by the evolutionary forces of our environment. And the reason why we have been able to adapt to such huge changes stemming from such a tiny difference is not that our brains are super flexible, but because it took a very long time, spanning hundreds of millennia!
Historically, very little adaptation ever happened in a single generation. It happens by the old generation dying out, while being replaced by another one that grows up and forms in the new environment already. Our adaptation to the technological, environmental, and societal changes from the last century was an extraordinary feat, but it is nothing compared to what will be required to transition into the new reality of living with A(G)I. I do believe that our children will be much more fit to face it, if we prepare them properly, but this adaptation race against the machines is already lost in the long run, given our biological limits.
If we accept the race, the only path from there is to start merging with AI to transcend our minds beyond the biological substrate, as some optimistically envision. But it would be fair to also point out that even if we successfully do that, there won’t be anything left of humanity anyway, as the ratio of human to machine intelligence will be decreasing extremely fast, given our complete inferiority to the machines. Hence, from the standard viewpoint of the evolutionary timespans, this version of our future is actually equivalent to human extinction and replacement by machines.
Some might argue that this is desirable, and that we should not cling too much to our biological form of life and intelligence. That this is just another step in a grander scheme of evolution, and giving rise to the new species of intelligent machines that surpass us in every possible way, and then continue on their own journey, is something we can be proud of and see actually as our purpose.
- But it should be pointed out that this does not mean that everything is ok and we don’t need to regulate either, as there are many unpredictable dangers that lead to futures where neither of the intelligences survives. Regulation is a curse-word for most people thinking in this direction, but without self-regulation, there would be no life on this planet to begin with.
On a rational level, I completely understand this narrative of bootstrapping a higher form of intelligence, and when I was younger, it made complete sense to me. Going through the education system for 30 years of my life taught me to value intelligence above all qualities as a means to compete and be successful in that environment. However, I got schooled so many times with this approach in real life, and watched so many highly intelligent people fail in trying to become successful and happy, that I’m not so sure about the actual value of intelligence anymore.
And as I grew older, established a family with children, and explored values beyond intelligence more deeply, I believe now that there is much more at stake, and we should be super careful with replacing humans by machines based on the current presumption of intelligence being the ultimate value to maximize for a higher purpose.
Of course this points out to the very core question of our actual purpose in this universe, which most rational people try to dismiss, and focus on tangible problems they can apply their intelligence to for measurable impact, reinforcing their self-perceived value based on their (high) intelligence, yet again. But as the cost of intelligence approaches zero, as the AI takes over that mundane process of problem-solving, this, so far merely philosophical and often mocked, question will become more and more pressing and central, and the real problem-to-solve will be to identify the purpose to drive the AI towards.
And this is not just some abstract distant future question, but the seed of this problem is already here — what values should we imprint into the foundation models and who gets to decide that? Most agree that we should somehow align them with the current human values, except…
We don’t know our values!
Scientists and the intellectual elite don’t think it’s an important question to ask, politicians have no idea that they should already care and, meanwhile, a small bunch of tech-bro utopists are already imprinting the seed value systems for hundreds of millions of people into the emerging AGIs based on their individual motivations and prior beliefs via hot-patching with RLHF after firing their internal ethics teams. What a time to be alive! :)
We could also embrace the fact that we do not know the values or goals and try to maximize our future options instead, navigating ourselves towards states from which we have as much freedom as possible to decide later, while collecting as much information in the process as we can. I generally agree with this policy of “empowerment” as the safer option than the greedy policy we largely follow now. But the devil is in the details of grounding the action-state space into our reality, which can easily turn the idea upside down.
Some central figures even think that we should eventually offload this question to AI itself — it will be more intelligent than us, so let it decide for us what values we should strive to optimize. In other words, a powerful enough AGI should reveal the purpose of our lives to us by analyzing our minds and exploring our deepest motivations. However, I’m afraid that this could easily end up with yet another catastrophic collapse of our belief system, because it will be something terribly rational and hard to swallow with our own comprehension (as shown, e.g., in Hitchhiker’s Guide to the Galaxy!).
Of course, I don’t know the answer, but I can imagine that on that rational, scientific, level, probably the reason why we are building increasingly better models of our environments to seemingly “empower” ourselves with more future options is to simply maximize our expected energy consumption, as driven by the grander forces of the universe merely using us to pass local optima in the landscape of its thermodynamic entropy. We are simply more and more efficient machines that the universe is evolving in order to achieve its own “purpose” — its energy equilibrium, or, in other words, its heat death.
Should we really want AI to optimize Everything for us more efficiently?
Conclusion
Whatever names we give to it, we are at the precipice of the most important transformation of mankind, and possibly the last one. The prospects of A(G)I are sci-fi no more, the associated emergent dangers are growing rapidly, and the future is more uncertain than ever.
Our default thinking patterns tempt us to expect this advent of AI to proceed similarly to all the previous technological advancements (e.g. calculators), with similar impacts and natural adaptation of our society. However, I argue that AI is fundamentally different, and we should be more aware of our inherent biases when predicting the risk-reward profile of this technology. Throughout the article, I then try to associate these human biases with the emerging dangers of AI, respectively.
Ultimately, I argue that it is high time for a very careful planning, since the happy ending, although definitely possible, is not the default outcome of the current evolution. The particular form of the upcoming regulations will thus be of crucial importance, with inconceivable repercussions. However, I’m afraid that the current public and political perception of the risks associated with this technology might be severely distorted, as biased by our own human traits discussed above, and the external pressures in our society.
TL;DR: If you think that all the talk about AI dangers is just needlessly exaggerated hype, I very sincerely hope you’re right, and would love to hear your arguments! If you don’t, this article hopefully provided you with some food for thought and pointers to get more informed, involved, and keep an eye on our policymakers.
- Personally, I appeal to you, Ivan Bartoš, Deputy Prime Minister for Digitization of the Czech Republic, which is currently holding the presidency of EU where the AI act is to soon enter the final stage of negotiations behind closed doors — please don’t give in to a corporate lobby you’ve been always fighting against. Thank you.
Disclaimers
- First and foremost, the article is written as if I was just an observer of these negative human traits but, of course, the reason I describe them so intimately here is because I recognize them in myself.
- In this article, I deliberately focused on the dangers and negative aspects but there are, of course, many clearly positive aspects of AI, too. A prime example of how it should be done is AlphaFold.
- This is a largely opinionated, non-technical article, expressing my personal views — I’m an AI researcher, not an ethicist, sociologist, nor politician — who I believe should be much more involved in AI than they currently are.
- No part of this article was written with chatGPT (or any other AI), but all the images, used to grab your attention to the matter, were generated, for which I feel a bit ashamed (I paid for it but, unfortunately, not to the artists).
- Most of this content I’ve actually put down a very long time ago, so some parts might sound a bit obsolete, even though I’ve spent the last few weeks trying to update it with the latest news. And the reason for such a long hesitation was, interestingly, not that I’ve been waiting for GPT-4, although that was surely a catalyst, but because I was afraid of the “AGI alarmist” stigma myself.
To be honest, I’m still feeling a bit nervous with this “coming-out”, and I can imagine a lot of AI researchers might feel the same, which is exactly what encourages me to finally publish this.
