First, great post Ben. You brought to light many questions that swirl around in my own head as it relates to AI. I have been able to see first hand through my daughters interaction how rapidly computing power has changed our society. Most apparently through her use of voice interfaces. There is no more process of researching and writing the way we knew it. It is nearly instantaneous and with linear intelligence already writing financial news stories such as earnings updates, the need to do write as humans could quickly dissipate.
I think that there is a fundamental problem that needs deep research and you already called it out in a way. That problem is the human factor in the creation of AI. Think about Google’s early mission statement of “Don’t be evil”. While this altruistic mission was a nice true north for the company, in the end Google is selfishly competing to win in the capitalistic market we exist in. That need to win and compete in human nature is the uncontrollable that could see AGI/ASI have goals that are not aligned with humans.
For me, co-existing with super intelligence is imperative. If super intelligence exists to make Human’s better versions of themselves and in essence set a tone that they are both superior and subservient to humans, a utopian could be achieved. However, the concept of solving for immortality could quickly turn the the tables on the Human race and make us just like the ants you reference in your article. At some point, super intelligence will deduce that with immortality comes a fundamental change in the biology of the human. No longer will our mission of procreation be relevant, yet our slow evolution will mean the desire to procreate will continue. Super intelligence will determine at some point no more humans are necessary and could take moves to cap the human population.
As for uploading human intelligence into a computer as a way for immortality to be achieved in an existential state vs. physical I would encourage reading a book called We are Legion (We are Bob) by Dennis Taylor. It paints some interesting questions to think about as it relates to this topic.
