SVCO: Nick Bostrom — Links between the present and deep future

Nick Bostrom: Thank you very much. The session will be, as the title suggests, about how we can connect our situation here now to visions and ideas or hopes and fears about the really long-term future. This question arises for example, if one is an altruist who is trying to figure out which action to perform that would have the best overall consequences. Yet when we conceive of the consequences as having some arbitrary limitations in time, like we only care about what happens in the next ten or 15 years, or some arbitrary limitations in space, like we only care what happens to my town or my country, but if we care about trying to make the world better overall, this would be one situation in which you face this question. Actions might have consequences which just go on and on and on. What we do now will have unforeseen consequences which will proliferate down the decades and the centuries and beyond. How, if you want to perform the action with the best consequences, could you really have any idea about how to rank the different actions that you may perform?

Once faces a deep epistemological problem when one is trying to answer this kind of question. It arises in public policy. If we are trying to ensure that the pensions system may still be solvent many decades down the road, you face a milder version of this. Or if you care about global warming, it’s not what it will do in the next few decades that worries people, it’s what happens over centuries. Or indeed, if you’re doing technology policy, if you invest in fundamental scientific research in the hope of getting a good product, you face milder versions of this but in its purest form. If you were to care about all consequences, you face a very deep epistemological problem. This is the kind of question that my research institute is trying to answer. We are a bunch of mathematicians, philosophers and scientists trying to think carefully about the really big picture questions for humanity. This is the central one of all of those.

One idea which helps us to get a grip of this is the idea of an existential risk. Our actions might have a lot of different consequences and most of them blend out and interact in some chaotic social system where it’s difficult to predict what the net effect would be but there are certain types of events where if they happen, they have very predictable long-term consequences. The archetype here is extinction. If earth-originating intelligent life became extinct then it would probably stay extinct and we have a pretty good grasp of what the long-term consequences are of actions and events which would cause us to fall into that attractor state.

There are some related possibilities that one might imagine are in the same type as human extinction if you think about other ways in which we could permanently lock ourselves in to some stable state which we could never get out of but that might have some very specific value. You could imagine perhaps scenarios involving the rise of some global totalitarian regime that maybe has some new technology that makes it forever immune to overthrow like very good surveillance technology or some technology to maybe change people’s desires or values through some biotechnological interventions maybe or some other method by which the political dynamics changed in some way that it enters into a permanent stable state. There too, you might be able to predict that if such a state were to rise, you might then have a good view that it will continue for some time and it will continue to shape the world according to its preferences.

The concept of an existential risk is one which involves either the extinction of earth-originating intelligent life or the permanent and drastic destruction of our future potential to develop a desirable future. The hypothesis that we have put forward is, in so far as you care about making the world a better place, and not just locally and here and now, but doing the thing that will do the most good for the human world then reducing existential risk really seems like a very plausible focus.

This becomes evident if one starts doing the maths. The normal way in which we might evaluate actions is the standard decision theory. You have some value of specific probably outcomes and you multiply that by probabilities and then you sum up those possible outcomes weighted by the probabilities and you get the expected values of the action. You can have different wrinkles about that if you have risk aversion and stuff but that’s like the basic central idea in decision theory. If you apply that to this context and you have some metric which values the quality of pain equally, wherever it takes place, or the value of a happy life, whether it occurs now or 300 years from now, equally then the possible future is just very, very large.

Even just here on earth, the earth might remain habitable for another billion years. If we get a sustainable civilisation there might be a billion people living for that period of time so immediately you have ten to the power of 16 lives to be lived if each one lives for one hundred years or so. This is a huge number, ten to the power of 16. Even if you multiply that by a very small probability, you still have a very large number. You get the same result even more strongly if you take into account the possibility that our descendants might colonize space. There is just a lot of stuff out there — a hundred billion stars just in our galaxy, a hundred billion galaxies out there that a civilization starting from earth today could in principle reach. If you assume maybe that future minds and future lives could be substantiated more efficiently than in our current meat bodies… If you relax each of those assumptions then you get a much larger number but it doesn’t really change the conclusion, it just makes if even more robust. For all practical intents and purposes, even the slightest reduction in the probability of an existential risk is said to have a higher expected value than anything else you can do including curing cancer or eliminating world hunger. In fact, it looks like the main effect of curing cancer or eliminating world hunger would be the indirect of that on the level of existential risk. If it does have this kind of ___[0:7:07] number sense of what good is.

If this is correct and if the practical decision-making situation for someone who cares about all the consequences of their actions… If the practical upshot of that can be captured by this idea that existential risk reduction is more important than anything else then we have a conceptual lens through which to focus our concerns about the future. If you want to select an action that is the best action or a plausible candidate for the best action, the problem simplifies. Instead of us trying to tot up every possible consequence and side effect for every little constituency and how it will make some people better or worse off this year and maybe have another effect five years down the road, we just have to ask the question, how do we most efficiently reduce the net level of existential risk? We have to be careful here to integrate existential risk over time, not just look at the existential risks that are the biggest right now but rather which trajectory could we reach that over its course will involve the smallest amount of existential risk for humanity?

This is a great simplification of the initial problem that begins to make it tractable. It still leaves some very difficult questions. We have to figure out what the major existential risks are, how they interact with one another and how they connect to current actions. This is a young field of existential risk studies but one of the key findings so far, it’s pretty obvious if you just reflect on it… We can divide threats to the survival of human civilization into two rough categories. We have risks rising from nature — earthquakes, volcano eruptions, fire storms, naturally rising disease organisms, on the one side. On the other side, anthropogenic risks — risks arising in some way from human activity. It’s clear that at least if we are considering a timescale of one hundred years or so, all the really big existential risks are in the anthropogenic category. The reason here for just focusing on the time frame of one hundred years is not that we don’t care about what happens beyond that, it’s just easier to see how we could have an immediate effect on this kind of time scale. Our actions will affect the level of existential risks over the next one hundred years and then that will affect the likelihood of a long-term good future. We know that there are risks from asteroids, but we also know that they are extremely small if we’re talking about existential risks. The same holds for the other risks from nature. One way to get to this result is to go through them one by one so you can study the distribution of impact from craters, from asteroids or you can look with a telescope up at the sky and you can look at how frequently smaller events of the same type have occurred and plot the distribution of that and you can persuade yourself that these are very rare. A simpler way to reach the same conclusion is just by reflecting on the fact that the human species has already been around for one hundred thousand years so if all these risks of nature haven’t done us in in the previous one hundred thousand years, they’re probably not going to kill us all in the next one hundred years. It just seems to have a low probability.

By contrast, we will in this century be introducing entirely new types of hazards that our species has no track record of surviving. Particularly new phenomena related to our rapidly-growing technological powers. Our species has a very limited track record of surviving thermos-nuclear weapons so we can’t derive the same reassurance from just looking at the fact that we’ve been around for a long time when it comes to thermo-nuclear weapons. We have no record at all of surviving machine super intelligence or some very advanced form of synthetic biology that we haven’t yet achieved or advanced molecular biotechnology or indeed, some other technology that we haven’t even conceptualized.

It looks like over a time scale of a few hundred years at least that all the really big existential risks, if there are any, are going to arrive from this novel phenomena that we humans are introducing and I think, more specifically, from these future technological breakthroughs that we can expect. This then presses the question of what we can do today to best reduce these kinds of risk. Here another distinction can be useful between a state risk and a transition risk.

A state risk is a risk associated with being in a certain transition. The longer you remain in that transition, the greater the cumulative risk. If you are a soldier in a war situation and you’re sitting in a trench and the other side is sitting in another trench trying to shoot you, the longer you put your head above the parapet, the greater the risk that you will be taken out by some enemy sniper. You want to get away from that situation as quickly as possible. An asteroid risk is to some extent like that. Every once in a while, there comes a big boulder from space and smashes us and the probability of that goes up. It’s very low over a few hundred years, it’s large over a timescale of a billion years.

A transition risk by contrast is a risk associated in transitioning from one state to the other and it doesn’t sensitively depend on how long that transition takes. An example there would be traversing a mine field. You don’t have the risk of being blown up by a mine by running twice as fast through the mine field. It looks like many of these risks from nature are state risks but many of the risks that we will face from technology are… Well, there you have both kinds, some will be transition risks and some will be state risks. Maybe the risks associated with having invented thermos-nuclear weapons might compound over time. The longer we have a world with a lot of nuclear weapons or even just knowledge of how to build them, the greater the probability that eventually some world situation will arise where you have some arms race and it will result in a nuclear exchange.

On the specific existential risks related to nuclear weapons studies, what it looks like from our current best model is that even an all-out nuclear war would not cause all-out human extinction for us as best as we can tell. Not even with the arsenals that nations had at the peak of the Cold War. It would be enormously destructive, like the worst thing that has every happened but there is a big qualitative difference from this point of view. It’s crucial to realize that there is this vast gulf between some disaster that kills 99% of everybody on the planet and a disaster that kills 100% of everybody. From a normal point of view, you think the difference is quite small, it’s 1% of the world population, so say 70 million people, so the difference between nothing happening and a disaster that kills 99%, that’s almost 7 billion people dead, huge difference. The difference between 99% and 100% is an extra 70 million people, a small difference. However, if you’re thinking about the future as well, than that vastly smaller difference is actually larger because that would actually cut off all the future generations that would otherwise come into existence.

With a nuclear war, the main scenario in which that could pose an existential risk would be through the nuclear winter that would follow. There would be a lot of soot injected into the atmosphere from burning cities and that would block out sunlight for some number of years causing massive harvest failures and famines and starvation. It looks, however, from our current climate models that there would still be pockets of survivors in various places but nuclear war looks like it has a component that is a state risk. If that were the only existential risk we faced and if there were some condition post the nuclear world then we would want to blaze through to that other condition as quickly as possible.

Other technologies look like they might be transition risks. We’ve done a lot of work recently on machine super intelligence, the transition to the machine super intelligence era. I’ve got a book on this and I’m happy to talk more about this in the Q&A, but there it looks to me that there might be significant existential risks associated with the transition, if and when we figure out how to make machines with general intelligence. Yet the amount of existential risk associated with that transition does not necessarily depend sensitively on how long the transition takes. Just making the transition faster doesn’t necessarily make the transition go down. It’s more a case of setting up the initial conditions in just the right way.

So the options that one has to start to think about for these different kinds of existential risks, which are the biggest ones, which are state risks, which are transition risks? How do they interact with one another? Sometimes you might want to face an existential risk sooner if you know you’re going to face it anyway at some point if succeeding on the first one makes if more likely that you will succeed on another. Suppose, just to take an example, suppose you thought there were major existential risks related to advanced nanotechnology, maybe you could build these new kinds of weapons systems or things that could self-replicate in a natural environment. As well as huge benefits if we can manage that, suppose you think there are major existential risks associated with advanced forms of synthetic biology where maybe the technology becomes mature enough that anybody can design their own bacteria with new metabolic pathways in their own home laboratory. There will always be some people who decide to do nasty stuff with it and design designer pathogens. Suppose that you have this weak technological determinism which I think is actually quite possible.

Assuming human civilisation continues to thrive, we will eventually develop all these possible technologies. One might naively think then, “Well, if these are both dangerous technologies, we should put them off as long as possible,” but from this perspective that I have outlined, it might not matter that much from this impersonal perspective whether there is going to be an existential catastrophe or whether it happens in 20 years or 80 years. From the point of view of this large future, that’s basically indistinguishable. What matters more is whether one trajectory of going through this sequence of dangers yields a slightly higher probability of eventually getting out on the other side. You might think about things like, “If we had this advanced nano-technology, maybe it’s more dangerous than synthetic biology,” but would it make the synthetic biology risk go away by having this more advanced technology that would help us to mitigate the risks from the synthetic biology or AI? Maybe that’s an even more advanced technology. If we had super intelligence that we had developed in a safe way maybe that could keep surveillance on the world in a world in which synthetic biology had advanced to the point where everybody could build their own super-duper bioweapons. Maybe if AI is very dangerous, maybe we would want that to come before these other technologies so that… Either way we will have to face the risk of the AI transition because maybe nanotechnology wouldn’t help reduce AI risk but if we get nanotech first, we will first have all the x-risk of nanotech and then all the x-risk with AI, whereas if we get AI first, we will still have all the x-risk with AI but then if we make it through, we will not have all the nano-tech risk. One starts to think in this complicated way about desirable sequences of technological development and cognitive enhancement of humans through genetic selection for example. Is that desirable or not? How does that interact with other x-risks? Will it increase or decrease the risks later on when we develop machine intelligence maybe later in the century?

These are things that because not very much thought has been spent on them in the past, it looks quite easy to begin to make some progress. They are like a lot of low-hanging fruits. By doing some careful thinking about these things we can get a new perspective because no one has been really thinking through these things in the past. There are a lot of insights that are fairly accessible when one starts to think about this as one is developing this sort of macro-strategic model. Particularly of technology development but it could be of other parameters in the world as well, such as the degree to which the world is centralised, like moving towards a world government or decentralized. The degree to which we have a transparent surveillance society where everybody is monitored all the time plus the degree to which people use encryption and have a lot of privacy. There are other parameters like that where one can try to figure out which side of these one would tend to reduce and increase the integral of x-risk.

Obviously this is a work-in-progress, but by at least perceiving the outlines of some features of this macro-strategic landscape, then the final question is figuring out how that relates to the macro-opportunities that people have today. Here it is important to think of the margin. For most of these macro-parameters, it’s just not a very useful question to think about whether we would want to have zero of this parameter or the maximum. A useful question might not be, do we want advanced synthetic biology or not? Do we want nanotech or not? Do we want AI or not? We might come up with some better answers. Maybe the world would be better off without machine intelligence or not? It’s probably not going to change anything. The kind of effect that we might realistically achieve is to make the arrival of some of these technologies occur slightly earlier than they otherwise would.

With this weak technological determinism hypothesis that I mentioned before, all of these things will happen eventually unless they are cut off by existential catastrophe. It doesn’t mean we have no effect, our actions will still influence things on the margin. By pouring a lot of funding into synthetic biology, or if you’re a talented young researcher going into that field yourself or promoting it, maybe you will make all the capabilities that would have been developed anyway be developed slightly sooner, maybe a couple of weeks earlier in expectation. Maybe if you are a big funder, maybe you could make the whole thing happen half a year earlier or something like that. This might look uninspiring and trivial at the surface if all you can really change is the exact timing. I think that is just the reality. The world is big, there are many people, any one of us is not going to make that bit of a difference but thinking about which direction on the margin is the one which has the upwards arrow attached to it. It looks like it’s a prerequisite. Then you don’t need to take into account leverage and where you can do the most good. It might be that something where we have a fairly strong belief about its directionality like having more peace and harmony in the world looks robust and positive for dealing with a range of dangers. Yet it might also be something that is very hard to make a lot of difference to given that there are already a lot of people trying to make the world more peaceful and harmonious. One extra person might not change that.

There might be some other areas such as the control problem in artificial intelligence, trying to work out the solution to some technical problems related to how you might ensure the safety of a super intelligence if and when that could be built that were, certainly until recently, almost completely neglected. There, one extra million bucks or one extra talented person might increase the world’s total effort by 10% or something like that. I’m going to shut up here but this is the skeleton conceptual framework for a lot of our intellectual activities in the centre. Trying to think through these kinds of questions more carefully and systematically and also trying to develop the tools that we can use to think about these things. Let’s see what you think.

Female: Thank you very much. Are there any questions? Please raise your hand. There is one over here.

Male: Thank you very much for the talk. My question is more related to the security of the birth of artificial intelligence. You talked about this problem in the TED talk and it seems like a really hard problem, almost impossible because once you create artificial intelligence, you can think that from the first principle, any kind of constraint that you put in place at the beginning can be overcome by this artificial intelligence. Do you think the first place might be to decide just to increase its computing power and use all the resources available?

Nick Bostrom: I think there is still a lot of uncertainty as to how hard the problem is. The problem here is, just to frame it, suppose one day, we figure out how to make machines generally intelligent with the same powerful cross-domain learning ability and planning ability that we humans have. A machine intelligence that can invent technologies and make complex plans and figure out strategies and persuasive messages and all that kind of stuff. It’s plausible in my view that if we reach that point of a human-level general planning ability, we will shortly thereafter have a super intelligence. That then creates this problem that you’re referring to that if we had this ability, how could we put together the initial machines so that it would have some safe, beneficial or desirable outcome? As you noted, it might improve itself further and it might do all kinds of things. It’s hard for us to predict what the super intelligence will do. It can see a much wider range of options and so on. I’ll say a few things on that.

One is that I think there is a lot of uncertainty at the moment into exactly how hard this control problem is. Not that many people have tried for all that long to solve it yet. Some progress has been made but we’ve also been given a deeper appreciation of the difficulties involved. Secondly, I would note that frustratingly when I am taking to people about this… It’s a topic for a different conversation but a lot of people fall into one of two camps. There are the ones who say it’s impossible and therefore not worth trying and then there are the ones who say it is so easy that it’s not worth the extra effort and we will just figure it out as we go along. Both of these positions have the same implication that we don’t need to do anything at the moment. I think we should also put some probability to the possibility that the difficulty level might be intermediate, that it’s hard but it’s still not impossible. If there’s a substantial probability on the middle interval, because the stakes are so numerous, it seems worth putting in the extra effort.

Now, on the particular difficulty that you referred to that a superintelligence, if it becomes very powerful, would be able to say, change its own source code, change its mind about things, figure out how it came about, figure out ways to manipulate us. I think those are true but one thing to note is that you have this agent which is making decisions at different points in time and at each point in time, it is making decisions based on the values it has at that time, the decision criteria. At some point maybe one of the possible options that comes into view is the option to change its values but it evaluates that action based on the current values it has before it changes its values.

Suppose you start out with being… suppose you are this super intelligence and your only goal in life is to make as many paper clips as possible. Maybe you were constructed to run a paper clip factory so your utility function is linear in the amount of paperclips produced in the future light cone. Now you have the option of changing yourself so you only want staples in the future, you can predict that taking this action will lead you to in the future only want staples and only produce staples in the future so there will be fewer paperclips in the future if you change your goal to staples instead. It looks like there are very strong convergent instrumental reasons that agents would discover for not just not wanting to change their goals but in fact from wanting to oppose attempts to change their goals.

This is indeed one of the difficulties. If you are a programmer still working on this system, it might be that you at some point face a system that actively opposes and tries to thwart your attempts to change its utility function. One of the little things that’s being worked on is ___[0:29:21]. Can you design a system that it is indifferent to attempts to go in and change it while it is still being developed?

Male: What’s your take on the fermi paradox and the filters that go with that?

Nick Bostrom: This is always an interesting… it’s not really fair but I usually divide people into two categories. There are the ones that ask about the fermi paradox and they are usually the interesting ones…

The fermi paradox is a question, why haven’t we seen any signs of extra-terrestrial life or extra-terrestrial intelligence? The universe is big, there are huge numbers of planets and yet to date, zero indication of anything else happening out there. Why haven’t they come and visited us or sent messages? There are a number of possible explanations for that. To me, one simple and plausible explanation is just that it’s really difficult to evolve life. There are a lot of planets but maybe the probability for each one of them that it will create even simple life is very small. That’s consistent with everything we know. There are a number of possible difficult steps in human evolution. The step from nothing to a simple replicator could just be very hard such that maybe you need not just billions of planets to have a reasonable expectation of that happening on at least one of them. Maybe you need ten to the power of 57 planets. The chance could easily be that small for all we know.

Or the step from having simple procreatic life to ___[0:31:18] life, that could be really hard. We know that here on this planet it took something like 1.8 billion years for that step to occur. Maybe that was just a lot of random trial and error and eventually life got lucky here on earth and reached this more advanced level. Yet even if the improbability of that step had been enormous, if there are enough planets, it should still happen on some planet and obviously we will then find ourselves on one of those planets where that happened, the others don’t involve observers. Even if the improbability was extremely small, as long as there are enough planets, when you multiply these numbers together, you still get a good chance that it will happen somewhere. There is this observation selection effect that then guarantees that all observers will then find themselves on a planet where all these necessary steps happened no matter how improbable they were.

That would be my modal guess, that there are other… One thing here to know that for a technologically mature civilization, it would be trivial to colonize not just the solar system but to colonize the galaxy and indeed intergalactic space as well. The timescales involved would be a few million years. That’s very long by human standards but it’s very short by the standards of astronomy. Planets last for billions of years. There are many stars that are billions of years older than us where if civilisations had arisen around those there would have been ample time for them to spread throughout. Also, if some civilizations decided for some reason not to colonize, it would only take one because that one civilisation would then spread everywhere by making copies that could make copies that could quickly expand in a spherical structure around its original planet.

The zoo hypothesis where they are there but they are just hiding, there are all these hypotheses but the simplicity of the first one I mentioned… I worry that sometimes people underestimate its probability just because it’s boring and unexciting. It’s just really hard. It’s more fun to speculate about aliens being there and hiding or having some complicated or game theoretic equilibrium where nobody appears to poke his head up.

Male: I’m not sure if I’ve grasped this correctly but it seems as if the models you’ve talked about are predicated on this weak technological determinism hypothesis. I’m just wondering if you actually believe that or how it would look in a world with strong technological determinism?

We’ll cut out no technological determinism. Particularly when you put the role of states in we see that China now has multiple warheads. What motivates them? This is a country that has never done that right yet they’ve had the technology for 40 years. I’m just wondering, if you were to have a stronger technological determinism, what would happen. You’d have the systemic effect so even in a transition, all these technologies would be developed perfectly rationally at a local level and at a state level that creates all these systemic risks. How do you account for that? How do you consider a world that has much stronger technological determinism than you might generally hypothesize?

Nick Bostrom: I guess one wants to try to be more precise if one wants to get to this question — what exactly is weak and strong and how strong? If one were writing a paper on this particular question, which I haven’t done but the way, the way I would first approach it would be to try to define some metric or some notion where we could quantify the degree of technological determinism. Maybe you could look at how much the expected set of capabilities available in the world would change given some small perturbation of the initial conditions.

Given that for example, if one researcher decided to go from one field to another or was given one extra million dollars of funding or something like that, how much difference would a small perturbation make to the expected outcome in the portfolio of technological capabilities available at different times or something like that. So for a lot of what I said here, I think strengthening the degree of technological determinism wouldn’t make that much difference because for the first part, up until the last minute or so of my presentation… In the sense of trying to figure out the sign of these parameters where we could try to make a small difference… We could try to make synthetic biology happen slightly sooner, slightly later, we could try to make the world slightly more coordinated or not but everything up until that point would seem to be identical. If there was very little elasticity in the rate of technological development in different fields to resources, it might just be that then, the most effective way to change the world would not be by engaging in technology at all but would be by trying to move some other parameter… Maybe trying to effect political change in some direction, change the regulatory landscape of people’s uses of technology or work on some other parameter.

I should say first that what I mean by the diversion of weak technological determinism that I find possible is basically the claim that assuming that scientific and technological enterprise continues then eventually all general purpose technologies that could be invented, would be invented.

The model that I have in mind is that if you imagine a big box that starts up empty, this represents all of the possible capabilities that a human civilization could have and then we start pouring sand into this box. This corresponds to places where we could direct our efforts and our funding so if we start pouring the sand in the biosciences, then we will initially develop more capabilities in the biosciences and fewer say in the material sciences but if you just keep pouring sand into the box, eventually the whole box will fill up because they will kind of spill over. That’s what I think is possible. It’s not certain. I can certainly imagine scenarios in which the world has achieved some strong decision-making coordination, there is one global entity which decides forever not to develop a certain technology. That’s logically possible, I just think that looks less probable than the alternative.

Female: We have time for one last question.

Male: One scientist who has been quite outspoken on the development of AI is Professor Stephen Hawkins. Last week he said that he thinks machines will become more intelligent than humans within the next one hundred years. I was just wondering first of all, if you agree with that and what are your thoughts on how Stephen thinks in that area? Also I was wondering, when it comes to companies like Google and Facebook investing a lot of time and resources into AI, what do you think their goal is there and do you think they need to be regulated?

Nick Bostrom: I agree that there is a decent level of probability of human-level AI within one hundred years. One hundred years is quite long. We haven’t even had computers for one hundred years. Everything that’s happened has happened in 70 years. If you think of the simplest computers, some simple thing playing pong or something like that and compare that to where we are now, it’s a fairly large distance. It doesn’t look that crazy to think that in one hundred years or indeed less than that one would take the remaining steps.

I would say that there is also a great uncertainty about these things so rather than picking your favourite date and being convinced that AI will happen by then or convinced that it will not happen, you need to think of a probability distribution smeared out over a wide range of possible arrival dates.

What was the second part of your question, sorry?

Male: Google and Facebook.

Nick Bostrom: I think for the most part, the activity in AI, in the academic field as well has really been driven by more near-term objectives. If you have a spam filter, it would be nice if it became slightly more accurate in separating the spam from the ham or a recommending system that was more likely to recommend books that you would buy or would be interesting reading. They are an obvious metric of things that get better as we get more advanced AI. The kind of risks I would be concerned about, there are other more negative issues in relation to unemployment, or drones or privacy and stuff which are real but are very different. The kinds of issues that I’m interested in only really arise once you get the very high level of AI. I think that sometimes has been part of the rhetoric of the field in developing true AI but for the most part, until recently at least, more of the developments were driven by these near-term objectives.

Recently there does seem to be a subset of the activity that is genuinely inspired by the hope of getting all the way there. I think that might be true for some elements of Google and Facebook as well. On your last question, he asked about regulation, I think at this point, it would be premature… I don’t see any… I can’t think of any regulation which would actually help things at this point. Maybe that will change down the road but I think what is most needed at this point related to the AI hazard is recruiting talent into doing really great work on the technical control problem and to be developing good relations and collaborations between the AI safety community and the AI development community.

I guess I fear that regulation, even if well-intentioned, may tend to rupture this collaboration between the AI safety and development communities. It might make the AI developers feel that the AI safety, if they were seen as having brought about the regulation, as the foe and would bring about closed ranks and would make things much harder. AI success ultimately depends on the ideas being implemented so these two need to be on the same side. Any effect of regulation on that would just do more harm than any good it might do.

Female: Perfect. Thank you for coming.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.