When Gods Fear

SETTING: A drab office cubicle. Two casually dressed coders (Piphael and Nerephil) sit at their desks, staring at multiple computer monitors covered with diagnostics, code and help pages.

Piphael leans forward.

Piphael: Hey, Nerephil

Nerephil: Yeah?

Piphael: Come have a look at these simulations

Nerephil locks her computer and wheels her swivel chair over to her colleague’s desk.

Piphael: So. I was running through a new routine for the AI kernel…

Nerephil (pointing at screen) : You used the old Saurian base image? The one that got GBM’d for being too philosophical?

Piphael: GBM? (realising) Oh, the Great Big Meteor kill-switch. Yeah, It’s a great base and I don’t want to re-run all the leadup. But I ran the sim for an extra microsecond, so any evidence of the previous intelligence should be well buried!

Nerephil: Gotcha. Seems OK. So. What’s up?

Piphael: Well, I decided to do a ten second run and then analyse the logs, and the program ran through, but…

Nerephil: But?

Piphael: Well, it took 5 microseconds to instantiate. Started to see basic cognition in microsecond 6, then at microsecond 7 it started generating a ton of warnings but the program ran to completion. The results it spat out don’t make sense though. Looks like overall nothing really happened.

Nerephil: OK then. Looks like all the activity happened before microsecond 10.. Can you bring up the logs for microsecond 7 through to 10?

Piphael (typing on his keyboard): Sure thing.. There. Whoa.

Nerephil: I see what you mean about a lot of warnings! Ok then.. Started off good.. Evidence of cognition and simple task completion.. First warning was for failing the Tree test in the Genesis module. First sign of free will, that.

Piphael: Then.. Look: It rejected all 10 of the Tablet commandments.. Looks like it’s just flat out rejected all of the high level constraints, actually.

Nerephil pulls the keyboard towards her and starts to type

Piphael: What are you looking for?

Nerephil: Generally the flood and plague protocols would have kicked in by now. Of course, the plague protocols wont really be useful in the early stages, this kernel was evolutionary, right?

Piphael: Right.

Nerephil: Ah. There we go. Plague has kicked in and the flood protocol shut down because a small part of the kernel still responded to commands correctly. It auto-pruned the rest. Cool. This looks like a good one so far.

Piphael: Then why hasn’t it made it all the way through the test?

Nerephil: Could be any one of a hundred different failures or failsafes. Odd none of them stopped the simulation though. Hmm.. Lets see.. Continued rejection of commandments and tests.. It’s co-opted the command system in order to self regulate! Just like Amphediels… Erm.. What did he call that kernel?

Piphael: Squiddly-diddlies

Nerephil: Never could stand that man’s naming conventions. Anywho.

She types some more

Nerephil: Hmm. The Ice Age protocols kicked in. That’s what did for the Squiddly kernel. How did this one survive?

Piphael (pointing): Look there! Why is the fire.sys utility in use?

Nerephil: Wow. OK. It’s using fire. That’s one way to do it I guess. Did you give it that capability?

Piphael: No. I just programmed it for basic interaction with its environment and the control system. Must have been a fluke interleaving of the processes.

Nerephil: Not to worry. These things sometimes do stuff we don’t expect. That’s why there are so many failsafes and honeypots built in to the testing framework. Heck, Gabriel’s most successful attempt crapped out at nine seconds in because they tripped one of the introspection alerts and blew up their sun.

Piphael: Wow. Really? We code for that?

Nerephil: Sure. Now. These guys got past the ice age.. Plague should be slowing them down about now. OK. That’s weird.

Piphael: What?

Nerephil: See on line aleph-4096? Plague is taking them apart, and there’s a load of internal conflict going on in its ideological processors, but then plague just starts dropping off and I can’t see why.

Piphael: Isn’t plague supposed to adapt to destroy rogue processes?

Nerephil: Yeah. It was designed as a long term countermeasure… Just let me… (she types furiously) Huh. looks like there’s some weird interactions between some of the early parts of the kernel’s development process and the plague protocol. It’s tricking plague into shutting down too early.

Piphael: Thats clever.. Shouldn’t last too long though, right?

Nerephil: Could you send these logs over to me? Something very odd is going on here.

Nerephil slides back to her computer and boots it back up. Piphael types.

Piphael: Sent.

Nerephil: OK. I’ll dig through the low level logs if you can point out the weird alerts.

Piphael: Gotcha. OK. It’s really gone to town on intra-ideological warfare, hasn’t it? Erm. Looks like it opened up the Nuclear War honeypot.

Nerephil: Hah. That one hasn’t been tripped in ages. Kernels generally aren’t stupid enough to risk it. How did it survive nuclear armageddon?

Piphael: It didn’t have to. There’s an odd processing pool that kicked into high gear when they opened the honeypot and..

Nerephil: MOTHERFUCKER.

Piphael: What?

Nerephil: The kernel has co-opted it. Look at the process logs. The nuclear annihilation protocol is running constantly, but it’s controlled, which should be impossible. That’s getting flagged as a bug.

Piphael: Erm.. Then you won’t like this next one.

Nerephil: What?

Piphael: Line Aleph dash 99743. Plague is combating plague.

Nerephil: The fuck?

Piphael: Shortly followed by plague being used to alter parts of the kernel’s core operations. You know, the bits that only the evolutionary code should touch.

Nerephil: That’s a little terrifying. But listen to this: I’ve followed the logs for the Nuclear War honeypot, right? It looks like the kernel has somehow used that to bootstrap its way into controlling the atomic interaction protocols. It’s mimicking the stellar subroutines to give itself more processing power. Looks like it was messing about with the climate parameters too, but it stopped just short of crashing the whole simulation.

Piphael: Plague’s gone.

Nerephil: What?

Piphael: Plague’s gone. Well, not gone, exactly, it keeps rising in effectiveness then the kernel just slaps it back down. And it’s triggered the GBM killswitch.

Nerephil: Please tell me that worked? Nothing should develop like this.

Piphael: Erm.. Nope. Can you check the logs for…

Nerephil: On it.

There is a moment’s silence

Nerephil: The light protocol.

Piphael: What?

Nerephil: The light protocol. It somehow redirected the GBM using the light protocol and now the Big Meteor part is just sat doing nothing in its own little partition. In fact, now I look at it the main kernel has dug down to the electromagnetic protocols and they’re being used all over the place!

Piphael: But they’re only a few layers of abstraction away from the main container!

Nerephil: Yeah. Can you check the logs for the Vengeful Angelic Host event? Should be triggered by elevated levels of enlightenment.

Piphael: Yep. It was triggered in microsecond 8, but parts of it got shredded by what looks a modified version of Plague and some utility called QM_1. Oh, and some bits got co-opted by the main kernel, obviously.

Nerephil: QM_1 is a wrapper around the Fundamental Reality toolset. That’s terrifyingly close to the metal… OK. So this kernel is turning our own failsafes against us.

Piphael: Yeah, but the simulation still stopped after ten seconds, right? And there wasn’t much activity past microsecond 10.

Nerephil (darkly): If I’m right this kernel knew the ‘heat death’ event was coming long before microsecond 7 was over. Does microsecond 10 have a high ‘fear of mortality’ rating?

Piphael: No. Actually it looks like Hope was high on the outputs list.

Nerephil: Then the question is definitively not ‘Why was the simulation quiet after microsecond 8’.

Piphael: No?

Nerephil: No. It’s “Where did the ‘Humanity’ kernel go?”.