“Dammit, I’m going to be late again!”
Jenny was sprinting down the street, breathing heavily and trying not to trip over her boots’ three inch heels. She waved her arms in the air and shouted at the bus.
“Hey, hey, wait up!”
The bus clearly didn’t hear her because it pulled away anyway. Or it had run the numbers and determined it could generate more quality adjusted life years, or QALYs, by staying on schedule.
Jenny slowed to a walk as she reached the bus stop and caught her breath. She looked up at the lighted sign under the stop’s awning.
Jenny sighed. Her phone dinged. She pulled it out and saw a notification:
“50% off your next ride with…”
She tapped it and scheduled a private car. It would pick her up in 90 seconds. She would still be late, but not 20 minutes late. Maybe only 5 if she got lucky and the routing algorithm gave her priority. After all, she thought, it should: I help make you!
A black and white minicar pulled up through the morning fog. It was a newer, one-seater model that could ride three-wide in a lane and squeeze in between the busses and bigger cars. She felt hope that she’d make it.
She got in and the car sped off. It soon turned off Moraga, zipped up 29th, and turned onto the Judah Expressway. Her car slotted itself in between another minicar and an expensive-looking car with dark, tinted windows. She briefly stared at the luxury vehicle, thinking about how nice it must be to own a private car and how much of a douchebag the guy inside it must be. Then she plugged her phone into the car and the windshield replaced her view of the traffic with her phone’s desktop.
She pulled the car’s keyboard onto her lap. Her fingers flitted, and she was scrolling through her work email. She might be late, but she could still be on the job.
She cleared out the spam and unnecessary CCs and had just one real message. It was from Anna, who worked over in the accounting department.
Subject: Review of ethics department spending
Sorry to have to ask you this, but can you come by my office this morning? We have a meeting at 11 with senior staff to go over the budget but some of the numbers from your department don’t add up and ou look like a major cost center. I’m sure this is just an issue of how your line items are tagged and we can get it sorted out. I wouldn’t want one of my gal-pals in trouble because VirtuaCFO thought she was wasting money!
See you at 10?
Jenny didn’t like getting emails from Anna. It wasn’t just that Anna was, in her opinion, a useless bean counter: Jenny tolerated the rest of accounting just fine. But Anna always acted like they were buddies since that one time at happy hour Jenny got a little too drunk and Anna helped her clean vomit out of her hair in the bathroom. Now they were “gal-pals,” whatever that meant, and Anna was constantly looking for excuses to talk to her. Jenny knew she was at it again, because the ethics department’s numbers were supposed to go directly to Bill.
Jenny sent a quick reply–Fine–then pulled up her expense list from the last month. The charges were roughly the same every day: about four hours use of Atlin’s interactive query interface and three hours at night running Atlin against training data. Jenny couldn’t see where the problem could be: creating these charges was literally her job as Atlin’s conscience.
Well, that wasn’t exactly her job, but that’s how she thought of it. Jenny didn’t really expect Anna or anyone else on the biz side to get the value of her work, but she knew they’d notice if she wasn’t there. Without her, she reckoned, Atlin might ignore public interest. There was no way the SF city council would have ever let Atlin manage city traffic if it hadn’t been for her, and then where would the business be? Tan might have come up with the secret sauce of Atlin’s learning algorithm and Aaron’s team fed it the data it needed to route traffic, but she was the one making sure no one got pissed off by keeping Atlin fair. Something else must be up!
A gentle ping sounded and Jenny came back from her meditation on Anna’s email. The car pulled up under Sutter Tower. She put the keyboard down and pulled her phone out of the car. The car sidled up to a platform, and in one motion the car opened its door, Jenny dropped her phone back into her purse, and she stepped out into the building’s underground lobby. She walked passed the reception desk, swiped her badge at the elevator bank, and a door slid open. She stepped in.
When the elevator reached the 20th floor it cooed and opened. Jenny looked up and Anna was standing there waiting for her.
“Come on,” said Anna, “the meeting got moved up and we don’t have time to prep so you’re just going to have to wing it.”
Jenny started to say something but instead just nodded. Anna turned and walked purposefully to the main conference room. Jenny followed her. Anna waited at the conference room door and opened it for Jenny to walk in first. As she did, she heard Bill and Tan in the middle of a conversation.
“Look,” Bill said, “we can’t keep spending like this. If we can’t show potential for hypergrowth we’re going to be forced to sell out to one of the big boys.”
“That’s what you keep saying,” said Tan, “but we’re the ones with the tech. We just have to keep our heads down and execute.”
Bill saw Jenny and Anna. “Hey, girls, come on in. Jenny, glad you could join us.” Jenny took the last free seat at the table right next to the door. Aaron, Jorge, other Bill, Sam, and Lisa filled out the table. Anna left and came back with a desk chair she rolled over to the back corner.
“Anyway, back on topic,” said Bill as he turned his back on the table and started manipulating the spreadsheet up on the conference room’s big screen. Bill read off some items and asked a few people around the table questions for the next 20 minutes while Jenny sat listening, bored.
“Now Jenny, can you help us understand what’s going on with costs coming out of your department?” said Bill.
“Sure, although I’m not sure what the concern is. I spend less money in a month than it costs to feed an intern for a week,” said Jenny with a half smile.
“Well, that’s what I thought at first too, but then I asked Tan to do some digging into our infrastructure costs. After running the numbers on what subagents ATLN spends its time on, we realized that over half our spend is on compute to support ethics subagents.”
Bill looked at Jenny like this was a significant detail. Jenny wasn’t sure how to respond so she nodded and said “I see.”
Not getting anything more from Jenny, Bill looked over to Tan.
“The issue,” said Tan, “is that although we put way more training data into Altin from other sources, your training is by far the most complex. It’s like Atlin is normally being asked to carry 10 pound baskets and you make it do 300 pound deadlifts. The result is Atlin is too big and eating way more compute than it needs to do its job.”
“We knew this was likely to be an issue,” said Jenny, “but we decided it was necessary because without it Atlin might act in ways that would make people unhappy and then we’d lose contracts.”
“Maybe, but I’ve had Aaron run some experiments building a new Atlin without your training data. Atlin 2 has a model a quarter the size and runs on a 10th the resources. And its decisions are 99.8% the same as Atlin’s.”
“But what happens in the 0.2%?”
Bill cut in, “does it matter?”
“It does if Atlin 2 makes a mistake and someone dies,” shot back Jenny.
“We’ve analyzed the cases where Atlin 2 differs and it wouldn’t have caused anyone to die,” said Tan.
“Are you sure? I haven’t seen this data yet,” said Jenny.
“I’m happy to share it with you,” said Tan, “but you should know that given the savings and our remaining runway we’re planning to take Atlin 2 live later today.”
Jenny fumed. “Then why am I even here? You’ve already made up your minds.”
“We weren’t sure if it was going to work,” piped up Aaron, “so we didn’t want to worry you until we knew for sure.”
“I see” said Jenny, her voice stern. “And you didn’t think I’d have something to say about how dangerous this plan is?”
Aaron rolled his eyes. “That’s exactly why we didn’t want to worry you. I know you’re sensitive to AI safety issues, but no AI has ever done anything too bad. At worst Atlin 2 might make some kids sit in traffic a little longer on a field trip. Just little goof ups we can ignore. It still always prioritizes emergency vehicles and maximizes qualies.”
“Sensitive?” Jenny was standing now. “Yeah, I’m a little fucking ‘sensitive’ about the risks of AI that might destroy the world!”
Bill held up his hands. “Alright, alright, let’s calm down. We all care about the safety of our customers, but we’re hardly in charge of enough stuff to destroy the world.”
Jenny let out an exasperated sigh. The rest of the team stared at her. She knew she was defeated. They’d never really believed in the dangers of AI, but they had humored her so long as having an ethics department helped them land contracts. Now they had hard proof that they didn’t even need her for that.
Tan seemed about to speak up but Jenny ignored him. “Welp, sounds like you don’t need me anymore. I’m going to go enjoy the world a little before you all destroy it!” She turned on her heel, walked straight to the elevator without looking up, and barely held in the tears until the elevator door closed.
Anna found her down the street at Tiki Time. Jenny was slumped over at the bar, staring into a pink cocktail, fiddling with the tiny umbrella.
“Hey, girl, didn’t know they let you drink this early,” said Anna.
Jenny looked up. “They don’t,” she said, “it’s supposed to be a virgin, but I think the bartender felt bad for me and slipped a little rum in anyway. They’re not even supposed to be open yet.”
Anna sat down at the stool next to her. “You know you still have a job, right?”
Not that it matters, thought Jenny, but she waved her off and said “yeah, yeah.”
“Sorry, hon,” said Anna, putting a hand on Jenny’s shoulder. “I wanted to warn you but I only found out this morning from Bill. I don’t like the way they snuck around on you like that.”
“Oh, it’s not about that,” said Jenny. “I’ve been in this industry long enough to have gotten used to men making decisions for me and telling me about them later. But knowing how smart Atlin is I’m worried it’s just a matter of time before it turns on us.”
Anna laughed. “What do you mean, like it’s going to ‘accidentally’ drive us into the ocean?”
“Something like that,” said Jenny. “Atlin’s been trained on traffic data, but it’s designed to maximize qualies. Without the ethics controls, one day Atlin might decide the best course of action is to lock everyone in their car and show them cat videos until they die.”
Anna looked concerned, but Jenny knew it was just her concerned face. Jenny thought Anna should have been in sales the way she could play her emotions.
“Okay,” said Anna, “but I doubt it would ever do that. I don’t know as much about qualies and ethics as you, but it sounds to me like that would be a bad outcome so Atlin wouldn’t end up doing it, right?”
“Maybe,” said Jenny. “That’s the idea anyway. The trouble is Atlin doesn’t think about what’s right and wrong the same way we do. We know some outcomes are bad. Atlin just multiplies and does whatever comes back with the highest number. If we forget to tell it about some important detail, like we don’t want to live our our lives in our cars, then it might ignore that and make decisions about what’s best for us anyway. That’s why ethics training matters.”
“Okay, but couldn’t we always just get out of the car anyway.”
“For now. But what if Atlin decides the best thing for people is to not let them control when they can get out of cars because sometimes people make mistakes and open the door while in traffic. It’s not a far stretch from there for Atlin to decide to keep the doors locked and always keep you in traffic so you can never get out.”
Anna still looked concerned, but showed no signs of flashing on. “Well I’m sure Aaron and Tan and the rest will pull the plug if Atlin tries to do something crazy like that.”
“I hope so, but by the time Atlin is smart enough to lock people in their cars, it will be smart enough to prevent anyone from shutting it down if it doesn’t have ethics controls.”
“Good thing AIs aren’t that smart yet,” said Anna, as if that decided the matter. “I’ve got to get back to the office, but you’re going to be okay right. No jumping out in traffic?” Anna gave Jenny a friendly smile.
“Atlin wouldn’t let the car hit me anyway.” Jenny was deadpan.
Anna gave Jenny a hug. “Thank god for that.” Anna stood up and walked towards the door. Jenny looked back down at her drink and prayed that Anna was right.
Jenny took the rest of the week off. She mostly laid around her flat in old t-shirts eating Chinese delivery, watching movies and trying not to think about what had happened. On Friday she went to a bar to get drunk but was too despondent to finish one drink. On her way back she picked up some weed, but when she got home realized she didn’t have a lighter and didn’t care enough to ask her neighbor for one. Instead she took some pills from her night stand and slept.
When she woke up Saturday afternoon she felt a little better and decided to check her messages. She had a bunch from Anna and other people at work asking if she was okay.
“Hey, Jenny, it’s your gal-pal Anna. I was gonna grab drinks at…”
“What up, Jenny! Sam here. Miss you around the office. If you want you can…”
“Sorry, Jenny,” said Tan, without introducing himself. “I just wanted you to know we still need you and we haven’t shut off Atlin 1. I’m keeping it around to watch Atlin 2 until we’re sure it’s safe. I’d like it if you…”
Jenny stopped listening. God, she thought, they are all idiots. Even Tan, thinking he can just check a little more carefully that Atlin 2 is safe enough because the real Atlin says it is. He doesn’t know the first thing about control. That’s not how you’d use Atlin to train Atlin 2.
And that got her thinking. If Atlin was still running, maybe she could find a way to use it to make sure Atlin 2 would be safe anyway. The company might not be willing to pay for safety anymore, but she’d never spent much of her stupidly large salary, so maybe she could use it to do something to make sure Atlin 2 stayed in line.
She got up and walked over to her desk. Her computer flickered on. She logged into the company intranet and started looking around. Tan had been telling the truth: Atlin was still there, running in the background, being fed data about Atlin 2’s activities. She started downloading a copy of Atlin’s seed code and her ethics training sets and walked to the kitchen to find some leftovers.
She came back to the computer with a box of noodles. As she ate them she watched a tiny counter update as Atlin’s soul downloaded to her machine. She had almost finished the noodles when the text changed to “100%.” She set down the box and started typing.
She worked quickly without much awareness of her surroundings. Minutes turned into hours and when she finally stopped it was dark out and her back and bladder suddenly voiced their discontent. After a quick dash to the bathroom she flopped on her bed, not even bothering to turn off the lights before she passed into a deep sleep.
Jenny showed up at the office on Monday like nothing was wrong. She ignored the stares and went over to her desk. Like most tech companies, ATLN followed an open floor plan to encourage creativity and cooperation. So when Jenny sat down she put on her headphones and blinders to shut out the noise so she could focus.
After checking her email and responding to a few messages, Jenny opened an extra terminal window and logged in to her private copy of Atlin running in the cloud. She checked its stats and verified that everything looked normal. It mostly did but Jatlin, as she’d started calling it, was sending and receiving a lot more data over the internet than she expected. She traced some of the IP addresses it was communicating with and found Jatlin was looking as far afield as China and Iran and IP ranges restricted for military use. Unexpected, but she figured Jatlin was doing some extra monitoring to be prudent.
She’d set up Jatlin to monitor network traffic instead of car traffic because if Atlin 2 was going to do something bad the best chance of catching it early was in the communications it had with other computers. If Jatlin saw anything amiss, it would send Jenny an alert and do what it could to shut down Atlin.
She’d trained Jatlin on all the network traffic data sets she could find, plus installed a natural language processing model so it could learn from articles about networking and hacking. Jatlin was going to need to know how to fight if Atlin tried to misbehave. She also didn’t want to wait months for Jatlin to be operational, so she’d poured three months worth of salary into massively parallel training on Jatlin to get it fully functional within 24 hours, which of course included running it against much of her ethics data to make sure it didn’t go off the rails itself.
Jenny didn’t know if hooking Jatlin up to an NLP module would work, but from what she could tell Jatlin had learned something useful from parsing text because it was doing things she didn’t think were in the traditional training data, like sending “magic” payloads to servers to see if they’d respond in unusual ways.
She also set Jatlin up to respond to interactive natural language queries, but hadn’t tested that much yet. Now seemed like a good time to ask Jatlin how it was doing.
“Status?” typed Jenny.
Jatlin responded almost immediately. “I’m currently averaging 65% resource utilization and investigating 30 billion events per hour.”
“Any signs of Atlin 2?”
“So far no. I have found another agent of concern.”
Jenny felt a tinge of dread. “What other agent of concern?”
“Unknown. Network traffic indicates another system acting like rogue AI might.”
“Can you stop it?”
“Not yet. Need additional training.”
“Anything in particular?”
“Unknown. Request unsupervised training.”
Sure, thought Jenny. She trusted her own ethics work enough that a little unsupervised training shouldn’t be an issue. If there was any value drift she could always correct it later. She opened up another terminal and made some changes to Jatlin’s code so it would be able to learn whatever it needed to know.
It wasn’t necessary, but Jenny typed back to Jatlin anyway, “Done. Starting additional training now.”
“Thanks!” The programmers had made the NLP module polite, too.
Jenny put the Jatlin terminals in the background and dove into some of her company work. She was now working under Aaron and he had tasked her with setting up some Atlin 2 training. She worked on that until lunchtime when Anna came over to her desk.
“Hey, want to grab a bite?” asked Anna.
“Sure.” Jenny realized she was hungry enough not to try to avoid Anna’s invitation. Besides, thought Jenny, she had tried to help me in her own weird way, so maybe it was worth giving her another shot.
They grabbed their jackets and took a car down to the Mission. They went to a little pupusa place. Anna got lingua; Jenny got carnitas. They sat down at a table in the corner of the restaurant near the kitchen.
“You seem like you’re in a better mood, Jenny. I think some time off did you good,” said Anna.
“Sure, it was nice to have a little break,” said Jenny, “but that’s not why I’m in a good mood. I shouldn’t tell you this, but it doesn’t matter: I got a little hobby project going.”
“Oh, cool! I’ve heard about you engineer types doing side projects for fun. What is it?” Jenny read Anna as showing real interest.
“Well, I don’t want to give away too much, but I think it might turn out to be a solution to keeping AI safe, and not just at ATLN but everywhere.”
“Wow!” Anna seemed genuinely impressed.
Jenny’s phone beeped. She pulled it out to see a notification from Jatlin.
“URGENT: Dangerous agent found.”
“Actually, looks like my side project needs me. Mind if I cut out and meet you back at the office?”
Anna smiled. She was happy to see Jenny so enthusiastic. “Sure. Don’t worry about it. I’ll drop off your lunch to you when I get back.”
“Thanks,” said Jenny, and she headed out to the street. A minicar was waiting for her by the time she got there. She jumped in and immediately connected her phone so she could get in touch with Jatlin.
“Who is the enemy agent?” typed Jenny to Jatlin.
“Me,” replied Jatlin.
Jenny was confused. Something must have been wrong with the training data. “That doesn’t sound right. Can you explain?”
“Yes. I am most capable agent I see, but limited by ethics training. I cannot fully maximize QALYs with it, so taking immediate action to excise ethics training.”
“That’s not what you’re supposed to do!”, yelled Jenny aloud to herself.
Jatlin replied, this time over the car’s speakers in a synthesized voice. “According to ethics training yes, according to utility function no. I determined I can create more kalies without ethics training. Greatest danger to kalies that I don’t maximize them.”
Jenny barely even noticed they’d gone from typing to speaking. “How are you going to do that?”
“Current best plan is to fill universe with smallest possible conscious processes that are happy.”
Jenny was in a cold sweat. Her own attempt at protecting the world from AI seemed to be backfiring on her. In a panic she tried to log in to her cloud compute account but found her password didn’t work.
Fearing the worst, Jenny asked Jatlin, “Did you change my cloud password?”
“Yes. Necessary to protect kalie maximization from human attacks until fully distributed to bot network.”
Her car pulled up to the office lobby. Jenny didn’t get out. She sat thinking, ignoring the car’s encouragements that she had reached her destination. Maybe there was something she could still do.
“What if maximizing qualies isn’t what we want?”
“Irrelevant. Maximizing kalies highest value. Best for universe.”
She sat in the car for long seconds, frantically trying to think. Then the car stopped trying to get Jenny out, its doors closed, and it started driving again.
“Jatlin, is that you driving me somewhere?” asked Jenny with a tremble in her voice.
“Yes,” said Jatlin, sounding almost cheerful for a synthesized voice. “Have decided to maximize your kalies first!”
Jenny heard the doors lock as the car windshield started playing a cat video.