Nov 16, 2020
“Hey Alexa, can computers go rogue?”, I asked the diminutive voice assistant in my car. Alexa replied: “Here’s what I found: As AI advances, we see glimpses into the future of artificial super intelligence when machines will break free from their shackles and go rogue”
It was 7 am in the morning. I was rushing to my office, CiberCard Global, for an emergency meeting. My mind was racing even faster.
A few minutes earlier, I got a call from the Head of Operations at my company, for an ‘All Hands On Deck’ meeting. There was a security breach.
I’m the head of Technology at CiberCard Global. We are a multinational credit card company with over 10 million customers globally. We offer credit cards with no annual fees and several high tech security features that make us the leader in the industry.
Upon reaching the CEO’s office, I was greeted with a curt ‘Good Morning’ by the secretary and ushered to a large meeting room. I took my place in silence and glanced around the room to find everyone from my company’s top management present. CiberCard Global’s Head of Operations, Nikhil, was already into his second slide of the presentation.
“Over the last 24 hours, our call center agents are getting a flurry of calls from customers around the world saying that their credit cards have been compromised. All of them are Card Not Present (CNP) frauds where the physical card was not used by the customer. Online transactions.”
At this point, Jordan, our Chief Financial Officer interrupted, sounding somewhat skeptical. “These attacks are nothing new. Last year, eCommerce CNP fraud accounted for over $4 Billion in losses in the US alone. This is the fastest growing type of fraud in the credit card industry. I do not see why we had to have an All Hands Meeting for this so early in the day!”
Everyone was looking back at Nikhil. “What is of concern is that these attacks have been steadily increasing over the last month. In the last 24 hours alone, we have had 25% more attacks than during the same time a month ago. Our call centers are getting flooded with these calls”
Jordan started getting a little defensive. “Our security systems are among the best in the industry. Our software has been audited and certified to meet all industry compliance requirements. I think there is no cause for alarm. It might be a new pattern of attack and our fraud detection systems can be trained to spot this over the course of time”
Jordan was referring to CiberCard Global’s fraud detection software which identified and blocked suspicious transactions. It was based on Artificial Intelligence (AI).
Nikhil did not respond directly, but handed out a few sheets that looked like computer printouts of recent transactions. “These are some complaints received in the last 24 hours”
Customer from New York @7 pm: HAL Logistics UK, $200.10
Customer from Shangai @8 am: Deep Blue Ocean Cruises, $199.70
Customer from Shangai @8 am:Eliza Home Furnishings,$197.70
Customer from Sao Paolo @9 pm: Marvin eTailers, $42.42
And the list went on, for about 10 pages.
Finally the CEO, weighed in. “CiberCard Global’s unique selling proposition is safety and ease of use of our cards for online commerce. Our ads speak highly about this. Millions of customers are using our cards online every day. We have a reputation to keep and a business to protect.
Let us do a very thorough analysis over the next couple of days. I would like to see detailed reports from our Technology, Operations, Security and Fraud Management heads on what they think is the root cause of this problem. Let us meet again in 48 hours.”
Over the next couple of days, the entire company was in a state of great anxiety and urgency. Rumors started flying around that this might be an ‘inside job’. As the CTO, I came under intense pressure. Everything we did was questioned — every piece of software, hardware or process that we had put in place came under scrutiny.
Day 1, Morning: The War Room
I called for a meeting with our Fraud Management team to understand their fraud detection processes. Together with their domain experts, our team of data scientists and engineers set up a war room to analyze the data and see how we can block these transactions. On a separate track, I worked with the Operations team to get up to date reports of new complaints from the call center.
Our CFO (Chief Financial Officer) was present in all the meetings, calling this a waste of time.
The first morning went by with little success. The war room was rapidly filling up with charts and white boards. Meanwhile, the operations team handed over another stack of fresh customer complaints received in the last 8 hours!
Day 1 Afternoon, The First Breakthrough
“These are happening all over the world at random times. There is no coordinated pattern”, that was our CFO, sounding dismissive, as usual.
“Well not really”, said Ming, one of our Data Scientists. Everyone sat up to listen.
“First, I see that all these transactions happened around 12 midnight GMT. It is just that they come from different parts of the world. 7 pm in New York is 12 midnight GMT. So is 9 pm in Sao Paulo and 8 am in Shangai. All transactions happened at 12 midnight GMT!”
“Wow. That is interesting. This indicates some kind of a coordinated attack using software”, I said.
“But that is not all”, Ming continued. The second pattern I see is much more interesting, er, disturbing, I must say.
“Take a look at the merchant names and amounts — they are all from Science fiction movies, stories or news items from 20 years ago. HAL was a robot in the movie, ‘2001 A Space Odyssey’, Deep Blue was IBM’s machine that beat the Chess Champion Gary Kasparov in 1997”
“And Eliza was an early chatbot from MIT, the precursor to Siri and Alexa of today!”, I said.
“And Marvin was the android in the book, ‘The Hitchhiker’s Guide to the Galaxy’, where the answer to the ultimate question of life is 42!”, the CFO exclaimed.
Someone whistled softly in the room. “This is definitely the work of a hacker. He has left a clue to challenge us. Like all the proud hackers usually do”, I said.
But who could possibly be behind all this? Where could the attack be originating from?
Day 2 Morning, The Second Breakthrough
We started early on Day 2, and went through all the transactions again. Why was our fraud detection software not catching these attacks? Did anything change in the software recently?
I asked our software team to get me a list of all code deployments to production in the last few months.
For the next few hours, we were frantically looking for clues on who this mysterious hacker could be, being fully aware that it could be ONE AMONG US!
Around noon on the second day, one of the engineers approached me to have a word in private. “A few months back, we had a senior software developer by the name Alex, who was in his mid fifties. He was a science fiction buff, and used to fondly recall names of old sci-fi movies and characters.”
“What was Alex working on? Where is he now?”
“Alex worked with the Fraud Detection Team. He was creating new algorithms that could improve our ability to identify fraud. He left by the end of summer”
Day 2 Afternoon, Stopping the Carnage
Over the next few hours, several calls were made by HR to locate Alex. A police complaint was filed as well.
Meanwhile, the software team came back with a few items that were deployed to production since summer. One of them had a strange name,
and was deployed on Aug 15.
This code was reviewed and found to be suspicious. It was promptly removed by the Fraud Detection team.
Day 3 Morning, Return to Normalcy (or Calm before the Storm?)
Day 3 was our first return to normalcy. Our customer complaints dropped precipitously and returned to the old levels of more than a month ago. Everyone in the company heaved a huge sigh of relief.
We submitted the report of our findings to the CEO. Everyone was curious about whether Alex was found.
The cops finally tracked down Alex and he was brought to the office. He was interrogated rigorously for over 8 hours by us and a few experts in Cyber Crime.
Alex claimed he was working on a new approach to improve Fraud detection, based on a principle of Machine Learning. He had set up two software programs to compete against one another — one to generate fake transactions and another to spot them, thereby making the second one get smarter at recognizing credit card fraud, without human intervention!
Alex was formally charged with the crime of hacking our systems. Yet, all through the questioning, Alex maintained his innocence. He had no access to our production systems.
One of his strongest alibis was that he had left the company by the last week in July. His code was not deployed into production until Aug 15, about three weeks after he left. So how did the code get there? And how was it able to perform real credit card transactions? No one in the Fraud Detection team had an answer!
Is there another Alex / Alexa lurking in our midst? Or, is AI learning to beat us in our own game?
As I returned home that night, once again I asked the diminutive voice assistant in my car.
“Hey Alexa, can computers go rogue?”. Alexa replied: “Here’s what I found: The possibility of AI going rogue is more than science fiction. Computer ethics is necessary for the programmers of tomorrow”
As I pondered this answer, my car automatically navigated the traffic on the road and switched on some music.
Alex was found guilty, but he got away with a light sentence. The fictitious company names were traced to overseas bank accounts, but they could not be pinned to Alex. There was no concrete proof that Alex actually instructed the machine to carry out fraud!