Friends, let’s talk tech and magic. This is a technical story but I put in some magic to dazzle so bare with me. It’s important.
To start with, the central premise of computer science has always been the same: what can be efficiently automated. Computers don’t mirror the human soul, they don’t mirror human memory in the least, they don’t even add like we do. With this in mind, we can begin to analyze computer and network problems without feeling like they are a reflection of us or there is something inherent to them. We get to decide how they work. Very explicitly, every damn step of the way.
With that understood, we all know the media has been obsessed with Cambridge Analytica. I understand the fascination: lots of actually un-understood and interesting ideas such as data mining are involved. But if no one is going to say it, I will:
You have been tricked by this misdirection. And like the masked magician, I will reveal the trick to you.
For some credentials, I am a developer. I’m not the best by any means, but I’ve seen a lot in the industry. And with that knowledge, I’m going to reveal the real deception. It’s a far cleverer trick than data mining, hacking the DNC, and something about Wikileaks. As I said, those were all the misdirection, and importantly, so are the apologies.
See really neat tricks are always ones that need no real magic.
One of the best pieces of advice I’ve heard from a teacher was: “If you want to do something wrong, act like you’re supposed to be doing it.” Granted, this was how his roommate stole beer signs from bars during college, but the adage still bears merit.
So let’s get started on how tech companies have been acting like they’ve supposed to have been doing what leads to these apologies. Keep in mind, that you’ll have to follow along with some boring stuff as this is a technical story, which is how you got tricked in the first place.
So, years back in 2013, some engineer was given a prototype iPhone4, got drunk, and left it on the counter in the bar he was in. Somebody found it and started publishing pictures with it.
Apple has a reputation for secrecy and big reveals. We’ve, or maybe just me and my friends, have all lost phones, wallets, and cards at bars. So I do not doubt this was an accident, and believe it was not some weird publicity stunt.
Mistakes do happen. But some mistakes should raise an eyebrow.
Now comes the turn.
Years prior, a Sprint employee lost a laptop at a coffeeshop with millions of customer records on it (if someone can source this, I cannot as “sprint stolen laptop” gives me way too many support links). Now, I’ve seen plenty of people not pay attention to their laptops at coffeeshops, leaving them out when they hit the bathroom. Certainly irresponsible, but not unrealistic.
When I first heard the story, it was years before I was pro. So I shrugged and thought “Wow, someone got fired today.”
But that’s the question — who? why?
Now we’ll get into some technical details. Up front though, nobody has millions of records of customer info incidentally on a laptop.
Developers typically are given two types of customer data for work:
- “A. Arronson” lives at “123 Fake St”
- A selected pool of customer data, either recent or a curated group, maybe about 10,000 records.
No one would ever be given a laptop with that much information. Even the most irresponsible companies I worked for never allowed that info to be — keep with me — locally hosted on a single laptop.
You might say, “Couldn’t they just have some sort of access to the servers that held the big amount of data?”
Absolutely astute reader. However, if I go to the bathroom at Starbucks and come out and find my highly valuable laptop missing. What would you, my astute reader, assume my first action would be?
For those not following, the first action would be to call literally anyone with security controls and cancel any access my account may have had. Shut it down.
So what is my tinfoil hat conclusion — they wanted it stolen.
Often times developers performance test software with large amounts of data. Sure. But a laptop with that data on it isn’t going to a useful example. Lot of times sales or marketing teams need to review customer data. Again, sure. But millions? Heck, maybe even they were data mining. But with a laptop? What a waste of time. Go back to the office with your coffee and use a more powerful machine.
Sounds paranoid, I know. After all, Edward Snowden walked out with millions of records of NSA data. Yet, we know his name and that he stole it for, ostensibly, political purposes.
Why did this person have millions of records of customer data on hand when no existing development company would ever on a single day allow that to happen?
Unless it was a Snowden-like situation. But instead, Sprint presented it as a “whoops” incident.
My conclusion is that it was no coincidence. Know this sounds conspiratorial , but stay with me.
Many of you probably expected this — Sony.
Sony was hacked. No conspiracy here. It happened. Hell, it was funny watching everyone freak out about Ghostbusters emails (really Paul Feig, alien ghosts is a billion dollar idea? Yeah, he actually wrote that to his bosses. But, go ahead and prove me wrong Paul).
Back to details. The Sony issue was that the hackers were able to access databases that had customer info that contained passwords in what we call plaintext.
To prepare us, a database is a really efficient spreadsheet. However, consumer data is typically encrypted to some degree. This is importantly passwords but also other stuff depending upon the company. Always passwords though. This is why there should be no point in giving a customer service person your password. They can’t see it anyways.
Now, patient reader, you may think that the reason you encrypt passwords is to keep out the hackers. Nope. No company would ever consider exposing password info even to their own developers.
Why? As Independence Day taught — plausible deniability.
Let’s say somebody’s account gets hacked and the victim learns your company has access to their password, which just happened to be the same as their bank account password, which just saw a huge loss of funds, and oh their jealous ex-boyfriend works at your company with access to that database. No, as one of the biggest tech companies in the world, such as Sony, you don’t do that. Ripe for lawsuits.
But it gets even worse.
Let’s go through the numbers on how free out-of-the-box encryption works:
- First, you do some cool math thing that does a replacement of the characters of your password (basically, but indeed more complex as ‘a’=1, ‘b’=2….). This is the hash you’ll often hear about on the news that the NSA or hackers are using quantum computers to break.
- Then you add a salt. This is a randomly generated string of characters that is inserted typically into the head and tail of the encrypted password. This makes it harder to read even if you know the hashing technology.
- Then you do a resort of the string generated from the previous steps based upon a really specific criteria, say the creation of an account’s date, which again, can be encrypted as well.
Hashing tech for free tools standard and typically easy to break. The salt is generated randomly by even free web tools like Wordpress or Drupal and the resort is typically something internal that is variated within divisions of companies like Sony. Much harder for automated tools to break these.
Boy — you gotta do a lot to undo that process.
You might ask a simple question: yeah but Corey, if you know the hash tool, you have access to the database, you know the salt, you know the algorithm to resort and you might even know the SSL key (the green little box in the top of your browser bar that further encrypts communication), as well as the date of sign up, I mean you could just reverse engineer all that, right?
I…suppose, but it would be stupid. Someone at the company would probably notice me up to no good, most companies keep that all stuff separate anyhow like the two guys who have to turn the keys at the same time to launch nuclear missiles in the movies. Hell, you think the CEO of Sony even knows that level of detail?
In other words, even if I was the highest level person at Sony, nearly everyone below would have (and should have) asked a simpler question: why are we not doing normal development stuff? Why are you, most likely, undoing normal development stuff?
As above, I’ll suggest a theory — cause they wanted to easily and quickly read your data.
And a rougher point: where did that data go? As I said, lots of people use the same password for bank accounts as they do media accounts. Why else purposefully not do the most standard of security activities and leave passwords in plaintext? The hackers got the data, but what was Sony doing with it beforehand such that they need it readable?
Now we come to recent hated and cry-me-a-single-tear company — Twitter.
Twitter, supposedly, did encrypt their data when saving it to their databases. But then it unfortunately got into a log.
So let’s discuss logs. Most folks probably know what a log generally is, so I’ll dispense with the explanation. Generally though:
- Logs are good
- Logs are typically keeping track of incoming web requests (loading a web page, including login pages) and help companies protect themselves, mainly tracking intrusive behavior or for debugging.
- Logs are like a database written out chronologically with lots of additional info like the source of the request, type of device, browser, etc.
- There are entire teams devoted to logs at large companies and they are generally the most hardass ballbusters in literally any company I’ve been in.
- Logs also tend to end up in databases…
- And so that means…logs are subject to all the rules about password encryption I mentioned above, and as such no one would have not noticed.
So we know what that means: somebody purposefully undermined common practices. Then someone got caught or, more likely, threatened to go to the media and Twitter fessed up.
Now that we’ve discussed logs, let’s talk about developer logs. No company worth its salt doesn’t keep track of its code. You know that Word doc from college you had where you kept writing “final draft” over and over again. Developers have serious tools for that, namely git and Subversion (aka svn):
This stuff is like a CSI blacklight. You can find everything that was shipped out to the world. So — simple question: what do those logs look like? Not the ones where all the passwords were held. The developer logs. Cause I got news for anyone dumb enough to believe that this was a a “whoops” situation, there is evidence of the developers actions and, as before, who did it, and when they did it, and how long the code was implemented. It’s totally possibly it was a mistake, somebody’s first day for example, and I’m sure Twitter has servers on their CDNs all over the world so code updates take hours.
But simple question: instead of saying “sorry,” can you tell us how it went down? Otherwise, you’re hiding something. You got the developer logs.
It’s like how you never believe a drunk when they say they slipped up and had two beers. Oh, they had at least two. And most likely many more. The question is always — what specifically did you do? Not — what are you willing to tell me you did?
Now, all this really has nothing really to do with our data. Seriously folks, general advice, use password tools, you can fix setup them up in an afternoon, and most banks in the US at least have pretty generous reimbursement policies for fraud.
The problem, and the matter of this article’s frustration, is the trick.
Well, let’s be clear — the lie.
All of these issues are obvious. You would have be as thick as bricks to not notice them as a developer at these specific companies and so many more.
The tech world has been amazing for myself and most of the world (except you know, all those people dying when building iPhones so Words with Friends can happen). For many of us, it’s been the opportunity to get nice homes in nice neighborhoods, saved our lives with its generous health benefits, people patted us on the backs and said we were smart, hell, we even got free take out on Tuesdays and a single beer one Friday every month.
But there are so many people looking the other way when obvious stuff happens because the pay is good. Cambridge didn’t need to employ all this elaborate machine learning and data mining to manipulate us. Google has been reading your emails directly for years and then modifying search results based on verbatim reading your actual words with friends.
And then we said the following — Google is the best tech company in the world. No. They were the powerful, and the ones on which, including me, we were the most dependent.
Cambridge is awful by all means. But it’s like drinking a single beer then injecting meth and blaming your ensuing problems on the one beer.
Astute reader, I’ve revealed the trick. To my friends in the tech world, did you not realize you were a part of it? Or was that Jimmy John’s sandwich just that delicious each Tuesday? You were so afraid to lose that nice lease on your car, house, and who knows what else, such that you couldn’t say a word.
Yeah, Facebook is sorry. Twitter is sorry. Google will probably be sorry. But like any good magic trick, it’s more about doing one thing with your right hand and doing the real trick with your left.
Tech companies have for years as evidenced above been exposing and most likely selling your data. Then when found out say “sorry, computers are hard, amiright?” Then their employees stayed tight lipped.
In nearly ten years in the industry, my loud mouth has learned it does not literally pay to have a loud mouth.
Call me paranoid (I am), but until we see the git logs, I call bullshit. Just like with a good magic trick, I don’t know how it was done, but I know magic doesn’t exist.
Tech companies are sorry cause they got caught.
No more apologies. Clap your hands, and show nothing up your sleeves.