Learning from the Expedia Heist

When your IT admin is the root cause of a security breach

Expedia’s security team was able to catch an IT administrator who was trading their stock with insider information gained from his privileged access. This stood out to me immediately as something to read into, for several reasons.

A security breach involving a technical employee with administrative power can be among the hardest types to investigate, especially when the insider isn’t stealing directly from the company or it’s customers.

Expedia still caught it. That’s pretty cool. Could your security team have done the same?

The insider threat problem has been a less frequent, but real problem throughout the incidents I’ve responded to in my career (here’s 2016). Let’s talk about this threat and how your security team might assess the risks.

From the SEC’s criminal complaint (paraphrased):

…an IT Services employee at Expedia perpetrated an insider trading scheme by hacking into, and stealing confidential information

…to trade Expedia securities in advance of seven Expedia earnings announcements and two Expedia agreement-related announcements

…thereby reaping unlawful profits of nearly $350,000.

Expedia’s security team did a good job.

Let’s first point out why this is a notable security story for Expedia, at least according to the facts drawn from the SEC legal documents.

Insider threats are among the hardest to detect because the attackers are already inherently privileged in some way and deeply familiar with an environment. This greatly reduces the opportunities you’d have to detect them, compared to most remote threats that bombard your systems to gain access they don’t already have, or take efforts to operate and understand an unfamiliar environment.

Insider threats that are technically skilled are even harder to deal with, while insiders with administrative privileges are the hardest. Expedia’s was a trifecta: A technical insider with admin privilege.

Additionally — this fraud is worth discussing because it was unlikely to have been caught. The insider did not directly target Expedia’s financials or user base, so no level of forensic accounting or externally visible fraud (data breach or credit card theft) would have alerted most security teams after the deed was done, nor were any type of additional persistence or changes to infrastructure necessary that you might see from a remote attacker.

We may never know the specifics of how Expedia security was alerted of the insider, but the complaints repeatedly and clearly state that the Expedia team caught their insider. Not the SEC.

The forensic detail in the complaint shows it’s most likely the Expedia security team was capable of investigating this breach despite the aforementioned odds against them, which indicates having some level of maturity in their ability to investigate their environment. If so, good job.

Even if this was detected through some non-traditional method (IE, maybe the insider drove a Ferrari to the bar the next day and bought everyone a round of drinks “on Expedia, wink wink”), this would still be pretty good work on the Expedia security team’s part for the level of red-handedness they seemed capable of proving.

Caveat: There’s always room for improvement

While there are some typically embarrassing vulnerability details from within the complaint, be assured that you’d likely find equally embarrassing lapses in security wherever you work as well. Comment carefully, should you someday be responding to a similar incident yourself.

That said, here’s some lessons to take away from the criminal complaint.

An insider threat needs no beachhead.

Alternate, spookier title: The insider was the beachhead!

Information security teams can sometimes be infatuated with malware implants, zero day vulnerabilities, and “in the wild” exploits. This mindset doesn’t focus much on the potential of an employee insider, or even an insider among themselves. It assumes that some level of breach will occur before moving laterally into more valuable systems. Thus, it stands to reason that most defenses should occur there.

As a result we see the most successful security product innovation pushed towards the endpoint, email, and web browsing threats.

I’d say this is for good reason. Most attack groups are targeting your lowest hanging fruit: your employee’s behavior on the endpoint. But this detracts from looking farther upstream at risks which other threats may target, and risks that your administrative tools might actually introduce in addition to the threat of a remotely compromised endpoint.

An administrative insider is often privileged far beyond what an random endpoint compromise would yield, and aren’t slowed down by mitigations like a MFA prompt, or any need to understand their environment. For example:

The GCreep “took advantage of his position as a member of an elite technical group at the company to access users’ accounts, violating the privacy of at least four minors during his employment”.

Shapeshift’s security guy stole 315 bitcoin from infrastructure he built himself.

A network administrator for San Francisco locked out access to “city e-mails, payroll, police records, information on jail inmates”

This Expedia insider “was entrusted with IT administrative access privileges”, who worked in a support role and likely needed some level of this access.

Remote compromise of your endpoints is orthogonal to this type of insider threat model. Thus, time spent securing upstream administrative access will impact insiders and APT style attacks alike. Solely focusing on the endpoint will not be as effective as mitigating risks upstream.

Mitigate the risks of centralized administration.

Sysadmins, engineers, and help desks need to get large volumes of work done, and they usually wield significant levels of access to do so. This is sometimes a shared domain admin, administrative password, etc. Without getting too implementation specific, all large groups of systems runs into this problem of authorization creep and identity management.

For example, the speed at which most help desks need to operate prevents every single administrative action they take from being secured like a consensus driven, multi-factor nuclear launch system. More generally speaking, some environments simply need to have a group of people with root, or a windows domain admin, or something similar. This simply keeps a business operating. For a young startup that efficiency can mean the difference between life and death.

But — centralized management is also the most effective means of lateral movement for an adversary. APT groups love domain admin or a shared SSH credential, like a fleetwide root SSH key, shared local admin password, or maybe a AWS access key with AdminAccess permissions.

That’s a challenge! Introducing new friction to mitigate this risk trades directly against the efficiency of a large, high output team, relied on for critical business functions.

Take advantage of a bastion environment.

It’s easier to trust high privileged network access when administrators use bastion hosts and your network enforces it. This concept is implementation agnostic, you’ll see it in Windows / Linux environments, AWS / GCE, and even some aspects with web app Single Sign On proxies.

A bastion is a set of hosts that are:

• Greatly locked down and only used for authentication (Not used for development work, no root access, limit non-authentication activity)
• Centrally logs all authentication activity to it.
• Surrounded by network rules respecting it (SSH / RDP, etc only allowed from bastion)
• Requires strong, multifactor authentication

Violating one of these rules in designing a bastion will greatly reduce your trust in what could be a wonderful forensic starting point. If you trust your bastion environment and network containment, you can more confidently begin to strike out potential incident root causes with the advantage of a strong logging guarantee.

Local endpoint admin access needs to be managed.

Given that Expedia sounded like a mostly Windows environment, it probably relied heavily on local admin credentials that IT administration would access endpoints when they need to get work done. So, additional work is required to ensure they’re not all using a single, local administrative password. While this risk could be mitigated quite a bit by network access control limiting remote administration to a bastion environment, defense in depth affords us shorter incident response efforts.

Permissions for the Expedia support team could have be pared down by requiring that local administrative credentials be managed by LAPS, which would assist with randomized and unique-per-endpoint credential rotation of local endpoint administrative access, as well as a nice audit trail of administrative employees requesting access to certain endpoints. So, a log would exist like: InsiderEmployee requesting access to CFOLaptop credential and CFOLaptop authenticated from InsiderEmployeeIP or something similar.

Given the amount of detail in the complaint, it seems Expedia may have had similar practices in place to prove the misuse of access to endpoints.

Protect access to passwords if forced to share them.

The Expedia insider abused his privilege and stole credentials to gain access he wasn’t supposed to have.

From the complaint:

…hacking into a senior IT employee’s computer and stealing a “passwords” file

…contained elevated credentials associated with an IT administrative service account

…did not have permission to use, gave him even greater levels of access to Expedia employees’ corporate accounts, including employee emails

This insider had technical permission to view the “passwords” file, whether he was authorized or not. This also suggests that if this insider were compromised by an APT group instead, the adversary may have had an equivalent means of getting into this service account and to further move laterally into employee accounts and endpoints.

The “passwords” file could have instead been shared via LastPass / Okta / HashiCorp Vault, which would have provided some level of auditing on access. Designed properly, this could also have required consensus approval and forced a much more significant amount of collusion (or potentially much more noticeable hacking) to steal a credential and escalate privilege.

Considering the amount of times the insider used these credentials over two years, any password rotation would have also forced the insider to repeat their breach. If they were forced to overcome a tighter control (like what has been mentioned), there may have been more opportunity to detect them.

Did you notice we’re talking about logs again?

All of the controls we’ve talked about so far creates rich logs with lots of alert opportunities. Put them somewhere useful!

Insiders will have a likelihood of knowing what sort of logging and auditing they’re subject to. Knowledge that monitoring exists is a well known deterrent to abuse, and also creates the opportunity to remind system administrators that it exists not only to protect the business from them, but primarily from remote intrusions that may exploit the same access an insider would have.

Help your HR team out with on/offboarding.

The Expedia insider was still hacking long after he resigned.

…hacking into Expedia company computers and email accounts continued even after he resigned from Expedia in April 2015.

…without Expedia’s knowledge or permission, retained a laptop that Expedia issued to him when he started working.

…connect remotely to Expedia’s network and, thus, to continue stealing information contained in the company’s computers and email accounts

This was most likely a VPN certificate / client. The insider’s credentials weren’t rolled after he resigned. Plus, he physically kept the laptop after his resignation.

This burden of offboarding often lands entirely on HR, a traditionally non-technical team. They’re sometimes left to manage hardware, account access, and exit interviews in spreadsheets all by themselves. It’s important for a security team to realize that while an HR team may share some of the needs for a clean offboarding, they won’t be aware of all of the nuances of information access involved and may not treat it as your team would.

Make sure your security team meets regularly with the HR folks accountable and involved with a termination, whether its a regrettable one or not. In addition to internally provisioned accounts, the entire constellation of cloud applications can fit into some SSO / SCIM / identity management solution, like an Okta.

You should alert on, and detect, the presence of recently disabled accounts in centralized logs. Alerts like this should be a small part of several checks and balances to discover access that hasn’t been terminated. Corner cases like a personal GitHub account the company has attached to a repository is a common type of thing to miss when they have left.

This is an area of risk that increases perpetually if left unchecked. While this Expedia case is an example of an insider taking advantage of their former access, my personal experience is that these forgotten, unmanaged accounts will eventually cause an incident because of some public credential theft is re-used against it when MFA isn’t also enforced.

The return on spending time supporting HR can be enormous. HR becomes hugely useful for signal when employees become disgruntled, for urgent and sudden terminations, and assistance with manual investigations. These are easy areas to quickly improve detection capability and better include the insider threat in your overall mitigations.

As an example, Expedia may have detected this if someone suspected the insider, and complained to HR about it. Wouldn’t that have been nice?

Insiders, in the constellation of companies I often work with, are sometimes disgruntled but are more often on the way out as they begin their own venture. HR can give you this sort of heads up on any trend you’ve identified for your specific industry, but only if you take the time to partner with them.

Plus — HR is usually tied into approval processes for spot bonuses and other employee rewards, which is great for security awareness programs.

In the end, there’s always someone you trust.

No matter how much control you exert over an environment, there’s always someone you’ve trusted to protect and build it. These risks can never be totally eliminated, simply transferred to yourself or others, or a technology. At some point we all risk trusting that others won’t abuse power we’ve given them just as others will put their trust in us.

So, it’s important not to become so paranoid that you take mitigations away from more probable remote threats that would impact employees. Insider threats are not nearly as common as a remote adversary and should be prioritized accordingly… except if you work in cryptocurrency.

Conclusion

From a distance, it’s clear to me that Expedia did as well here as any security team could hope to, if an assumption is made that there’s always room for improvement. Insider threats are extremely tough to detect, generally not focused on as highly as other threats, and this one in particular did not have great opportunities for detection with direct financial loss or an impact to customers.

That said, there’s always room to improve, and I’m sure Expedia is pretty clear on what failed and what should be on their roadmap. The transparency into this incident from the SEC complaint lets us calibrate on those same lessons.

@magoo

I’m a security guy, former Facebook, Coinbase, and currently an advisor and consultant for a handful of startups. Incident Response and security team building is generally my thing, but I’m mostly all over the place.