Investigating Account Takeover
How a security team identifies ATO threats
Users of your product have been attacked and have lost control of their accounts. We’ll discuss how a security team would approach the situation. Each attacker method has its own forensic properties to observe. This will guide your response.
If you’re looking to prevent account takeover altogether, read this instead:
Instead, we’ll focus on response to hacked users in this article.
If you respond without becoming familiar with these different types of attacks, you may botch the cleanup and see your users suffer repeated account takeovers. If you’re experiencing a breach at scale, you will only want to clean this up once, and need to identify it correctly the first time.
Here are a few things that will improve your ability to troubleshoot account takeover:
Start logging user authentications
…And make sure they’re usable enough to investigate.
In order to investigate any type of account breach, you’ll need to have access to a decorated authentication log for your users. You’ll need to observe the actions they’ve taken on the site (login, logout, adding payment instruments, adding / removing things), IP address associated with the action, and Browser or User Agent, which is hopefully a unique string associated with their client. Most security teams only log the actual User Agent string, but it’s better to have something like a randomly generated, unique string in a cookie to compare against many other similar clients for these many breach scenarios.
You will need logging around password resets and additional registration email changes, including the previous and new email you have for the users.
Build Session Investigation and Password Reset Tools
You’ll need the ability to pivot between IP addresses and known users accessing your product. This is an extremely common use case:
- Which IP address compromised this user? (Search for a malicious single IP from one User)
- What other Users did they compromise from this IP? (Search for many Users by a single malicious IP)
- How can I reset the passwords of all these users, and destroy their sessions?
You don’t need to build a big sexy UI for this, but the ability to run circular lookups in a console or web UI will make this experience much nicer. Keep in mind there are infinite other troubleshooting scenarios you’ll be able to use this workflow for, it won’t be used solely for incident response.
These are the types of attacks that would cause a user’s account to exhibit malicious behavior. We’ll start with malware.
MALWARE
If a user is continuously doing bad things despite a password reset… then there is likely some form of persistent malware harming them. In almost all malware cases, the attacks will appear to be coming from the victimized user’s personal machine and IP address, as opposed to a remote foreign IP address they’ve never used (or could not possibly use due to their physical location).
You very likely don’t have access to the user’s machine itself. Near the bottom is appendix of help articles companies use to instruct the user through these steps that you can pull from for your own FAQ
BROWSER EXTENSIONS
People have the habit of installing tons of browser extensions. These extensions have limited, but similar capabilities of executable code running on a users machine.
These can be found by digging into a victims loaded extensions. Chrome Extensions especially can’t hide from an investigation very well unless they’ve managed to pivot from and somehow install other software outside of the browser.
Also, you can explore the permissions of the extension which will give you a roadmap for other forensic response, like modified bookmarks or proxies. You have to eliminate any tampering that occurred.
There is a trend that makes these attacks more common, and that’s previously innocent extensions sold to bad guys to be repurposed as attack tools. An extension a victim may have been holding for a while can suddenly become a blackhat extension.
After being sold to a malware author, new code is pushed down to cause your exploit your users and misbehave on your product.
Mitigation: Try to discover which extension is being used with a single victimized user. Investigate permissions of the extension and what may have changed within the browser. Advise all impacted users to delete extension and clear local storage, exit browser. Destroy all known server side user sessions. Reset the password.
CLICKJACKING
A clickjacked user can make it appear as if their computer taken over by a bad guy, because they will have taken a small amount of actions that they simply won’t remember from their own machine. Except, they’re not infected with malware, despite all the signs of a local compromise.
Clickjacking (also known as a User Interface Redress) is a method where an adversary manipulates a user into clicking in a specific way over objects floating on top of your own application. When the clicks take place, they are actually interacting with your own site.
For example: The user may think they’re playing a silly game like “put the ball in the hoop for a free gift card”. However, underneath this game, they are unknowingly clicking on your product’s user interface and harming themselves from their own machine, as if they’ve gone temporarily insane.
Reviewing the user’s activity will quickly show that they don’t have any foreign logins from strange places, and maintained the same user agent. This will make it appear as if they were fully aware of their actions, when in fact they weren’t. Upon an interview, they may remember a spam email or otherwise remember playing some kind of game, but rarely. They may also have the clickjacking attack in their browser history around the timeframe of the malicious action.
Mitigation: This is sometimes simple, and only requires some browser headers to prevent your product from being iframed by another domain. If you require your product to be iframed (for instance, a digg button), it will get more complicated.
Write up a help document and send it to users that were impacted. If clickjacking caused the user to install malware or add another user to the account, that is a separate issue.
SELF XSS
Similar to Clickjacking, Self-XSS is another example where a user exhibits temporary insanity without actually losing their credentials or seeing a malicious login. The attack is performed by convincing a user to run javascript within their browser by a copy paste into the URI bar or the debugging console. The result is a browser fully under the control of the attacker, very similar to the result of a traditional XSS attack.
Facebook and Google has been impacted so often by these attacks that they throw a warning in the Chrome console when users would be one step away from hurting themselves:
Mitigation: The primary defense against Self-XSS is awareness and education (plus a warning like the above helps as well). Browsers have also made adjustments to add warnings when a user is about to shoot themselves in the foot, but are incomplete.
To clean up a self-XSS’d user, advise the user to clear out all local storage and reset the browser to avoid any obscure methods of persistence that attackers look to maintain. Destroy all known user sessions. It’s not necessary to reset their password, but if attacks appear to be phishing the user as well, it may be necessary.
MAN IN THE BROWSER
Malware will sometimes manipulate the browser itself and interfere with the DOM a user is interacting with. Meaning, a user will be looking at your website and firmly believe you’re showing them a GUI that does not actually exist, except in their own browser.
The family of malware called Zeus has long been a toolkit for malware authors to perform these sorts of attacks, mostly against banking sites but also against social media.
Mitigation: This is just traditional malware removal. You can advise the user to use antivirus (lazy), give them per-malware instructions (high effort), or tell them wipe their machine (expensive and rage inducing).
Destroy all known user sessions. Reset the user’s password.
CREDENTIAL THEFT
Previously we covered a persistent threat on a users machine that causes repeat compromises. In these cases, a users password is stolen and remotely used away from their computer, usually from suspicious IP address.
PHISHING
Many people have a generic definition for phishing, but mine is very specific. Phishing is an attack that leverages a spoofed landing page of your product to convince a victim so share their credentials in the wrong place.
Phishing has lots of characteristics that will reveal itself and is among the most well known types of account breaches you can investigate.
A phishing attack will grab a users password and try to log in remotely from a totally unknown IP address. So, you’ll see records of that. The attacker might spoof the User Agent of the victim if they were able to phish them over a browser (the phisher can see it), but it will be impossible for them to spoof other unique cookies unless they found some way to violate the same origin of the browser.
While the victim is interacting with a phishing site, you’ll likely see Referral data coming back to your domain. This is hard to avoid unless the phishing site is well made. Finding this evidence will not only be a dead giveaway that your hacked user is a result of a phishing attack, it will give you a website to report for abuse.
A huge benefit to interactive phishing attacks is the ability to steal a second factor. This will explain any two factor bypasses that you see. However, it’s only temporary, as the attacker will only have access to one session.
Mitigation: Destroy all known user sessions. Reset the users password. Inform them that if they are using the password anywhere else (especially with the email address they registered with) they are at risk and should reset those too. They should sign up again with a unique password. If two-factor was stolen, they don’t need to reseed their app or anything like that.
DATABASE BREACH
Companies are breached all the time in ways that result in large amounts of passwords becoming available on the internet. Attackers with credentials from a database breach behave very differently than attackers with credentials from a phishing site, mainly because they don’t have a victim engaged with a website of their own.
Namely, they don’t know the victim’s user agent, and the attacker is unable to capture a second factor like a OTP. Also, the attacker will have credentials for possibly thousands of other users that are not on your platform, causing tons of false alarms as they race through login attempts. A targeted phishing site, however, will have very rate of successful logins, comparably, with their stolen credentials.
Mitigation: Destroy all known user sessions. Reset the users password. Inform them that if they are using the password anywhere else (especially with the email address they registered with) they are at risk and should reset those too. They should sign up again with a unique password.
KEYLOGGER
Keyloggers are usually a form of malware that persists on a users’s computer, but can also come in hardware form. You’ll almost always find this in a software flavor, but hardware attacks are possible in situations where a victim was directly targeted by a physical adversary. This is rare, but here’s a picture anyway:
Keyloggers are unique because they attack a user while at keyboard on their own laptop, but usually ship the passwords they discover remotely to an adversary elsewhere. This is a very deceptive behavior while trying to troubleshoot because it looks so similar to other attacks that happen remotely, like a database breach or a phishing attack.
The key difference here is that the user will have their password stolen and used remotely repeatedly, despite password resets. In a database breach or traditional phishing attack, one password reset for the user is enough for mitigation, whereas a keylogger will repeatedly steal their new passwords.
Mitigation: Advise the user through malware cleanup. Destroy all known user sessions. Reset the users password and make sure they weren’t using the password anywhere else, especially with the email address they registered with.
EMAIL COMPROMISE
Identifying a user who has lost control of their registration email address is pretty simple.
- User is compromised and behaving badly
- Your customer support team resets their password, assuming a phishing attack
- The “good” user logs in and all is well
- Another password reset occurs without the “good” user’s knowledge
- User is hacked again
Upon step five, you may be thinking “keylogger?” if you didn’t notice the password reset or have a log of it taking place.
The repeat back and forth email resets are indicative that the maclious actor is able to access the password reset emails to the user, and is triggering them to regain access at will. They’re in the victims email.
Mitigation: This whole writeup needs to be followed (again) with the users email address in mind. Unfortunately, you probably don’t work at that email provider to do the troubleshooting, so this is extra hard as most of the evidence will be out of your line of sight.
Fortunately, most email providers have some sort of “recent activity” you can observe to help isolate the issue, along with your own password reset logs.
Additionally, make sure the user isn’t forwarding email elsewhere, which would give remote access to password reset emails without access to the password.
ADMINISTRATIVE BREACH
The most rare case to troubleshoot is also the hardest, and that’s when almost all of the above scenarios have none of the evidence to point to them. A user’s account is misbehaving, and no amount of troubleshooting can point to an issue with the user, their credentials, or their system. No malware, no phishing, no attack in the browser…. nothing.
In this case, it’s possible your underlying systems or employees have been breached and this is causing the misbehavior. There will be little to no evidence of this behavior unless you’re specifically logging administrative actions (not user actions), because your adversary is accessing administrative tools directly, or even an underlying data at rest on a server.
Here are some examples:
SENDGRID BREACH
SendGrid employees were reportedly compromised via social engineering, which modified user accounts and allowed the “bad guys” to attack Bitcoin companies. Their administrative tools would bypass anything a typical user could do, which would be tough to troubleshoot.
TWITTER BREACH
A Twitter employee used the password ‘happiness’ for administrative access to accounts, and this was discovered and used by a hacker to tweet as Barack Obama. Again, a review of Barack Obama’s account likely wouldn’t have had any evidence associated with it, as an employee’s access was leveraged.
GOOGLE BREACH
Google was breached in 2010 and very publicly blamed China for the incident. Their systems were used to gain access to the Gmail accounts of Tibetan activists. The activists themselves probably saw no activity in their “Recent Activity” and a password reset would have been irrelevant .
Ai Wei Wei, an activist reportedly targeted in this attack, noted how password resets weren’t helpful for him given the user was compromised in other ways.
The irony of this situation is the tools used to troubleshoot account takeover could end up being used by bad guys. This is a whole different discussion and probably a smart thing to look over Starting Up Security.
Mitigation: This sort of scenario can go as far as a total corporate compromise, in which I would recommend reading Security Breach 101.
Conclusion
There are several takeaways that you should see in this article:
- Robust logs are good for IR. The user, the IP, the browser, and user actions like a manual reset. You’re blind without them and you’ll have no answers.
- Build ways to destroy and reset users sessions and passwords easily.
- Understanding the method of breach will guide your response
- Don’t let an internal compromise happen. It’s the worst.
@magoo
I’m a security guy, former Facebook, Coinbase, and currently an advisor and consultant for a handful of startups. Incident Response and security team building is generally my thing, but I’m mostly all over the place.
