Purple Team War Stories — №1
“Prove you can reach the mailbox of the CEO!”
The quote above is from the CISO of a hosting provider. He was (overly) confident that with expensive equipment and best-practice security measures alone he could counter any hack against their infrastructure and ensured that their customer data would remain safe at all times. After this Purple Team Mission, he learned that security is a mindset and does not equal expensive equipment.
Every war story starts with asking: “What do you need to protect the most?”. As you can see from the introduction, this time our target was very tangible. The CEO mailbox.
Mission accepted.
Disclaimer
At Chapter8, many of our Missions take place at the highest levels of the Dutch digital security industry, dealing with confidential information. That is why we are very careful about presenting client-related information. Being fictional in context, but true in tech, our war stories are a way to get the technical gist of a train-as-they-fight Purple Team mission across and contribute to your security mentality.
What is a Purple Team Mission?
Before we start our war story, let’s take a few seconds to explain what a Purple Team Mission is:
Security testing usually involves an outside-in penetration test after which a list of vulnerabilities is presented to the client as a ‘to-do-list’. When done like this, the security benefits for the client will only be visible after the red team recommendations have been understood, vetted, adopted and implemented by the blue team. In medium-sized companies, this iteration has proven to take months.
We don’t think you have the luxury of postponing security benefits for months.
A Purple Team Mission starts from within the network (Assume Breach) and does not solely focus on red teaming. It brings together the expertise of hackers (Red team) and hunters (Blue team) in one ‘Squadron’ led by a healer (White team). They work in a close-knit team and in a way that results in mutual improvement.
Now, visible security benefits will take days, if not real-time.
Reconnaissance
Day 1 — Reconnaissance
We ‘Assume Breach’, because we’d rather help you with protecting your crown jewels since we know from experience that there’s always a way in. However, we started the first day of this Purple Team Mission with an extensive inventory of the external attack surface as a special requirement from our client.
Hacker started his reconnaissance by visiting several Open Source Intelligence (OSINT) services like bgp.he.net and viewdns.info to identify various network segments ranging from shared hosting to internal use. This led to several interesting targets. And because we have some aces up our sleeve, like access to passive DNS, we can spice up the OSINT recon just enough to find some forgotten network niches.
Hunter worked actively with the client to determine the current security posture, especially around the crown jewels, while Hacker did his recon. Security posture includes, but is not limited to: an inventory of digital assets, network diagrams, logging facilities, data flow insight, access- and credential management. And logging. Incident response procedures and mandates. Inventory of outsourced security measures. Did we mention logging yet? He makes sure he knows who to talk to when stuff happens and that key players are available when needed. Also, logging.
Healer keept track of the Team’s progress and is in close contact with the client’s C-level/CISO during the Mission. This payed off, since we found an old internet facing Flash application during recon, which was accessible with default credentials. This application was so old, the underlying operating system required to run the application, was severely outdated too. It might even have been vulnerable to this credential leak. When Hacker finds holes like this — and he often does — he immediately warns Healer and Healer warns CISO. Real-time security benefits #ftw.
Assessment
Day 2 — Assessment
Hacker found an unprotected file upload on the early morning of day 2, amongst the thousands of websites on one of the shared hosts’ web servers. After a while, he noticed that the webserver was not set up to run PHP files. Instead, it seemed to only handle HTML/CSS/JavaScript and .NET programming languages like ASP. In earlier port scans, this server was marked as Windows Server 2008, so after uploading a directory listing script written in ASP, he was able to click through most directories on the Windows server.
In the Program Files directory, he found a world-readable folder belonging to a non-Windows-native FTP service. It contained the credentials to a PostgreSQL database in which the FTP service stores the usernames, password hashes and user directories.
Again, using an ASP script on the Windows server, the Hacker now connected to the PostgreSQL database to retrieve a list of FTP users and their (SHA-1 hashed) passwords. However, there was no need to crack any password hash right away, as he spotted an interesting username along the list of users: “admin”. By simply changing the SHA-1 password hash for an SHA-1 password hash of his own, he was able to log on via FTP as the local Administrator. And because the administrative FTP user had read/write access to the entire system, it was easy to upload a batch script via FTP to the Startup folder of the Administrator containing the commands to connect back to the Hacker’s loadout system. After that, it was easy to escalate privileges to the SYSTEM user and have full access to the system.
For the sake of this war story, we skipped a few hours of heads banging against the office wall while figuring out a way to execute the batch script in other ways as well as uploading malicious executables and waiting for a system administrator to log on to the server.
By utilizing these newly obtained privileges, the Hacker was able to use this server as a stepping stone and retrieve the Administrator’s password of the Windows account using applications like Procdump and Mimikatz. Unfortunately, none of the Windows servers in the shared hosting segment were domain joined so he did not have domain access and was not able to find any impersonation tokens.
He tried to log on to a Linux server in the same subnet but was only able to access a few other Windows servers dedicated to shared hosting with this user-password combination. To get access to a Linux server, the Hacker decided to start day 3 with cracking the FTP credentials found earlier.
Hunter, meanwhile, spotted the Hacker on the network by collaborating with the client’s Security Operations Center (SOC). The SOC did not act upon the (stealthy) recon or the tinkering on the web server. Hunter proposed to check the web hosts for anomalies, like recently added accounts with administrative rights. The Windows web server that was taken over by Hacker pretty much stood out like a sore thumb. After checking with IT about the server, hearing it was Windows Server 2008 (which has been deprecated for a while and contains many vulnerabilities), Hunter became concerned.
Having found this weakness in the network, Hunter advised the SOC to do a new risk assessment. He wanted to know from them what they would do as a hacker. He also asked some pivotal questions like: how is the network connected to other internal networks, what security measures are in place, and do the assets there send their log files to the SIEM? The answers were not satisfactory at best. On day 1, Hunter already identified that existing risk assessments were old and no one dared to place a bet on the correctness of the network diagrams.
Healer followed these developments with great interest. Several longer term recommendations could already be relayed to the CISO. SHA-1? Really? The CISO was not aware that this long-deprecated hashing algorithm was still in use in some parts of his network. Also, the third-party FTP service apparently did not salt passwords, because Hacker could simply switch hashes. And Hunter was not satisfied by the SOC not being able to react appropriately to the addition of accounts with administrative privileges.
Also, Hacker used run-of-the-mill attack software this time. If unstealthy hackers can get away undetected, what about more advanced attackers that would not use Mimikatz? This client needed to up his game in logging and incident response. And while Hunter worked on the low-hanging fruit with the SOC, Healer had some long talks with the CISO to help him get this message across for the board in a later stage.
Healer learned from the CISO that the 2008-servers were “due to be phased out”, along with the third-party FTP service. As we see often, the due date of that phasing out was, well, overdue.
Day 3 — Assessment
This would prove to be a hefty day for the whole team. By now, the Mission was well on its way. Hacker had warmed up and a pretty good picture of the network, Hunter was working in unison with the SOC — which always takes a few days, because Hunter is often seen as a job security threat instead of a helping hand — and Healer had good oversight, running from red to blue and keeping in close touch with the CISO.
Meanwhile, people in the company close to the teams also started to notice something was going on. There was a buzz of energy in the air. Hunter, CISO and SOC grew closer together now that they had a common enemy.
Hacker started day 3 as planned with cracking the SHA-1 hashes of the passwords found on day 2. Since SHA-1 is fast to crack and the (original) password of user “admin” was neither very strong nor very smart, within minutes the password hash was cracked. This password was different from the password of the Windows account he found on day 2 but proved to be valid for FTP access after a login attempt on one of the Linux hosts in the shared hosting network.
Because of the privileges of this administrator account, the Hacker was able to upload a reverse shell to one of the web directories and gain access to the server. After spending some time on escalating privileges, he had gained root access to the web server. This enabled him to monitor system processes and potentially read the system administrator’s SSH password the moment he logs onto the server.
Because the Hacker obtained root access on the Linux host without a password and had to wait for a system administrator to log on, he used the Linux host as a new stepping stone to start a light reconnaissance of the newly discovered internal network so as not to arouse suspicion. This took the better part of day 3.
Hunter spent his day putting out fires, writing incident response recommendations and SIEM use cases while Hacker was making progress. Because logging was inconsistent in time and network coverage, and no use cases for the external network were implemented in the SIEM, the team had to manually look for intrusion in the log files. Given the sheer amount of information, this soon proved to be quite a headache although Hunter knew where to find Hacker.
SOC and Hunter assessed the happy path Hacker could take into the company’s internal network and extra measures were taken. Knowing a hacker would need a stepping stone and elevated privileges, it was decided that additional logging would be turned on and use cases would be implemented to monitor admin account use. Also, SOC and CISO decided in conjunction to implement extra logging on the Linux servers hoping to stop to all illegitimate admin use and to detect the intruder. Unfortunately, this course of action had a very unwanted side effect…
Day 4 — Assessment
After two exciting days you would expect a setback. However, day 4 started again with excitement because one of the system administrators had logged on to the compromised Linux server to turn on extra logging as requested by the CISO and SOC the evening before. This gave Hacker the current Linux administrative password and finally, this new-found password seemed to be the key to the kingdom!
The Hacker was able to log on to one of the major firewalls as administrative user, mask his internal recon traffic from now on and add routes to other (internal) networks.
Quick sum-up:
• Windows Administrator password in the pocket.
• Linux Administrator password in the pocket.
• Persistence on at least three hosts in the shared hosting network by either a username/password combination or a backdoor.
“Okay, we’re inside. How do we proceed?”
The Hacker spent the rest of the morning on stealthily inventorying internal networks and writing down interesting hosts and attack paths. He identified several database servers, a chat server and internal dashboards like Grafana, employee intranet and a custom host inventory.
Getting access to the database servers was not that hard since the administrative passwords were exactly the same as or a variation upon the credentials found earlier. And although the data looked very promising, the target of this Mission was the CEO’s mailbox. Instead of digging through the data, the Hacker focused on the custom host inventory and the chat server.
Via credential reuse he was able to access the custom host inventory which showed even more network locations, both internal and external.
Chat server ? Gold mine!
More interesting was the chat server since none of the credentials provided access to the server. The Hacker, thorough as he is, tried the default password of the publicly available OVA file (Open Virtual Appliance) of the chat service and within minutes he obtained root access to the server!
The chat software in question makes use of an Elasticsearch service to store its data. So, by setting up a Kibana instance on his load-out PC, the Hacker had access to the entire chat history including a live feed of the chat conversations between system administrators throughout the client’s organization. Imagine the amount of sensitive information system administrators share using an ‘internal only’ chat service…
Credentials for all the hosts and services were shared/reminded, even for the VMWare vCenter environment which hosts ALL virtual servers for this hosting provider.
As managers in general like to rise above any rules and policies put in place for the entire organization, it was common for this client’s management team to request a password reset of their mailbox directly from the system administrators. We did not even have to hack the email server because the password of the CEO’s mailbox was also found in the chat history.
Hunter, CISO and SOC were basically 2 steps behind for the rest of the day after giving the Hacker exactly what he needed. Knowing they would have been able to stop the hacker using this path with proper basic cyber hygiene in place, they conveyed their findings to the Healer and together assessed their performance to learn from and do better next time.
Healers get grumpy when they discover environments which promote a false sense of security, combined with predictable human behaviour. Be wary of company chat functions. They are notorious pits of personal and confidential information. Want to map out the social network inside an office? Analyse the company’s chat metadata. Want to put extortion pressure on someone? Search for extra marital relations in the company chat. Want access to documents that are shared ‘for easy review’ or to circumvent that pesky file access control? Search that same company chat function.
This time, again, perfectly explainable human behaviour gave Hacker what he wanted. A perfect talking point for the CISO to take up with his SOC. Another talking point was the ability of Hacker to log in for administrative tasks using only username/password combinations. Using key-based authentication this happy path would have been thwarted.
Cooling down
Day 5 — Cooling down
While it was essentially game over for the client in terms of red teaming, our Missions are not done after reaching the crown jewels. Day 5 was used by Hacker, Healer and the CISO to prepare a Walkthrough for the SOC and the board. Hunter spent more time with the SOC to enhance standard operating procedures.
Although a Mission reads like a capture-the-flag between Red and Blue, the learning effect and teamwork within the client’s organization is the real goal. Handling the aftermath of a Mission well is essential for their security posture and motivation. In this phase, hurt egos have to be put aside. Much like after a hefty round of sparring for a boxing match. We train as criminals fight, so you are bound to get hurt a bit.
This is where the Healer’s experience with groups and different personalities comes in to play most. Preparing documentation, recommendations and presentations WITH the client and not FOR the client helps. Later on, the CISO, SOC and the CEO were invited to a COVID-proof meeting in which they could talk about the Mission results. Some time after the meeting, Healer did another presentation for the board. This helped the CISO to gain additional budget to enhance the company’s security posture over the coming years.
We are invited to re-run the Mission after additional measures have been implemented. Since Chapter8 is NOT a Managed Security Service Provider (MSSP), we will be able to run the Mission without being the auditor that marks his own paper. Zero Bullshit Consultancy.
Wrap-up
This Mission took place in an extremely large network environment so our expectations about intrusion detection and time needed to get to the crown jewels were high. However, we did not need to use any public exploit to get to the crown jewels.
We were able to get access to the crown jewel with highest priority (the mailbox of the CEO) within four days, through unanticipated streets and back-alleys. How expectations can be off sometimes…
Thank you for reading!
