Hacking into an AWS Account — Part 3: Kubernetes
I’m excited to dive right into this one because it’s a hot topic, as so many businesses are starting to work with it in connection with their AWS accounts!
So in this instalment, I’ll give you an overview of the potential weak points of Kubernetes and show you how I uncovered real-life AWS credentials during my last research run.
Hold up Dawg — What is Kubernetes again?
First thing’s first: a short overview of what Kubernetes is and how you typically use it in connection with AWS accounts. If you know this part, skip ahead to the next chapter.
Kubernetes (k8s) is an application that can automate the deployment, scaling and management of containerized applications. A container is a set of processes that is isolated from the rest of the system, and while it’s deployment isn’t rocket science, making it operational and scalable can be quite complicated!
This is where orchestration tools like Kubernetes come in — they help you use your containers in concert with microservices and cloud providers. On AWS, for example, we frequently use EKS (Elastic Kubernetes Service) for our complex microservice architectures.
Did you know?
Kubernetes (/koo-ber-nay’-tace/) comes from Greek and means “sailing master”.
So now that the plain-English definition is out of the way and we’re all on the same page, let me put on my white hat and see where the common weak points of this thing are.
In the Beginning, There was the Configuration
Managed Kubernetes like EKS usually have APIs secured with mutual TLS, IAM roles and API-level RBAC (Role-based Access Control). These generally constitute a pretty secure setup.
But when developers configure Kubernetes, it can happen that these security settings are overruled in favor of a less complicated setup. As a result, the API gets exposed.
Working with public Cloud providers doesn’t mean you should have public APIs. And yet, this is one of the most common issues when it comes to Kubernetes.
These k8s APIs give anyone with access the opportunity to look into different parts of the environment configuration.
All Good (and Potentially Vulnerable) Things Come in Threes
A Kubernetes API is the looking glass through which we can discover a plethora of information, some of which may be of interest to a hacker. What follows are three ways to get to that juicy meat.
1. /api/v1/secrets
Sensitive info, such as passwords and keys, is stored in the aptly named Secrets.
These are encoded with base64, which doesn’t really have anything to do with security but rather serves to avoid characters escaping in requests/yaml files.
$ echo ZXhhbXBsZQo= | base64 -d
If set up properly, they are encrypted with AES. But the encryption is being handled transparently by the API, so if you’re given the appropriate permissions in RBAC, you’re able to decrypt and read the secrets.
For example, you could find GIT Repository or database credentials, .dockerconfig files and AWS access keys stored here. You shouldn’t be able to, but I’ve seen it happen in my research.
2. /api/v1/configmaps
ConfigMaps is basically the content of various configuration files which are used in your application — instead of keeping them in your repository.
Within it, the “annotations” parts can give you access to email addresses and project names, while the “config file contents” could even harbor plaintext secrets!
3. /api/v1/pods
You can get access to plaintext secrets through “arguments” and entry point “command”.
Reading about the theory can be fun and good, but it doesn’t hold up to a solid, real-life example.
Showtime!
How I Easily Accessed an AWS Account of a Huge Multinational
I’ll walk you through my last research step by step — from the way I found a “victim”, uncovered their AWS credentials through /api/v1/secrets to finding the contact e-mail of the person responsible for the AWS account.
All actual references to the account owner are anonymized for their own protection.
1. Finding the Victim
In my first blog post on hacking into AWS accounts through Atlassian Apps, I mentioned one of the tools that helps me find exposed data: Binaryedge.io. It scans the entire public internet to uncover exposures to help you secure them.
So as a first step, I switch on the VPN and simply search for an unsecured machine.
My eyes fall on some AWS EC2s running in the US, so I go into that list and just pick one at random.
2. Uncovering Secrets
The most straightforward path can be the most effective, so that’s the one I start with. I simply go to:
/api/v1/secrets
Now I can see the entire configuration details of the Kubernetes — and this is just after a few clicks. There ain’t no magic tricks!
I can see some tokens, certificates… and suddenly I hit the jackpot!
The credentials are right there, just waiting to be base64-decoded. And lo and behold! I am now in possession of the aws_access_key_id and the aws_secret_access_key.
3. Striking Gold
And now the lucrative part begins. I open my terminal and configure a new AWS profile with the access keys.
Then I run my script to check the access level of the account by asking for IAM users, S3 buckets, last month’s costs, etc…
After a few seconds waiting time, I’m in — and can see all details pertaining to the AWS account, complete with number of users, associated costs, S3 buckets and the AWS Organization master account e-mail!
In the account summary, I can see that it’s got 11 Users, 57 policies and 91 roles attached to it, so it’s not huge but not small either. And there’s the total cost of this infrastructure for the last month.
4. Finding the owner
Now that I have all this information, I want to find the owner in order to inform them of the exposure.
I already have the AWS Organization master e-mail account (which has some revealing info in the domain name) but I want to get one more just to be on the safe side.
Using the same AWS profile, I can extract a root e-mail from a recent AWS support case — if there is any. The account owner e-mail domain was the same as the previously found one, so I know that I have the owner.
Free House
As you can see by the example above, it can be very easy to get into an AWS account through Kubernetes. But it should never be that easy.
The exposed account belonged to a large multinational company. After I contacted them with the info, they managed to fix everything within a few hours.
This could have been avoided from the start. Let’s see how in our conclusion.
The Usual Suspects
As is often the case with these types of vulnerabilities, they can be avoided by implementing sound security measures.
Here are three recommendations on how to secure your Kubernetes.
- Keep your k8s API secured
Don’t disable the default settings, which have RBAC and mTLS. If you need to do this for any reason, at least limit inbound connections to your office IPs only. - Use IAM Roles instead of access keys
There is no need to generate an access key for your application running on EKS. Use roles for your service accounts used by your pods. - Limit the privileges of your application
If your application needs to talk to S3, you can limit it to a single bucket. It really doesn’t need an admin-like policy to work! This also applies to k8s RBAC. It’s a common mistake to assign cluster-admin roles to users and service accounts.
Bonus tip: Another common problem is that rights to exec into containers and/or debug containers are too open. Combined with a missing seccomp/apparmor setup (which aren’t on by default) — this makes it possible to escape container and access host and other container resources.
Cyber security will always play a major role on the stage of business in the digital age. Don’t take the topic lightly and make mistakes that can be avoided.
If you are uncertain of how to configure your Kubernetes, ask your security experts — and listen to them. On that note, I’ll leave you with a great quote from American writer Joanna Ruocco:
“Without the opinion of an expert, there’s no such thing as certainty. “
Originally published at https://www.pgs-soft.com.
