My AWS Pentest Methodology
Why write this?
As I’ve been learning about AWS, I’ve found a need for comprehensive documents for offensive-focused practitioners. There are courses and offerings from pentesting firms, but fewer free resources are easy to find, in my experience. I wanted to write down how I typically approach pentesting AWS specifically and create a framework from which one could operate.
While I would love to write an in-depth technical manual that is far outside my time, resources, and skill level. The person reading this document should ideally be familiar with the basics of cloud computing and traditional security fundamentals. An excellent way to quickly build those would be to take some of the free training courses from AWS.
Whoami
My name is Lizzie, and I’m working as a pentester in the Seattle area. I used to do pentest project management at Rhino Security Labs in 2018, where I helped the late Spencer Gietzen run his “AWS Post-exploitation” projects. At the time, this was a particularly novel assessment offering. There was a lot of outstanding research from Rhino that is still highly relevant today. At the time of writing, a “cloud configuration review” is the most common offering from pentesting firms and, in some cases, is misrepresented as a comprehensive penetration test.
After I left Rhino, I spent more time as a project manager in the penetration testing space (at Leviathan Security Group). A significant responsibility of mine was to interface with a large cloud provider and ensure we delivered several services at scale. As a result, I’ve managed over 800 Pentests, proofread their reports, and sometimes presented findings to executives of major companies. Most of those projects were hosted in cloud environments, giving me a high-level understanding of techniques but less hands-on experience. Over the last year, I became a pentester, and I’ve played cloud CTFs, read a lot of research, and signed myself up for projects where I could get real-world experience beyond configuration reviews.
All of this to say — I am still learning in a field in its infancy. There will be limits to my knowledge, and I consider this a disclaimer that I might get something wrong. Please take the valuable parts and build off them for yourself. If you have feedback, I would absolutely love to hear from you.
What is the goal?
When we’re talking about AWS Pentests, a few approaches are typically used. One is a review of the external facing posture, which you can consider adjacent to a traditional External Network Penetration Test. Another is a review of the AWS environment’s static configurations, which can be performed with Read-only permissions. Exploitation is not involved; we call these account configuration reviews or a “cloud security audit,” as I’ve seen others refer to it.
The last and more in-depth penetration test is a dynamic test from the perspective of an attacker who obtained an initial foothold in an AWS account. Whether this is from keys accidentally published to GitHub, a disgruntled employee, or a server hosted on AWS was compromised, and credentials were obtained. This type of assessment aims to simulate an attacker trying to escalate privileges to get sensitive information from an account. That is the sole focus of this post.
I should also add that some people feel that AWS Cloud Configuration reviews provide more value than a pentest because of how large environments are and the associated cost for pentester time. This is a valid criticism and should be something you discuss with a client. As a professional, you should strive to bring clients the most value. A more mature client might have legitimate needs or compliance requirements a configuration review would not satisfy. I believe you should start with a configuration review for clients and, if a pentest makes sense, scope it reasonably. I want to use an analogy here: vulnerability assessment vs. a traditional pentest. A vulnerability assessment can provide a lot of value initially and efficiently for many companies. Still, while costlier, a pentest can identify deeper issues and satisfy compliance drivers for a company. Do not fall into the trap of thinking that what is valuable to some companies is valuable to all and vice versa. Each client’s needs and circumstances may vary, and it is important to tailor the assessment approach accordingly.
What should you ask clients for?
My project management background demands that I include this. If you’re considering performing an AWS penetration test, you’ve likely got clients who require it. You should always have legal permission and proof of ownership of the accounts you’re pentesting. If you’re at a pentesting firm, you likely have a Signature of Work, Professional Services Agreements, or other documents that specify the specifics. If you’re going the freelance route, I highly encourage you to seek an actual lawyer for advice to protect you legally. Also, read this and follow the AWS pentesting rules. Back in my day, AWS required clients to ask for permission to have a pentest, both ways, uphill and in the snow. Now if you follow those guidelines, you don’t need special permissions.
After you have those documents, clients will likely ask, “What do you need to get started?”
I recommend the following per account you assess:
- An access key pair with ReadOnly permissions & username/password for the console with:
- Permissions a typical developer might have, OR
- Permissions an application might have on EC2, Lambda, or another execution service. - An architecture diagram of the environment
We want the ReadOnly key for running tools like ScoutSuite or Prowler with access to everything in an environment. It can be a waste of time and effort not to know what exists inside an environment or to be able to inspect specific policies when you run into issues exploiting misconfigurations. We will use this key to establish familiarity with the target environment. If you are doing red team-style work, you don’t need or want this level of access unless your client is trying to find ways to reduce the cost of an engagement. For those who have done pentesting in traditional environments, you can consider this level of access as making the assessment a white box assessment.
For the other keys we ask for, we want to ask ourselves what is more likely, a developer’s key being exposed through some arbitrary method or application, or a compute resource being exploited so an attacker can obtain credentials for an execution role. These are by far the most common ways that keys are exposed.
There are other ways access might be acquired by attackers, such as unauthenticated Cognito Identity pools, but those depend on a specific configuration in a single service. However, I’ve found that clients rarely have a threat model that includes such scenarios. You can always ask your client if they have different ideas of a foothold position they want to be explored.
Mapping the account usage
Now that we can access an account, it’s time to take the foothold and see what is accessible. I recommend starting by enumerating information about the principal attached to your key.
- Create an AWS CLI profile associated with your key
- Perform a whoami to see if there’s a name associated with the principal the key is for
- Use a tool designed to enumerate information, such as Pacu, CloudFox, or enumerate-iam
a. You can do this manually for learning purposes. Still, AWS’s sheer amount of permissions means you’d be doing that for a while. - Create a list of services your key has permission to use. Individual permissions matter less at this point since we’re looking to create a mind map of an account.
- Use the ReadOnly key to describe the policies attached to your principal at each logical level, especially inline policies, which can often be overlooked.
a. Policies attached to User
b. User inline policies
c. Policies attached to Groups
d. Group inline policies
e. Policies attached to Roles
f. Role inline policies - Check if the account is a management or member account in an organization.
a. This is to see if a Service Control Policy (SCP) might be applied that you need to consider. If you see access denied errors referencing an SCP, this implies the account is in an Organizational Unit (OU). - Create a list of services with permissions granted and explicitly denied.
- Compare the list of services you enumerated dynamically against the ReadOnly results.
a. By creating these two lists and checking their differences, you can be reasonably assured that you are getting all the attack surfaces in the environment your principal can interact with.
Additionally, we can obtain a solid idea of the types of attacks we wish to perform by using contextual clues about the intended use of your principal. If you have multiple principals, you have keys for you can rinse and repeat this section, especially if you’ve moved laterally.
General configuration scanning
In the previous section, we enumerated valuable information about the principal you’re using. In this section, you’re going to leverage some automation.
Earlier, I mentioned that account configuration reviews are often sold and mislabeled as comprehensive pentest. Notice I did not say that they are without value. In fact, they can point out some low-hanging fruit, and you can consider them the Nessus of the cloud. Like how low-quality penetration testing vendors will “scan and rebrand” Nessus reports as a “pentest,” this trend also exists in the cloud with ScoutSuite and Prowler. If you get a low-price tag for an AWS “Pentest,” it’s probably too good to be true, and you should ask what their methodology will be to confirm it suits your needs.
To explain these tools if clients ask (they often do), I use the following copy-and-pastable explanation, “It’s a benchmark configuration tool for your AWS environment. It will check your configurations for different resources, principals, and settings and compare them to ‘best practices.’ They do this with Read permission to be unobtrusive, but it will generate a good deal of API calls. They can flag many false positives, so we manually review the report it generates and write up the valid issues.”
To run these tools, you can do the following:
- Clone the repositories for ScoutSuite and Prowler
- Create an AWS CLI profile for the ReadOnly key you have.
a. Their wikis list more specific permissions you can technically request, but I found ReadOnly to be sufficient. - Run said programs using the new profile
- Find the HTML versions of the report
a. These are easily consumable and easier for human readability - Find the JSON versions of the report
a. I recommend opening these in a program with good find-and-replace features. Personally, I use VS Code.
Another word of wisdom is that your note-taking will be challenged as some environments can have thousands and thousands of resources in use. I had seen a ScoutSuite report in a single AWS account consuming over 20 gigs of RAM when the HTML report was opened in Chrome. If you want to work with larger environments developing a note-taking system that scales is essential… and beyond the scope of this blog post.
I recommend using both ScoutSuite and Prowler. The reason for this is that when I dug into the actual default rulesets, and realized there was a difference in checks that were performed (there’s also some overlap). There are also a lot of AWS services that do not have checks at the time of writing, which is why it’s so important to rely on more than these tools.
In the previous section, we compiled a simple list of the services in use for a key. You should specifically denote which services are not covered by these tools, as you could easily miss sometimes critical severity misconfigurations.
There will likely be a lot to go over in these reports, so I’d recommend spending at least a day reviewing these tool reports in-depth and writing up valid issues. In larger environments, I’ve spent several days writing up valid issues. Telling you how to review these reports to remove false positives is a horrible approach as it ages poorly and doesn’t consider the context of the usage of these accounts. A finding in one account might be a false positive in another. I believe experience and critical thinking skills will serve you better than checklists — don’t be afraid to research and learn as you go.
Single Service Inspection
Now we’re past the low-hanging fruit and scanner-identified findings. You might have flagged some valid issues from the previous step, or you might be trying to review a service with no automated checks even written.
If I were to describe every service and the vulnerabilities you might encounter, I would never finish this blog post. Instead, I will describe the mindset you can apply to assess services in an isolated fashion. By this, I mean we look at a service, its configuration, publicly known issues to look out for, and how I find solid research.
For example, if we were to review the Lambda service, we would not want to look (yet) at Lambda’s interactions with an EC2 instance. Instead, we could review the checks in ScoutSuite & Prowler related to the Lambda service. We could then research ways people have found privilege escalation in Lambda or otherwise abused misconfigurations.
When I do that research, I typically use a few resources (aside from Google):
- https://hackingthe.cloud
a. Nick Frichette’s blog contains helpful and organized writing. - https://www.cloudvulndb.org/
a. A database of publicly known issues with Cloud Provider services, usually with a blog post associated with it - https://www.breaches.cloud/
a. Chris Farris compiled information related to security incidents and breaches with customers operating in the major cloud providers. Using the tags helped search for service-specific TTPs. - Honorable mention: coworkers and smart people in Cloud Security
From using these resources, suppose hypothetically I might have found that a privilege escalation can happen if a Lambda execution role is overly permissive and has an allow action for iam:PutUserPolicy in combination with your principal having permissions to modify & invoke that lambda function. I might even be so lucky that proof of concept code is available to try. I would review the IAM policy associated with that execution role to see if it is overly permissive and if I can even modify and invoke the lambda function. Since we did that enumeration earlier, we should be able to review our notes to check our hypothesis.
Another helpful thing is to refer to the API documentation for the service to see what actions are possible and familiarize yourself with the service's features.
It’s also worth mentioning in this section that every service is different, but they can often rhyme if you need help finding good information about a rarely used service. Look to other services in similar service categories and see if you can research those instead. Eventually, you might even identify something not publicly written about, but you saw something in a similar service. If that’s the case, report it to AWS beforehand and get permission to publish a blog post if you can.
Cross-service inspection
We’ve reviewed services in isolation, but now how do things go wrong when they interact with each other? To do this, we need to consider the account’s usage. In some scenarios, you might have access to design diagrams or developers to ask questions directly. Otherwise, you’ll need to reverse-engineer how the account is being used.
Again, there are too many permutations to go to a significant depth. However, it’s almost expected that a service uses other services and interacts with them. Consider a hypothetical situation in which an application is deployed on Elastic Beanstalk that relies on data stored in an Amazon S3 bucket and accessed by an AWS Lambda function. I would need to inspect the security configurations of all three services (Elastic Beanstalk, S3, and Lambda) to ensure they are appropriately configured individually.
Let’s assume that I know I have access to list & upload files to that S3 bucket and notice that it is hosting the Lambda’s code. I could attempt to target the runtime of the Lambda function by uploading a “new” and malicious version of that S3 object (the Lambda code). Assuming I could code the Lambda runtime to give myself temporary credentials from environment variables if invoked, I could obtain a new principal (the Lambda execution role) in the target account and can further attempt privilege escalation. This is a simple cross-service attack chain, but many attack chains are possible within any AWS environment.
Cross-account inspection
So far, we’ve had a simplified version of an AWS assessment. It’s standard for a company to have multiple AWS accounts. This is because AWS provides a range of services, and different services may have additional security and compliance requirements. Such as accounts for development, production, testing, or other business units or projects. Multiple accounts can also reduce the blast radius if there’s a potential security breach. To do this, usually (read: hopefully), a company will use the AWS Organizations service.
The AWS Organizations service allows customers to centrally manage and govern their multiple AWS accounts to which they can apply policies. For example, they might make a service control policy (SCP) that defines a policy that states an account cannot use a specific service.
For that reason, we need to identify all linked accounts. Again, I always recommend striking up a conversation with your nearest point of contact who is knowledgeable about the environment you’re in. Or… you can determine all the linked accounts associated with the target account by reviewing the AWS Organizations service with our previously requested ReadOnly principal on the AWS Console. When you identify or obtain access to a new account, I recommend following the previous steps for each account (if time permits).
Now that we know multiple accounts are in use and how many, we will also want to see what relationships exist between accounts and how they interact at a high level.
- Review all cross-account trust relationships: ensure they are correctly configured according to your context. This can be done by reviewing the IAM service and the trust policies, specifically, the account IDs mentioned in the ARNs.
- Check cross-account permissions by reviewing permissions in those policies granted to external principals.
- Review cross-account roles: ensure they are correctly configured for cross-account AssumeRole.
- Check cross-account networking: Review all cross-account networking configurations to ensure they are appropriately configured. Look at Security Groups, ACLs, and Service-specific configurations.
- Review cross-account security controls such as AWS Config, AWS CloudTrail, and AWS GuardDuty.
How can I learn more?
First and foremost, find people who are passionate about Cloud Security. Whether that’s going to conferences like fwd:CloudSec or re:invent (say hi to me!) or going onto Discords designed for learning cloud security (shout out to Rhino Security Labs’ discord). Being surrounded by and immersing yourself in an environment will always be the fastest way to learn.
As for specific resources, there are courses and certifications offered by AWS. Even though I currently do not have them, I have heard great things about them from people I respect.
There are also hands-on CTFs such as CloudGoat, http://flaws.cloud/, and https://pentesting.cloud/. These are my preferred method for hands-on experience.
I hope this blog post was helpful or, at the very least, interesting.
