fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Error trying to copy a file from an S3 bucket: Possible MFA bug

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: Bugs | AWS Security | Secure Code

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is a very unhelpful error message, isn’t it?

Fix: Align the error message with the actual fix the user needs to make.

There are two sides to S3 permissions. One is the permission to take S3 actions at all which is defined in the IAM Permissions for the user, a group the user is in, or a role the user has assumed. Navigate to IAM, click on policies on the left, and then create a policy that grants S3 permissions. Assign that to the user, group or role that can’t access the S3 bucket.

Creating IAM policies

Writing IAM Policies: How to Grant Access to an Amazon S3 Bucket | Amazon Web Services

The second side is permission via the S3 bucket policy. By default you should have access to a bucket via the bucket policy in your own account. However, if you want cross-account access you’ll need to add that permission to your bucket policy.

Provide cross-account access to objects in Amazon S3 objects

Troubleshoot issues copying an object between S3 buckets

In my case, I was trying to give a user access to any bucket in a particular OU.

Use IAM to share your AWS resources with groups of AWS accounts in AWS Organizations | Amazon Web…

It doesn’t seem to be working as of yet. When I follow the above instructions, AWS IAM says the policy grants no permissions. The problem was that I forgot the * in the policy below:

"ForAnyValue:StringLike": {
 "aws:PrincipalOrgPaths": "[org-id]/[root-id]/[ou-id]/*" 
}

Tip: Get the organization id, root id, and OU ID from the AWS console on the organizations page or query it using the AWS CLI.

Even after addressing that problem I still couldn’t access the bucket. I had some other possible issues but to resolve the problem I simply granted full read access to s3 in my IAM Policy. Then I was able to download the file. Of course that’s not what I want but this is what people do when error messages are not helpful.

Fix: In this case neither the S3 error message nor the IAM error message are very useful. It seems like AWS could address the fact that I need a /* here. The same is true for similar problems in S3 bucket policies where some commands require a /* at the end of the bucket name and other commands apply directly to the bucket. Additionally, some actions require pre-requisite actions. Help a user out and provide more information.

Then I navigated to my role and click on the IAM access analyzer tab. Here ai can see that AWS S3 was called. But which action?

I click on Amazon S3 and I only see one action even though I downloaded a file from another account:

So it appears here that the cross-account access is not covered by the IAM Access Analyzer.

FIX: This tab needs to show cross account access and permissions used in that case. There should be a GetObject operation here.

Since this role doesn’t exist in the other account I can’t use the AWS IAM access analyzer over there. I can check CloudTrail logs. However, in CloudTrail I can only see the AssumeRole action.

In order to see the S3 actions you have to turn on S3 Data Events — something that was critical in the Capital One Breach aftermath and a topic I cover in my cloud security classes. Of course, that will cost you extra money.

So to get this working I both disabled the organizational restriction and the MFA requirement. Try adding both of them one at a time and see which one causes failure, if they do.

Aha. Enforcing MFA on this request causes the bucket access to fail — even though I am authenticated with MFA.

Fix: This seems like a bug. This should work with assume role using MFA and MFA required in the IAM policy to call the S3 commands.

Let’s remove the condition that requires MFA and add the organizational unit. That fails as well. The thing is that the account making the request in the OU. The restriction to the OU might be on the caller being in the designated OU rather than allowing access to resources in the OU. Well, I’m not sure but it doesn’t work either.

Do any conditions work? Let’s try IP address.

I remove all conditions and once again, I can access my file.

Nope. That causes the same error. So it appears that with cross-account access you cannot put any condition in your IAM Policy that is used for cross-account access.

BUG
Or incomplete implementation. This should work.
When teams test features there should be a standard list of test cases that they go through to verify all possible paths work correctly. Something doesn't work here or at the very least the error message needs to be more specific.
Also any documentation related to conditions, OU resource access, etc. should clearly call out any limitations or known issues.
Code should address common misconfigurations such as a missing * and ask the user if they meant something different that might work (as long as it does not introduce security problems.)
NOTE: There is one potential issue and that is a lag in the time from which you implement an IAM policy and it actually takes effect. The AWS IAM console and CLI should really have a way to verify that the role is, in fact, in effect otherwise testing is too much of a guessing game due to eventual consistency issues.

OK moving on for the moment…will revisit this later to see if it gets fixed.

So I already mentioned above that my attempt at granting access to an entire OU might not work for various reasons. However, I tested this against a bucket in the same account and figured out that the MFA required to assume the role does not show up in the request made by the role after that point. That means you can’t enforce MFA in conditions with assume role if I understand correctly. I think it would be a good idea to change this behavior so you could use MFA conditions the way I am trying to do above.

mfaAuthenticated with AWS Assume Role

Here’s a related issue with PutObject for a local account S3 bucket.

upload failed: ./[xyz]

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2022

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab