Assumed role on AWS falling back to underlying user while running commands

Couldn’t quite pin it down, but either way, be careful

I’ve been testing something on AWS with an assumed role and noticed that I’m getting inconsistent behavior. In my script, I have a user that has permissions to assume another role in another account.

Before I assume the role I verify the caller identity with this command:

aws sts get-caller-identity

That shows me the original user whose credentials are used to assume the external role.

After assuming the role, I call the role again and I get the assumed role as the caller identity.

Then things get weird and inconsistent.

Initially I had the user assuming the role set up with zero permissions except to assume the role it is supposed to use to run some tests. So when that user tried to describe ec2 anything, it would fail until assuming the role.

That was by design so I was sure that I was getting data from the expected account and would notice if I had made a mistake. The user also didn’t need those permissions to do what I was trying to do, in accordance with zero trust security policies.

Trust is overrated

Don’t be fooled

medium.com

Because things were working inconsistently, I added permissions to read information about EC2 instances and networking in the account I’m trying to run my tests from. I tested my scripts locally in that account to verify they work correctly. I transferred the commands to the script where the assume role command is used to access an external account to run the same queries.

Essentially, I’m getting a list of public IP addresses.

Now that my user has access to read ec2 data, here’s what happens.

I realized, after I gave the local user permissions to query IP addresses in the local account, is that the script is randomly failing back to the local user — AFTER sts get-caller-identity says I’ve assumed the role. So instead of getting failures in every region, I sometimes get the remote IP addresses and sometimes get the local addresses.

What could be causing this?

I assumed the role about a minute ago so the role session should not be timing out.

I thought maybe I had mistyped the MFA code but if that was the case, my attempt to assume the remote role would fail. Additionally, second STS command would show the current user, not the remote role after I assumed the role correctly. Maybe I just looked at it all wrong. (It happens.)

I’m looping through regions to get IP addresses so to pinpoint this problem further and verify I have not simply mistyped the MFA code somehow, I started running the STS call before every single command in every single region.

Since I’ve done that, I have not been able to reproduce the problem, so maybe I simply mistyped the MFA code because that’s really the only variable in this case.

Regardless of what caused this, it points to the need to be aware that this can happen — somehow — and to always validate the role is what you think it is before taking sensitive actions that could affect the wrong account. This also points to the need for zero trust roles throughout your IAM design.

Teri Radichel

If you liked this story please clap and follow:

Medium: Teri Radichel or Email List: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests services via LinkedIn: Teri Radichel or IANS Research