Know Accessing S3 Bucket privately without IAM Creds/Roles and Use Cases

We all have used IAM credentials to access our S3 buckets. But it’s not a very safe or recommended practice to keep our Access keys and Secrets stored in a server or hard code them in our codebase.
Even if we have to use keys, we must have some mechanism in place to rotate the keys very frequently (eg: using Hashicorp Vault). Another widely adopted method is to use IAM roles attached on the EC2 instance or the AWS service accessing the bucket.

But, what if we need access to the bucket from an on-premise Data Center where we can not attach an IAM role?

Yes, we can obviously use IAM credentials and secret tokens with the rotating mechanism. But setting up the key rotation mechanism itself could be another overhead if we do not have one already in place. What if we do not require keys or roles without making the bucket public?

In this blog, I will make an attempt to cater to this problem with another alternate and easy solution.

Let’s first learn how we can access an S3 bucket without IAM credentials or IAM roles. Then we can move to the part of accessing it from the Data Center.

We need to make two small changes in order to achieve the same. In this example, I have created a bucket named — s3-access-test-techdemoswith all the default settings.
Now, we can see as I have no AWS credentials configured, hence I am not able to list or access the s3 bucket.

Step 1:

I will edit the S3 bucket policy and change it. The bucket policy below is to allow accessing the bucket from my ISP’s router public IP address. You can change it to your’s own. (Run curl ifconfig.me or visit https://whatismyipaddress.com/)
The bucket policy would look something like below:-

Code link is given here

With this bucket policy we are allowing access to publicly ( “Principal”: “*”) and hence do not require IAM credentials or roles. But at the same time restricting access from only IpAddress: 45.64.225.122

Step 2:

Go to CLI and update the command by appending --no-sign-request

aws s3 ls <your-bucket-name> --no-sign-request

As you can see the bucket has been listed. Similarly, we can upload or download files to S3.

When we run an AWS CLI command, in the backend a request URL is generated using the AWS credentials we provide, which determines whether we have access to the bucket or not. Usually, it is called the signed URL.
In our case, we haven’t configured any credentials.
Basically, -no-sign-request (boolean) means AWS CLI will not sign the requests when a request URL is generated. i.e credentials will not be loaded if this argument is provided. We are doing this because our bucket is always accessible from my public IP → 45.64.225.122.

This IP could be our Data Center end router, Ip. but that will not be a very recommended method to access buckets.

Now that we are able to access the bucket we can think of its use cases.

Since in this article, we are not interested in accessing our S3 bucket/s using IAM credentials/roles from an on-premise data center, hence we will use solutions like AWS Direct Connect to connect the AWS Cloud services from on-premise.

After AWS Direct Connect connections have been established, we can establish access to Amazon S3 in the following ways:

• Use a public IP address (Public VIF) over Direct Connect
• Use a private IP (Private VIF) address over Direct Connect (with an interface VPC endpoint)

See this doc here to know how to connect to S3 over Direct Connect.

(i) Using Public VIF we can access all Public AWS services. Eg: S3, EC2 using public Ip addresses.

See the document here to learn more.

Our bucket policy might look something like this -

{
    "Version": "2012-10-17",
    "Id": "Policy1415115909152",
    "Statement": [
        {
            "Sid": "Access-to-specific-PublicIP-only",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::s3-access-test-techdemos",
                "arn:aws:s3:::s3-access-test-techdemos/*"
            ],
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": "<local peer Ip address>"
                }
            }
        }
    ]
}

(ii) If we use Private VIF (which is basically used to access an Amazon VPC using private Ip addresses) we will have to use an AWS VPC Interface endpoint in between to access S3. It is because — we will use the Amazon private link for S3 to access S3 rather than the public prefix lists of S3 which we did in the earlier case.

Note: On-premises traffic can’t traverse the Gateway VPC endpoint.

Interface endpoints are actually one or more elastic network interfaces (ENIs) that are assigned private IP addresses from subnets in your VPC. Requests that are made to interface endpoints for Amazon S3 are automatically routed to Amazon S3 on the Amazon network. And we can access interface endpoints in our VPC from on-premises applications through AWS Direct Connect or AWS Virtual Private Network (AWS VPN).

See these articles to know more — Secure hybrid access to Amazon S3 using AWS PrivateLink and Interface VPC endpoints (AWS PrivateLink)

Since now we will access S3 through the private link, our bucket policy should allow the VPC Interface endpoint rather than any IP. It would look like below -

{
    "Version": "2012-10-17",
    "Id": "Policy1415115909152",
    "Statement": [
        {
            "Sid": "Access-to-specific-VPCE-only",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::s3-access-test-techdemos",
                "arn:aws:s3:::s3-access-test-techdemos/*"
            ],
            "Condition": {
                "StringEquals": {
                    "aws:sourceVpce": "vpce-0809dxxxxyyyy0045"
                }
            }
        }
    ]
}

Next,

When we use Interface endpoints for S3 access points we would need to modify our API requests slightly.

See Accessing Amazon S3 interface endpoints

So our final request Url from the On-premise server/application would look something like below -

aws s3 -no-sign-request — region ap-south-1 — endpoint-url https://bucket.vpce-1a2b3c4d-5e6f.s3.us-east-1.vpce.amazonaws.com ls s3://my-bucket/

(again --no-sign-request would be added since we would not be using Aws IAM Credentials)

2. Accessing bucket from a private subnet in a protected zone:-

We can also access S3 buckets securely and privately without traversing through the public internet from a private subnet in a protected zone (which has no access to the internet i.e no NAT gateway and can be only SSH from a jump server/bastion host).

( Note: since our instance would not have access to internet we must launch the instance from an AMI which has AWS CLI pre-installed)

Next, to access S3 from a private subnet in the protected zone we can use the Gateway Interface endpoint for S3. We could also use an Interface endpoint but Gateway endpoints are not-chargeable and the former is chargeable.

Follow doc hereIf we allow the gateway-vpce in our bucket policy and append --no-sign-request in our API request, then we can access the bucket privately even without attaching an IAM role or putting any IAM credentials.

Thanks for reading. Hope you found it useful!

Originally published at http://blog.opstree.com on March 15, 2022.