Creating VPC Endpoint for Amazon S3 using Terraform

This blog has moved from Medium to blogs.tensult.com. All the latest content is available there. Subscribe to our newsletter to stay updated.

Amazon Web Services has a history of coming up with new features and upgrades regularly by analyzing customer requirements and experience. VPC Endpoint for S3 was introduced by AWS sometime in the middle of 2015. It’s a helpful feature that lets you connect your VPC to supported AWS services and VPC endpoint services privately. A private connection from your VPC to your AWS services is a much more secure way compared to giving internet access to your instances or using a NAT device. You can read more about VPC endpoints and VPC endpoints for S3 here and here, respectively.

As the title of this blog suggests, I am writing this to share my experience of using Terraform to build an Amazon network consisting of a VPC which comprises 2 subnets(one private and public) and a VPC Endpoint for S3 services to be accessed from the private subnet. I chose to work with Terraform for this task because of the interactive CLI it has which helps us understand the working of the scripts as well. We at Tensult love to work with Terraform as an automation tool so feel free to check out our Github page here for other scripts for AWS which we have worked on.

A simple architecture showing the functioning of VPC Endpoint with S3.

Input provider values and Create a VPC

Talking about our topic here, let’s have a look at how we start configuring our Terraform script for VPC endpoint for S3. As a standard process, we have 2 variables each for region and profile. Later, we create a VPC named My_VPC using the resource command.

Create 2 subnets- a public and a private subnet.

Next, we create a private and a public subnet in our VPC (My_VPC). Kindly refer to the screenshot provided here for the code for that. Notice how we set the map_public_ip_on_launch value to “false” for only private instance because we do not want this instance to be accessible from the public internet. It’s always a good idea to give tags so that you can recognize your subnets on the AWS management console once they are initialized.

What do you think we need to create next? As a security best practice, it’s always better to create a security group as per the requirement. So let’s create 2 for each subnet.

Creation of Private security group

Creation of public security group

So, we are done with security groups for both subnets. Kindly note here that the inbound access for the private subnet is only from our public subnet. We can consider the latter to be for the office(s) or if you want to add access for remote users then hardcoding their IPs is the way to go. You can put more restrictions by following the Terraform commands given here. For testing purpose, we have intentionally left out some optional commands here which you can use for building a secure infrastructure.

Next, we create an internet gateway for the VPC so that we can communicate with our public subnet from the internet. Later, we create a route table for the VPC, create internet access by providing the destination IP address and associate the route table with the public subnet.

IGW, Route Table, Internet Access and Route table association

Now, where do we want to connect from the private subnet? An S3 bucket. So we create an S3 bucket with a little Terraform code as below :

S3 bucket creation

Below is the code for the creation of VPC Endpoint and associating it with VPC route table :

Make sure you give the correct service_name for endpoint as per the region you are working in. For this setup, we are in the Mumbai region, hence ap-south-1 is mentioned in the service_name.

Next, we create EC2 instances in each of the subnets that we have.

Code snippet for spinning up EC2 instances in respective subnets

If you are already learning Terraform, then you must know that we are done here. Yes, it’s that simple. Run terraform plan and then terraform apply command in the terminal to initialize the service in AWS. All you need to do now is to login to the public instance from your computer, then jump onto the private instance and after that access the S3 bucket service from there.

So what did we achieve here? We were able to access S3 service from our private instance, which was not earlier possible without using a public network. Here, thanks to VPC Endpoint, we didn’t have to leave the Amazon network for doing the same work. Pretty amazing, right?

I would like to add here that all the variables used in the above script were declared in a separate file in the same folder. It’s a neat way to declare variables and that also helps you get a clear picture of the entire script. We also have a similar Terraform script for SSM automation using VPC Endpoints. Feel free to check that out. That script helps you run SSM automation for your instances in a secure way without using the public internet and hence building a network which is more reliable and secure.

Thanks to Sandeep and Jawad for the inputs.