Amazon S3 File Share Storage Gateway with VPC endpoint and EC2 Instance
What is storage gateway?
Storage gateway is a hybrid cloud solution that helps customers overcome hybrid cloud storage challenges, and bridge the gap between their on-premises environments and the cloud. Storage Gateway enables on-premises applications to use cloud storage by providing low-latency data access over standard storage protocols. A local cache stores your most recently used data on premises, and the cloud provides scalability in addition to industry-leading data protection, durability, availability, security, and performance.
There are three different gateway types
File Gateway: The Storage Gateway acts as a file server that delivers applications SMB and NFS protocol access while storing all the data in Amazon S3 and keeping all the file system structure.
Volume Gateway: This gateway acts as an iSCSI target. There are two deployment options:
- Cached volumes keep a local copy of frequently accessed data
- Stored volumes are where the dataset resides locally in the gateway and frequent snapshots are taken and stored in S3 for possible recovery needs.
Tape Gateway: This type of gateway acts as a VTL that interacts via iSCSI with your tape backup software, keeping the data locally and then asynchronously moving the tape backups to Amazon S3, Amazon S3 Glacier, and S3 Deep Archive.
Below are the deployment methods of above 3 Gateway
- On-prem as a VM on ESXi
- On-prem as a Microsoft Hyper-V.
- On-prem as a hardware appliance.
- As a VM on VMWare Cloud in AWS.
- As an Amazon EC2 instance on AWS.
Architecture to follow
You may have privacy and security concerns with sending and receiving data across the public internet. In this case, you can use AWS PrivateLink, which provides private connectivity between Amazon VPC and other AWS services. We will create 2 VPC endpoint for this solution. (S3 and storage gateway)
Prerequisite
- AWS Accounts
- VPC and subnets
- S3 bucket
- AWS storage gateway VPC endpoint
- S3 VPC endpoint
Steps to follow
We are going to create a S3 file storage gateway and file share. The storage gateway will be deployed on EC2 instance. The storage gateway service will be accessible internally using storage gateway endpoints.
Step 1: Verify S3 bucket creation
Create a simple S3 bucket.
Step 2: Verify the S3 and storage gateway VPC endpoint
Create VPC endpoint under VPC section. search service name as storage gateway.
Follow to Create storage gateway VPC endpoint.
Please find below storage gateway VPC endpoint
Storage gateway VPC endpoint security group. Make sure to open below ports from your on-premise NFS/SMB file share range. (TCP 443, TCP 1026, TCP 1027, TCP 1028, TCP 1031, TCP 2222)
Similarly create S3 Interface vpc endpoint.
Step 3: Create Amazon S3 file gateway
Navigate to Storage gateway, Click on create Gateway.
Provide the name and choose Amazon S3 file Gateway as Gateway type.
Choose Amazon Ec2 as host platform and choose customize your settings under Ec2 instance creation.
Click on Launch Instance, It will open a new window. Provide the name of Ec2 instance and AMI will be auto populated (similar to this aws-storage-gateway-FILE_S3–1.24.4).
Choose m5.xlarge as instance type. select the desired VPC and subnet from the dropdown.
Choose the created security group for Ec2 instance. Make sure you have these ports open. For Inbound security group rules, add firewall rules to allow clients to connect to your instance. Amazon S3 File Gateway requires TCP port 80 to be open for inbound traffic and one-time HTTP access during gateway activation. After activation, you can close this port.
- If you plan to create NFS file shares, you must open TCP/UDP port 2049 for NFS access, TCP/UDP port 111 for NFSv3 access, and TCP/UDP port 20048 for NFSv3 access.
- If you plan to create SMB file shares, you must open TCP port 445 for SMB access.
For Configure storage, choose Add new volume to add storage to your gateway instance. You must add at least one Amazon EBS volume for cache storage with a size of at least 150 GiB, in addition to the Root volume. For increased performance, we recommend allocating additional EBS volumes for cache storage. Make sure to select “Add New Volume” as this is going to be needed for the gateway’s local cache and upload buffer.
Click on launch instance.
Come back to storage gateway console and click on below checkbox and proceed next.
Choose IP address in connection option and provide Public IP address of your EC2 instance.
Click on VPC hosted as endpoint options and choose the storage gateway VPC endpoint created earlier.
Review the changes and click on Activate Gateway
Now configure cache storage, choose the previously allocated 150Gib volume and create a log group.
click on configure. Now storage gateway has been created.
Step 4: Create a NFS file share
Navigate to file share under storage gateway. Click on create file share
Choose the storage gateway from the dropdown and choose the s3 bucket created earlier.
Then click on customize configuration. Provide a name, click on use VPC endpoint for s3 checkbox. Choose the Interface VPC endpoint from the dropdown.
keep rest all settings default and click on next. Provide a valid IP address or a valid CIDR block for allowed clients. you can restrict to only single IP as well.
Now click on create file share.
The file share is available and ready to share the files.
Step5: Mount your NFS file share on your client
Now you mount your NFS file share on a drive on your client and map it to your Amazon S3 bucket.
Windows: NFS, turn on Services for NFS in Windows.
For Windows clients, type the following command at the command prompt (cmd.exe).
mount –o nolock -o mtype=hard [GatewayVMIPAddress]:/[FileShareName] [WindowsDriveLetter]
Linux:
For Linux clients, type the following command at the command prompt.
sudo mount -t nfs -o nolock,hard [GatewayVMIPAddress]:/[FileShareName] [ClientMountPath]
MacOS:
For MacOS clients, type the following command at the command prompt.
sudo mount -t nfs -o vers=3.0,rsize=1048576,wsize=1048576,hard,nolock -v [GatewayVMIPAddress]:/[FileShareName] [ClientMountPath]
You can use Public or Private IP of your storage gateway to mount. Make sure the IP address is reachable from the on-premise NFS file share.
Step6: Test your S3 File Gateway
You can copy files and folders to your mapped drive. The files automatically upload to your Amazon S3 bucket.
To upload files from your Windows client to Amazon S3
- On your Windows client, navigate to the drive that you mounted your file share on. The name of your drive is preceded by the name of your S3 bucket.
- Copy files or a folder to the drive.
- On the Amazon S3 Management Console, navigate to your mapped bucket. You should see the files and folders that you copied in the Amazon S3 bucket that you specified.
You can see the file share that you created in the File shares tab in the AWS Storage Gateway Management Console.
Let's summarize what we have created so far.
- VPC and subnets to host Ec2 instance and endpoints
- VPC endpoints for S3 and Storage gateway
- S3 bucket to store files from on-premises
- Storage gateway that host on Ec2 instance.
- Multiple Security groups for communication. (VPC endpoint , Ec2)
- Created file share and allowed desired client.
- Mount the file share with storage gateway.
Conclusion:
Storage Gateway is a hybrid storage solution for on-premise and cloud. It helps to share simple file share or volume gateway or backup data with EBS snapshots or tape libraries. To set production-ready configuration we have to follow best practices like the use of VPC endpoints, correct configuration on security groups, least privilege IAM permissions, and correct client configuration.
