How to create a File Gateway | AWS Storage Gateway

The main objective of this blog is to

1. Create a Storage Gateway( using EC2 instance) and File System which can be connected to your local system
2. Automate the retrieval archived objects automatically using Lambda and Cloudwatch logs

Create a Storage Gateway( using EC2 instance) and File System which can be connected to your local system:

1. To create a storage gateway :

Select File Gateway

Select EC2 instance in host platform and click launch instance

This above step will open a new tab and redirect you to EC2 creation page which has selected a pre-defined AMI containing File Gateway requirements

In ‘ CHOOSE INSTANCE TYPE’ select AWS Recommended for Storage gateway i.e. “Compute Optimized” m4.xlarge instance type

In CONFIGURE INSTANCE keep everything as default.

In ADD STORAGE along with Root add other EBS with the recommended size of 150 GB for the smooth working of Storage gateway

In ADD TAGS add any tag if you want. Not compulsory

In ADD SECURITY GROUP open up the following ports by creating a new security group:

Security Group

NOTE: You should keep Source restricted to your IP address only so that it won’t be open to the world for access. This is the best security practice. I have kept here 0.0.0.0/0 just for the demonstration purpose but I highly recommend to follow best security practices

Next Review and Launch. You can review all the configuration you have selected for the gateway instance and then click Launch.

If you don’t have a Key Pair, Create a new Key Pair, Download it, and then launch the instance.

Once the instance is launched copy the public IP address of that instance. We will use this IP later.

Now again go back to the Storage Gateway console tab and click next.

Keep Endpoint type “Public”. You can configure FIPS or VPC if you want more secure transition of items. Here for demonstration purpose, we will select public.

Click Next, and paste the IP address of the instance that you just launched above. This means that you are configuring that instance as the storage gateway. Click Connect to Gateway.

Now you can activate the gateway by giving your timezone and a gateway name. Click Activate Gateway

On the next page, wait for a few seconds until it shows the EBS volume of our instance. Once it comes up you can check whether that EBS volume is selected to Cache. If not, select cache:

Click Save and continue. So we have successfully created our file gateway and linked with our account.

Next Create an S3 bucket for File Share

1. Create an S3 bucket from the S3 Console

NOTE: S3 bucket must be created in the same region as that of Storage Gateway.

2. Upload one file from your local system to the bucket and remember it should be in the Standard Storage class.

3. Now upload another object but this time keep the Storage class as Glacier or Glacier Deep Archive.

Create a File Share

1. Now in the Storage Gateway console, from left panel go to File Shares and click Create File Share
2. In Amazon S3 bucket name add the S3 bucket name which you created above. Leave the prefix blank. If you have created any folder in the bucket and then uploaded those files in those folder, then mention that folder name in the prefix. If it is in the root of the bucket you can leave it blank.
3. Select Access Objects using NFS. You can also choose SMB but NFS founds to be more efficient and faster than SMB. Though
4. In Gateway select the gateway which you created previously
5. Keep everything else as default and click next.

6. In Storage class for new objects choose S3 standard. Keep everything else as default. Click next.

7. Review everything and click on create File Share.

Now once the File Gateway status is available we can mount the file share to our local on-premise client.

Mount your NFS file share:

• For Linux clients, type the following command at the command prompt.
• sudo mount -t nfs -o nolock,hard [Your gateway VM IP address]:/[S3 bucket name] [mount path on your client]
• For MacOS clients, type the following command at the command prompt.
• sudo mount_nfs -o vers=3,nolock,rwsize=65536,hard -v [Your gateway VM IP address]:/[S3 bucket name] [mount path on your client]
• For Windows clients, type the following command at the command prompt.
• mount –o nolock -o mtype=hard [Your gateway VM IP address]:/[S3 bucket name] [Drive letter on your windows client]

For example, suppose that on a Windows client your VM’s IP address is 123.123.1.2 and your Amazon S3 bucket name is test-bucket. Suppose also that you want to map to drive T. In this case, your command looks like the following:

mount -o nolock -o mtype=hard 123.123.1.2:/gateway T:

Once done you can check on your client system that it will be showing the files which you have uploaded to S3.

So now we can say that we created a file share with our client system and Amazon S3 bucket.

Same content which we uploaded to S3 bucket

Now when you will try to access the files which were in Standard Storage class you will able to open those. So we can say that those objects that will be stored in Standard Storage class can be accessed quickly. Also if you add any files in the local client system drive which you mounted, it will be reflected in the S3 bucket( and that as a standard storage class). This way you don’t need to worry about storage capacity and will able to scale storage due to S3’s unlimited storage service.

But now when you try to access the object which are in Glacier or in Glacier Deep Archive, it won’t open as you can’t retrieve the object from a glacier storage class directly. So now if someone wants to access those files for audit purposes what can we do? If we keep all in standard it will incur more storage cost. So we have to keep those old files in Glacier only and when tried to retrieve, had to make it available using glacier retrieval ( Expedited, Standard and Bulk). Also, we don’t want to do manual retrieval, every time a file is asked. So we can automate this task.

Refer this blog to automate this task