How to incorporate S3, EC2, and IAM in a CloudFormation template

Here’s how you can use CloudFormation to have EC2, IAM, and S3 work together

This is the sixth article in our Infrastructure as Code blog series. You can read the first five here:

• Infrastructure as Code: 5 Reasons Why You Should Implement IaC Now
• Infrastructure as Code: 6 best practices to get the most out of IaC
• 15 Infrastructure as Code tools you can use to automate your deployments
• What is AWS CloudFormation and how can it help your IaC efforts?
• How AWS CloudFormation works (and how to create a virtual private cloud with it)

To be notified when future posts go live, click here.

In our last article, we dug deep into how AWS CloudFormation works and provided an analysis of a VPC template we created.

Our next template example is that of SFTP Gateway, a product that we sell on the AWS Marketplace that makes it easy to transfer files via SFTP to Amazon S3. We have over 1000 customers using the product, so it’s a useful tool!

We’ll incorporate S3, EC2, IAM, Security Groups, and more to facilitate this file transfer.

Here’s the template in its entirety:

AWSTemplateFormatVersion: 2010-09-09
Mappings:
  RegionMap:
    ap-northeast-1:
      AMI: ami-0d1a9e6b
    ap-northeast-2:
      AMI: ami-0ca50362
    ap-south-1:
      AMI: ami-989fd6f7
    ap-southeast-1:
      AMI: ami-db7a25b8
    ap-southeast-2:
      AMI: ami-8c14e0ee
    ca-central-1:
      AMI: ami-3d13a859
    eu-central-1:
      AMI: ami-fd6fe192
    eu-west-1:
      AMI: ami-e8922c91
    eu-west-2:
      AMI: ami-09223c6d
    eu-west-3:
      AMI: ami-9563d4e8
    sa-east-1:
      AMI: ami-eb4f0887
    us-east-1:
      AMI: ami-c599febf
    us-east-2:
      AMI: ami-6a1c350f
    us-west-1:
      AMI: ami-99b983f9
    us-west-2:
      AMI: ami-a7ca10df
Resources:
  SFTPGatewayInstance:
    Type: AWS::EC2::Instance
    Metadata:
      AWS::CloudFormation::Init:
        config:
          commands:
            setup:
              command: /usr/local/bin/sftpgatewaysetup
    Properties:
      IamInstanceProfile: !Ref RootInstanceProfile
      ImageId: !FindInMap
        - RegionMap
        - !Ref AWS::Region
        - AMI
      InstanceType: !Ref EC2Type
      BlockDeviceMappings:
        - DeviceName: /dev/xvda
          Ebs:
            VolumeSize: !Ref DiskVolumeSize
            VolumeType: gp2
      KeyName: !Ref KeyPair
      SecurityGroupIds:
        - !Ref SFTPGatewaySG
      SubnetId: !Ref SubnetID
      Tags:
        - Key: Name
        Value: SFTPGateway Instance
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash
          yum update -y aws-cfn-bootstrap
          /opt/aws/bin/cfn-init --stack ${AWS::StackName} --resource SFTPGatewayInstance \n --region ${AWS::Region}
  SFTPGatewayBucket:
    DeletionPolicy: Retain
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub sftpgateway-${SFTPGatewayInstance}
    DependsOn:
      - SFTPGatewayInstance
  RootInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles:
        - !Ref S3WritableRole
  S3WritableRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole
      Path: /
  RolePolicies:
    Type: AWS::IAM::Policy
    DependsOn:
      - SFTPGatewayInstance
    Properties:
      PolicyName: SFTPGatewayInstancePolicy
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: 's3:*'
            Resource: '*'
      Roles:
        - !Ref S3WritableRole
  SFTPGatewaySG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: SFTPGateway Security Group
      VpcId: !Ref VPCIdName
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 22
          ToPort: 22
          CidrIp: 0.0.0.0/0
  IPAddress:
    Properties:
      Domain: vpc
      InstanceId: !Ref SFTPGatewayInstance
    Type: AWS::EC2::EIP
Parameters:
  EC2Type:
    Description: SFTPGateway Instance Type
    Type: String
    Default: t2.micro
    AllowedValues:
      - t2.micro
      - t2.small
      - t2.medium
      - t2.large
      - m3.medium
      - m3.large
      - m3.xlarge
      - m4.large
      - m4.xlarge
      - c3.large
      - c3.xlarge
      - c4.large
      - c4.xlarge
      - r3.large
      - r3.xlarge
  KeyPair:
    Description: EC2 KeyPair
    Type: AWS::EC2::KeyPair::KeyName
    ConstraintDescription: Existing EC2 KeyPair.
  DiskVolumeSize:
    Default: 32
    Description: Disk volume size in GB. Must be at least 32.
    ConstraintDescription: Must be a number greater or equal to 32
    MinValue: 32
    Type: Number
  VPCIdName:
    Description: Select the VPC to launch the SFTPGateway into
    Type: AWS::EC2::VPC::Id
  SubnetID:
    Description: Subnet ID
    Type: AWS::EC2::Subnet::Id
Outputs:
  ElasticIP:
    Value: !Ref IPAddress
    Description: Elastic IP address

You can download the SFTP Gateway template here.

There’s a lot going on in the template, so we’ll just give a “brief” overview of what’s happening and point out some interesting syntax that you might use for your own projects.

Create an EC2 instance and S3 bucket

SFTP Gateway uses an EC2 server to upload files to S3. So we start off with an EC2 instance and S3 bucket:

Resources:
  SFTPGatewayInstance:
    Type: AWS::EC2::Instance
    Metadata:
      AWS::CloudFormation::Init:
        config:
          commands:
            setup:
              command: /usr/local/bin/sftpgatewaysetup
    Properties:
      IamInstanceProfile: !Ref RootInstanceProfile
      ImageId: !FindInMap
        - RegionMap
        - !Ref AWS::Region
        - AMI
      InstanceType: !Ref EC2Type
      BlockDeviceMappings:
        - DeviceName: /dev/xvda
          Ebs:
            VolumeSize: !Ref DiskVolumeSize
            VolumeType: gp2
      KeyName: !Ref KeyPair
      SecurityGroupIds:
        - !Ref SFTPGatewaySG
      SubnetId: !Ref SubnetID
      Tags:
        - Key: Name
          Value: SFTPGateway Instance
        UserData:
          Fn::Base64: !Sub |
          #!/bin/bash
          yum update -y aws-cfn-bootstrap
          /opt/aws/bin/cfn-init --stack ${AWS::StackName} --resource SFTPGatewayInstance --region ${AWS::Region}
  SFTPGatewayBucket:
    DeletionPolicy: Retain
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub sftpgateway-${SFTPGatewayInstance}
    DependsOn:
      - SFTPGatewayInstance

Here are a few things worth noting:

• The EC2 instance has a Metadata section in addition to its properties. We’ll cover these in more detail below.
• The S3 bucket has a Deletion Policy of “Retain”. This means you keep the S3 bucket if you delete the CloudFormation stack.
• The S3 BucketName uses an intrinsic function called “!Sub”, which lets you do string interpolation. The syntax “${SFTPGatewayInstance}” gives you the EC2 instance ID, just like the “!Ref” function.

This is what we have so far:

Let’s take a closer look at the EC2 instance metadata and properties:

SFTPGatewayInstance:
    Type: AWS::EC2::Instance
    Metadata:
      AWS::CloudFormation::Init:
        config:
          commands:
            setup:
              command: /usr/local/bin/sftpgatewaysetup
    Properties:
      IamInstanceProfile: !Ref RootInstanceProfile
      ImageId: !FindInMap
        - RegionMap
        - !Ref AWS::Region
        - AMI
      InstanceType: !Ref EC2Type
      BlockDeviceMappings:
        - DeviceName: /dev/xvda
        Ebs:
          VolumeSize: !Ref DiskVolumeSize
          VolumeType: gp2
      KeyName: !Ref KeyPair
      SecurityGroupIds:
        - !Ref SFTPGatewaySG
      SubnetId: !Ref SubnetID
      Tags:
        - Key: Name
          Value: SFTPGateway Instance
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash
          yum update -y aws-cfn-bootstrap
          /opt/aws/bin/cfn-init --stack ${AWS::StackName} --resource SFTPGatewayInstance --region ${AWS::Region}

There’s a lot going on here:

“CloudFormation::Init” — This is a powerful tool that lets you define config files and commands. In this case, we run a command called “sftpgatewaysetup” to initialize the software.

“ImageId” — This uses the “!FindInMap” intrinsic function that looks up an AMI ID based on the current region. The AMI mappings are located in the Mappings section of the CloudFormation template.

“InstanceType” — This refers to a parameter that we named “EC2Type” which gives you a drop-down list of common EC2 instance types.

“BlockDeviceMappings” — This sets the disk drive type to solid state (gp2). It also points to a parameter named “DiskVolumeSize” which allows the user to define disk size at stack creation.

“KeyName” — This refers to an SSH key that you use to log into the server.

“UserData” — This lets you run bash commands on server launch. In this case, we use “cfn-init” to read the “CloudFormation::Init” metadata we defined earlier.

A lot of the properties above reference parameters. Below is a snippet of the Parameters section of the template, which includes the “EC2Type”, “DiskVolumeSize”, and “KeyPair” parameters mentioned earlier:

.
.
.
Parameters:
  EC2Type:
    Description: SFTPGateway Instance Type
    Type: String
    Default: t2.micro
    AllowedValues:
      - t2.micro
      - t2.small
      - t2.medium
      - t2.large
      - m3.medium
      - m3.large
      - m3.xlarge
      - m4.large
      - m4.xlarge
      - c3.large
      - c3.xlarge
      - c4.large
      - c4.xlarge
      - r3.large
      - r3.xlarge
  KeyPair:
    Description: EC2 KeyPair
    Type: AWS::EC2::KeyPair::KeyName
    ConstraintDescription: Existing EC2 KeyPair.
  DiskVolumeSize:
    Default: 32
    Description: Disk volume size in GB. Must be at least 32.
    ConstraintDescription: Must be a number greater or equal to 32
    MinValue: 32
    Type: Number

Parameters let you pass dynamic values to make your template more flexible. Here are a few things to note:

• “AllowedValues” — This presents the user with a drop-down, so you don’t have to worry about form validation.
• “AWS::EC2::KeyPair::KeyName” — This is a special type that automatically presents the user with a list of key pairs in their AWS account.
• “MinValue” — This provides form validation that gives an error if the user puts in a value that is too small.

Create an IAM role

In order for our EC2 instance to access S3, we need to grant it permissions using an IAM role.

The architecture looks like this:

The diagram shown above simplifies what’s actually happening. If you look at the CloudFormation template, you’ll see that there’s more to it:

Resources:
.
.
.
  RootInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles:
        - !Ref S3WritableRole
  S3WritableRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole
      Path: /
  RolePolicies:
    Type: AWS::IAM::Policy
    DependsOn:
      - SFTPGatewayInstance
    Properties:
      PolicyName: SFTPGatewayInstancePolicy
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: 's3:*'
            Resource: '*'
      Roles:
        - !Ref S3WritableRole

There are three resources involved when assigning permissions to an EC2 instance:

• “InstanceProfile” — You can’t assign an IAM role directly to the EC2 instance, but you can assign an instance profile, which passes role information to the EC2 instance.
• “IAM::Role” — The EC2 instance can assume a role and inherit any permissions from the role, via the instance profile.
• “IAM::Policy” — This contains the actual permissions. The policy is associated with the role.

Using an existing public subnet

The EC2 instance needs to be in a public subnet so that end users can access it via SFTP. This CloudFormation template doesn’t create this public subnet. Rather, you select an existing subnet and pass it as a parameter to the template.

This happens here:.

.
.
.
Parameters:
.
.
.
  VPCIdName:
    Description: Select the VPC to launch the SFTPGateway into
    Type: AWS::EC2::VPC::Id
  SubnetID:
    Description: Subnet ID
    Type: AWS::EC2::Subnet::Id

Here’s what’s going on:

• “AWS::EC2::Subnet::Id” — This is a special parameter type that lists existing subnets in your AWS account. The EC2 instance is provisioned in this subnet.
• “AWS::EC2::VPC::Id” — This is another special parameter type and it lists existing VPCs. In the next section, we will define a security group that gets provisioned in this VPC.

The architecture looks like this:

Incorporate Security Group settings

In order for SFTP users to access the server, we use a Security Group to expose port 22 for specific IP addresses.

Here’s the relevant code:

Resources:
.
.
.
  SFTPGatewaySG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: SFTPGateway Security Group
      VpcId: !Ref VPCIdName
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 22
          ToPort: 22
          CidrIp: 0.0.0.0/0

The security group has a property called “SecurityGroupIngress”, which accepts an array of rules. Here we have a single rule that allows all traffic (0.0.0.0/0) on TCP port 22.

Adding an elastic IP

Finally, we need to create a static IP so that our public IP address doesn’t change each time the server shuts down.

Resources:
.
.
.
  IPAddress:
    Properties:
      Domain: vpc
      InstanceId: !Ref SFTPGatewayInstance
    Type: AWS::EC2::EIP

Here’s a brief explanation:

• “Domain” — This is set to “vpc”, since we’re using VPC instead of classic networking.
• “InstanceId” — This is the instance ID of the EC2 server that receives the IP address

We wind up with this final wonderful architecture:

Outputs section

The Outputs section lets you display concise information for easy access. Here, we show the public IP address to make it easier to connect for the first time.

Outputs:
  ElasticIP:
    Value: !Ref IPAddress
    Description: Elastic IP address

Conclusion

Now you can easily and securely upload your files to Amazon S3 via SFTP! Give the product a try by visiting the SFTP Gateway AWS Marketplace page.

You’ve also learned how to incorporate EC2, IAM, S3, Security Groups, and more to facilitate this file transfer.

We hope this has been helpful! If you liked this post, please share it with the share buttons to the left. Please let us know your thoughts in the comments.

Our next CloudFormation template analysis will show you how to create a Redshift stack. Thanks for reading!

Like this post? It likes you too. :)

If you enjoyed this post, please give it some claps so others can discover it more easily!

This article was originally published on ThornTech.com.