AWS EKS: Managed setup with CloudFormation

8 min readOct 26, 2019

Overview

In this post, we will review how to setup the AWS EKS service on your AWS account. This will be created in a single AWS region but you can reuse the material we go over here on a per region basis. We will be working with the CloudFormation service which will help us build and maintain the cluster itself as well as the networking infrastructure required to logically contain our resources. Finally, we will build the control plane within the same template and run it all as one single stack.

The EKS IAM Role

First and foremost, we are required to create a new IAM role that will allow the EKS service to assume role to your AWS account to help manage the associated resources.

Resources:
  EKSIAMRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - eks.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      RoleName: !Ref EKSIAMRoleName
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
        - arn:aws:iam::aws:policy/AmazonEKSServicePolicy

Briefly going over the above template extract, we are creating a new IAM “role” resource and allowing the EKS service to use the role. Additionally we have attached two (existing) managed IAM policies that are required for the EKS service to properly manage the upcoming cluster. A managed policy is a built-in policy already available within AWS.

The EKS network infrastructure

Next, we are required to build the EKS network infrastructure that the EKS cluster will reside in. This should be created next in-line and a fully configured template for this is already available on the official AWS documentation website. We will link it here for good measure.

https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2019-10-08/amazon-eks-vpc-private-subnets.yaml

The three things you are normally required to make a note of as a result of the network build are the following:

VPC ID
SecurityGroup
Subnet IDs

These values will be required when it comes time to generate the actual EKS cluster.

In our case though, we will be skipping even the need to collect those details as we will just be referencing all of this in the CloudFormation template. Just note that the template is built in such a way that the order of operations is to create the IAM role and the network infrastructure before finally creating the EKS cluster itself.

EKS Cluster: Control Plane

Creating the control plane piece of this is probably the easiest part. Why? Because it is just the culmination of the previous resources as we noted in the last section. Since we’re using a CloudFormation template though, we are going to let the template do all of the work for us in specifying the associated subnets, security group, and IAM role we want to use for the cluster creation.It’s annoying to have to write those values down somewhere for later as the official documentation suggests.

We will simply append the following to our template:

EKSCluster:
  Type: AWS::EKS::Cluster
  Properties:
    Name: !Ref EKSClusterName
    RoleArn:
      "Fn::GetAtt": ["EKSIAMRole", "Arn"]
    ResourcesVpcConfig:
      SecurityGroupIds:
      - !Ref ControlPlaneSecurityGroup
      SubnetIds:
      - !Ref PublicSubnet01
      - !Ref PublicSubnet02
      - !Ref PrivateSubnet01
      - !Ref PrivateSubnet02
  DependsOn: [EKSIAMRole, PublicSubnet01, PublicSubnet02, PrivateSubnet01, PrivateSubnet02, ControlPlaneSecurityGroup]

At this point, we now have a full and complete Cloudformation template to build the IAM Role, the network infrastructure, and the master node/control plane for our EKS cluster. Next up is the worker nodes.

Control Plane template (final):

Here is our final CloudFormation template to build the IAM Role, the network infrastructure, and the control plane for our new EKS cluster. Remember this is a combination of our IAM Role template, the EKS Cluster template, and the AWS provided network template.

---
AWSTemplateFormatVersion: '2010-09-09'Parameters:  EKSIAMRoleName:
    Type: String
    Description: The name of the IAM role for the EKS service to assume.  EKSClusterName:
    Type: String
    Description: The desired name of your AWS EKS Cluster.
    
  VpcBlock:
    Type: String
    Default: 192.168.0.0/16
    Description: The CIDR range for the VPC. This should be a valid private (RFC 1918) CIDR range.  PublicSubnet01Block:
    Type: String
    Default: 192.168.0.0/18
    Description: CidrBlock for public subnet 01 within the VPC  PublicSubnet02Block:
    Type: String
    Default: 192.168.64.0/18
    Description: CidrBlock for public subnet 02 within the VPC  PrivateSubnet01Block:
    Type: String
    Default: 192.168.128.0/18
    Description: CidrBlock for private subnet 01 within the VPC  PrivateSubnet02Block:
    Type: String
    Default: 192.168.192.0/18
    Description: CidrBlock for private subnet 02 within the VPCMetadata:
  AWS::CloudFormation::Interface:
  ParameterGroups:
    -
      Label:
        default: "Worker Network Configuration"
      Parameters:
        - VpcBlock
        - PublicSubnet01Block
        - PublicSubnet02Block
        - PrivateSubnet01Block
        - PrivateSubnet02BlockResources:
  EKSIAMRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
          Principal:
            Service:
              - eks.amazonaws.com
          Action:
            - 'sts:AssumeRole'
      RoleName: !Ref EKSIAMRoleName
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
        - arn:aws:iam::aws:policy/AmazonEKSServicePolicy  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock:  !Ref VpcBlock
      EnableDnsSupport: true
      EnableDnsHostnames: true
    Tags:
    - Key: Name
      Value: !Sub '${AWS::StackName}-VPC'  InternetGateway:
    Type: "AWS::EC2::InternetGateway"
    VPCGatewayAttachment:
    Type: "AWS::EC2::VPCGatewayAttachment"
    Properties:
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref VPC  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
    Tags:
    - Key: Name
      Value: Public Subnets
    - Key: Network
      Value: Public
  PrivateRouteTable01:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
    Tags:
    - Key: Name
      Value: Private Subnet AZ1
    - Key: Network
      Value: Private01
  PrivateRouteTable02:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
    Tags:
    - Key: Name
      Value: Private Subnet AZ2
    - Key: Network
      Value: Private02  PublicRoute:
    DependsOn: VPCGatewayAttachment
    Type: AWS::EC2::Route
      Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway  PrivateRoute01:
    DependsOn:
    - VPCGatewayAttachment
    - NatGateway01
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTable01
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NatGateway01  PrivateRoute02:
    DependsOn:
    - VPCGatewayAttachment
    - NatGateway02
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTable02
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NatGateway02  NatGateway01:
    DependsOn:
    - NatGatewayEIP1
    - PublicSubnet01
    - VPCGatewayAttachment
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt 'NatGatewayEIP1.AllocationId'
      SubnetId: !Ref PublicSubnet01
    Tags:
    - Key: Name
      Value: !Sub '${AWS::StackName}-NatGatewayAZ1'  NatGateway02:
    DependsOn:
    - NatGatewayEIP2
    - PublicSubnet02
    - VPCGatewayAttachment
    Type: AWS::EC2::NatGateway
    Properties:
    AllocationId: !GetAtt 'NatGatewayEIP2.AllocationId'
    SubnetId: !Ref PublicSubnet02
    Tags:
    - Key: Name
      Value: !Sub '${AWS::StackName}-NatGatewayAZ2'  NatGatewayEIP1:
    DependsOn:
    - VPCGatewayAttachment
    Type: 'AWS::EC2::EIP'
    Properties:
      Domain: vpc  NatGatewayEIP2:
    DependsOn:
    - VPCGatewayAttachment
    Type: 'AWS::EC2::EIP'
    Properties:
      Domain: vpc  PublicSubnet01:
    Type: AWS::EC2::Subnet
    Metadata:
      Comment: Subnet 01
    Properties:
      AvailabilityZone:
        Fn::Select:
        - '0'
        - Fn::GetAZs:
          Ref: AWS::Region
        CidrBlock:
          Ref: PublicSubnet01Block
        VpcId:
          Ref: VPC
    Tags:
    - Key: Name
      Value: !Sub "${AWS::StackName}-PublicSubnet01"  PublicSubnet02:
    Type: AWS::EC2::Subnet
    Metadata:
      Comment: Subnet 02
    Properties:
      AvailabilityZone:
        Fn::Select:
        - '1'
        - Fn::GetAZs:
          Ref: AWS::Region
        CidrBlock:
          Ref: PublicSubnet02Block
        VpcId:
          Ref: VPC
    Tags:
    - Key: Name
      Value: !Sub "${AWS::StackName}-PublicSubnet02"  PrivateSubnet01:
    Type: AWS::EC2::Subnet
    Metadata:
      Comment: Subnet 03
    Properties:
      AvailabilityZone:
        Fn::Select:
        - '0'
        - Fn::GetAZs:
          Ref: AWS::Region
        CidrBlock:
          Ref: PrivateSubnet01Block
        VpcId:
          Ref: VPC
    Tags:
    - Key: Name
      Value: !Sub "${AWS::StackName}-PrivateSubnet01"
    - Key: "kubernetes.io/role/internal-elb"
      Value: 1  PrivateSubnet02:
    Type: AWS::EC2::Subnet
    Metadata:
      Comment: Private Subnet 02
    Properties:
      AvailabilityZone:
        Fn::Select:
        - '1'
        - Fn::GetAZs:
          Ref: AWS::Region
        CidrBlock:
          Ref: PrivateSubnet02Block
        VpcId:
          Ref: VPC
    Tags:
    - Key: Name
      Value: !Sub "${AWS::StackName}-PrivateSubnet02"
    - Key: "kubernetes.io/role/internal-elb"
      Value: 1  PublicSubnet01RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet01
      RouteTableId: !Ref PublicRouteTable  PublicSubnet02RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet02
      RouteTableId: !Ref PublicRouteTable  PrivateSubnet01RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PrivateSubnet01
      RouteTableId: !Ref PrivateRouteTable01  PrivateSubnet02RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PrivateSubnet02
      RouteTableId: !Ref PrivateRouteTable02  ControlPlaneSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Cluster communication with worker nodes
      VpcId: !Ref VPC  EKSCluster:
    Type: AWS::EKS::Cluster
    Properties:
      Name: !Ref EKSClusterName
      RoleArn:
        "Fn::GetAtt": ["EKSIAMRole", "Arn"]
        ResourcesVpcConfig:
          SecurityGroupIds:
          - !Ref ControlPlaneSecurityGroup
          SubnetIds:
          - !Ref PublicSubnet01
          - !Ref PublicSubnet02
          - !Ref PrivateSubnet01
          - !Ref PrivateSubnet02
    DependsOn: [EKSIAMRole, PublicSubnet01, PublicSubnet02, PrivateSubnet01, PrivateSubnet02, ControlPlaneSecurityGroup]Outputs:
  SubnetIds:
    Description: Subnets IDs in the VPC
    Value: !Join [ ",", [ !Ref PublicSubnet01, !Ref PublicSubnet02, !Ref PrivateSubnet01, !Ref PrivateSubnet02 ] ]  SecurityGroups:
    Description: Security group for the cluster control plane communication with worker nodes
    Value: !Join [ ",", [ !Ref ControlPlaneSecurityGroup ] ]  VpcId:
    Description: The VPC Id
    Value: !Ref VPC

EKS Cluster: Worker nodes

Before we have a fully working cluster, we obviously need to create the actual worker nodes! This is required to be a separate template because the creation of these nodes has the pre-requisite of the cluster actually existing first! While technically we could leverage the “DependsOn” attribute that we’ve been using, you must remember that you may also wish to update the stack at a later date and you don’t want to mess with the foundations of your cluster as we’ve created above if you can avoid it.

The worker node template that AWS has built is pretty straight forward and I would recommend leveraging it as a separate stack. Here’s the link to that YAML file:

https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2019-10-08/amazon-eks-nodegroup.yaml

We will go over deploying this template as well after we’ve stood up our Control Plane stack.

Deploying your CloudFormation template(s)

A quick reminder on how to deploy your template within your AWS account. The following steps apply:

Login to your AWS account and select the appropriate region you will be working out of at the top right.
On the services search bar, type “cloudformation” and select the CloudFormation service.
Leave the default “template is ready” selected and then click “Upload a template file” to select your YAML file.

4. Fill in the required fields on the next page and leave the defaults for the ones that are populated if desired.

5. Proceed to create the stack and ensure that you specify and agree to the fact that it will create IAM resources on the account.

6. Once the stack has completed, you should see a “CREATE_COMPLETE” status for the newly deployed stack.

7. You can browse to the EKS service to see your newly deployed resources for the control plane.

Now it’s time to deploy that worker stack that we talked about earlier.

One step we need to do with the worker node template is create an EC2 key pair that we will need to specify on the template creation page. This key will allow you to be able to SSH into your worker nodes for troubleshooting purposes. AWS currently does not support creating a key pair with CloudFormation so it has to be done elsewhere. In this case, we will create a key pair through the AWS console. Follow the steps below to do so:

Browse to the EC2 service
Select “Key Pairs”

Select “Create Key Pair” and provide the key pair with a name.

At this point, you can follow the same steps as before but this time you can reference the S3 link in the stack creation page.

Step through the fields in the template, this time you will need to specify a name for the node group as well as select the associated VPC and subnets for the worker group.

Create the stack in the same manner as before and at the end, you will have two CloudFormation stacks. One for the control plane and one of the worker nodes.

Enjoy! Next time we will go over how to authenticate to your new cluster!

References:

https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html