Automating running GitLab on AWS with Terraform, Packer, and Ansible

17 min readApr 18, 2018

This is Part 2 of the Comprehensive Guide to Running GitLab on AWS. In part 1 of this series, we discussed the high level architecture of running a highly available GitLab on AWS. In this post, we’re going to discuss how we can use modern tools like Terraform, Packer, and Ansible, to automate installing and running GitLab.

This guide is aimed at people who already have some knowledge of these tools as well as an understanding of networking and permissions in AWS, so we will not be covering the basics. In this guide, we will cover creating:

RDS Postgres instance
Elasticache Redis cluster
NFS server backed by an EBS RAID array
GitLab application servers in an AutoScaling group

Getting Started

Prerequisites

You must have an AWS account and Packer (1.2.0), Ansible (2.4.3), and Terraform (0.11.7) installed (this guide was tested with these versions, your mileage may vary with others).

In order for Terraform to deploy these resources in your AWS account, you will need to set the AWS credentials for a user with the proper permissions. My preferred method is to create a profile in the AWS credentials file at ~/.aws/credentials.

// ~/.aws/credentials[gitlab-tutorial]
aws_access_key_id=(your access key id)
aws_secret_access_key=(your secret access key)

When interacting with AWS, you can now specify a profile of gitlab-tutorial and these credentials will be used.

Finally, open your terminal and create a folder to house all of your files.

$ mkdir -p ~/gitlab-aws/{ansible,packer,terraform} && cd ~/gitlab-aws

VPC

To start, you’ll need a VPC in which to create all your resources. Alchemy has a public terraform module to create a VPC you can use, so create a terraform/main.tf file and add the following.

// terraform/main.tfvariable "name" {
  default     = "gitlab-tutorial"
  description = "Used to prefix all created resource names."
}variable "cidr" {
  default = "10.30.0.0/16"
}variable "availability_zones" {
  default = ["us-east-1a", "us-east-1b"]
}provider "aws" {
  region  = "us-east-1"
  profile = "gitlab-tutorial"
}module "vpc" {
  source             = "git@github.com:alchemy-aws-modules/terraform-aws-vpc"
  version            = "0.1"
  name               = "${var.name}"
  environment        = "development"
  cidr               = "${var.cidr}"
  availability_zones = ["${var.availability_zones}"]
  private_subnets    = ["10.30.0.0/19", "10.30.64.0/19"]
  public_subnets     = ["10.30.32.0/20", "10.30.96.0/20"]
  use_nat_instances  = true
}module "security_groups" {
  source      = "git@github.com:alchemy-aws-modules/terraform-aws-security-groups"
  version     = "0.1"
  name        = "${var.name}"
  vpc_id      = "${module.vpc.id}"
  environment = "development"
  cidr        = "${module.vpc.cidr_block}"
}output "vpc_id" {
  value = "${module.vpc.id}"
}

This will create the basic resources for a VPC across two availability zones: two private and public subnets, corresponding NAT instances, etc, with all the proper route tables set up for you. In addition, it will use the Alchemy security groups module to create some common security groups you’ll need.

You also need to create a bastion host so you can “jump” into your VPC and access other resources in private subnets. Alchemy has another public terraform module to create a bastion you can use, so add the following to main.tf.

// terraform/main.tfvariable "key_name" {
  description = "Name of the EC2 key pair to assign to instances."
}variable "ssh_private_key" {
  description = "Local filepath to the ssh private key associated with var.key_name"
}module "bastion" {
  source          = "git@github.com:alchemy-aws-modules/terraform-aws-bastion"
  version         = "0.1"
  name            = "${var.name}"
  environment     = "development"
  security_groups = "${module.security_groups.external_ssh}"
  key_name        = "${var.key_name}"
  subnet_id       = "${element(module.vpc.public_subnets, 0)}"
}output "bastion_ip" {
  value = "${module.bastion.public_ip}"
}

You’ll see two new variables declared here: key_name and ssh_private_key. key_name is the name of an EC2 key pair to assign to instances, and ssh_private_key is the local file path to the associated private key so you can ssh into the instances.

If you do not already have a key pair in your AWS account, you can create one by following these instructions.

Because the variables do not have a default value, Terraform will prompt you to enter a value every time you run it. This gets very tedious, so you can create a file to hold the values to your variables. Create a terraform.tfvars file in the terraform directory and add the following:

// terraform/terraform.tfvarskey_name = "your_key_name"ssh_private_key = "~/.ssh/my_key.pem"

terraform.tfvars is a special file where Terraform looks for values to variables. You can specify them here so you won’t be prompted for them each time you apply terraform. You can also specify the value for variables on the command line, but I find this method easier to manage.

Now, go to the terraform directory in your terminal and run terraform init. This will download the modules from GitHub and initialize a working directory for Terraform.

$ terraform init
Initializing modules...(...)Terraform has been successfully initialized!

Then run terraform apply and enter “yes” and the prompt. After a few minutes, you should see a success message.

ATTENTION: In the real world, you should always inspect the output of terraform before applying.

$ terraform apply(...)Plan: 28 to add, 0 to change, 0 to destroy.Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.Enter a value: yes(...)Apply complete! Resources: 28 added, 0 changed, 0 destroyed.

Postgres and Redis

Postgres and Redis will be managed by the AWS services RDS and Elasticache, so you can create these resources easily. Add the following to your main.tf file.

// terraform/main.tfvariable "gitlab_postgres_password" {
  default = "supersecret"
}resource "aws_db_instance" "gitlab_postgres" {
  allocated_storage      = 50
  storage_type           = "gp2"
  engine                 = "postgres"
  engine_version         = "9.6.6"
  instance_class         = "db.m4.large"
  multi_az               = true
  db_subnet_group_name   = "${module.vpc.default_db_subnet_group}"
  name                   = "gitlabhq_production"
  username               = "gitlab"
  password               = "${var.gitlab_postgres_password}"
  vpc_security_group_ids = ["${module.security_groups.internal_psql}"]
}resource "aws_elasticache_subnet_group" "gitlab_redis" {
  name       = "${var.name}-redis-subnet-group"
  subnet_ids = ["${module.vpc.private_subnets}"]
}resource "aws_elasticache_replication_group" "gitlab_redis" {
  replication_group_id          = "${var.name}"
  replication_group_description = "Redis cluster powering GitLab"
  engine                        = "redis"
  engine_version                = "3.2.10"
  node_type                     = "cache.m4.large"
  number_cache_clusters         = 2
  port                          = 6379
  availability_zones            = ["${var.availability_zones}"]
  automatic_failover_enabled    = truesecurity_group_ids = ["${module.security_groups.internal_redis}"]
  subnet_group_name  = "${aws_elasticache_subnet_group.gitlab_redis.name}"
}output "gitlab_postgres_address" {
  value = "${aws_db_instance.gitlab_postgres.address}"
}output "gitlab_redis_endpoint_address" {
  value = "${aws_elasticache_replication_group.gitlab_redis.primary_endpoint_address}"
}

Run terraform apply and wait until the resources are created (sometimes RDS can take awhile to provision, so be patient).

$ terraform apply(...)Plan: 3 to add, 0 to change, 0 to destroy.Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.Enter a value: yes(...)Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Network File System

Creating an NFS server won’t be as easy as Postgres and Redis, but you can use one of my favorite tools, Ansible, to make the process much easier. You will create a CentOS server and use Ansible to install and configure standard Linux packages to share files on the network.

Start by creating a CentOS server.

// terraform/main.tfresource "aws_security_group" "nfs" {
  vpc_id      = "${module.vpc.id}"
  name_prefix = "${var.name}-gitlab-nfs-"  ingress {
    from_port   = 2049
    to_port     = 2049
    protocol    = "tcp"
    cidr_blocks = ["${var.cidr}"]
  } ingress {
    from_port   = 111
    to_port     = 111
    protocol    = "tcp"
    cidr_blocks = ["${var.cidr}"]
  }  lifecycle {
    create_before_destroy = true
  }
}data "aws_ami" "centos" {
  owners      = ["aws-marketplace"]
  most_recent = true  filter {
    name   = "product-code"
    values = ["aw0evgkw8e5c1q413zgy5pjce"]
  }
}resource "aws_instance" "nfs_server" {
  ami                    = "${data.aws_ami.centos.id}"
  instance_type          = "t2.micro"
  key_name               = "${var.key_name}"
  subnet_id              = "${element(module.vpc.private_subnets, 0)}"
  vpc_security_group_ids = ["${module.security_groups.internal_ssh}", "${aws_security_group.nfs.id}"]
}

This will use a data resource to look up the id of the most recent CentOS AMI and use that id to create an EC2 instance. It also creates a security group that opens up the standard ports for an NFS server and applies it to the instance.

If you recall from Part 1 of this series, you will remember that adding EBS volumes and creating a RAID array will greatly increase the performance of an NFS server, so let’s create some EBS volumes.

// terraform/main.tflocals {
  device_names = ["/dev/xvdf", "/dev/xvdg", "/dev/xvdh"]
}resource "aws_ebs_volume" "gitlab_nfs" {
  count             = "${length(local.device_names)}"
  availability_zone = "us-east-1a"
  size              = 128
}

Run terraform apply and wait for the resources to be created.

$ terraform apply(...)Plan: 5 to add, 0 to change, 0 to destroy.Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.Enter a value: yes(...)Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

At this point, you have a server and some EBS volumes. But, the EBS volumes are not actually attached to the instance yet. You may be thinking let’s add a few aws_volume_attachment resources, and I did at first as well. However, I encountered a lot of issues with that resource causing timeout issues and making Terraform fail when making changes to the attached instance. Many people are also experiencing this issue, and you can track its progress here. The root of the issue seems to be that the aws_volume_attachment doesn’t actually represent anything in EC2, it just runs the attach volume command when creating it, which is hard for Terraform to track and keep dependencies in order.

So how are we going to attach the volumes? I have had success with having the instance attach the volumes to itself when it starts, and we can use Ansible to easily accomplish this as well as run the rest of the tasks to bootstrap an NFS server.

Create an ansible-playbook file nfs-servers.ymlin the ansible directory.

$ touch ansible/nfs-servers.yml

And add the following:

// ansible/nfs-servers.yml- hosts: "{{ nfs_server_hosts }}"
  become: true
  vars:
    ansible_ssh_user: centos
    ansible_ssh_common_args: '-o ProxyCommand="ssh -W %h:%p -q {{ bastion_user }}@{{ bastion_host }}"'
    ansible_ssh_extra_args: '-o StrictHostKeyChecking=no'  pre_tasks:    - easy_install:
        name: pip
        state: latest    - pip:
        name: boto    - name: Attach EBS volumes
      ec2_vol:
        id: '{{ item.0 }}'
        instance: '{{ instance_id }}'
        state: present
        device_name: '{{ item.1 }}'
        region: '{{ region }}'
      with_together:
        - '{{ volumes }}'
        - '{{ devices }}'    - name: Configure NFS to use more concurrent processes
      lineinfile:
        path: /etc/sysconfig/nfs
        line: "RPCNFSDCOUNT=16"
        regexp: "^#RPCNFSDCOUNT=16"    - name: Configure NFS to use more resources
      copy:
        content: |
          sunrpc.tcp_slot_table_entries = 128
          sunrpc.tcp_max_slot_table_entries = 128
          net.core.rmem_default = 262144
          net.core.rmem_max = 16777216
          net.core.wmem_default = 262144
          net.core.wmem_max = 16777216
          net.ipv4.tcp_rmem = 4096 262144 16777216
          net.ipv4.tcp_wmem = 4096 262144 16777216
          net.ipv4.tcp_window_scaling = 1
          net.ipv4.tcp_syncookies = 0
          net.ipv4.tcp_timestamps = 0
          net.ipv4.tcp_sack = 0
          net.ipv4.ip_local_port_range = 1024 65000
          fs.inode-max = 128000
          fs.file-max = 64000
        dest: /etc/sysctl.d/30-nfs.conf    - name: Apply NFS configuration from previous task
      command: sysctl --system  roles:    - role: aloisbarreras.ebs-raid-array
      raid_dev: /dev/md0
      raid_level: 0
      raid_name: gitlab-data
      raid_mount: /gitlab-data
      raid_devices: '{{ devices }}'    - role: geerlingguy.nfs
      nfs_exports:
        - "/gitlab-data {{ cidr }}(rw,sync,no_root_squash)"  post_tasks:    - name: Create directories
      file:
        path: "{{ item }}"
        state: directory
        recurse: yes
      with_items:
        - /gitlab-data/git-data
        - /gitlab-data/.ssh
        - /gitlab-data/uploads
        - /gitlab-data/shared
        - /gitlab-data/builds

This playbook does several things.

Installs pip and boto. These packages are required to use the Ansible ec2_vol module. We use the ec2_vol module to attach the EBS volumes created in Terraform.
Configures some settings that allows NFS to be able to use more resources. The settings are taken from CloudBees’ tips for running production NFS servers, which you can find here.
Runs the aloisbarreras.ebs-raid-array role to create and mount a RAID array from attached block devices. In this case, it will create a RAID 0 array using the specified block devices and mount it onto the /gitlab-data folder.
Runs the geerlingguy.nfs role to install the nfs packages and export the specified folder /gitlab-data.
Creates some directories on the RAID array that the NFS clients will mount.

In order to use the above mentioned community Ansible roles, you first need to install them.

$ ansible-galaxy install aloisbarreras.ebs-raid-array,0.1.1 \
geerlingguy.nfs

Ansible is now almost ready to be run. Attaching EBS volumes to an instance requires IAM permissions, so the final step is to create an IAM role that allows the instance to access the required services.

// terraform/main.tfresource "aws_iam_role" "nfs_server" {
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}resource "aws_iam_role_policy" "nfs_server_ebs" {
  role = "${aws_iam_role.nfs_server.id}"  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": "*"
    }
  ]
}
EOF
}resource "aws_iam_instance_profile" "nfs_server" {
  role = "${aws_iam_role.nfs_server.name}"
}

This creates an IAM role that allows EC2 instances to assume it and adds a policy that grants it all EC2 permissions. Finally, it creates an instance profile so our NFS server can assume the role. Now we need to add the instance profile to our existing NFS server. Note that you are only adding the lines in bold to the existing resource block.

// terraform/main.tfresource "aws_instance" "nfs_server" {
  ami                    = "${data.aws_ami.centos.id}"
  instance_type          = "t2.micro"
  key_name               = "${var.key_name}"
  subnet_id              = "${element(module.vpc.private_subnets, 0)}"
  vpc_security_group_ids = ["${module.security_groups.internal_ssh}", "${aws_security_group.nfs.id}"]
  iam_instance_profile = "${aws_iam_instance_profile.nfs_server.id}"
}

Now that the instance will have the proper permissions, Ansible is ready to be run. We will orchestrate running Ansible from Terraform, so add the following to main.tf:

// terraform/main.tfresource "null_resource" "nfs_server_bootstrap" {
  triggers {
    nfs_server_id = "${aws_instance.nfs_server.id}"
  }  provisioner "remote-exec" {
    inline = [
      <<EOF
        while [ ! -f /var/lib/cloud/instance/boot-finished ]; do
          echo -e "\033[1;36mWaiting for cloud-init..."
          sleep 1
        done
      EOF
      ,
    ]  connection {
      user         = "centos"
      host         = "${aws_instance.nfs_server.private_ip}"
      private_key  = "${file(pathexpand(var.ssh_private_key))}"
      bastion_host = "${module.bastion.public_ip}"
      bastion_user = "centos"
    }
  }  provisioner "local-exec" {
    command = <<EOF
      ansible-playbook ../ansible/nfs-servers.yml -i "${aws_instance.nfs_server.private_ip}," \
      -e nfs_server_hosts="${aws_instance.nfs_server.private_ip}" \
      -e bastion_user=centos \
      -e bastion_host=${module.bastion.public_ip} \
      -e instance_id=${aws_instance.nfs_server.id} \
      -e '{ "volumes": ${jsonencode(aws_ebs_volume.gitlab_nfs.*.id)} }' \
      -e '{ "devices": ${jsonencode(local.device_names)} }' \
      -e region=us-east-1 \
      -e cidr=${module.vpc.cidr_block}
    EOF
  }
}

This is a lesser known Terraform resource known as the null_resource. According to Terraform:

The null_resource is a resource that allows you to configure provisioners that are not directly associated with a single existing resource.

We’re using it here to run Ansible and configure the server after it is created. The remote-exec provisioner waits for ssh to become available, then the local-exec provisioner runs the local ansible-playbook executable and runs the playbook you created earlier in ansible/nfs-server.yml.

Run terraform init, then terraform apply and wait until the resources are created. After the server is created, you will be able to see the output from Ansible as Terraform will automatically run it using the null_resource.

$ terraform apply(...)Plan: 4 to add, 1 to change, 0 to destroy.Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.Enter a value: yes(...)Apply complete! Resources: 4 added, 1 changed, 0 destroyed.Outputs:bastion_ip = 34.239.213.176
nfs_server_private_ip = 10.30.9.102

Note the two outputs for bastion_ip and nfs_server_private_ip.

SSH into your NFS server and make sure that the volumes were attached and formatted successfully. Replace the IP addresses in the following command with the address of your bastion host and NFS server private IP address.

$ ssh -A -t centos@bastion_ip ssh centos@nfs_server_private_ip

Your terminal should now be connected to a session running on your NFS server. Run df -h to see what drives are mounted.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      8.0G  887M  7.2G  11% /
devtmpfs        476M     0  476M   0% /dev
tmpfs           496M     0  496M   0% /dev/shm
tmpfs           496M   13M  483M   3% /run
tmpfs           496M     0  496M   0% /sys/fs/cgroup
/dev/md0        252G   61M  239G   1% /gitlab-data
tmpfs           100M     0  100M   0% /run/user/1000

You should see a drive mounted on /gitlab-data of the appropriate size and block device you specified earlier. If you see output similar to above, your NFS server is up and running.

GitLab

The final piece of the architecture to create is the GitLab application servers. You will use packer to create an AMI that serves as the base image for the servers. Navigate to the packer directory in your terminal and create a gitlab.json file.

// packer/gitlab.json{
  "variables": {
    "name": null,
    "profile": null,
    "aws_region": "us-east-1"
  },
  "builders": [
    {
      "type": "amazon-ebs",
      "region": "{{user `aws_region`}}",
      "profile": "{{user `profile`}}",
      "source_ami_filter": {
        "filters": {
          "product-code": "aw0evgkw8e5c1q413zgy5pjce"
        },
        "owners": [
          "aws-marketplace"
        ],
        "most_recent": true
      },
      "ami_name": "{{user `name`}}-{{timestamp}}",
      "ami_description": "CentOS 7 with gitlab-ee installed",
      "ena_support": "true",
      "instance_type": "t2.large",
      "ssh_username": "centos",
      "tags": {
        "OS Version": "CentOS 7"
      }
    }
  ],
  "provisioners": [
    {
      "type": "ansible",
      "playbook_file": "../ansible/gitlab.yml"
    }
  ]
}

This instructs packer to use CentOS as the base image, and uses the Ansible provisioner to configure the machine. Create a gitlab.yml file in your ansible directory to hold the Ansible instructions on how to configure the GitLab servers.

// ansible/gitlab.yml- hosts: all
  become: true
  tasks:    - name: Create mount point for gitlab-data
      file:
        path: /gitlab-data
        state: directory
    
    - name: yum update
      yum:
        name: '*'
        state: latest    - name: Install required packages
      yum:
        name: "{{ item }}"
        update_cache: yes
      with_items:
        - curl
        - policycoreutils-python
        - openssh-server    - name: Start and enable sshd
      systemd:
        name: sshd
        enabled: true
        state: started    - name: Add gitlab repo
      shell: curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.rpm.sh | bash    - name: Install gitlab-ee
      yum:
        name: gitlab-ee
        update_cache: yes

This Ansible playbook will install some GitLab dependencies and GitLab itself. Navigate to the packer directory in your terminal and build the AMI. Note the output from Packer. You will need the AMI id.

$ cd ~/gitlab-aws && packer build \
  -var name=gitlab-tutorial \
  -var profile=gitlab-tutorial \
  gitlab.json(...)==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs: AMIs were created:
us-east-1: ami-3a9c3f45

Once packer has built your AMI, you can use it to start an autoscaling group. This will allow GitLab to scale up and down as traffic changes as well as ensure there is always a minimum number of servers running. Add the following to main.tf, entering your AMI id as the value for the gitlab_application_ami variable.

// terraform/main.tfvariable "gitlab_application_ami" {
  default = "ami-3a9c3f45" // Your AMI here
}resource "aws_launch_configuration" "gitlab_application" {
  name_prefix     = "${var.name}-gitlab-application-"
  image_id        = "${var.gitlab_application_ami}"
  instance_type   = "t2.large"
  security_groups = ["${aws_security_group.gitlab_application.id}", "${module.security_groups.internal_ssh}"]
  key_name        = "${var.key_name}"  lifecycle {
    create_before_destroy = true
  }
}resource "aws_autoscaling_group" "gitlab_application" {
  launch_configuration = "${aws_launch_configuration.gitlab_application.name}"
  min_size             = 1
  max_size             = 1
  vpc_zone_identifier  = ["${module.vpc.private_subnets}"]  lifecycle {
    create_before_destroy = true
  }
}resource "aws_elb" "gitlab_application" {
  subnets         = ["${module.vpc.public_subnets}"]
  security_groups = ["${module.security_groups.external_elb}"]  listener {
    instance_port     = 80
    instance_protocol = "http"
    lb_port           = 80
    lb_protocol       = "http"
  }  health_check {
    healthy_threshold   = 2
    unhealthy_threshold = 5
    timeout             = 3
    target              = "HTTP:80/-/readiness"
    interval            = 30
  }  lifecycle {
    create_before_destroy = true
  }
}resource "aws_autoscaling_attachment" "asg_attachment_gitlab" {
  autoscaling_group_name = "${aws_autoscaling_group.gitlab_application.id}"
  elb                    = "${aws_elb.gitlab_application.id}"
}resource "aws_security_group" "gitlab_application" {
  vpc_id      = "${module.vpc.id}"
  name_prefix = "${var.name}-gitlab-application-"  ingress {
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = ["${module.security_groups.external_elb}"]
  }  lifecycle {
    create_before_destroy = true
  }
}output "gitlab_dns_name" {
  value = "${aws_elb.gitlab_application.dns_name}"
}

The above code:

Creates an aws_launch_configuration using the AMI you created with Packer
Uses the launch configuration to create an autoscaling group
Creates an Elastic Load Balancer (ELB) and attaches it to the autoscaling group
Creates a security group that allows HTTP traffic from the ELB to the GitLab application instances

There is one final step we need to finish before starting the server. We need a way to mount the NFS server inside all of the GitLab applications as well as specify the configuration to use our Postgres and Redis instances. This is a great case to use AWS user data to run some commands on our instances when they launch.

Create a templates folder inside the terraform directory, and create a new file gitlab_application_user_data.tpl.

// terraform/templates/gitlab_application_user_data.tpl#cloud-configmounts:
  - [ "${nfs_server_private_ip}:/gitlab-data", /gitlab-data, nfs4, "defaults,soft,rsize=1048576,wsize=1048576,noatime,lookupcache=positive", "0", "2" ]
  - [ /gitlab-data/git-data, /var/opt/gitlab/git-data, none, bind, "0", "0" ]
  - [ /gitlab-data/.ssh, /var/opt/gitlab/.ssh, none, bind, "0", "0" ]
  - [ /gitlab-data/uploads, /var/opt/gitlab/gitlab-rails/uploads, none, bind, "0", "0" ]
  - [ /gitlab-data/shared, /var/opt/gitlab/gitlab-rails/shared, none, bind, "0", "0" ]
  - [ /gitlab-data/builds, /var/opt/gitlab/gitlab-ci/builds, none, bind, "0", "0" ]write_files:
  - content: |
      # Prevent GitLab from starting if NFS data mounts are not available
      high_availability['mountpoint'] = ['/var/opt/gitlab/git-data', '/var/opt/gitlab/.ssh', '/var/opt/gitlab/gitlab-rails/uploads', '/var/opt/gitlab/gitlab-rails/shared', '/var/opt/gitlab/gitlab-ci/builds']
      # Disabe built-in postgres and redis
      postgresql['enable'] = false
      redis['enable'] = false
      # External postgres settings
      gitlab_rails['db_adapter'] = "postgresql"
      gitlab_rails['db_encoding'] = "unicode"
      gitlab_rails['db_database'] = "${postgres_database}"
      gitlab_rails['db_username'] = "${postgres_username}"
      gitlab_rails['db_password'] = "${postgres_password}"
      gitlab_rails['db_host'] = "${postgres_endpoint}"
      gitlab_rails['db_port'] = 5432
      gitlab_rails['auto_migrate'] = false
      # External redis settings
      gitlab_rails['redis_host'] = "${redis_endpoint}"
      gitlab_rails['redis_port'] = 6379
      # Whitelist VPC cidr for access to health checks
      gitlab_rails['monitoring_whitelist'] = ['${cidr}']
    path: /etc/gitlab/gitlab.rb
    permissions: '0600'runcmd:
  - [ gitlab-ctl, reconfigure ]output: {all: '| tee -a /var/log/alchemy-cloud-init-output.log'}

This is a Terraform template file that we will use to generate the user data for our GitLab application servers. You can see there are various ${variables} declared throughout the file, and we can use the Terraform template_file data source to interpolate the actual values into this file. Add the following to main.tf.

// terraform/main.tfdata "template_file" "gitlab_application_user_data" {
  template = "${file("${path.module}/templates/gitlab_application_user_data.tpl")}"  vars {
    nfs_server_private_ip = "${aws_instance.nfs_server.private_ip}"
    postgres_database     = "${aws_db_instance.gitlab_postgres.name}"
    postgres_username     = "${aws_db_instance.gitlab_postgres.username}"
    postgres_password     = "${var.gitlab_postgres_password}"
    postgres_endpoint     = "${aws_db_instance.gitlab_postgres.address}"
    redis_endpoint        = "${aws_elasticache_replication_group.gitlab_redis.primary_endpoint_address}"
    cidr                  = "${module.vpc.cidr_block}"
  }
}

Now we need to take the rendered value of the template_file and supply it as user data to our GitLab application instances. Edit the aws_launch_configuration we created earlier and add the new line highlighted in bold.

// terraform/main.tfresource "aws_launch_configuration" "gitlab_application" {
  name_prefix     = "${var.name}-gitlab-application-"
  image_id        = "${var.gitlab_application_ami}"
  instance_type   = "t2.large"
  security_groups = ["${aws_security_group.gitlab_application.id}", "${module.security_groups.internal_ssh}"]
  key_name        = "${var.key_name}"
  user_data       = "${data.template_file.gitlab_application_user_data.rendered}"lifecycle {
    create_before_destroy = true
  }
}

When the server first starts, this user data file instructs the server to mount some the NFS drive onto /gitlab-data and create some bind mounts to map some GitLab directories to use folders on the shared drive. We use the bind mounts because it’s simpler to export one directory from the NFS server rather than managing exporting 5 directories.

Next the user data script copies a gitlab.rb configuration file that specifies the configuration to use our Postgres and Redis instances instead of the built-in ones.

Finally, the script runs gitlab-ctl reconfigure which will apply the configuration to GitLab.

Run terraform apply and wait until the resources are created.

$ terraform apply(...)Plan: 5 to add, 0 to change, 0 to destroy.Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.Enter a value: yes(...)Apply complete! Resources: 5 added, 0 changed, 0 destroyed.Outputs:(...)gitlab_dns_name = tf-lb-20180417223748890000000002-1613259304.us-east-1.elb.amazonaws.com(...)

Once Terraform finishes creating the resources, you can log in to your AWS console and watch the GitLab server become active. However, the server will not become active just yet.

This is because GitLab expects tables and schemas to be present that do not exist yet. Therefore, we need to seed the database so we can bootstrap Postgres. Log into your AWS console and find the the private IP address of your GitLab application server.

You can find the private IP in the bottom right

Use this IP address to ssh into your your GitLab application server.

$ ssh -A -t centos@bastion_ip ssh centos@gitlab_private_ip

Once logged into the box, run sudo gitlab-rake gitlab:setup. Enter “yes” at the prompt to seed the database.

Once that finishes, you can look at your load balancer again and watch the instance come into service.

ELB Status of GitLab application servers

If you see that 1 of 1 instances are in service, then the server is ready to start receiving traffic. Take the gitlab_dns_name output from Terraform and enter it into your address bar in your web browser.

Congratulations! You now have GitLab running on AWS. Enter a password for your root user then log in and start using GitLab.

Where to go from here?

This is a great start, but there are still many things we need to do before we are production ready. For example, we don’t even have SSL yet! In the next post in this series, we will cover adding security such as SSL, how to add more durability and reliability by automating backups, and how to recover in case an availability zone goes down.

Alchemy is always hiring great engineers! If you are excited about building great software with some of the world’s largest brands, email alois@alchemy.codes.