Automating running GitLab on AWS with Terraform, Packer, and Ansible
This is Part 2 of the Comprehensive Guide to Running GitLab on AWS. In part 1 of this series, we discussed the high level architecture of running a highly available GitLab on AWS. In this post, we’re going to discuss how we can use modern tools like Terraform, Packer, and Ansible, to automate installing and running GitLab.
This guide is aimed at people who already have some knowledge of these tools as well as an understanding of networking and permissions in AWS, so we will not be covering the basics. In this guide, we will cover creating:
- RDS Postgres instance
- Elasticache Redis cluster
- NFS server backed by an EBS RAID array
- GitLab application servers in an AutoScaling group
Getting Started
Prerequisites
You must have an AWS account and Packer (1.2.0), Ansible (2.4.3), and Terraform (0.11.7) installed (this guide was tested with these versions, your mileage may vary with others).
In order for Terraform to deploy these resources in your AWS account, you will need to set the AWS credentials for a user with the proper permissions. My preferred method is to create a profile in the AWS credentials file at ~/.aws/credentials
.
// ~/.aws/credentials[gitlab-tutorial]
aws_access_key_id=(your access key id)
aws_secret_access_key=(your secret access key)
When interacting with AWS, you can now specify a profile of gitlab-tutorial
and these credentials will be used.
Finally, open your terminal and create a folder to house all of your files.
$ mkdir -p ~/gitlab-aws/{ansible,packer,terraform} && cd ~/gitlab-aws
VPC
To start, you’ll need a VPC in which to create all your resources. Alchemy has a public terraform module to create a VPC you can use, so create a terraform/main.tf
file and add the following.
// terraform/main.tfvariable "name" {
default = "gitlab-tutorial"
description = "Used to prefix all created resource names."
}variable "cidr" {
default = "10.30.0.0/16"
}variable "availability_zones" {
default = ["us-east-1a", "us-east-1b"]
}provider "aws" {
region = "us-east-1"
profile = "gitlab-tutorial"
}module "vpc" {
source = "git@github.com:alchemy-aws-modules/terraform-aws-vpc"
version = "0.1"
name = "${var.name}"
environment = "development"
cidr = "${var.cidr}"
availability_zones = ["${var.availability_zones}"]
private_subnets = ["10.30.0.0/19", "10.30.64.0/19"]
public_subnets = ["10.30.32.0/20", "10.30.96.0/20"]
use_nat_instances = true
}module "security_groups" {
source = "git@github.com:alchemy-aws-modules/terraform-aws-security-groups"
version = "0.1"
name = "${var.name}"
vpc_id = "${module.vpc.id}"
environment = "development"
cidr = "${module.vpc.cidr_block}"
}output "vpc_id" {
value = "${module.vpc.id}"
}
This will create the basic resources for a VPC across two availability zones: two private and public subnets, corresponding NAT instances, etc, with all the proper route tables set up for you. In addition, it will use the Alchemy security groups module to create some common security groups you’ll need.
You also need to create a bastion host so you can “jump” into your VPC and access other resources in private subnets. Alchemy has another public terraform module to create a bastion you can use, so add the following to main.tf
.
// terraform/main.tfvariable "key_name" {
description = "Name of the EC2 key pair to assign to instances."
}variable "ssh_private_key" {
description = "Local filepath to the ssh private key associated with var.key_name"
}module "bastion" {
source = "git@github.com:alchemy-aws-modules/terraform-aws-bastion"
version = "0.1"
name = "${var.name}"
environment = "development"
security_groups = "${module.security_groups.external_ssh}"
key_name = "${var.key_name}"
subnet_id = "${element(module.vpc.public_subnets, 0)}"
}output "bastion_ip" {
value = "${module.bastion.public_ip}"
}
You’ll see two new variables declared here: key_name
and ssh_private_key
. key_name
is the name of an EC2 key pair to assign to instances, and ssh_private_key
is the local file path to the associated private key so you can ssh into the instances.
If you do not already have a key pair in your AWS account, you can create one by following these instructions.
Because the variables do not have a default value, Terraform will prompt you to enter a value every time you run it. This gets very tedious, so you can create a file to hold the values to your variables. Create a terraform.tfvars
file in the terraform directory and add the following:
// terraform/terraform.tfvarskey_name = "your_key_name"ssh_private_key = "~/.ssh/my_key.pem"
terraform.tfvars
is a special file where Terraform looks for values to variables. You can specify them here so you won’t be prompted for them each time you apply terraform. You can also specify the value for variables on the command line, but I find this method easier to manage.
Now, go to the terraform directory in your terminal and run terraform init
. This will download the modules from GitHub and initialize a working directory for Terraform.
$ terraform init
Initializing modules...(...)Terraform has been successfully initialized!
Then run terraform apply
and enter “yes” and the prompt. After a few minutes, you should see a success message.
ATTENTION: In the real world, you should always inspect the output of terraform before applying.
$ terraform apply(...)Plan: 28 to add, 0 to change, 0 to destroy.Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.Enter a value: yes(...)Apply complete! Resources: 28 added, 0 changed, 0 destroyed.
Postgres and Redis
Postgres and Redis will be managed by the AWS services RDS and Elasticache, so you can create these resources easily. Add the following to your main.tf
file.
// terraform/main.tfvariable "gitlab_postgres_password" {
default = "supersecret"
}resource "aws_db_instance" "gitlab_postgres" {
allocated_storage = 50
storage_type = "gp2"
engine = "postgres"
engine_version = "9.6.6"
instance_class = "db.m4.large"
multi_az = true
db_subnet_group_name = "${module.vpc.default_db_subnet_group}"
name = "gitlabhq_production"
username = "gitlab"
password = "${var.gitlab_postgres_password}"
vpc_security_group_ids = ["${module.security_groups.internal_psql}"]
}resource "aws_elasticache_subnet_group" "gitlab_redis" {
name = "${var.name}-redis-subnet-group"
subnet_ids = ["${module.vpc.private_subnets}"]
}resource "aws_elasticache_replication_group" "gitlab_redis" {
replication_group_id = "${var.name}"
replication_group_description = "Redis cluster powering GitLab"
engine = "redis"
engine_version = "3.2.10"
node_type = "cache.m4.large"
number_cache_clusters = 2
port = 6379
availability_zones = ["${var.availability_zones}"]
automatic_failover_enabled = truesecurity_group_ids = ["${module.security_groups.internal_redis}"]
subnet_group_name = "${aws_elasticache_subnet_group.gitlab_redis.name}"
}output "gitlab_postgres_address" {
value = "${aws_db_instance.gitlab_postgres.address}"
}output "gitlab_redis_endpoint_address" {
value = "${aws_elasticache_replication_group.gitlab_redis.primary_endpoint_address}"
}
Run terraform apply and wait until the resources are created (sometimes RDS can take awhile to provision, so be patient).
$ terraform apply(...)Plan: 3 to add, 0 to change, 0 to destroy.Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.Enter a value: yes(...)Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
Network File System
Creating an NFS server won’t be as easy as Postgres and Redis, but you can use one of my favorite tools, Ansible, to make the process much easier. You will create a CentOS server and use Ansible to install and configure standard Linux packages to share files on the network.
Start by creating a CentOS server.
// terraform/main.tfresource "aws_security_group" "nfs" {
vpc_id = "${module.vpc.id}"
name_prefix = "${var.name}-gitlab-nfs-" ingress {
from_port = 2049
to_port = 2049
protocol = "tcp"
cidr_blocks = ["${var.cidr}"]
} ingress {
from_port = 111
to_port = 111
protocol = "tcp"
cidr_blocks = ["${var.cidr}"]
} lifecycle {
create_before_destroy = true
}
}data "aws_ami" "centos" {
owners = ["aws-marketplace"]
most_recent = true filter {
name = "product-code"
values = ["aw0evgkw8e5c1q413zgy5pjce"]
}
}resource "aws_instance" "nfs_server" {
ami = "${data.aws_ami.centos.id}"
instance_type = "t2.micro"
key_name = "${var.key_name}"
subnet_id = "${element(module.vpc.private_subnets, 0)}"
vpc_security_group_ids = ["${module.security_groups.internal_ssh}", "${aws_security_group.nfs.id}"]
}
This will use a data resource to look up the id of the most recent CentOS AMI and use that id to create an EC2 instance. It also creates a security group that opens up the standard ports for an NFS server and applies it to the instance.
If you recall from Part 1 of this series, you will remember that adding EBS volumes and creating a RAID array will greatly increase the performance of an NFS server, so let’s create some EBS volumes.
// terraform/main.tflocals {
device_names = ["/dev/xvdf", "/dev/xvdg", "/dev/xvdh"]
}resource "aws_ebs_volume" "gitlab_nfs" {
count = "${length(local.device_names)}"
availability_zone = "us-east-1a"
size = 128
}
Run terraform apply
and wait for the resources to be created.
$ terraform apply(...)Plan: 5 to add, 0 to change, 0 to destroy.Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.Enter a value: yes(...)Apply complete! Resources: 5 added, 0 changed, 0 destroyed.
At this point, you have a server and some EBS volumes. But, the EBS volumes are not actually attached to the instance yet. You may be thinking let’s add a few aws_volume_attachment resources, and I did at first as well. However, I encountered a lot of issues with that resource causing timeout issues and making Terraform fail when making changes to the attached instance. Many people are also experiencing this issue, and you can track its progress here. The root of the issue seems to be that the aws_volume_attachment doesn’t actually represent anything in EC2, it just runs the attach volume command when creating it, which is hard for Terraform to track and keep dependencies in order.
So how are we going to attach the volumes? I have had success with having the instance attach the volumes to itself when it starts, and we can use Ansible to easily accomplish this as well as run the rest of the tasks to bootstrap an NFS server.
Create an ansible-playbook file nfs-servers.yml
in the ansible directory.
$ touch ansible/nfs-servers.yml
And add the following:
// ansible/nfs-servers.yml- hosts: "{{ nfs_server_hosts }}"
become: true
vars:
ansible_ssh_user: centos
ansible_ssh_common_args: '-o ProxyCommand="ssh -W %h:%p -q {{ bastion_user }}@{{ bastion_host }}"'
ansible_ssh_extra_args: '-o StrictHostKeyChecking=no' pre_tasks: - easy_install:
name: pip
state: latest - pip:
name: boto - name: Attach EBS volumes
ec2_vol:
id: '{{ item.0 }}'
instance: '{{ instance_id }}'
state: present
device_name: '{{ item.1 }}'
region: '{{ region }}'
with_together:
- '{{ volumes }}'
- '{{ devices }}' - name: Configure NFS to use more concurrent processes
lineinfile:
path: /etc/sysconfig/nfs
line: "RPCNFSDCOUNT=16"
regexp: "^#RPCNFSDCOUNT=16" - name: Configure NFS to use more resources
copy:
content: |
sunrpc.tcp_slot_table_entries = 128
sunrpc.tcp_max_slot_table_entries = 128
net.core.rmem_default = 262144
net.core.rmem_max = 16777216
net.core.wmem_default = 262144
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 262144 16777216
net.ipv4.tcp_wmem = 4096 262144 16777216
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 0
net.ipv4.ip_local_port_range = 1024 65000
fs.inode-max = 128000
fs.file-max = 64000
dest: /etc/sysctl.d/30-nfs.conf - name: Apply NFS configuration from previous task
command: sysctl --system roles: - role: aloisbarreras.ebs-raid-array
raid_dev: /dev/md0
raid_level: 0
raid_name: gitlab-data
raid_mount: /gitlab-data
raid_devices: '{{ devices }}' - role: geerlingguy.nfs
nfs_exports:
- "/gitlab-data {{ cidr }}(rw,sync,no_root_squash)" post_tasks: - name: Create directories
file:
path: "{{ item }}"
state: directory
recurse: yes
with_items:
- /gitlab-data/git-data
- /gitlab-data/.ssh
- /gitlab-data/uploads
- /gitlab-data/shared
- /gitlab-data/builds
This playbook does several things.
- Installs
pip
andboto
. These packages are required to use the Ansibleec2_vol
module. We use theec2_vol
module to attach the EBS volumes created in Terraform. - Configures some settings that allows NFS to be able to use more resources. The settings are taken from CloudBees’ tips for running production NFS servers, which you can find here.
- Runs the aloisbarreras.ebs-raid-array role to create and mount a RAID array from attached block devices. In this case, it will create a RAID 0 array using the specified block devices and mount it onto the
/gitlab-data
folder. - Runs the geerlingguy.nfs role to install the nfs packages and export the specified folder
/gitlab-data
. - Creates some directories on the RAID array that the NFS clients will mount.
In order to use the above mentioned community Ansible roles, you first need to install them.
$ ansible-galaxy install aloisbarreras.ebs-raid-array,0.1.1 \
geerlingguy.nfs
Ansible is now almost ready to be run. Attaching EBS volumes to an instance requires IAM permissions, so the final step is to create an IAM role that allows the instance to access the required services.
// terraform/main.tfresource "aws_iam_role" "nfs_server" {
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}resource "aws_iam_role_policy" "nfs_server_ebs" {
role = "${aws_iam_role.nfs_server.id}" policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:*"
],
"Resource": "*"
}
]
}
EOF
}resource "aws_iam_instance_profile" "nfs_server" {
role = "${aws_iam_role.nfs_server.name}"
}
This creates an IAM role that allows EC2 instances to assume it and adds a policy that grants it all EC2 permissions. Finally, it creates an instance profile so our NFS server can assume the role. Now we need to add the instance profile to our existing NFS server. Note that you are only adding the lines in bold to the existing resource block.
// terraform/main.tfresource "aws_instance" "nfs_server" {
ami = "${data.aws_ami.centos.id}"
instance_type = "t2.micro"
key_name = "${var.key_name}"
subnet_id = "${element(module.vpc.private_subnets, 0)}"
vpc_security_group_ids = ["${module.security_groups.internal_ssh}", "${aws_security_group.nfs.id}"]
iam_instance_profile = "${aws_iam_instance_profile.nfs_server.id}"
}
Now that the instance will have the proper permissions, Ansible is ready to be run. We will orchestrate running Ansible from Terraform, so add the following to main.tf
:
// terraform/main.tfresource "null_resource" "nfs_server_bootstrap" {
triggers {
nfs_server_id = "${aws_instance.nfs_server.id}"
} provisioner "remote-exec" {
inline = [
<<EOF
while [ ! -f /var/lib/cloud/instance/boot-finished ]; do
echo -e "\033[1;36mWaiting for cloud-init..."
sleep 1
done
EOF
,
] connection {
user = "centos"
host = "${aws_instance.nfs_server.private_ip}"
private_key = "${file(pathexpand(var.ssh_private_key))}"
bastion_host = "${module.bastion.public_ip}"
bastion_user = "centos"
}
} provisioner "local-exec" {
command = <<EOF
ansible-playbook ../ansible/nfs-servers.yml -i "${aws_instance.nfs_server.private_ip}," \
-e nfs_server_hosts="${aws_instance.nfs_server.private_ip}" \
-e bastion_user=centos \
-e bastion_host=${module.bastion.public_ip} \
-e instance_id=${aws_instance.nfs_server.id} \
-e '{ "volumes": ${jsonencode(aws_ebs_volume.gitlab_nfs.*.id)} }' \
-e '{ "devices": ${jsonencode(local.device_names)} }' \
-e region=us-east-1 \
-e cidr=${module.vpc.cidr_block}
EOF
}
}
This is a lesser known Terraform resource known as the null_resource
. According to Terraform:
The null_resource is a resource that allows you to configure provisioners that are not directly associated with a single existing resource.
We’re using it here to run Ansible and configure the server after it is created. The remote-exec
provisioner waits for ssh to become available, then the local-exec
provisioner runs the local ansible-playbook
executable and runs the playbook you created earlier in ansible/nfs-server.yml
.
Run terraform init
, then terraform apply
and wait until the resources are created. After the server is created, you will be able to see the output from Ansible as Terraform will automatically run it using the null_resource
.
$ terraform apply(...)Plan: 4 to add, 1 to change, 0 to destroy.Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.Enter a value: yes(...)Apply complete! Resources: 4 added, 1 changed, 0 destroyed.Outputs:bastion_ip = 34.239.213.176
nfs_server_private_ip = 10.30.9.102
Note the two outputs for bastion_ip and nfs_server_private_ip.
SSH into your NFS server and make sure that the volumes were attached and formatted successfully. Replace the IP addresses in the following command with the address of your bastion host and NFS server private IP address.
$ ssh -A -t centos@bastion_ip ssh centos@nfs_server_private_ip
Your terminal should now be connected to a session running on your NFS server. Run df -h
to see what drives are mounted.
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 8.0G 887M 7.2G 11% /
devtmpfs 476M 0 476M 0% /dev
tmpfs 496M 0 496M 0% /dev/shm
tmpfs 496M 13M 483M 3% /run
tmpfs 496M 0 496M 0% /sys/fs/cgroup
/dev/md0 252G 61M 239G 1% /gitlab-data
tmpfs 100M 0 100M 0% /run/user/1000
You should see a drive mounted on /gitlab-data
of the appropriate size and block device you specified earlier. If you see output similar to above, your NFS server is up and running.
GitLab
The final piece of the architecture to create is the GitLab application servers. You will use packer to create an AMI that serves as the base image for the servers. Navigate to the packer
directory in your terminal and create a gitlab.json
file.
// packer/gitlab.json{
"variables": {
"name": null,
"profile": null,
"aws_region": "us-east-1"
},
"builders": [
{
"type": "amazon-ebs",
"region": "{{user `aws_region`}}",
"profile": "{{user `profile`}}",
"source_ami_filter": {
"filters": {
"product-code": "aw0evgkw8e5c1q413zgy5pjce"
},
"owners": [
"aws-marketplace"
],
"most_recent": true
},
"ami_name": "{{user `name`}}-{{timestamp}}",
"ami_description": "CentOS 7 with gitlab-ee installed",
"ena_support": "true",
"instance_type": "t2.large",
"ssh_username": "centos",
"tags": {
"OS Version": "CentOS 7"
}
}
],
"provisioners": [
{
"type": "ansible",
"playbook_file": "../ansible/gitlab.yml"
}
]
}
This instructs packer to use CentOS as the base image, and uses the Ansible provisioner to configure the machine. Create a gitlab.yml
file in your ansible
directory to hold the Ansible instructions on how to configure the GitLab servers.
// ansible/gitlab.yml- hosts: all
become: true
tasks: - name: Create mount point for gitlab-data
file:
path: /gitlab-data
state: directory
- name: yum update
yum:
name: '*'
state: latest - name: Install required packages
yum:
name: "{{ item }}"
update_cache: yes
with_items:
- curl
- policycoreutils-python
- openssh-server - name: Start and enable sshd
systemd:
name: sshd
enabled: true
state: started - name: Add gitlab repo
shell: curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.rpm.sh | bash - name: Install gitlab-ee
yum:
name: gitlab-ee
update_cache: yes
This Ansible playbook will install some GitLab dependencies and GitLab itself. Navigate to the packer
directory in your terminal and build the AMI. Note the output from Packer. You will need the AMI id.
$ cd ~/gitlab-aws && packer build \
-var name=gitlab-tutorial \
-var profile=gitlab-tutorial \
gitlab.json(...)==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs: AMIs were created:
us-east-1: ami-3a9c3f45
Once packer has built your AMI, you can use it to start an autoscaling group. This will allow GitLab to scale up and down as traffic changes as well as ensure there is always a minimum number of servers running. Add the following to main.tf
, entering your AMI id as the value for the gitlab_application_ami variable.
// terraform/main.tfvariable "gitlab_application_ami" {
default = "ami-3a9c3f45" // Your AMI here
}resource "aws_launch_configuration" "gitlab_application" {
name_prefix = "${var.name}-gitlab-application-"
image_id = "${var.gitlab_application_ami}"
instance_type = "t2.large"
security_groups = ["${aws_security_group.gitlab_application.id}", "${module.security_groups.internal_ssh}"]
key_name = "${var.key_name}" lifecycle {
create_before_destroy = true
}
}resource "aws_autoscaling_group" "gitlab_application" {
launch_configuration = "${aws_launch_configuration.gitlab_application.name}"
min_size = 1
max_size = 1
vpc_zone_identifier = ["${module.vpc.private_subnets}"] lifecycle {
create_before_destroy = true
}
}resource "aws_elb" "gitlab_application" {
subnets = ["${module.vpc.public_subnets}"]
security_groups = ["${module.security_groups.external_elb}"] listener {
instance_port = 80
instance_protocol = "http"
lb_port = 80
lb_protocol = "http"
} health_check {
healthy_threshold = 2
unhealthy_threshold = 5
timeout = 3
target = "HTTP:80/-/readiness"
interval = 30
} lifecycle {
create_before_destroy = true
}
}resource "aws_autoscaling_attachment" "asg_attachment_gitlab" {
autoscaling_group_name = "${aws_autoscaling_group.gitlab_application.id}"
elb = "${aws_elb.gitlab_application.id}"
}resource "aws_security_group" "gitlab_application" {
vpc_id = "${module.vpc.id}"
name_prefix = "${var.name}-gitlab-application-" ingress {
from_port = 80
to_port = 80
protocol = "tcp"
security_groups = ["${module.security_groups.external_elb}"]
} lifecycle {
create_before_destroy = true
}
}output "gitlab_dns_name" {
value = "${aws_elb.gitlab_application.dns_name}"
}
The above code:
- Creates an
aws_launch_configuration
using the AMI you created with Packer - Uses the launch configuration to create an autoscaling group
- Creates an Elastic Load Balancer (ELB) and attaches it to the autoscaling group
- Creates a security group that allows HTTP traffic from the ELB to the GitLab application instances
There is one final step we need to finish before starting the server. We need a way to mount the NFS server inside all of the GitLab applications as well as specify the configuration to use our Postgres and Redis instances. This is a great case to use AWS user data to run some commands on our instances when they launch.
Create a templates
folder inside the terraform
directory, and create a new file gitlab_application_user_data.tpl
.
// terraform/templates/gitlab_application_user_data.tpl#cloud-configmounts:
- [ "${nfs_server_private_ip}:/gitlab-data", /gitlab-data, nfs4, "defaults,soft,rsize=1048576,wsize=1048576,noatime,lookupcache=positive", "0", "2" ]
- [ /gitlab-data/git-data, /var/opt/gitlab/git-data, none, bind, "0", "0" ]
- [ /gitlab-data/.ssh, /var/opt/gitlab/.ssh, none, bind, "0", "0" ]
- [ /gitlab-data/uploads, /var/opt/gitlab/gitlab-rails/uploads, none, bind, "0", "0" ]
- [ /gitlab-data/shared, /var/opt/gitlab/gitlab-rails/shared, none, bind, "0", "0" ]
- [ /gitlab-data/builds, /var/opt/gitlab/gitlab-ci/builds, none, bind, "0", "0" ]write_files:
- content: |
# Prevent GitLab from starting if NFS data mounts are not available
high_availability['mountpoint'] = ['/var/opt/gitlab/git-data', '/var/opt/gitlab/.ssh', '/var/opt/gitlab/gitlab-rails/uploads', '/var/opt/gitlab/gitlab-rails/shared', '/var/opt/gitlab/gitlab-ci/builds']
# Disabe built-in postgres and redis
postgresql['enable'] = false
redis['enable'] = false
# External postgres settings
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_encoding'] = "unicode"
gitlab_rails['db_database'] = "${postgres_database}"
gitlab_rails['db_username'] = "${postgres_username}"
gitlab_rails['db_password'] = "${postgres_password}"
gitlab_rails['db_host'] = "${postgres_endpoint}"
gitlab_rails['db_port'] = 5432
gitlab_rails['auto_migrate'] = false
# External redis settings
gitlab_rails['redis_host'] = "${redis_endpoint}"
gitlab_rails['redis_port'] = 6379
# Whitelist VPC cidr for access to health checks
gitlab_rails['monitoring_whitelist'] = ['${cidr}']
path: /etc/gitlab/gitlab.rb
permissions: '0600'runcmd:
- [ gitlab-ctl, reconfigure ]output: {all: '| tee -a /var/log/alchemy-cloud-init-output.log'}
This is a Terraform template file that we will use to generate the user data for our GitLab application servers. You can see there are various ${variables} declared throughout the file, and we can use the Terraform template_file
data source to interpolate the actual values into this file. Add the following to main.tf
.
// terraform/main.tfdata "template_file" "gitlab_application_user_data" {
template = "${file("${path.module}/templates/gitlab_application_user_data.tpl")}" vars {
nfs_server_private_ip = "${aws_instance.nfs_server.private_ip}"
postgres_database = "${aws_db_instance.gitlab_postgres.name}"
postgres_username = "${aws_db_instance.gitlab_postgres.username}"
postgres_password = "${var.gitlab_postgres_password}"
postgres_endpoint = "${aws_db_instance.gitlab_postgres.address}"
redis_endpoint = "${aws_elasticache_replication_group.gitlab_redis.primary_endpoint_address}"
cidr = "${module.vpc.cidr_block}"
}
}
Now we need to take the rendered value of the template_file
and supply it as user data to our GitLab application instances. Edit the aws_launch_configuration
we created earlier and add the new line highlighted in bold.
// terraform/main.tfresource "aws_launch_configuration" "gitlab_application" {
name_prefix = "${var.name}-gitlab-application-"
image_id = "${var.gitlab_application_ami}"
instance_type = "t2.large"
security_groups = ["${aws_security_group.gitlab_application.id}", "${module.security_groups.internal_ssh}"]
key_name = "${var.key_name}"
user_data = "${data.template_file.gitlab_application_user_data.rendered}"lifecycle {
create_before_destroy = true
}
}
When the server first starts, this user data file instructs the server to mount some the NFS drive onto /gitlab-data
and create some bind mounts to map some GitLab directories to use folders on the shared drive. We use the bind mounts because it’s simpler to export one directory from the NFS server rather than managing exporting 5 directories.
Next the user data script copies a gitlab.rb configuration file that specifies the configuration to use our Postgres and Redis instances instead of the built-in ones.
Finally, the script runs gitlab-ctl reconfigure
which will apply the configuration to GitLab.
Run terraform apply
and wait until the resources are created.
$ terraform apply(...)Plan: 5 to add, 0 to change, 0 to destroy.Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.Enter a value: yes(...)Apply complete! Resources: 5 added, 0 changed, 0 destroyed.Outputs:(...)gitlab_dns_name = tf-lb-20180417223748890000000002-1613259304.us-east-1.elb.amazonaws.com(...)
Once Terraform finishes creating the resources, you can log in to your AWS console and watch the GitLab server become active. However, the server will not become active just yet.
This is because GitLab expects tables and schemas to be present that do not exist yet. Therefore, we need to seed the database so we can bootstrap Postgres. Log into your AWS console and find the the private IP address of your GitLab application server.
Use this IP address to ssh into your your GitLab application server.
$ ssh -A -t centos@bastion_ip ssh centos@gitlab_private_ip
Once logged into the box, run sudo gitlab-rake gitlab:setup
. Enter “yes” at the prompt to seed the database.
Once that finishes, you can look at your load balancer again and watch the instance come into service.
If you see that 1 of 1 instances are in service, then the server is ready to start receiving traffic. Take the gitlab_dns_name
output from Terraform and enter it into your address bar in your web browser.
Congratulations! You now have GitLab running on AWS. Enter a password for your root user then log in and start using GitLab.
Where to go from here?
This is a great start, but there are still many things we need to do before we are production ready. For example, we don’t even have SSL yet! In the next post in this series, we will cover adding security such as SSL, how to add more durability and reliability by automating backups, and how to recover in case an availability zone goes down.
Alchemy is always hiring great engineers! If you are excited about building great software with some of the world’s largest brands, email alois@alchemy.codes.