Spring Boot CI/CD on Kubernetes using Terraform, Ansible and GitHub: Part 4
Part 4: Configuring servers using Ansible
This is part of a series of articles that creates a project to implement automated provisioning of cloud infrastructure in order to deploy a Spring Boot application to a Kubernetes cluster using CI/CD. In this part we use Ansible to configure the Virtual Private Servers (VPSs) we created previously using Terraform.
Follow from the start — Introduction
Having created our three VPS nodes in the previous article, we will now set up a base level configuration using Ansible.
You can find the code for this part here: https://github.com/MartinHodges/Quick-Queue-IaC/tree/part4
Ansible
Like Terraform, Ansible works from a set of declarative configuration files that tell it the desired state you want for each of your servers (target hosts). We will create these files in the ansible
folder you created earlier.
These declarative configuration files form what is known as a playbook. A playbook can have one or more plays within them. It executes these plays against each of the target hosts it is told to configure.
Each play can include one or more tasks. Ansible uses a set of plugin modules with an extensive and useful set built in to the tool itself. Each module performs a task on the target host and can, generally, perform the action on a number of different Operating Systems (OS).
Some plays are specific to an OS and there are ways you can tell Ansible to carryout different play for different OSs. In this way, you can configure a set of different hosts using the same playbook. All the nodes we have created are Debian nodes.
Defining Your Target Hosts
The target hosts are defined in the inventory
file. This is created by Terraform and looks like this:
[k8s_master]
103.4.235.452
[k8s_node]
112.213.316.55
110.532.114.134
Fake IP addresses are shown here but you can use domain names if you have set up a suitable DNS accessible from the VPC.
The names between [ ] allows you to group target hosts, allowing you to refer to a number of hosts by their group name. You can use groups to identify your hosts as particular types (eg: db
, web
, app
etc). A host can appear in more than one group.
When applying a playbook against a set of target hosts, Ansible has a number of ways of directing all or part of the playbook to one or more of the hosts.
For instance, you can tell Ansible to only configure the k8s_master
hosts by using --limit=k8s_master
on the command line. This will tell Ansible only to run the playbook against your one master node. If you used --limit=k8s_node
it will configure both of the worker nodes (which are configured identically).
Alternatively we will see how we can assign a play to one or more groups.
Configuring Servers using Ansible
Whilst a single playbook can configure all your servers, it is sometimes useful to configure your servers in two steps:
- Bootstrapping the Servers with identical configuration
- Configuring the Servers for your specific application
In this article, we will bootstrap our servers to give them a common configuration on which we can build our Kubernetes cluster.
Remember, you can run your playbook against one or more of your target hosts. I recommend limiting your play to the single k8s_master
host until you get it working and then apply it to all in a group or all in the inventory.
Bootstrapping the Servers
All hosts need an initial configuration, such as updating the Operating Systems, adding a user etc. Rather than adding these plays to every playbook you create, it is easier to have it as a standalone playbook containing your standard set of plays.
This application of an initial playbook is known as bootstrapping the servers and it can be done using Ansible once you have added the servers to the known_hosts
file (see the previous article).
In your ansible
folder, add a subfolder called bootstrap
. This is where we will create the configuration files for Ansible.
The first thing to do is to provide a configuration for Ansible itself.
Create bootstrap/ansible.cfg
:
[defaults]
inventory = ../inventory
private_key_file = ~/.ssh/qq_rsa
remote_user = root
This tells Ansible to use the inventory file created by Terraform to define the target hosts. It also says which ssh key and user to use to connect to the targets. At the bootstrap stage, root is used as there are no other users on the servers.
A playbook is now needed to carry out the bootstrap. Ansible playbook files are written in the yaml format, so be careful of indentation errors.
Create bootstrap/bootstrap.yml
:
---
- hosts: all
become: yes
pre_tasks:
- name: install updates (Debian)
tags: always
apt:
upgrade: dist
update_cache: yes
when: ansible_distribution == "Debian"
- name: install sudoers for Debian servers
tags: debian
apt:
name:
- sudo
state: latest
when: ansible_distribution == "Debian"
- hosts: all
become: true
tasks:
- name: create kates user
tags: always
user:
name: kates
groups: root
shell: "/bin/bash"
- name: Set authorized key taken from file
authorized_key:
user: kates
state: present
key: "{{ lookup('file', '~/.ssh/qq_rsa.pub') }}"
- name: add sudoers file for kates
tags: always
copy:
src: sudoer_kates
dest: /etc/sudoers.d/kates
owner: root
group: root
mode: 0440
- name: generate sshd_config file from template
tags: ssh
vars:
ssh_users: kates
template:
src: "sshd_config.j2"
dest: /etc/ssh/sshd_config
owner: root
group: root
mode: 0644
notify: restart_sshd
handlers:
- name: restart_sshd
service:
name: sshd
state: restarted
We will break this down into sections.
---
- hosts: all
become: yes
pre_tasks:
This tells Ansible to apply this section (ie: this play) to all hosts except where you might limit it on the command line as mentioned earlier (note that ---
is a YAML standard that allows you to define multiple files in one file, if desired).
The become: yes
tells Ansible to execute the play as root. The command allows you to define other users but the yes
indicates root and currently we have no other users on the server.
Finally, whilst these configuration files are declarative, you may need to define dependencies. The pre_tasks
option tells Ansible that these configurations must be set before others.
- name: install updates (Debian)
tags: always
apt:
upgrade: dist
update_cache: yes
when: ansible_distribution == "Debian"
In this section, we see a task within our play. As the play is within the pre_tasks
option, its tasks (including this one) will get carried out before anything else.
First the task is given a name (install updates (Debian)
), which will be displayed on the console when the play is attempted. This is followed by a tag
. This allows you to define which type of tasks you want Ansible to run. In this case we say always
, which means always apply this task, regardless of any command line preference.
The next line, apt
, is an Ansible module that will execute the task. Modules come with their own documentation, which you can find here:
https://docs.ansible.com/ansible/latest/collections/ansible/builtin/apt_module.html
A module performs an action on the host to create a particular configuration. They each come with their own, specific set of parameters and arguments. In this case we are asking it to update its cache of packages (update_cache: yes
) and to upgrade the OS (upgrade: dist
).
Finally, the when
tells Ansible when it should run this tak. Although all our servers have a Debian OS, I have added the when
clause to show how it works. In this case Ansible will only run this task for servers with a Debian OS.
Before running a playbook against a host, Ansible extracts all the information it needs about the host first. The when
clause can target any of these fields.
- name: install sudoers for Debian servers
tags: debian
apt:
name:
- sudo
state: latest
when: ansible_distribution == "Debian"
This next task, still within the pre_tasks
play, requests for the latest verison of the sudoers
package to be loaded.
This is where the power of Ansible can be seen. As a declarative file, Ansible will only carryout this task if the latest version of sudoers has not been installed.
You need to remember this. Whilst it is tempting to think of tasks as an imperative set of instructions, they actually define what you want and not how to get there.
- hosts: all
become: true
tasks:
- name: create kates user
tags: always
user:
name: kates
groups: root
shell: "/bin/bash"
In this section we define a new play. Again it is applied to all hosts and is run as root.
The task that is shown is adding the user kates
to the OS and adds this user to the root
group. It also sets the login shell to be /bin/bash
.
Again, Ansible will only create this user if they do not already exist. This means you can run this playbook as many times as you like.
- name: Set authorized key taken from file
authorized_key:
user: kates
state: present
key: "{{ lookup('file', '~/.ssh/qq_rsa.pub') }}"
This task adds your Quick Queue ssh key to the kates user. This allows other ansible operations to be done under that user.
- name: add sudoers file for kates
tags: always
copy:
src: sudoer_kates
dest: /etc/sudoers.d/kates
owner: root
group: root
mode: 0440
As we want to add more configuration using the kates
user, we add the user to the sudoers file by copying over the permissions file (sudoer_kates
). The Ansible copy module looks for the source files in a subfolder called files
.
Create a folder called files
. Then create the sudoers permissions file files/sudoer_kates
:
kates ALL=(ALL) NOPASSWD: ALL
This gives the kates
user the same privileges as root
using sudo
. This is useful as it enables the rest of the Ansible config to happen under this user, rather than under root
. Later we will see that we will remove root
from being able to login via ssh.
Back in the bootstrap.yml
file, there is the next section:
- name: generate sshd_config file from template
tags: ssh
vars:
ssh_users: kates
template:
src: "sshd_config.j2"
dest: /etc/ssh/sshd_config
owner: root
group: root
mode: 0644
notify: restart_sshd
In this section we use the template module to create a file in the target’s filesystem based on a j2
template file. In this case the file provides a default sshd configuration that prevents password and root
user login. The module looks for the file in the templates
folder. It replaces user defined fields like {{ ssh_users }}
with values from the vars
section of the task definition.
Create templates/sshd_config.j2
:
# $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.
Include /etc/ssh/sshd_config.d/*.conf
AllowUsers {{ ssh_users }}
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key
# Ciphers and keying
#RekeyLimit default none
# Logging
#SyslogFacility AUTH
#LogLevel INFO
# Authentication:
#LoginGraceTime 2m
#PermitRootLogin prohibit-password
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10
#PubkeyAuthentication yes
# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2
#AuthorizedPrincipalsFile none
#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody
# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes
# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no
# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no
# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes
#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none
# no default banner path
#Banner none
# Allow client to pass locale environment variables
AcceptEnv LANG LC_*
# override default of no subsystems
Subsystem sftp /usr/lib/openssh/sftp-server
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# PermitTTY no
# ForceCommand cvs server
In the last line of this section, there is a line:
notify: restart_sshd
This line instructs Ansible to notify a handler (restart_sshd
) that a change has been made and that Ansible needs to undertake an additional task. Multiple tasks can notify a handler to undertake that task.
Handlers are tasks that are run after all other plays have finished. They should be defined in your playbook as I have done here (note that there are better ways to build your configuration files using roles, which I will describe in the next article):
handlers:
- name: restart_sshd
service:
name: sshd
state: restarted
In our case this handler restarts the sshd daemon whenever a change is made to its configuration (as is happening in this task).
This is simple. Using the service
module, we are requesting that the state of the sshd service is restarted
. Ansible will now restart that service as a result of the notification it receives.
Applying the Bootstrap Config
You should now have a folder structure that looks like this:
ansible
bootstrap
files
sudoers_kate
templates
sshd_config.j2
ansible.cfg
bootstrap.yml
inventory
terraform
...
The inventory file should have been created by the Terraform script.
If you destroyed your infrastructure at the end of the previous article, you can now create it again with the following command from the terraform
folder:
terraform apply
Although Binary Lane returns the server information quickly, you should allow a minute for them to be created. Once you have created and prepped your servers, you are now ready to configure them.
As Ansible has been given its configuration through the ansible.cfg
file and we want to apply this bootstrap to all ours servers, all you need enter (from the ansible/bootstrap
folder) is:
ansible-playbook bootstrap.yml
This will now go through a series of steps:
PLAY [all] **************************************************
Shows you are applying this playbook to all reasources
TASK [Gathering Facts] **************************************
Shows it is gathering existing details of the servers so it can work out what to do.
TASK [install updates (Debian)] *****************************
Shows it is installing updates to the OS. You may have noticed by now that these messages come from the task names in the playbook.
TASK [install sudoers for Debian servers] *******************
PLAY [all] **************************************************
It shows this again as the first play was the pre-tasks it was running. This is the next play and it is applying to all servers.
TASK [Gathering Facts] ****************************************
Each play gathers facts about its target hosts as the targets may be different to the previous play.
TASK [create kates user] **************************************
TASK [Set authorized key taken from file] *********************
TASK [add sudoers file for kates] *****************************
After all tasks and plays are finished, it provides you with a recap of what it did:
PLAY RECAP ****************************************************
ip: ok=8 changed=5 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
ip : ok=8 changed=5 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
ip : ok=8 changed=5 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
ok
says how many tasks were found and attempted
changed
says how many tasks caused a change
failed
says how many tasks could not be carried out as the host could not be reached or logged in to
skipped
says how many tasks were not executed because of their when
clause
rescued
says how many rescue tasks were executed (tasks that are marked to be run when another task fails)
ignored
says how many tasks failed but the error was ignored due to a ignore_errors: true
clause
Because of the declarative nature of Ansible, you can run the ansible-playbook
command as many times as you like.
You now have three servers that are bootstrapped, ready for the next stage of configuration. This next stage will setup Kubernetes on our newly configured servers.
Summary
In this article we configured Ansible to provide a base layer of configuration on our servers. We used an Ansible bootstrap configuration which we can use with any of our servers.
In the next article, we will apply a Kubernetes configuration on top of these bootstrapped servers.