Spring Boot CI/CD on Kubernetes using Terraform, Ansible and GitHub: Part 4

11 min readNov 6, 2023

Part 4: Configuring servers using Ansible

This is part of a series of articles that creates a project to implement automated provisioning of cloud infrastructure in order to deploy a Spring Boot application to a Kubernetes cluster using CI/CD. In this part we use Ansible to configure the Virtual Private Servers (VPSs) we created previously using Terraform.

Having created our three VPS nodes in the previous article, we will now set up a base level configuration using Ansible.

You can find the code for this part here: https://github.com/MartinHodges/Quick-Queue-IaC/tree/part4

Ansible

Like Terraform, Ansible works from a set of declarative configuration files that tell it the desired state you want for each of your servers (target hosts). We will create these files in the ansible folder you created earlier.

These declarative configuration files form what is known as a playbook. A playbook can have one or more plays within them. It executes these plays against each of the target hosts it is told to configure.

Each play can include one or more tasks. Ansible uses a set of plugin modules with an extensive and useful set built in to the tool itself. Each module performs a task on the target host and can, generally, perform the action on a number of different Operating Systems (OS).

Some plays are specific to an OS and there are ways you can tell Ansible to carryout different play for different OSs. In this way, you can configure a set of different hosts using the same playbook. All the nodes we have created are Debian nodes.

Defining Your Target Hosts

The target hosts are defined in the inventory file. This is created by Terraform and looks like this:

[k8s_master]
103.4.235.452

[k8s_node]
112.213.316.55
110.532.114.134

Fake IP addresses are shown here but you can use domain names if you have set up a suitable DNS accessible from the VPC.

The names between [ ] allows you to group target hosts, allowing you to refer to a number of hosts by their group name. You can use groups to identify your hosts as particular types (eg: db, web, app etc). A host can appear in more than one group.

When applying a playbook against a set of target hosts, Ansible has a number of ways of directing all or part of the playbook to one or more of the hosts.

For instance, you can tell Ansible to only configure the k8s_master hosts by using --limit=k8s_master on the command line. This will tell Ansible only to run the playbook against your one master node. If you used --limit=k8s_node it will configure both of the worker nodes (which are configured identically).

Alternatively we will see how we can assign a play to one or more groups.

Configuring Servers using Ansible

Whilst a single playbook can configure all your servers, it is sometimes useful to configure your servers in two steps:

Bootstrapping the Servers with identical configuration
Configuring the Servers for your specific application

In this article, we will bootstrap our servers to give them a common configuration on which we can build our Kubernetes cluster.

Remember, you can run your playbook against one or more of your target hosts. I recommend limiting your play to the single k8s_master host until you get it working and then apply it to all in a group or all in the inventory.

Bootstrapping the Servers

All hosts need an initial configuration, such as updating the Operating Systems, adding a user etc. Rather than adding these plays to every playbook you create, it is easier to have it as a standalone playbook containing your standard set of plays.

This application of an initial playbook is known as bootstrapping the servers and it can be done using Ansible once you have added the servers to the known_hosts file (see the previous article).

In your ansible folder, add a subfolder called bootstrap. This is where we will create the configuration files for Ansible.

The first thing to do is to provide a configuration for Ansible itself.

Create bootstrap/ansible.cfg:

[defaults]
inventory = ../inventory
private_key_file = ~/.ssh/qq_rsa
remote_user = root

This tells Ansible to use the inventory file created by Terraform to define the target hosts. It also says which ssh key and user to use to connect to the targets. At the bootstrap stage, root is used as there are no other users on the servers.

A playbook is now needed to carry out the bootstrap. Ansible playbook files are written in the yaml format, so be careful of indentation errors.

Create bootstrap/bootstrap.yml:

---

- hosts: all
  become: yes
  pre_tasks:

  - name: install updates (Debian)
    tags: always
    apt:
      upgrade: dist
      update_cache: yes
    when: ansible_distribution == "Debian"

  - name: install sudoers for Debian servers
    tags: debian
    apt:
      name:
        - sudo
      state: latest
    when: ansible_distribution == "Debian"

- hosts: all
  become: true
  tasks:

  - name: create kates user
    tags: always
    user:
      name: kates
      groups: root
      shell: "/bin/bash"

  - name: Set authorized key taken from file
    authorized_key:
      user: kates
      state: present
      key: "{{ lookup('file', '~/.ssh/qq_rsa.pub') }}"

  - name: add sudoers file for kates
    tags: always
    copy:
      src: sudoer_kates
      dest: /etc/sudoers.d/kates
      owner: root
      group: root
      mode: 0440

  - name: generate sshd_config file from template
    tags: ssh
    vars:
      ssh_users: kates
    template:
      src: "sshd_config.j2"
      dest: /etc/ssh/sshd_config
      owner: root
      group: root
      mode: 0644
    notify: restart_sshd

  handlers:
  - name: restart_sshd
    service:
      name: sshd
      state: restarted

We will break this down into sections.

---

- hosts: all
  become: yes
  pre_tasks:

This tells Ansible to apply this section (ie: this play) to all hosts except where you might limit it on the command line as mentioned earlier (note that --- is a YAML standard that allows you to define multiple files in one file, if desired).

The become: yes tells Ansible to execute the play as root. The command allows you to define other users but the yes indicates root and currently we have no other users on the server.

Finally, whilst these configuration files are declarative, you may need to define dependencies. The pre_tasks option tells Ansible that these configurations must be set before others.

- name: install updates (Debian)
    tags: always
    apt:
      upgrade: dist
      update_cache: yes
    when: ansible_distribution == "Debian"

In this section, we see a task within our play. As the play is within the pre_tasks option, its tasks (including this one) will get carried out before anything else.

First the task is given a name (install updates (Debian)), which will be displayed on the console when the play is attempted. This is followed by a tag. This allows you to define which type of tasks you want Ansible to run. In this case we say always, which means always apply this task, regardless of any command line preference.

The next line, apt, is an Ansible module that will execute the task. Modules come with their own documentation, which you can find here:

https://docs.ansible.com/ansible/latest/collections/ansible/builtin/apt_module.html

A module performs an action on the host to create a particular configuration. They each come with their own, specific set of parameters and arguments. In this case we are asking it to update its cache of packages (update_cache: yes) and to upgrade the OS (upgrade: dist).

Finally, the when tells Ansible when it should run this tak. Although all our servers have a Debian OS, I have added the when clause to show how it works. In this case Ansible will only run this task for servers with a Debian OS.

Before running a playbook against a host, Ansible extracts all the information it needs about the host first. The when clause can target any of these fields.

- name: install sudoers for Debian servers
    tags: debian
    apt:
      name:
        - sudo
      state: latest
    when: ansible_distribution == "Debian"

This next task, still within the pre_tasks play, requests for the latest verison of the sudoers package to be loaded.

This is where the power of Ansible can be seen. As a declarative file, Ansible will only carryout this task if the latest version of sudoers has not been installed.

You need to remember this. Whilst it is tempting to think of tasks as an imperative set of instructions, they actually define what you want and not how to get there.

- hosts: all
  become: true
  tasks:

  - name: create kates user
    tags: always
    user:
      name: kates
      groups: root
      shell: "/bin/bash"

In this section we define a new play. Again it is applied to all hosts and is run as root.

The task that is shown is adding the user kates to the OS and adds this user to the root group. It also sets the login shell to be /bin/bash.

Again, Ansible will only create this user if they do not already exist. This means you can run this playbook as many times as you like.

  - name: Set authorized key taken from file
    authorized_key:
      user: kates
      state: present
      key: "{{ lookup('file', '~/.ssh/qq_rsa.pub') }}"

This task adds your Quick Queue ssh key to the kates user. This allows other ansible operations to be done under that user.

  - name: add sudoers file for kates
    tags: always
    copy:
      src: sudoer_kates
      dest: /etc/sudoers.d/kates
      owner: root
      group: root
      mode: 0440

As we want to add more configuration using the kates user, we add the user to the sudoers file by copying over the permissions file (sudoer_kates). The Ansible copy module looks for the source files in a subfolder called files.

Create a folder called files. Then create the sudoers permissions file files/sudoer_kates:

kates ALL=(ALL) NOPASSWD: ALL

This gives the kates user the same privileges as root using sudo. This is useful as it enables the rest of the Ansible config to happen under this user, rather than under root. Later we will see that we will remove root from being able to login via ssh.

Back in the bootstrap.yml file, there is the next section:

  - name: generate sshd_config file from template
    tags: ssh
    vars:
      ssh_users: kates
    template:
      src: "sshd_config.j2"
      dest: /etc/ssh/sshd_config
      owner: root
      group: root
      mode: 0644
    notify: restart_sshd

In this section we use the template module to create a file in the target’s filesystem based on a j2 template file. In this case the file provides a default sshd configuration that prevents password and root user login. The module looks for the file in the templates folder. It replaces user defined fields like {{ ssh_users }} with values from the vars section of the task definition.

Create templates/sshd_config.j2:

# $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

Include /etc/ssh/sshd_config.d/*.conf
AllowUsers {{ ssh_users }}

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem sftp /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# PermitTTY no
# ForceCommand cvs server

In the last line of this section, there is a line:

notify: restart_sshd

This line instructs Ansible to notify a handler (restart_sshd) that a change has been made and that Ansible needs to undertake an additional task. Multiple tasks can notify a handler to undertake that task.

Handlers are tasks that are run after all other plays have finished. They should be defined in your playbook as I have done here (note that there are better ways to build your configuration files using roles, which I will describe in the next article):

  handlers:
  - name: restart_sshd
    service:
      name: sshd
      state: restarted

In our case this handler restarts the sshd daemon whenever a change is made to its configuration (as is happening in this task).

This is simple. Using the service module, we are requesting that the state of the sshd service is restarted. Ansible will now restart that service as a result of the notification it receives.

Applying the Bootstrap Config

You should now have a folder structure that looks like this:

ansible
  bootstrap
    files
      sudoers_kate
    templates
      sshd_config.j2
    ansible.cfg
    bootstrap.yml
 inventory
terraform
  ...

The inventory file should have been created by the Terraform script.

If you destroyed your infrastructure at the end of the previous article, you can now create it again with the following command from the terraform folder:

terraform apply

Although Binary Lane returns the server information quickly, you should allow a minute for them to be created. Once you have created and prepped your servers, you are now ready to configure them.

As Ansible has been given its configuration through the ansible.cfg file and we want to apply this bootstrap to all ours servers, all you need enter (from the ansible/bootstrap folder) is:

ansible-playbook bootstrap.yml

This will now go through a series of steps:

PLAY [all] **************************************************

Shows you are applying this playbook to all reasources

TASK [Gathering Facts] **************************************

Shows it is gathering existing details of the servers so it can work out what to do.

TASK [install updates (Debian)] *****************************

Shows it is installing updates to the OS. You may have noticed by now that these messages come from the task names in the playbook.

TASK [install sudoers for Debian servers] *******************

PLAY [all] **************************************************

It shows this again as the first play was the pre-tasks it was running. This is the next play and it is applying to all servers.

TASK [Gathering Facts] ****************************************

Each play gathers facts about its target hosts as the targets may be different to the previous play.

TASK [create kates user] **************************************

TASK [Set authorized key taken from file] *********************

TASK [add sudoers file for kates] *****************************

After all tasks and plays are finished, it provides you with a recap of what it did:

PLAY RECAP **************************************************** ip: ok=8 changed=5 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 ip : ok=8 changed=5 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 ip : ok=8 changed=5 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

ok says how many tasks were found and attempted

changed says how many tasks caused a change

failed says how many tasks could not be carried out as the host could not be reached or logged in to

skipped says how many tasks were not executed because of their when clause

rescued says how many rescue tasks were executed (tasks that are marked to be run when another task fails)

ignored says how many tasks failed but the error was ignored due to a ignore_errors: true clause

Because of the declarative nature of Ansible, you can run the ansible-playbook command as many times as you like.

You now have three servers that are bootstrapped, ready for the next stage of configuration. This next stage will setup Kubernetes on our newly configured servers.

Summary

In this article we configured Ansible to provide a base layer of configuration on our servers. We used an Ansible bootstrap configuration which we can use with any of our servers.

In the next article, we will apply a Kubernetes configuration on top of these bootstrapped servers.

Series Introduction

Previous — Setting up the Cloud Infrastructure

Next — Creating the Kubernetes Cluster