devops4me
Published in

devops4me

Install Grafana-Prometheus-Node_Exporter Using Ansible

Introduction

You can accomplish so much with Ansible, and it’s so simple to get started with it. The more you use it, the more it becomes a part of you. Prometheus, Grafana, and Node-Exporter are all good options for a monitoring stack. Because you can use a small or large scale to practise a variety of standard modules.

I will guide you on how you can use the Ansible script to install Grafana and Prometheus in your Host/Main monitor server and install node_exporter in other servers that you would like to monitor.

Use-Case

Step 1: Set with static inventory file and Pinging the Target Nodes

  • As a best practice, I prefer to create a new folder for each project and create a config file in it.

Note: Remember that Ansible will process the below list and use the first file found, all others are ignored.

ANSIBLE_CONFIG (environment variable if set)
ansible.cfg (in the current directory)
~/.ansible.cfg (in the home directory)
/etc/ansible/ansible.cfg
  • So, make a directory named grafana-prometheus under the home directory and switch to it.
$ mkdir dynamic-inventory
$ cd dynamic-inventory

Create a file named inventory.txt

$ sudo vi inventory.txt
  • Paste the content below into the inventory.txt file.
[monitorserver]
db_server ansible_host=<YOUR-DB-SERVER-IP> ansible_user=ec2-user ansible_ssh_private_key_file=~/<YOUR-PEM-FILE>
[nodeservers]
server1 ansible_host=<YOUR-WEB-SERVER-IP> ansible_user=ec2-user ansible_ssh_private_key_file=~/<YOUR-PEM-FILE>
server2 ansible_host=<YOUR-WEB-SERVER-IP> ansible_user=ec2-user ansible_ssh_private_key_file=~/<YOUR-PEM-FILE>

Note: Don’t forget the change the IP addresses of the target nodes and the path of your pem key.

  • Create file named ansible.cfg under the the dynamic-inventory directory.
  • Paste the content below into ansible.cfg file.
[defaults]
host_key_checking = False
inventory=inventory.txt
interpreter_python=auto_silent
localhost_warning=false

Validate and check the inventory.

ansible-inventory --graph
  • Check the connectivity to the target nodes.
$ ansible all -m ping
  • So, that’s great! We have connected to the target nodes with a static inventory.

Step 2: Create a role to install node-exporter

To install a binary or a tarball with an IaC or configuration manager such as ansible, you must first master a basic case. A good example is the node exporter. First and foremost, the ansible-galaxy command aids in the construction of a role’s skeleton:

ansible-galaxy init roles/node-exporter

Then we can set some defaults variables in the default directory :

node_exporter_version: "1.1.2"
node_exporter_bin: /usr/local/bin/node_exporter
node_exporter_user: node-exporter
node_exporter_group: "{{ node_exporter_user }}"
node_exporter_dir_conf: /etc/node_exporter

And now the main file of the tasks directory :

- name: check if node exporter exist
stat:
path: "{{ node_exporter_bin }}"
register: __check_node_exporter_present
- name: create node exporter user
user:
name: "{{ node_exporter_user }}"
append: true
shell: /usr/sbin/nologin
system: true
create_home: false
- name: create node exporter config dir
file:
path: "{{ node_exporter_dir_conf }}"
state: directory
owner: "{{ node_exporter_user }}"
group: "{{ node_exporter_group }}"
- name: if node exporter exist get version
shell: "cat /etc/systemd/system/node_exporter.service | grep Version | sed s/'.*Version '//g"
when: __check_node_exporter_present.stat.exists == true
changed_when: false
register: __get_node_exporter_version

- name: download and unzip node exporter if not exist
unarchive:
src: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.linux-amd64.tar.gz"
dest: /tmp/
remote_src: yes
validate_certs: no
- name: move the binary to the final destination
copy:
src: "/tmp/node_exporter-{{ node_exporter_version }}.linux-amd64/node_exporter"
dest: "{{ node_exporter_bin }}"
owner: "{{ node_exporter_user }}"
group: "{{ node_exporter_group }}"
mode: 0755
remote_src: yes
when: __check_node_exporter_present.stat.exists == false or not __get_node_exporter_version.stdout == node_exporter_version
- name: clean
file:
path: /tmp/node_exporter-{{ node_exporter_version }}.linux-amd64/
state: absent
- name: install service
template:
src: node_exporter.service.j2
dest: /etc/systemd/system/node_exporter.service
owner: root
group: root
mode: 0755
notify: reload_daemon_and_restart_node_exporter
- meta: flush_handlers
- name: service always started
systemd:
name: node_exporter
state: started
enabled: yes

We need to create node_exorter.service.j2 file in templates directory :

[Unit]
Description=Node Exporter Version {{ node_exporter_version }}
After=network-online.target
[Service]
User={{ node_exporter_user }}
Group={{ node_exporter_user }}
Type=simple
ExecStart={{ node_exporter_bin }}
[Install]
WantedBy=multi-user.target

Finally the handler file in the handler direct

- name: reload_daemon_and_restart_node_exporter
systemd:
name: node_exporter
state: restarted
daemon_reload: yes
enabled: yes

Step 3: Create a role for Prometheus and its configuration

We can initialize a Prometheus role :

ansible-galaxy init roles/prometheus

We can set some defaults variables in the default directory :

prometheus_dir_configuration: "/etc/prometheus"
prometheus_retention_time: "365d"
prometheus_scrape_interval: "30s"
prometheus_node_exporter: true
prometheus_node_exporter_group: "all"
prometheus_env: "production"
prometheus_var_config:
global:
scrape_interval: "{{ prometheus_scrape_interval }}"
evaluation_interval: 5s
external_labels:
env: '{{ prometheus_env }}'
scrape_configs:
- job_name: prometheus
scrape_interval: 5m
static_configs:
- targets: ['{{ inventory_hostname }}:9090']

Since we define a part of the Prometheus configuration, the header exactly with prometheus_var_config.Now I create tasks in the main.yml file of the tasks directory :

- name: update and install prometheus
apt:
name: prometheus
state: latest
update_cache: yes
cache_valid_time: 3600
- name: prometheus args
template:
src: prometheus.j2
dest: /etc/default/prometheus
mode: 0644
owner: root
group: root
notify: restart_prometheus
- name: prometheus configuration file
template:
src: prometheus.yml.j2
dest: "{{ prometheus_dir_configuration }}/prometheus.yml"
mode: 0755
owner: prometheus
group: prometheus
notify: reload_prometheus
- name: start prometheus
systemd:
name: prometheus
state: started
enabled: yes

Then, we create the prometheus.yaml.j2 file:

#jinja2: lstrip_blocks: "True"
{{ prometheus_var_config | to_nice_yaml(indent=2) }}
{% if prometheus_node_exporter_group %}
- job_name: node
scrape_interval: 15s
metrics_path: /metrics
static_configs:
- targets:
{% for server in groups[prometheus_node_exporter_group] %}
- '{{ server }}:9100'
{% endfor %}
{% endif %}

And the prometheus.j2 file for the Prometheus CLI :

ARGS="--web.enable-lifecycle --storage.tsdb.retention.time={{ prometheus_retention_time }} --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries

Finally handlers of this Prometheus role;We have two handlers :

  • Restart with the systemd service
  • Reload with a curl on the prometheus API.
- name: restart_prometheus
systemd:
name: prometheus
state: restarted
enabled: yes
daemon_reload: yes
- name: reload_prometheus
uri:
url: http://localhost:9090/-/reload
method: POST
status_code: 200

Step 4: Create a role for Grafana

Now we can create a last role to install Grafana server package and start it. Just edit the main.yml file in the tasks directory :

- name: install gpg
apt:
name: gnupg,software-properties-common
state: present
update_cache: yes
cache_valid_time: 3600
- name: add gpg hey
apt_key:
url: "https://packages.grafana.com/gpg.key"
validate_certs: no
- name: add repository
apt_repository:
repo: "deb https://packages.grafana.com/oss/deb stable main"
state: present
validate_certs: no
- name: install grafana
apt:
name: grafana
state: latest
update_cache: yes
cache_valid_time: 3600
- name: start service grafana-server
systemd:
name: grafana-server
state: started
enabled: yes
- name: wait for service up
uri:
url: "http://127.0.0.1:3000"
status_code: 200
register: __result
until: __result.status == 200
retries: 120
delay: 1
- name: change admin password for grafana gui
shell : "grafana-cli admin reset-admin-password {{ grafana_admin_password }}"
register: __command_admin
changed_when: __command_admin.rc !=0

Don’t forget to set your Admin password for start and you can set it in default directory:

grafana_admin_password: "abc1234"

Step 5: Create Ansible Playbook

Create an Ansible playbook file like the below :

- name: install monitoring stack
hosts: monitorserver
become: yes
roles:
- prometheus
- grafana
- name: install node-exporter
hosts: nodeservers
become: yes
roles:
- node-exporter

Step 6: Testing

You can run this playbook file via the below command:

ansible-playbook -i inventory.txt playbook.yml```devops4me@ % ansible-playbook -i inventory.txt playbook.yml
/usr/local/Cellar/ansible/5.7.1/libexec/lib/python3.10/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
"class": algorithms.Blowfish,
PLAY [install monitoring stack] ****************************************************************************************************************************TASK [Gathering Facts] *************************************************************************************************************************************
ok: [18.136.207.64]
TASK [prometheus : update and install prometheus] **********************************************************************************************************
ok: [18.136.207.64]
TASK [prometheus : prometheus args] ************************************************************************************************************************
ok: [18.136.207.64]
TASK [prometheus : prometheus configuration file] **********************************************************************************************************
ok: [18.136.207.64]
TASK [prometheus : start prometheus] ***********************************************************************************************************************
ok: [18.136.207.64]
TASK [grafana : install gpg] *******************************************************************************************************************************
ok: [18.136.207.64]
TASK [grafana : add gpg hey] *******************************************************************************************************************************
ok: [18.136.207.64]
TASK [grafana : add repository] ****************************************************************************************************************************
ok: [18.136.207.64]
TASK [grafana : install grafana] ***************************************************************************************************************************
ok: [18.136.207.64]
TASK [grafana : start service grafana-server] **************************************************************************************************************
ok: [18.136.207.64]
TASK [grafana : wait for service up] ***********************************************************************************************************************
ok: [18.136.207.64]
TASK [grafana : change admin password for grafana gui] *****************************************************************************************************
ok: [18.136.207.64]
PLAY [install node-exporter] *******************************************************************************************************************************TASK [Gathering Facts] *************************************************************************************************************************************
ok: [13.215.254.70]
ok: [54.251.28.26]
TASK [node-exporter : check if node exporter exist] ********************************************************************************************************
ok: [13.215.254.70]
ok: [54.251.28.26]
TASK [node-exporter : create node exporter user] ***********************************************************************************************************
[WARNING]: 'append' is set, but no 'groups' are specified. Use 'groups' for appending new groups.This will change to an error in Ansible 2.14.
ok: [54.251.28.26]
ok: [13.215.254.70]
TASK [node-exporter : create node exporter config dir] *****************************************************************************************************
ok: [54.251.28.26]
ok: [13.215.254.70]
TASK [node-exporter : if node exporter exist get version] **************************************************************************************************
ok: [54.251.28.26]
ok: [13.215.254.70]
TASK [node-exporter : download and unzip node exporter if not exist] ***************************************************************************************
changed: [13.215.254.70]
changed: [54.251.28.26]
TASK [node-exporter : move the binary to the final destination] ********************************************************************************************
skipping: [54.251.28.26]
skipping: [13.215.254.70]
TASK [node-exporter : clean] *******************************************************************************************************************************
changed: [13.215.254.70]
changed: [54.251.28.26]
TASK [node-exporter : install service] *********************************************************************************************************************
ok: [13.215.254.70]
ok: [54.251.28.26]
TASK [node-exporter : meta] ********************************************************************************************************************************TASK [node-exporter : service always started] **************************************************************************************************************
ok: [54.251.28.26]
ok: [13.215.254.70]
PLAY RECAP *************************************************************************************************************************************************
13.215.254.70 : ok=9 changed=2 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
18.136.207.64 : ok=12 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
54.251.28.26 : ok=9 changed=2 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0

Improvement

We can improve this solution by apply the following steps:

1. Security:

1.1 Encrypt The Credentials/Password/Private key

Ansible Vault encrypts variables and files so you can protect sensitive content such as passwords or keys rather than leaving it visible as plaintext in playbooks or roles

2. Agility:

2.1 Continuous Integration and Continuous Delivery:

Integrate our Ansible script with CI/CD and apply it to other projects or needs.

2.2 Disable fact gathering :

When a playbook executes, each play runs a hidden task, called gathering facts, using the setup module. This gathers information about the remote node you're automating, and the details are available under the variable ansible_facts. But if you're not using these details in your playbook anywhere, then this is a waste of time. You can disable this operation by setting gather_facts: False in the play.

Conclusion

Finally, with all the above steps we have learned :

  1. How to create a Dynamic Ansible Inventory file.
  2. How to provision and installed Grafana & Prometheus using Ansible script
  3. How to automatically installed Node Exporter via Ansible

Additional:

You may find the full code for this post from here-> DevOps4Me Global Code

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Najib Radzuan

Najib Radzuan

DevOps | DevSecOps | Global DevOps Ambassador | CDF Ambassador | Digital Transformation [https://linktr.ee/devops4me]