Installing Cribl Steam in Distributed Deployment — Part 1 of 2

Installing One Leader node Running on Amazon Linux OS

6 min readJun 11, 2024

This is series of blog posts where I will cover the entire process sequentially with detailed procedure.

What is Cribl Stream?

Cribl Stream helps you process machine data — logs, instrumentation data, application data, metrics, etc. — in real time, and deliver them to your analysis platform of choice. It allows you to:

Add context to your data, by enriching it with information from external data sources.
Help secure your data, by redacting, obfuscating, or encrypting sensitive fields.
Optimize your data, per your performance and cost requirements.

Who Is Cribl Stream For?

Cribl Stream is built for administrators, managers, and users of operational/DevOps and security intelligence products and services.

What is Cribl Stream’s Leader node?

You use the Leader Node to centrally monitor and author configurations for the Worker Nodes in a Distributed deployment.

How does the distributed Architecture of Cribl Stream look like?

Below here is an overview of a distributed Cribl Stream deployment’s components. This has division of labor among components of the Leader Node and Worker Node.

Pre-requisites & Important Links

1. Here is the recommended Specs for choosing the host infra from Cribl Docs and link for downloading the Cribl Enterprise tar ball.
2. When creating the VM on Public Cloud — here are the Recommended AWS, Azure, and GCP Instance Types.
3. During installation of any Leader or Worker instance, firewalls on that instance’s host must enable outbound communication to https://cdn.cribl.io on port 443.
4. Version Control with Git — Cribl Stream requires git (version 1.8.3.1 or higher) to be available locally on the host where the Leader Node will run. Configuration changes must be committed to git before they’re deployed.
5. Enterprise grade Cribl Stream license from the Cribl sales team. :)
6. Internet Access would be required on the leader node to install Git package on Linux using the preferred package manager of your Linux distribution. In case this host is connected to satellite server for updates, then internet connection may not be required.
7. firewall port matrix requirement 1 — Inbound rule — for Cribl Leader Web UI — from 0.0.0.0/0 to this leader node over TCP port 9000
8. firewall port matrix requirement 2 — Inbound and Outbound rule — bidirectional communication between Cribl Stream Leader & worker nodes over TCP 9000 and TCP 4200 port for administration and heartbeat purpose.

There are various methods of installing any software, I have chosen a simple one among those. Without further ado, let’s start —

I have created two EC2 instances with Amazon Linux running on them.

https://docs.cribl.io/stream/deploy-planning/#requirements

https://docs.cribl.io/stream/scaling/#vms

2. First one will serve as Leader node and the second one will serve as Worker node.

3. We have got the private key(PuTTY Private Key File (.ppk)) created and downloaded from the aws portal to access these machines over SSH/MobaXterm securely without password.

4. Accessing the Leader machine and running below commands —

sudo useradd cribl
sudo usermod -aG cribl cribl
cd /opt

5. Post this we accessed https://cribl.io/download/ and downloaded the tar ball from here.

6. We used SCP tool e.g. MobaXterm, WinSCP to move this tarball to /tmp location in the Leader Machine.

As an alternative to this step of SCP, we can use the command to download this tarball on the Cribl Leader node itself. For that — During installation of any Leader or Worker instance, firewalls on that instance’s host must enable outbound communication to https://cdn.cribl.io on port 443.

7. Installing Cribl now using below command correctly defining the tarball name.

cd /tmp && ls -lrt
sudo tar xzvf cribl-4.7.1-* -C /opt

8. Running below set of commands for the remaining part of installation. Below screenshot shows the returned output corresponding to each command.

sudo chown -R cribl:cribl /opt/cribl
sudo /opt/cribl/bin/cribl boot-start enable -u cribl
sudo systemctl start cribl
sudo su cribl
/opt/cribl/bin/cribl status

9. Cribl is up and running on the CLI.

10. We used the provided “Public IPv4 DNS” at the AWS portal and Cribl UI is up and running in the browser.

11. We login with default username and password which is admin admin.

12. Post login, we get the prompt to change this password to something complex.

This is here we complete the installation of Cribl Stream on this node. This is important to note that currently, this Cribl stream instance is functioning as all in one instance.

13. For promoting this aio instance of Cribl Stream, we would need an purchased enterprise level/NFR internal & limited use license. Without this license the distributed deployment of Cribl Stream is not possible.

This license is to be applied under Account > Global Settings > Licensing.

14. You can promote the Cribl stream aio node to a Leader Node through the UI, through the instance.yml config file, or through the command line.

Using the UI: At Settings > Distributed Settings > General Settings, select Mode: Leader.

Next, on the Leader Settings left tab, confirm or enter the required Leader settings (Address and Port). Customize the optional settings if desired. Then click Save to restart.

15. For Cribl to be installed in the distributed mode, Leader node must have a git. Without git, it wont promote itself to leader node.

The Leader node uses git to:

Manage configuration versions across Worker Groups.
Provide users with an audit trail of all configuration changes.
Allow users to display diffs between current and previous config versions.

If you don’t have git installed, check here for details on how to get started.

[ec2-user@ip-172–x–x–x tmp]$ cat /etc/os-release
NAME=”Amazon Linux”
VERSION=”2023"
ID=”amzn”
ID_LIKE=”fedora”

[ec2-user@ip-172–x–x–x tmp]$ sudo dnf install git

To enable, download of Git packages from internet, i had to temporarily allow outbound internet access from this host.

16. With this done, when we now save the config on the UI as Mode: Leader, it restarts the node and promote it to Leader.

17. The Cribl Stream Leader node is up and running. Its now equipped with the locally installed git based version control mechanism as well.

18. Keep in mind — When you configure a node to work as the Leader, switching from a Single-instance to a Distributed mode, Cribl Stream generates a random auth token.

Follow second part of this blog series to install the worker nodes of this Cribl distributed 2-tier architecture.