Zarf Deployment with PLG Log Monitoring

Deploy a WordPress blog using Zarf and take advantage of Zarf’s integrated log monitoring functionality

In this tutorial, we will use the open source tool Zarf to deploy a sample WordPress blog into a Kubernetes cluster. Zarf offers optional deployment of the PLG (Promtail/Loki/Grafana) log monitoring stack. We will briefly dive into each component of the PLG stack and some of the commonly used monitoring features.

Prerequisites:

• kubectl installed
• k3d installed
• Git installed
• Zarf CLI installed

Objectives:

• Explore Promtail, Loki, and Grafana
• Spin up a k3d cluster
• Use Zarf to deploy a sample WordPress blog into the cluster
• Deploy the Zarf init package
• Create the Zarf WordPress package
• Deploy the Zarf WordPress package
• Connect to the WordPress blog
• Log into and explore Grafana

The PLG Stack

The PLG stack consists of Promtail, Loki, and Grafana and is a popular log monitoring tool. Promtail pulls data logs and sends them to Loki, where they are indexed and grouped via labels. Grafana dashboards can then be used to query and display the logs.

What is Promtail?

Promtail takes local log files or systemd journal logs and ships them to a private Loki instance. Promtail is first deployed to a Kubernetes cluster as a daemon set, where it performs service discovery on each node. After discovering services that emit log lines to files, it attaches labels to the log streams, then batches the target logs and ships them to Loki.

What is Loki?

Loki is a Grafana tool that aggregates logs efficiently via labels. After Promtail discovers and labels log streams, it ships them to Loki where they are indexed based on their labels. These logs can then be queried. Loki breaks up large queries into smaller requests, dispatching them in parallel (parallelization). This decreases response time, allowing a user to query large amounts of data without compromising speed.

What is Grafana?

Grafana is an open-source tool that can be used to visualize and analyze data, allowing you to create custom dashboards for easier interpretation. After connecting your database to Grafana, you can choose from one of the many prebuilt dashboards, or you can create your own to include metrics, logs, and tracings specific to your needs. Grafana can integrate with hundreds of popular plugins like Prometheus, Loki, and CloudWatch, just to name a few. You can set alerts for your data as well as easily generate PDF reports from your dashboard to be shared with others (ex. daily performance logs). Below is an example dashboard monitoring server requests, memory/CPU, and more. The various graphs and colors make interpretation of the data easier to visualize and understand.

Grafana sandbox dashboard

Many well-known and high profile companies use Grafana as part of their data monitoring stack, and case studies can be found here. Below are just a few advantages of using Grafana to optimize data interpretation:

• Provide real-time information on site performance
• Decrease mean time to repair by identifying issues early
• Measure performance to drive improvements
• Create a consistent observability experience for developers
• Provide fast feedback to developers
• Analyze security logs at scale to detect malicious activity

In this tutorial, we will deploy a sample WordPress blog into a Kubernetes cluster using Zarf. To learn more about Zarf, visit my previous blog posts here and here, or the official documentation here. Although Zarf can do a lot of really cool things, in its simplest form, it packages an application’s source code into a single tarball file, allowing that file to be easily deployed into a cluster. In the next few steps, we’ll spin up a cluster and deploy a sample WordPress blog with Zarf. Zarf gives you the option to deploy the PLG monitoring stack, which we will do. Then we’ll log into Grafana and checkout the Loki dashboard Zarf has deployed.

It is important to note that while Grafana is a visualization tool, the Loki dashboard deployed with Zarf is specifically looking at logs, not the state of your cluster. With Loki, we can query logs, but not CPU utilization, server requests, or other metrics, so the dashboard that we will see later in this tutorial will look a bit different than the vibrant one above.

Spin Up a k3d Cluster

If you already have some other Kubernetes cluster running, you are welcome to use it. If you don’t, k3d is a good option for local testing. To create a k3d cluster, first make sure k3d is installed. If you are using a Mac with Homebrew, simply run brew install k3d . If you are using a different machine, follow the k3d installation instructions here. Once k3d is installed, use the following command to spin up a cluster:

k3d cluster create <cluster_name>

The cluster should now be up and running but you can double check with the kubectl cluster-info command or look at your Docker Desktop if you have it installed.

Confirm the Zarf CLI is Installed

If you’ve used Zarf before, confirm the version with:

zarf version

If you don’t have the Zarf CLI installed yet, follow these instructions.

Clone the Zarf Repository to Your Local Machine

The Zarf repository has several application examples that can be used for learning and local testing. We will use the sample WordPress blog located in the repository at zarf/examples/wordpress . Clone the entire Zarf repository to your local machine:

git clone https://github.com/defenseunicorns/zarf.git

To learn more about how Zarf works and how to compose a zarf.yaml file, visit my previous blog articles on Zarf or the official Zarf documentation.

Deploy the Zarf Init Package

Zarf requires a special ‘init’ package whose source code is found at the root of the Zarf repository. The Zarf init package installs several core components needed for Zarf to function, as well as some optional ones. For this tutorial, it’s important to note that logging is one of the optional features you will be prompted to install. When logging is enabled, Zarf installs the Promtail/Loki/Grafana monitoring stack into the cluster. This is important because our focus here is to take a deeper look at this log monitoring tool that is built into Zarf.

To install the Zarf init package, first make sure you are in the root of the Zarf repository:

Then run the following command:

zarf init

You will be given a few prompts:

• Do you want to download this init package? Answer “y”
• Deploy this Zarf package? (y/N)Answer “y”
• Deploy the logging component?(y/N)Answer “y”
• Deploy the git-server component?(y/N)Answer “y”

The Zarf init package will take a few minutes to deploy. Once it is completed, you will see something like this:

Save these Usernames and Passwords somewhere as you’ll need them later to log into Grafana.

Create the Zarf WordPress Package

Now that our Zarf init package has been deployed, we’re ready to package up the code for our sample WordPress blog. Use cdto change your directory to examples/wordpress then ls to confirm the files in your directory:

With the next command we’ll use Zarf to package up our WordPress blog source code into a single tarball file:

zarf package create . 

You will be given a few prompts:

• Create this Zarf package? (y/N)
• Please provide a value for "Maximum Package Size" (0)

Answer y to the first prompt. For the second, simply hit Enter and this will override the max package size prompt, as it is not necessary for this tutorial. Once the application package is created, you will see this:

Check your local files again and you will see the tarball has been created:

Deploy the Zarf WordPress Package

Now that our tarball package is created, we want to deploy it to our Kubernetes cluster. We’re going to use K9s to watch this deployment. Zarf has the K9s tool built into it, so open a new terminal window and run:

zarf tools monitor

This opens the K9s tool for your current cluster, and you can see all of the pods that are currently deployed. Notice that the Zarf init package deployed several pods (agent-hook, zarf-loki-stack, etc) into a “zarf” namespace.

Place your two terminal windows side by side so you can see events in real-time as commands are run.

Now we’re ready to deploy the Zarf WordPress package (our tarball file). Use the following command to deploy the package:

zarf package deploy

You will be prompted Choose or type the package file [tab for suggestions] . Press the tab button and you should see the tarball that we created earlier. After confirming the correct tarball, press Enter .

You will then have some additional prompts:

Answer Yes to the first prompt, then enter values for the requested environment variables. These prompts are specific to the WordPress blog that’s being deployed, so if you were deploying a different application, you would not encounter them. For this example, I’m using a simple Username and Password that will be deleted once this tutorial is complete, but in the real word, you would want to use a more secure password. Once the deployment has finished, you will see this:

If you were watching the K9s window, you should have seen the WordPress pods being created.

Connect to WordPress Blog

Let’s connect to the sample WordPress blog we just deployed on our cluster. Zarf makes it easy and gives you the command to use:

zarf connect wordpress-blog

This opens a port-forward tunnel from your local machine, allowing you to access the WordPress blog on your cluster. Your default web browser will automatically open to the WordPress site:

Ok that was cool! Use ctrl + c to end the port-forward. What if I wanted to log into the backend of the WordPress site with the Username and Password environment variables I set? Run the following command:

zarf connect wordpress-admin

This opens another port-forward and your default browser, navigating you to the WordPress admin login page.

Use the Username and Password environment variables you set to log in.

Boom! Okay, enough of that. In the terminal, close the port with ctrl + c . Let’s move on to the point of this tutorial: log monitoring!

Log Into and Explore Grafana

When we deployed the Zarf init package earlier, we selected Y when prompted to install logging. Zarf installs the PLG stack, which includes Grafana. After installing the Zarf init package, you are given the Zarf command to connect to logging as well as the Username and Password. Remember I told you to save them? Go find them now.

To keep yourself organized, open a new terminal window and run the following command (you’ll see why you want it in a separate window):

zarf connect logging

The above command opens a port-forward tunnel and will open your default web browser to the Grafana login page.

Port-forward (see why you want it in its own window)

Use the “Logging” Username and Password you saved to log into the Grafana website.

Grafana login page

Your initial login should be met with a dashboard similar to this:

This is the Loki dashboard that Zarf has chosen for monitoring your data. In the Loki dashboard, logs are aggregated and indexed based on their labels that were assigned by Promtail.

Notice that the dashboard automatically opened to a query based on “{namespace=”zarf”}”. The histogram shows the quantity of new logs per timestamp. In the picture below, you can see there was a spike in log activity between the 07:07:30 timestamp and the 07:08:00 timestamp. The bars of the histogram are color-coded to indicate unknown, warning, info, error, or debug.

Under the histogram there are toggle buttons that allow you to optimize the output of your logs. For example, you toggle on/off if you want the timestamp to appear. To see a description of the None, Exact, Numbers, and Signature settings, simply hover over the word.

The Loki dashboard will display the most recent 1,000 logs. If you want to see more, scroll to the bottom of the displayed logs and click Older logs.

Grafana has made it easy to query logs without knowing a query language. You can search logs via labels by clicking the Log browser button on the top left, then selecting a label from the given options. We were previously looking at the logs in the zarf namespace. To look at logs in a different namespace, select the namespace label and then select from the three namespaces that are found in your cluster (I chose the wordpress namespace). Once you’ve selected your namespace, click Show logs.

You are now viewing logs for the wordpress namespace and notice how the histogram has changed.

You can evaluate an individual log by clicking on it. This will expand a window and give you detailed information about that log. The log below contains a warning message, and once expanded you can see the complete message that a datasource is outdated.

To narrow down your query, you can use piping to further filter your results. Grafana log query syntax explanation can be found here. In the log browser search bar, use the pipe character to add a filter. Maybe we only want to see the logs that have the word “error” in them. To do this, we would pipe for “error” by typing |= “error”then click Run query. Notice that now all of the logs listed are then ones that include the word “error.”

If you try to run a query filter and get an error like this:

copy your query filter, then start a new query (click Log browser and select what labels you want to query), then paste the filter expression in the log browser search bar instead of directly typing it out. Then click the Run query or refresh button. There is a known bug in Grafana at the time of this writing and is discussed here and here.

Conclusion and Cleanup

In this tutorial we used Zarf to deploy a sample WordPress blog into a Kubernetes cluster, then logged into Grafana and reviewed some of the monitoring capabilities of the Loki dashboard. Zarf has made monitoring of your logs easier by directing your logging to the Loki dashboard, instead of you having to build a dashboard from scratch. Grafana does allow you to create custom dashboards if needed, and videos of how to do this can be found on the Grafana website.

If you’re ready to clean up everything we created, use the following command to tear down the resources that were created with Zarf:

zarf destroy --confirm --remove-components

If you run zarf package list you should no longer see your init or wordpress packages.

To remove the k3d cluster you created, run the following command:

k3d cluster delete <cluster_name>

The last thing to do is remove the tarball file from your local directory:

rm <zarf_tarball_name>

Hopefully this tutorial was helpful in demonstrating how Zarf can be used to deploy packages as well as Zarf’s built in log monitoring capability. Thank you so much for following along and keep watching for more DevOps tutorials!