Getting Up and Running With Grafana Loki

Harsimran Singh Maan
Nov 6 · 6 min read

Grafana Labs recently launched the v2 of their log aggregation solution — Loki. Loki makes some interesting choices around log aggregation to support horizontal scaling and to keep the operational costs low. It is inspired by Prometheus. Loki supports querying the logs for errors, patterns and much more.

In this post, we’ll go over the setup for Loki on Kubernetes. Before we setup Loki, we’ll install some microservices that we can use to generate logs and events and then consume those via Loki. We’ll use FluentBit to forward logs to Loki and then use the Loki datasource on Grafana to analyze the logs.

Setup the sample application

We’ll use an Istio enabled namespace on a local Kubernetes cluster(docker-desktop to be precise) to install the microservices-demo application. If you don’t have Istio installed, you can follow the step from my previous post to add it to your Kubernetes cluster. Once the Istio setup is done, let’s create the new namespace

$ kubectl create ns demo
$ kubectl label namespace demo istio-injection=enabled #Enable istio sidecar injection

Next, let’s install the microservices-demo app.

# Clone the repo with all microservices to be deployed
$ git clone git@github.com:GoogleCloudPlatform/microservices-demo.git
$ cd microservices-demo# Install the services
$ kubectl apply -n demo -f ./release/kubernetes-manifests.yaml
# Setup istio gateway and virtual services for routing
$ kubectl apply -n demo -f ./release/istio-manifests.yaml

Once the installation finishes, quickly verify the running pods using . The demo application can now be accessed at the Istio gateway endpoint for your environment.

Microservice demo application
Microservice demo application
Microservices demo application

The microservices demo application is an online boutique backed my multiple services written in different programming languages. The application can be used to add items to a cart, change transaction currencies and perform a checkout. All these actions are powered by different services.

Install Loki and FluentBit

After the sample application is up and running, let’s begin installing Loki.

# Add the loki helm chart
$ helm repo add loki https://grafana.github.io/loki/charts
$ helm repo update

The above step would pull the Loki installation chart and it is now ready to be installed.

# Create the namespace to install loki
$ kubectl create ns loki
# Apply the loki helm chart
$ helm upgrade --install loki --namespace=loki loki/loki --set fluent-bit.enabled=true,promtail.enabled=false,grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=true,prometheus.server.persistentVolume.enabled=true

Ensure that is set to use FluentBit as the log forwarder.

Next, install FluentBit

$ helm upgrade --install fluent-bit --namespace=loki loki/fluent-bit --set loki.serviceName=loki.loki.svc.cluster.local

Check the running pods in the namespace with . You might see the loki pod crashing with an error . This is a known issue with value escaping on the FluentBit config map for Loki. The workaround is pretty straightforward

$ kubectl -n loki edit cm/fluent-bit-fluent-bit-loki
# Replace the value of BatchSize to 1048576 under the [Output] section of the config. Save the configmap

At this point you should see all services and pods in the namespace working fine. To make sure that things are working as expected, we’ll install Grafana to do some queries using Loki.

Install Grafana

Let’s pull the Grafana chart for helm

$ helm repo add grafana https://grafana.github.io/helm-charts
$ helm repo update

We’ll use the existing namespace to install Grafana although this is not a requirement.

$ helm upgrade --install grafana --namespace=loki grafana/grafana

You can use to verify that all pods are running as expected.

Image for post
Image for post
Checking if pods are running fine after Loki, FluentBit and Grafana have been installed.

Adding Loki to Grafana

To access the Grafana dashboard, port forward to the Grafana service and open the url in the browser.

$ kubectl -n loki port-forward svc/grafana 8080:80
# Navigate to http://localhost:8080 in the browser

You’ll be prompted for a username and a password. Use the username and follow the steps below to get the password

$ kubectl get secret --namespace loki grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
# The password for the admin user

Once you are logged in, we can add Loki as a data source. From the Left side-panel, select Configuration -> Data Sources

Image for post
Image for post
Select Datasources under configuration

In the Data sources view, click Add data source and then select Loki

Image for post
Image for post
Pick the Loki datasource

Give the datasource a name and then set the Loki URL. The Grafana instance should be able to access the Loki URL. In our case, since both Loki and Grafana are running in the same namespace and don’t need to use any authentication scheme, no other configuration is required. Click “Save & Test”. You should see a confirmation message like “Data source connected and labels found.”

Image for post
Image for post
Configuring the Loki data source

Queries with Loki

Now that Grafana has been connected to Loki, we can query the logs from our sample application deployed in the namespace to observe its behaviour. We can hunt for errors, lookup transactions by trace id and do many more powerful things including visualizing the patterns and trends. The query language for Loki is called LogQL.

In the Grafana application, select “Explore” from the side panel. Select the Loki data source added in the previous step. In the query box, add and hit “Run query”. You can see the logs being returned

Image for post
Image for post
A basic LogQL query using Loki in Grafana

You can add more filters and labels to narrow the logs you want to see. eg: To view the logs coming from the Istio-sidecars in the demo namespace, use the query . We can also look for http requests that don’t end with a desired status code

Image for post
Image for post
Querying a specific container

We can also perform some aggregations over time on the data from the logs

Image for post
Image for post
Aggregating http status codes in the demo namespace in the last 5 minutes. The green line above represents 302 status code and the yellow represents 200 status code. No other status codes in the last 5 mins.

You can do some advance operations and learn more about LogQL here. Your query history is saved so that you can revisit a query in the future if you’d like. The query inspector reports metrics about the query execution.

To generate some logs with a spread, you can use the sample application to perform some actions like add to card and checkout.

Lemme know here or on twitter about your Loki setup journey and about the use cases you are able to complement with Loki.

The Startup

Medium's largest active publication, followed by +732K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store