How To Log NestJS Applications in a Distributed System with the Opensearch Stack and Kubernetes — Part 4
Collecting Logs with fluentbit and showing them in Opensearch Dashboards in Kubernetes
In one of my latest articles, I explained the importance of Centralized Logging. I demonstrated how you could realize custom logging in a NestJs application using winston and described the central role of fluentbit.
In this article I want to show you how you can collect custom and stdout logs, push them into an Opensearch database, and visualize them in Opensearch-Dashboards.
Opensearch is the community offshoot of Elasticsearch and thus has a lot of similarities and commonalities with Elasticsearch.
Creating a Kubernetes Cluster for Local Use
To become more experienced with Kubernetes and improve our workflow, installing a local Kubernetes environment is key. For this, I use kind in most cases. Check out one of my articles to become familiar with kind.
You can check out the official documentation to come more familiar with kind.
We can use the following configuration file to configure a local Kubernetes cluster with one control-plane and three worker-nodes.
# kind.config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: nestjs-logging
nodes:
- role: control-plane
- role: worker
- role: worker
- role: workerNow we can start the Kubernetes cluster through kind with the following command.
kind create cluster --config=kind.config.yamlAfter initialization of the cluster, the .kubeconfig will automatically be appended to the profile directory, so we can run kubectl commands like kubectl get pods -A.
After we finish our work, we can delete the cluster with the following command.
kind delete cluster --name nestjs-loggingCreating a local Kubernetes cluster will be the basis for this article. We have to ensure that Docker and kind are installed, as well as kubectl CLI and Helm.
Collecting stdout logs with Opensearch Stack in Kubernetes
Collecting stdout logs is very simple with Opensearch and Helm, because with the helm charts we can easily configure all parts. So, opensearch, and opensearch-dashboards are already connected out of the box. For fluentbit we need to do some configurations.
After we have created a local Kubernetes cluster with kind in the first step, we can deploy the Opensearch charts on it using the following commands.
helm repo add opensearch-project-helm-charts https://opensearch-project.github.io/helm-charts
helm repo add fluent https://fluent.github.io/helm-charts
helm repo updateSince we need to store a configuration for fluentbit so that logs reach the Opensearch database, we can also make additional settings for the other two charts. Here we find a sample configuration:
# values.yaml
opensearch:
opensearchJavaOpts: "-Xmx1024M -Xms1024M"
replicas: 3
persistence:
enableInitChown: false
enabled: true
size: 10Gi
hostAliases:
- ip: "127.0.0.1"
opensearch-dashboards:
resources:
requests:
cpu: "100m"
memory: "512M"
limits:
cpu: "500m"
memory: "1024M"
fluent-bit:
config:
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
# multiline.parser docker, cri
parser docker
Tag kube.*
Mem_Buf_Limit 700MB
Skip_Long_Lines On
Refresh_Interval 5
[INPUT]
Name systemd
Tag host.*
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Read_From_Tail On
outputs: |
[OUTPUT]
Name opensearch
Match *
Host opensearch-cluster-master-headlessl
Port 9200
Index fluent_bit_kube
Logstash_Format On
Retry_Limit 10
Buffer_Size 512KB
Replace_Dots On
Suppress_Type_Name On
tls On
tls.verify Off
http_User admin
http_Passwd adminI prepared an Opensearch chart example for this configuration, which you can find in the sample Git Repository.
With this configuration, we can deploy the stack.
helm upgrade --install opensearch opensearch -f ./opensearch/values.yaml After a few seconds, we can see running pods in the default namespace with kubectl get pods.
Access the Opensearch Dashboards UI
To access the Opensearch-Dashboards UI, run the following command to forward Dashboards’ default port 5601 to port 8080 for local use.
kubectl port-forward service/opensearch-opensearch-dashboards 8080:5601Now we can hit http://localhost:8080/ in our browser to access Opensearch-Dashboards.
Default username and password for this chart is admin (for both).
After this, we get asked to select a tenant. If we choose Global, then we share our work with another user. The Private tenant is for personal use and work. For this scenario, it is irrelevant which one we select.
After entering the tenant, we reach the Starting Page.
Creating an Index Pattern
After opening the sidebar and navigating to Management > Stack Management > Index Patterns > Create index pattern, we can create an Index Pattern to filter incoming logs.
We could create a wildcard with logstash-* to view all incoming logs.
After this, we need to navigate to Discover in the sidebar. There we can see all the logs of our local Kubernetes cluster.
To filter by certain criteria, we can either use the filter bar on the left or familiarize ourselves with DQL to use the search bar.
For example, to evaluate the logs of a particular container, we can use kubernetes.container_name : “fluent-bit” as DQL. This is especially useful when there are multiple pods of the same container, as in a DaemonSets.
To delete all created dependencies, we can run the following command. For the custom-log approach below, we will build a separate helm chart.
helm uninstall opensearchCollecting custom logs with the Opensearch Stack in Kubernetes with a Sidecar
The other way is using a sidecar pattern and running a log-forwarding container next to the application container within the same pod. We need to use this pattern because winston writes its logs to the filesystem. Also, sidecars extend the functionality of a main container without changing it. The application logs will be transferred to the Opensearch database via a fluentbit sidecar container.
The logs of the main container are shared with the sidecar container via an emptyDir Volume.
apiVersion: apps/v1
kind: Deployment
# ...
spec:
template:
spec:
containers:
- name: main-container
# ...
volumeMounts:
- name: log-volume
mountPath: /usr/app/logs
- name: sidecar-container
# ...
volumeMounts:
- name: log-volume
mountPath: /usr/app/logs
# ...
volumes:
- name: log-volume
emptyDir: { }After we have created a local Kubernetes cluster with kind (see above), we can use a custom helm chart to deploy the Opensearch Stack and two NestJS microservices that generate some custom logs. The deployment can be found here.
To install this chart we need to run the following command.
helm upgrade --install opensearch-sidecar opensearch-sidecar -f ./opensearch-sidecar/values.yamlNow many different pods get spawned.
The connections between these microservices are shown in this architecture overview.
So, our frontend and backend service write logs to /usr/apps/logs in the filesystem. The task of our sidecar is to take these logs and send them on. For this we use a simple fluentbit container.
apiVersion: apps/v1
kind: Deployment
# ...
spec:
template:
spec:
containers:
- name: main-container-with-winston
# ...
volumeMounts:
- name: log-volume
mountPath: /usr/app/logs
# ...
- name: fluentbit
image: "fluent/fluent-bit:2.1-debug"
ports:
- name: metrics
containerPort: 2020
protocol: TCP
env:
- name: FLUENT_UID
value: "0"
volumeMounts:
- name: config-volume
mountPath: /fluent-bit/etc/
- name: log-volume
mountPath: /usr/app/logs
volumes:
- name: log-volume
emptyDir: { }
- name: config-volume
configMap:
name: fluentbit-sidecarLike a fluenbit DaemonSet the container needs a configuration mounted via a ConfigMap.
# backend Chart
# sidecar.configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: fluentbit-sidecar-backend
data:
fluent-bit.conf: |
[SERVICE]
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
Flush 5
Daemon Off
Log_Level debug
Parsers_File parsers.conf
[INPUT]
Name tail
Path /usr/app/logs/*.log
multiline.parser docker, cri
Tag custom.*
Mem_Buf_Limit 300MB
Skip_Long_Lines On
[FILTER]
Name parser
Parser docker
Match custom.*
Key_Name log
Reserve_Data On
Preserve_Key On
[FILTER]
Name modify
Match *
[OUTPUT]
Name opensearch
Match *
Host opensearch-cluster-master-headless.default.svc.cluster.local
Port 9200
Index fluent_bit_host
Logstash_Format On
Logstash_Prefix custom
Retry_Limit 10
Buffer_Size 512KB
Replace_Dots On
Suppress_Type_Name On
tls On
tls.verify Off
http_User admin
http_Passwd admin
parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
Decode_Field_As escaped_utf8 log do_next
Decode_Field_As json logAs we can see, the fluentbit container observes Paths usr/app/logs/*.log.
Very import is changing the [Output.Host] to the host of our needs. In this case, the host represents the Kubernetes Service of Opensearch with Port 9200.
We can further customize the output plugin by following the official documentation.
We could add more labels or label_keys. Or we could add a filter to add custom labels or service names.
[FILTER]
Name modify
Match *
Add service_name database-serviceAfter all pods are initialized, we can portforward the Opensearch-Dashboards container from default port 5601 to port 8080 for local use.
# Portforward
kubectl port-forward service/opensearch-sidecar-opensearch-dashboards 8080:5601After opening the sidebar and navigating to Management > Stack Management > Index Patterns > Create index pattern, we can create an Index Pattern to filter incoming logs. We could create a wildcard with custom-* to view all incoming logs.
After this, we need to navigate to Discover. There we can see the custom logs of our NestJS microservices.
We can also generate some error logs with the frontend and backend apps. For this, we need to portfoward the port of the frontend app to get access via localhost.
# portforward
kubectl port-forward service/opensearch-sidecar-frontend-service 8081:80
# Open UI
open http://localhost:8081This application gets some information about Star Wars from the backend app. To cause some errors, we need to hit the Cause an error button.
After refreshing the query in Opensearch-Dashboards, we can see the error logs.
Note, you can create custom dashboards in Dashboard to show only data fields with necessary information.
Opensearch & Opensearch-Dashboaards has so much going for it, which makes centralized logging far easier. All information about this can be found on the documentation page.
Conclusion
Logging has a central role in distributed systems, and in case of system failures, we want to have an overview to see which applications generate certain messages.
Fluentbit, Opensearch, and Opensearch-Dashboards help us to generate this approach. With fluentbit we have the possibility to customize our logs via the output plugin. We can add additional labels and tags.
But consider that audit logs can be very noisy, and it can be very expensive to log all actions. For this, we can generate custom logs collected via a sidecar to fine-tune this approach for our environment.
Thanks for reading! Follow me on Medium, Twitter, or Instagram, or subscribe here on Medium to read more about DevOps, Agile & Development Principles, Angular, and other useful stuff. Happy Coding! :)
