Dockerizing Logstash: A Step-by-Step Guide

If you’re here, I can safely assume you know what Docker and Elastic Search Logstash are. With that assumption, I’m confident that the following step-by-step guide will assist you in setting up and running Logstash jobs within a Docker container. By the end of this guide, you’ll be well on your way to achieving a production-ready environment.

Without further ado, let’s begin. For the purpose of this guide, we will overlook the internal implementation of the Logstash job itself and focus on configuring Logstash to run within a Docker container.

Step 1: Empty Project

In this guide, we will utilize a Dockerfile containing instructions to build an image that can be employed for both local running and deployment to a live environment.

Begin by creating an empty project directory and adding a file named: Dockerfile

Step 2: Logstash Image

Elasticsearch offers and manages Docker images for all of its stacks, including Logstash. Its official image is highly capable and suitable for fulfilling a wide range of production use cases.

As of now, the most recent version of Elasticsearch is v8.9. Let’s utilize the image from their Docker registry. https://www.docker.elastic.co/r/logstash

You can use any available version and architecture based on your preference.

Note: If you’re planning to use Logstash as output for your jobs, make sure to verify the compatibility of your Logstash version with the Elasticsearch version. https://www.elastic.co/support/matrix#matrix_compatibility

Update theDockerfile with below content.

FROM docker.elastic.co/logstash/logstash:8.9.0

Step 3: Structuring Configuration

The Logstash image comes with a default configuration file. To proceed, let’s delete this file from the directory /usr/share/logstash/pipeline/logstash.conf.

FROM docker.elastic.co/logstash/logstash:8.9.0

# delete default configuration
RUN rm -f /usr/share/logstash/pipeline/logstash.conf

When it comes to configuration there are 2:

1. Logstash Configurations
2. Pipeline Configurations

Let’s create a directory structure in the root directory of your project as below.

.(root)
./logstash/
./logstash/config/
./logstash/pipeline/
Dockerfile

Directory ./logstash/config/ going to host all your logstash related configurations. This includes logstash.yml, pipelines.yml, so on.

Directory ./logstash/pipeline/ going to host all your job pipeline configurations.

Step 4: Job Configuration

Let’s add a simple logstash pipeline job configuration under the directory ./logstash/pipeline/ with a file name your-pipeline-name.conf as below.

input {
}

filter {
}

output {
}

You can add your pipeline job configurations based on your requirement. But for now, I’m just adding an empty logstash pipeline.

Step 5: Logstash Configuration

Let’s add global logstash configuration logstash.yml file under the directory ./logstash/config/ with content below.

node.name: logstash_project_name
api.environment: dev

You can add additional configurations based on your requirement.

Let’s add pipeline configuration pipeline.yml in the same location as below.

- pipeline.id: your-pipeline-name
  path.config: "/usr/share/logstash/pipeline/your-pipeline-name.conf"

Here you need to define a unique id for the pipeline and the location of the pipeline job configuration that we created before. You may notice that the directory mentioned here is misleading. You’re right, but we are going to move the configuration to the location as such in the docker image.

Step 6: Configure Docker

So far we have created our project with logstash configuration and pipeline job configuration. Now we need to move them to the docker image so that when it is built and run logstash picks them up.

Let’s update the Dockerfile with a few more instructions to do so.

FROM docker.elastic.co/logstash/logstash:8.9.0

# delete default configuration
RUN rm -f /usr/share/logstash/pipeline/logstash.conf

# copy configurations
COPY logstash/pipeline/ /usr/share/logstash/pipeline/
COPY logstash/config/ /usr/share/logstash/config/

Step 7: Build and Run

Now the docker instruction and project configurations are ready. Let’s build the docker image and run it with the below commands in the terminal from the root project location.

# to build docker image
docker build -t logstash-job .

Once built successfully, let’s run it with below command in the terminal.

# to run docker image built before
docker run -t logstash-job

Conclusion

I hope this has provided you with useful and detailed instructions to successfully set up and run Logstash on a Docker container. Thank you for being a part of this journey in establishing Logstash and Docker integration. Happy coding!