UI-less Fleet Managed Elastic Agents: A guide

A guide on how to setup Fleet Server and Fleet managed elastic agents in Elasticsearch using only REST API calls, avoiding Kibana UI.

What are Fleet managed Elastic Agents?

Elastic Agents are software that collects logs, metrics and other data from services running on the same host machine.

These agents can also be centrally managed by Fleet in the Kibana UI, where you can create/update Agent Policies, upgrade agent versions, check the current health status of your fleet and add/update Elastic Agent Integrations for app specific behaviour when collecting of logs and metrics.

They also need to communicate through a Fleet Server, the service responsible for rolling out the changes that we apply through the Kibana UI to all relevant agents. It is also responsible for monitoring and reporting new enrollments and unenrollments.

Interactions between Elastic Agents, Fleet Server, Kibana and a Elasticsearch cluster

Why make this guide?

Elasticsearch provides great documentation on how to set up your own Fleet Server and Fleet managed Elastic Agents, but there are a few caveats:

• Reliance on the Kibana UI: When we want to experiment with Elasticsearch deployment and keep erasing its data for a clean state or we just want to automate things to remove human error, using the UI should be avoided as much as you can. By automating the set up process of the agents, it enables us to use best practices such as GitOps, integrating the set up steps into our CI/CD processes.
• Monitoring the host machine with the Fleet Server: Elastic Agents should be installed at the host level in order to get the most accurate metrics and applications running on the host machine. However, we might not want to monitor the host where the fleet-server is running on. The Kibana UI assumes that you will install the fleet server at the host level, which may or may not be possible or desirable. It might be less important or relevant for us to monitor the host machine the fleet server is running on, maybe because it is not as important as the other hosts (Like a “frontend” host machine that hosts frontend/UI applications and services like Kibana).
• Unclear complete setup for containerized Fleet Server and Elastic Agents: In the Elastic Documentation guide on Elastic Agents in a container, we get a nice step by step guide on how to run elastic-agents in containers. The catch is … it uses Kibana’s Fleet UI. It assumes you would like to create an agent policy and Elastic Agent integration through Kibana. And as stated before, sometimes we would like to avoid it, so that we can automate enrollment of Elastic Agents, the creation of new Agent Policies and the addition or editing of Elastic Agent integrations.

By setting up Fleet Server and Fleet managed Elastic Agents using REST API calls we create a consistent and reproducible state of Fleet, reducing human error and reproduce the same state consistently across multiple different environments(e.g.: Dev, Prod, QA, …).

Please note that Elasticsearch has documentation on the Fleet REST API, that gives us enough tools to set up our own Fleet Server and Fleet managed Elastic Agents as well as everything the Kibana UI can do with Fleet. There is also a declarative way to add Agent policies through the kibana.yml here, however, from my experience, updating the agent policies in an already set up Elasticsearch Cluster and Kibana was a bit “flacky”.

The problem lies in the lack of a streamlined step by step guide on setting up a Fleet Server and Fleet managed Elastic Agents using ONLY the provided Fleet REST APIs and not relying on the Kibana UI in ANY of the steps.

With the “why” out of the way, let’s get started with what is necessary to have a Fleet Server and Fleet managed Elastic Agents.

What we will need

In order to have our own Fleet Server and Fleet managed Elastic Agents, we will have to create:

• An Elasticsearch Cluster
• A Kibana instance
• Certificates and keys for all services (Elasticsearch, Kibana and Fleet Server)

First, let’s create our own self signed CA to create the certificates and keys so that the services that we will be using in this guide can authenticate themselves.

Self Signed CA and certificates

Elasticsearch has a nice tool for creating self signed CA’s and other certificates that work well on Elastic products. To configure SSL/TLS using this tool, you can check out their guide, but in this one we will be creating our own CA, certificates and certificate keys using openssl.

Let’s start with this example of creating our own CA certificate and key with an expiration of 10 years:

mkdir ca && \
openssl req -x509 \
-newkey rsa:4096 \
-keyout ca/ca.key \
-out ca/ca.crt \
-sha256 -days 3650 \
-noenc -subj "/C=PT/ST=Lisbon/L=Lisbon/O=Marionete/OU=IT/CN=marionete"

Now that we have our own self signed CA, we can create our own certificates and keys for the services.

In this guide, we will need create the following services: Elasticsearch, Kibana and a Fleet Server. We will create a key and a certificate for each service to authenticate with the other services.

In the next few steps, <service> can take the following values: elastic, kibana and fleet-server.

• Include the CA certificate which we just created in all services:

mkdir certs && cp ca/ca.crt certs

• Generate a key for the service

openssl genrsa -out certs/<service>.key 2048

• Create the CSR for the key

openssl req -new -key certs/<service>.key -out certs/<service>.csr -subj "/C=PT/ST=Lisbon/L=Lisbon/O=Marionete/OU=IT/CN=<service>"

• Sign the key with our self signed CA

openssl x509 -req \
-in certs/<service>.csr \
-CAkey ca/ca.key \
-CA ca/ca.crt \
-CAcreateserial \
-out certs/<service>.crt \
-days 3650 \
-sha256 \
-extfile <(printf "subjectAltName=DNS:<service>,DNS:localhost\nextendedKeyUsage=serverAuth,clientAuth\nkeyUsage=digitalSignature,keyEncipherment,keyAgreement\nsubjectKeyIdentifier=hash")

Once we have our certificates and keys for all services, we will deploy them using docker-compose.

Deploying Elastic, Kibana and a Fleet Server

We will now deploy a kibana instance, a single Elasticsearch node and a single fleet server using the following docker-compose.yml:

version: "2.2"

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.2
    container_name: elasticsearch-setup-container
    volumes:
      - ./certs:/usr/share/elasticsearch/config/certs:z
    user: "1000"
    command: >
      bash -c '
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca.crt https://elastic:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca.crt -u elastic:elastic -H "Content-Type: application/json" https://elastic:9200/_security/user/kibana_system/_password -d "{\"password\":\"kibana\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/ca.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  elasticsearch:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.2
    container_name: elastic
    user: "1000"
    volumes:
      - ./certs:/usr/share/elasticsearch/config/certs:z
      - ./elastic-data:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    environment:
      - node.name=elastic
      - cluster.name=fleet-elasticsearch
      - ELASTIC_PASSWORD=elastic
      - bootstrap.memory_lock=false
      - discovery.type=single-node
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/elastic.key
      - xpack.security.http.ssl.certificate=certs/elastic.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.http.ssl.client_authentication=optional
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/elastic.key
      - xpack.security.transport.ssl.certificate=certs/elastic.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.client_authentication=optional
    mem_limit: 2147483648 # ~ 2gb
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca.crt https://elastic:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      elasticsearch:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:8.12.2
    container_name: kibana
    volumes:
      - ./certs:/usr/share/kibana/config/certs:z
      - ./kibana-data:/usr/share/kibana/data
      - ./kibana.yml:/usr/share/kibana/config/kibana.yml:Z
    ports:
      - 5601:5601
    environment:
      - SERVER_NAME=kibana
      - ELASTICSEARCH_HOSTS=https://elastic:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=kibana
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca.crt
      - SERVER_SSL_ENABLED=true
      - SERVER_SSL_CERTIFICATE=config/certs/kibana.crt
      - SERVER_SSL_KEY=config/certs/kibana.key
      - SERVER_SSL_CERTIFICATEAUTHORITIES=config/certs/ca.crt
    mem_limit: 2147483648 # ~ 2gb
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -I -s --cacert config/certs/ca.crt https://kibana:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  fleet-server:
      depends_on:
        kibana:
          condition: service_healthy
        elasticsearch:
          condition: service_healthy
      image: docker.elastic.co/beats/elastic-agent:8.12.2
      container_name: fleet-server
      volumes:
        - ./certs:/certs:z
      ports:
        - 8220:8220
      restart: always
      user: "1000"
      environment:
        - FLEET_ENROLL=1
        - FLEET_SERVER_POLICY_ID=fleet-server-policy
        - FLEET_SERVER_ENABLE=1
        - KIBANA_FLEET_SETUP=1
        - KIBANA_HOST=https://kibana:5601
        - FLEET_URL=https://fleet-server:8220
        - FLEET_SERVER_ELASTICSEARCH_HOST=https://elastic:9200
        - FLEET_CA=/certs/ca.crt
        - KIBANA_FLEET_USERNAME=elastic
        - KIBANA_FLEET_PASSWORD=elastic
        - FLEET_SERVER_CERT=/certs/fleet-server.crt
        - FLEET_SERVER_CERT_KEY=/certs/fleet-server.key
        - FLEET_SERVER_ELASTICSEARCH_CA=/certs/ca.crt
        - KIBANA_FLEET_CA=/certs/ca.crt

This docker compose file deploys a single node elasticsearch cluster with security enabled, set up and deploy kibana and deploy a fleet server (Don’t forget to create elastic-data and kibana-data directories for the volumes).

Please note that this docker compose file should be used for demonstration purposes only. When deploying in production, secrets like certificate key passwords and user passwords should not be present statically in your docker compose file.

The kibana.yml file should contain at least the following content:

xpack.encryptedSavedObjects.encryptionKey: "random-string-above-32-or-more-characters"
server.host: "0.0.0.0"
xpack.fleet.packages:
  - name: fleet_server
    version: latest
  - name: system
    version: latest
xpack.fleet.agentPolicies:
  - name: Fleet-Server-Policy
    id: fleet-server-policy
    namespace: default
    monitoring_enabled: []
    package_policies:
      - name: fleet_server-1
        package:
          name: fleet_server

Once the services are deployed, we will set up Fleet Settings and enroll fleet managed Elastic Agents.

Fleet Managed Elastic Agents Without the UI

To deploy an Elastic Agent that is managed by Fleet we will need to:

• Create an agent policy for the elastic agent to enroll into.
• Add an Elastic Agent integration to our agent policy.
• Setup the Default output and fleet server hosts.
• Install the Elastic Agent on the machine we want to monitor.

All of these steps are documented by Elastic using the UI but we will be using REST API calls with curl and jq commands instead.

Let’s start with creating a new agent policy. According to the Elastic documentation, this step is rather straight forward. With the following command, we create a new agent policy named “Elastic-policy” with a custom identifier (this will come in handy in the next few steps).

curl -k -s -u "elastic:elastic" \
    -XPOST -H "kbn-xsrf: kibana" -H "Content-type: application/json" \
    "https://localhost:5601/api/fleet/agent_policies" \
    -d '{"id":"elastic-policy","name":"Elastic-Policy","namespace":"default","monitoring_enabled":["logs","metrics"]}'

If successful, the command will return a json of the created Agent Policy. After creating the agent policy with a custom identifier, we can add Elastic Agent integrations to it with this command:

curl -k -s -u "elastic:elastic" \
    -XPOST -H "kbn-xsrf: kibana" -H "Content-type: application/json" \
    "https://localhost:5601/api/fleet/package_policies" \
    -d '{"name":"Elastic-System-package","namespace":"default","policy_id":"elastic-policy", "package":{"name": "system", "version":"1.54.0"}}'

Next, we add a Fleet Server host:

curl -k -s -u "elastic:elastic" \
    -XPUT -H "kbn-xsrf: kibana" -H "Content-type: application/json" \
    "https://localhost:5601/api/fleet/settings" \
    -d '{"fleet_server_hosts": ["https://localhost:8220"]}'

To make all Elastic Agents send to our Elasticsearch cluster, we need to define the default output for fleet with the following command:

curl -k -s -u "elastic:elastic" \
    -XPUT -H "kbn-xsrf: kibana" -H "Content-type: application/json" \
    "https://localhost:5601/api/fleet/outputs/fleet-default-output" \
    -d '{"hosts": ["https://localhost:9200"], "config_yaml": "ssl.verification_mode: certificate\nssl.certificate_authorities: [\"/path/to/ca/ca.crt\"]"}'

Note that “/path/to/ca/ca.crt” will be the same for all Elastic Agents, so make sure the CA certificate has the same path on all machines that have an Elastic Agent installed.

Next, we download the appropriate Elastic Agent binary version for our host machine (in this case linux with x86_64 architecture) and unarchive it.

wget https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.12.2-linux-x86_64.tar.gz

tar xvzf elastic-agent-8.12.2-linux-x86_64.tar.gz

And finally, we get our agent policy enrollment token and use it when installing the Elastic Agent in our host machine.

ENROLLMENT_TOKEN=$(curl -k -s \
  -u elastic:elastic \
  https://localhost:5601/api/fleet/enrollment_api_keys | \
  jq -r '.items[] | select(any(.; .policy_id == "elastic-policy")) | .api_key')

sudo ./elastic-agent-8.12.2-linux-x86_64/elastic-agent install \
    --base-path=/path/to/install/dir \
    --url=https://localhost:8220 \
    --enrollment-token=${ENROLLMENT_TOKEN} \
    --certificate-authorities=/path/to/ca/ca.crt \
    --force

Note that “/path/to/install/dir” is where the Elastic Agent will store its logs and configurations. And since we are using sudo, only sudo users can access that directory and its contents.

Once the installation is complete, you should see a new Elastic Agent appear in the Fleet page and system metrics about the host machine in this dashboard.

Summary

By following this guide, you have set up and deployed a Fleet Server, set up and deployed Fleet managed Elastic Agents, deployed a single node Elasticsearch cluster with security, set up and deployed a Kibana instance without relying on the Kibana UI and using REST API calls.

By reducing the reliance of the Kibana UI, we reduce human error and allows us to reproduce the state of Agent Policies, Elastic Agent Integrations and Fleet Settings across different environments consistently.

It also enables us to integrate Fleet into our CI/CD processes, automating the set up and deployment of Fleet Servers and Elastic Agents.