ELK Stack with Fluent Bit instead of Logstash. Part 2— Index lifecycle management (ILM) without rollover policy.

The article was initially published here: https://epam.github.io/edp-install/operator-guide/kibana-ilm-rollover/

This flow was tested on Kibana 7.17. It should be identical for Kibana 8.

Part 1 of this guide describes index lifecycle with the rollover policy (index increment)

Part 2 of this guide covers an alternative way of the index lifecycle management without rollover policy (no index increment)

ILM Without Rollover Policy

It is also possible to manage index lifecycle without rollover indicated in the policy. If this is the case, this section will explain how to make the index follow this pattern: fluent-bit-kube-2023.03.18 and got deleted after a certain period of time.

The main drawback of this method is that the indices can be managed only by their creation date.

1. Create a Policy without rollover but with indices deletion:

PUT _ilm/policy/fluent-bit-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "7d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

2. Create an index template with the rollover_alias parameter:

PUT /_index_template/fluent-bit
{
  "index_patterns": ["fluent-bit-kube-*"],
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "fluent-bit-policy",
          "rollover_alias": "fluent-bit-kube"
        },
        "number_of_shards": "1",
        "number_of_replicas": "0"
      }
    }
  }
}

3. Change the Fluent Bit [OUTPUT] config to this one:

[OUTPUT]
    Name            es
    Match           kube.*
    Host            elasticsearch-master
    Port            9200
    HTTP_User       ${ES_USER}
    HTTP_Passwd     ${ES_PASSWORD}
    Logstash_Format On
    Logstash_Prefix fluent-bit-kube
    Logstash_DateFormat %Y.%m.%d
    Time_Key        @timestamp
    Type            flb_type
    Replace_Dots    On
    Retry_Limit     False
    Trace_Error     On

Restart Fluent Bit pods.

Fluent Bit will be producing a new index every day with the new date like in fluent-bit-kube-2023.03.18 . The index deletion will be performed according to the policy.