OpenSearch with Fluent Bit. Automation of index deletion.

This guide will help you to configure Fluent Bit integration with OpenSearch and automate index deletion after a certain period of time.

Fluent Bit

Configure Fluent Bit. Pay attention to the Output plugin configuration:

[OUTPUT]
    Name            opensearch
    Match           kube.*
    Host            opensearch-master
    Port            9200
    HTTP_User       ${OPENSEARCH_USER}
    HTTP_Passwd     ${OPENSEARCH_PASSWORD}
    Logstash_Format On
    Logstash_Prefix fluent-bit-kube
    Logstash_DateFormat %Y.%m.%d
    Time_Key        @timestamp
    Type            flb_type
    Replace_Dots    On
    Retry_Limit     False
    Trace_Error     On

fluent-bit-kube-date will be the name pattern of an index created by Fluent Bit.

Index State Management

To manage the index lifecycle, create an index policy via the OpenSearch Dev Tools Console:

PUT _plugins/_ism/policies/delete_index_policy
{
    "policy": {
        "description": "Delete indices",
        "error_notification": null,
        "default_state": "delete",
        "states": [
            {
                "name": "delete",
                "actions": [
                    {
                        "timeout": "7d",
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "fluent-bit-kube-*"
                ],
                "priority": 100
            }
        ]
    }
 
}

ism_template will check index names against index_patterns and that policy will be attached to those indexes automatically. Thus all new indexes with that policy will be removed automatically in 7 days.

Check the new index with the attached policy:

GET /fluent-bit-kube-*

OpenSearch Index State Management (ISM) is similar to Elastic Stack Index Lifecycle Management (ILM). The main difference here is that OpenSearch policy attaches automatically to indexes without the necessity of creating Index Templates like it is in Elastic Stack.