Stream multiple CloudWatch log groups in ElasticSearch(ELK)

You can configure a CloudWatch Logs log group to stream data it receives to your Amazon Elasticsearch Service (Amazon ES) cluster in near real-time through a CloudWatch Logs subscription.

Prerequisite:-

1. ES Cluster : https://docs.aws.amazon.com/cli/latest/reference/es/create-elasticsearch-domain.html

Subscribe cloudwatch log groups

step 01. Login aws console go to cloudwatch log groups and select your log group one by one and stream to Amazon Elasticsearch services.

Step 02. Find a new lambda was created with name of LogsToElasticsearch_ and find var indexName.

Existing code

// index name format: cwl-YYYY.MM.DD
     var indexName = [
         'cwl-' + timestamp.getUTCFullYear(),           // year
         ('0' + (timestamp.getUTCMonth() + 1)).slice(-2),  // month
         ('0' + timestamp.getUTCDate()).slice(-2)          // day
     ].join('.');

Replace this code

// index name format: cwl-YYYY.MM.DD
     var appName =payload.logGroup.toLowerCase();
     var indexName = '';
     var indexName = [
         'cwl-'+ appName + '-' + timestamp.getUTCFullYear(), // year
         ('0' + (timestamp.getUTCMonth() + 1)).slice(-2), // month
         ('0' + timestamp.getUTCDate()).slice(-2), // day
     ].join('.');

Save it.

Check ES indices and there is multiple indices with group name

step 03. Login kibana dashboard and create multiple index pattern.

it’s done.