(AWS )Elastic-Search + CloudWatch(logs) + Kibana

Setting up Elastic search cluster in AWS

Amazon ES is a managed service provided by AWS to deploy, secure, and run ES cost effectively at scale. We can build, monitor, and troubleshoot our applications using various tools . The service provides support for open source ES APIs, managed Kibana, integration with Logstash and other AWS services, and built-in alerting and SQL querying.

With Amazon ES we can also send logs from Cloud watch to ES cluster and visualize it in kibana. For this a lambda function is used to sends logs from cloud watch log groups to ES cluster.

Note: Here we will be referring ElasticSearch as ES in entire article.

There are mainly four steps required in creation of Amazon ES cluster:

Step 1 > Choose Deployment type : This is first step in creation of ES cluster and it has three options as Production, Development & testing and custom.

Deployment types specify common settings for our use case. After creating the domain, we can change these settings at any time.

We will be using production type as this meets our use case and also production type forces to use at least 2 AZ’s for fault tolerance and also a dedicated master node apart from data nodes. If we want only elastic search endpoint then we can use Development and testing option which has 1 AZ support.

Step 2 > Configure Domain : In this step we will provide domain name for our ES cluster and also data nodes configurations. Configurations like number of AZ’s instance type of our nodes and number of data nodes can be selected . Below is the link which give details about various instance types and costing

Amazon Elasticsearch Service Pricing — Amazon Web Services

Data Storage : Here we choose a storage type for your data nodes. If we choose the EBS storage type, multiply the EBS storage size per node by the number of data nodes in your cluster to calculate the total storage available to your cluster. It does not apply to any dedicated master nodes in the cluster. By default it is 10GiB we can increase it depending on our use case. I will just drop the link below to understand the EBS storage and IOPS which comes with it.

Amazon EBS volume types

Master Nodes : For production AWS forces us to have a dedicated master node in our cluster and which is restricted to either 3 or 5 for High Availability which follows the rule of quorum incase any master goes down. In clusters where we don’t have any dedicated master node then one of our data nodes will be acting as Master nodes/Data node. It is advisable to have dedicated Master node to avoid cluster going red in case of index spike.

Step 3 > Configure access and security : Here we select either internet or VPC access. To enable VPC access, we use private IP addresses from your VPC, which provides an inherent layer of security.

We control network access within your VPC using security groups. Optionally, we can add an additional layer of security by applying a restrictive access policy. Internet endpoints are publicly accessible.

If we select public access, we should secure your domain with an access policy that only allows specific users or IP addresses to access the domain.

Here we will be choosing public access but beware to restrict access it to specific ip’s or specific IAM roles or both. Here in pur setup we will restrict it to specific ip using access policy.

Fine–grained access control

Fine-grained access control provides numerous features to help us keep our data secure. Features include document-level security, field-level security, read-only Kibana users, and Kibana tenants. Fine-grained access control requires a master user.

Here we will create Master user and password which will be required for login into kibana dashboard. Once we have logged into kibana , then we can create separate users which can have only read access in kibana. A scenario can be were we want certain team members to only visualize the dashboard and not make any changes like crate index patterns or delete index for such use case we can leverage this feature with fine grained access control.

Access policy : Access policies control whether a request is accepted or rejected when it reaches the Amazon ES Service domain. If we specify an account, user, or role in this policy, we must sign your requests.

Here we will select JSON defined access policy and restrict traffic to our domain only from specific ip addresses. Below is the snippet of access policy which provides access to specific ip only.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "es:ESHttp*"
      ],
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "192.0.2.0/24"
          ]
        }
      },
      "Resource": "arn:aws:es:us-west-1:987654321098:domain/test-domain/*"
    }
  ]
}

Encryption : AWS ES’s also offers encryption-at-rest as a simple checkbox. In EC2, this is offered for EBS volumes, and they’ve extended that capability to AWS ES. Impressively, they offer the same simplicity even when you’ve selected instance stores for your data storage.

Step 4 > Review : On the review page we will review our options and check for any changes as some of the configurations like public access or VPC access can’t be changed once the cluster is up. Similarly if we have not enabled fine grained access control it can’t be enabled once the cluster is up and domain status is active.

So after very careful review we will confirm and wait for setup to be complete. Once we have confirmed we will get screen as below. Here we can see that Domain status is loading.

— 🤗 — 🥴 — 😵- ONE ETERNITY LATER — 😖 — 💀-☠️

HaHA……. Just joking it takes around 10 to 15 mins for this setup to complete and Domain status to change from pending to active status and ready to serve the users.

Streaming Logs from Cloud Watch log groups to Amazon ES cluster

Now we are done with cluster setup and we will streaming logs from cloud watch log groups to ES cluster.

To subscribe a log group to Amazon ES

1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
2. In the navigation pane, choose Log groups. then choose the name of the log group.
3. Choose Actions, Create Elastisearch subscription filter.

4. Choose whether you want to stream to a cluster in this account or another account.For Amazon ES cluster, choose the cluster you created in the previous step.

5. Under Lambda Function, for Lambda IAM Execution Role, we will seletc the IAM role that Lambda should use when executing calls to Amazon ES, and then choose Next.

6. The IAM role we will choose must fulfill these requirements. It must have lambda.amazonaws.com in the trust relationship. It must include the following policy:

If the target Amazon ES domain uses VPC access, the role must have the AWSLambdaVPCAccessExecutionRole policy attached. This Amazon-managed policy grants Lambda access to the customer’s VPC, enabling Lambda to write to the Amazon ES endpoint in the VPC.

7. For Log Format, choose a log format. For Subscription Filter Pattern, type the terms or pattern to find in your log events. This ensures that you send only the data you are interested in to your Amazon ES cluster. For more information, see Creating Metrics From Log Events Using Filters.

8. (Optional) For Select Log Data to Test, select a log stream and then choose Test Pattern to verify that your search filter is returning the results you expect. Choose Start Streaming. Once the streaming starts we can see the subscription create as below.

9. To verify go to Amazon ES console and click on indices option to see if indexes are being created or not. It gets created and has prefix as cwl-* example cwl-2020.07.12

Sending Multiple Log Groups log to same ES cluster

When we create subscription to send logs AWS created a lambda function which is written in node js to fetch logs from cloud watch and relay it ES cluster. But by default it can send logs from only one log group. To be able to send logs from multiple log groups lets do some code changes in lambda.

We will navigate to lambda and select our lambda function and go through the code and comment the following existing lines and replace it with lines below the commented line and save our lambda function. This will allow lambda to send logs from multiple cwl to ES cluster. So new indices name becomes as cwl + log_group name + Date

Congratulations ……….!! 😊 🤗 We are done with our ES cluster setup and streaming of logs.

In my next article I will be writing about kibana part of it. Few things we can expect is creating multiple kibana users based on access. Creating index patterns and visualize it on kibana dashboard. Configure kibana to be accessed by various IAM groups.

For any queries, suggestions and detailed discussions about this article and related technologies. please mail me at at yadavrajofficial442@gmail.com