Sumo Logic with AWS

Sumo Logic AWS Observability Solution — Install Metrics Collectors for EC2

The AWS Observability Solution uses EC2 Host metrics collected using Sumo Logic Host Metrics Source which is a part of Sumo Logic installed collector.

This is the 1st Step in the process of setting up the Sumo Logic AWS Observability Solution on AWS Control Tower Managed Accounts.

How an Installed EC2 Collector looks like on Sumo Logic Console

Method 1: Configure Collector on EC2 using bash

To install Metrics collector on Linux EC2 Instances, follow these steps—

1. Configure installer in a Linux EC2 Instance using the following EC2 User Script — This allows the collector to be configured as soon as the instance is launched. Alternatively, you can also execute this script on your Linux EC2 instances that are already running.
2. This sources.json file is used to configure the various parameters in the collector which is otherwise done using sumo logic UI. It is created and then used to configure Installed Metrics Collector in the script itself.

#!/bin/bash
apt-get update -y
apt-get install wget
yum update -y
yum install wget
cat << EOF > /etc/sources.json
{
  "api.version":"v1",
  "sources":[{
    "name":"EC2 Name",
    "category":"aws/observability/ec2",
    "automaticDateParsing":false,
    "multilineProcessingEnabled":false,
    "useAutolineMatching":false,
    "contentType":"HostMetrics",
    "forceTimeZone":false,
    "filters":[],
    "cutoffTimestamp":0,
    "encoding":"UTF-8",
    "fields":{
      "account":"mediumsample",
      "Namespace":"aws/production/EC2Collector"
    },
    "thirdPartyRef":{
      "resources":[{
        "serviceType":"HostMetrics",
        "path":{
          "type":"NoPathExpression"
        },
        "authentication":{
          "type":"NoAuthentication"
        }
      }]
    },
    "interval":300000,
    "metrics":["CPU_User","CPU_Sys","CPU_Nice","CPU_Idle","CPU_IOWait","CPU_Irq","CPU_SoftIrq","CPU_Stolen","CPU_LoadAvg_1min","CPU_LoadAvg_5min","CPU_LoadAvg_15min","CPU_Total","Mem_Total","Mem_Used","Mem_Free","Mem_ActualFree","Mem_ActualUsed","Mem_UsedPercent","Mem_FreePercent","Mem_PhysicalRam","TCP_InboundTotal","TCP_OutboundTotal","TCP_Established","TCP_Listen","TCP_Idle","TCP_Closing","TCP_CloseWait","TCP_Close","TCP_TimeWait","Net_InPackets","Net_OutPackets","Net_InBytes","Net_OutBytes","Disk_Reads","Disk_ReadBytes","Disk_Writes","Disk_WriteBytes","Disk_Queue","Disk_InodesAvailable","Disk_Used","Disk_UsedPercent","Disk_Available"],
    "sourceType":"SystemStats"
  }]
}
EOF

wget 'https://collectors.sumologic.com/rest/download/linux/64' -O SumoCollector.sh && sudo chmod +x SumoCollector.sh && sudo ./SumoCollector.sh  -q -Vsources=/etc/sources.json -VskipAccessKeyRemoval=true -Vephemeral=true -Vsumo.accessid=xxxxxxxxxxxxxx -Vsumo.accesskey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

The Above Script requires Sumo Logic Access ID & Key to be generated from the Sumo Logic console — steps for the same can be found here.

Once this script runs, the collector will be installed and registered successfully, and DONE!

I have also used default configuration options in the sources.json file. You can read more about the configuration options here. The documentation pretty much disperses the explanation for each of these config options, which are hard to find, so here’s what I customized so far —

1. api.version — Application Programming Interfaces (APIs) to interact with the Sumo Logic platform
2. sources — List of configured Sources for the collector, in this case, 1 per EC2 instance
3. name — Name of the collector that will appear in the Sumo Logic console. The maximum name length is 128 characters.
4. category — Category metadata to use later for querying, e.g. prod/web/apache/access. This data is queried using the ‘_sourceCategory’ key name.
5. fields— Metadata that can be used while searching metrics using Sumo Logic’s query language
6. interval (In milliseconds)— The frequency at which the Source is scanned. Selecting a shorter interval increases the message volume and could cause your deployment to incur additional charges.
7. metrics — Which Metrics should the collector monitor in the given EC2 instance. In my case, all the metrics.

Parameters for Command-Line Installer — ( find more here)

1. -VskipRegistration is only used in a startup script
2. -Vephimeral is used to delete the source from Sumo logic if EC2 is inactive for more than 12 hrs.
3. -Vsumo.accessid=[accessId] -Vsumo.accesskey=[accessKey] — Admins can generate an access ID/access key pair on the Preferences page in the Web application.

Method 2: Configure Collector using Sumo Logic Console — Setup Wizard

1. In Sumo Logic select Manage Data > Collection > Collection. Click on Setup Wizard

2. in the Setup Wizard, select Integrate with Sumo Logic.

3. In the Select Data Type step, Select Host Metrics

4. In the Set Up Collection Step, Select New Collector, and select the appropriate OS (here Linux). A Linux command will appear to install and register the Sumo Logic Collector.

5. Once you run this command on your EC2 Instance via SSH, the collector will be installed and registered to Sumo Logic, you then click on Next.

6. In Configure Source step, enter Source Category (naming convention here), the interval of Collection, and the metrics required to be collected ( CPU / Memory / TCP / Network / DISK), and click on Next.

7. It will take 10–15 minutes for the processing to finish depending on the size of your dataset.

And DONE!

Method 3: Configure Collector using Sumo Logic Console — Add Collector Option

1. In Sumo Logic select Manage Data > Collection > Collection. Click Add Collector, click Installed Collector

2. Then click the link for the appropriate collector to begin the download.

Once the Collector file is downloaded, you can ssh into the required EC2 Instance and move the Installer file into the Instance

3. Add execution permissions to the downloaded Collector file (.sh):

chmod +x SumoCollector.sh

4. Run the script with the parameters that you want to configure. See Parameters for the Command-Line Installer for a description of the parameters. By default, the Collector will be installed in either /opt/SumoCollector or /usr/local/SumoCollector.

sudo ./SumoCollector.sh -q -Vsumo.accessid=<accessId> -Vsumo.accesskey=<accessKey> -Vsources=<absolute_filepath> -Vcollector.name=<name>

In the above command, the -Vsources option can contain the absolute path to the sources.json file configured in method 1 . Later these configurations can also be customized on the Sumo Logic console, once the Collector is installed and running.

Installed EC2 Metrics Collector — Edit Options

Link to an interesting METHOD 4: Using SSM Run Command to Install EC2 Metric Collector

Commands to check the status of Collector

sudo ./collector start
sudo ./collector stop
sudo ./collector status
sudo ./collector restart

Uninstall Collector using the command line

1. In a terminal prompt, change the directory to the collector installation directory. By default, the collector will be installed in either /opt/SumoCollector or /usr/local/SumoCollector.

cd /usr/local/SumoCollector

2. Run the uninstall binary with the -q option. The -q option executes the command without presenting additional prompts.

sudo ./uninstall -q

Miscellaneous Notes :

1. At times after uninstalling, you will have to remove the source from the Sumo Logic Console as well, or it may appear in Error State.
2. If your Instance is offline for more than 12 hours, the collector will go into Error State, if you have used -Vephemeral=true while installing the collector, Sumo logic will delete the source.
3. category — This option is important if you are querying your metric data, as if you change it later, the metrics from that point of time will be stored in the new category and the old ones will remain in the original category.