Configure Monitoring and Alerting on Your Chainlink Node (AWS)

If you do not already have a Chainlink node running, then please follow the documentation available via the official docs.

Getting Started - Chainlink

We’re going to accomplish the following:
1. Configure log rotation for Chainlink node
2. Install AWS CloudWatch Agent
3. Configure AWS CloudWatch Agent
4. Configure Alerting
5. Monitor Host+Container Utilization with Grafana+Prometheus

Configure Log Rotation for Chainlink Node

The Chainlink container will store all log data in a file log.jsonl located in the directory set as your container’s volume. If following the official documentation, you will likely have it stored at ~/.chainlink-$NETWORK/log.jsonl.

~/.chainlink-ropsten/log.jsonl

Now that we know where the log file is located, we will want to use the logrotate application to keep this file to a reasonable size.
If you don’t already have it installed, we’ll want to get it installed with the following command:
sudo apt -y update && sudo apt -y install logrotate

We’ll want to create a configuration file in the /etc/logrotate.d/ directory that is specific to our Chainlink node’s log file. Let’s go ahead and change into that directory and see what’s there.
cd /etc/logrotate.d
ls

contents of /etc/logrotate.d

So now that we’re in the logrotate directory, let’s make a configuration file for our Chainlink logs.
sudo nano linklog
Once we have the file open, just enter the below (modifying it to fit your deployment — edit the $VALUES) into the file. ctrl+o to save and ctrl+x to close

/home/$USERNAME/.chainlink-$NETWORK/log.jsonl {
 daily
 rotate 14
 size 10M
 copytruncate
 delaycompress
 compress
 missingok
 notifempty
}

If you’d like to know what those values do, please read the logrotate manpage. :)

logrotate(8) - Linux man page

Nice. So now our log file will not grow out of control. Let’s move on to sending the logs to CloudWatch.

Install the AWS CloudWatch Agent

I’m not going to rewrite the directions, as they’re already sufficient.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/download-cloudwatch-agent-commandline.html

Once you have the agent installed, and the IAM role applied to your instance, we will want to create the config file for the agent to send the logs we want.

Configure the CloudWatch Agent

We’ll run the wizard to generate a configuration file.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Select the options that you like, and pay special attention to which user you select (root)* and which log file you enter (~/.chainlink-$NETWORK/log.jsonl).
*the CL container runs as uid=0

Once the configuration file has been made, start the agent with the new config

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json

Nice.
So if you followed directions correctly, you should be seeing logs in CloudWatch.
Now we need to configure alerting.

Configure Alerting on Job Errors

In AWS navigate to:
CloudWatch>Log Groups>

And click on the Metric Filters link next to your LogGroup

click on “0 filters”

Then select “Add Metric Filter”

On this page, you’ll create the filter to look for errored jobs.

The below Filter Pattern will work for you.
Just enter your JobID

{ $.status = “errored” && $.job = “$JOBID” }

Once you created the Filter Pattern, you’ll want to name it something specific to that jobId.

Create a new Filter for each of your jobIds.

Now that we have the filters set up, we’ll want to use Simple Notification Service (SNS) to alert us of these errors.

Configure Alerting

Navigate to Amazon SNS
Create a new “Topic”
I like to setup text alerts via email.
If you’d like to do this, this link has the domain to send the emails to
For example, 5553334444@advantagepaging.com

List Of Email-To-SMS Addresses - AVTECH

Be sure to confirm the endpoint in order to receive the alerts

Now you have that set up, now it’s time to associate that with an alert.

Now that you have your filters and SNS Topic set up, let’s make sure we get notified when something goes wrong.
Navigate to CloudWatch > Log Groups > $YOURGROUP > Filters

At the top right of all your filters you should see an option to “Create Alarm”. Click that.

Click “Create Alarm”

On this next page, we will want to set the threshold and the time span.

Set this however you like

Configure your Conditions and hit “Next”

On this page, we’ll configure the desired action.
Be sure to select your recently created SNS topic.

Once done, hit “Add notification”

Repeat this for each of your Filters.

Monitor Host+Container Utilization with Grafana+Prometheus

The below link is very straight forward.

stefanprodan/dockprom