Ethereum Node (EVM), Application (Geth) & Smart Contract — Monitoring
Introduction:
In recent days, I see news of corporations & financial institutions are deploying nodes on the Ethereum blockchain or on their own private blockchain for production applications. As I started my quest to understand smart contracts and deployment of the same on the blockchain, be it Hyperledger, Ethereum or others, there is a valid need to ensure that the corporate nodes that make up part of the blockchain are monitored properly. Not just up/down but hardware, software, application and application experience is monitored and can be reported on.
I thought this could be a good time to write a small document on my thoughts around monitoring these nodes and smart contracts. Mostly this is a reference for me to follow up on as I test and build complex blockchain ecosystem in my lab.
As this technology becomes more mainstream, I am sure there will be very specific tools and modules to existing application monitoring tools will start to develop. Until such time, here are some tips and findings I have found that could help you get a handle on monitoring an Ethereum Node (EVM) along with a smart contract that is running on the blockchain.
My thoughts were to monitor the server the same way I would monitor any other Microsoft Windows server or Linux server, but the application specific configuration would change where I would poll the server for health of the application (Ethereum Node) and status of the smart contract.
Hardware Monitoring (VMWARE ESXI Guest) -
For my testing, I used a Ubuntu 16.04 LTS virtual machine and installed EVM (GETH) to connect to the “testnet” blockchain. I deployed this on my lab ESXi system. One thing to keep in mind is that, you need to have enough disk space to download all the blocks. I set the disk space to 250 GB as recommended. Rest of the setup was standard.
The VMWARE ESXi is already being monitored via PRTG using SNMP and VMWARE API so I will not go into detail with that part.
System Monitoring — (OS Level) -
On the Ubuntu Server, I installed all the SNMP binaries and enabled SNMP daemon on the system to allow remote polling.
This is standard configuration and very well documented across the internet so follow the steps you find online.
For test system, I used my Lab PRTG application to remotely monitor the EVM node. I followed standard KPI’s for monitoring the node. Disk, Memory, CPU, etc. You should have a well defined strategy to monitor all your application servers and this EVM node is no different in that aspect.
EVM (GETH Application) monitoring -
Monitoring the health of the application that allows us to connect to the blockchain was the next step in my setup. I was using GETH to connect to the “testnet” blockchain (testnet). I after experimenting sending out custom traps and syslogs from the application for current blockchain status, I settled on syslog as my final facility to get the state of the block chain. I concluded that I need 2 different monitoring KPI’s to ensure my EVM was doing what it was supposed to do at the application health level.
1) The GETH process was running and was connected to the testnet blockchain
2) I was getting all the data that was written to the GETH application log file.
Item 1, ensuring the GETH process is running is fairly simple to setup. Every company will have different tools, agent or agent less to monitor the state of any processes, services or daemons on their Windows or Linux servers. In my case, I decided to use PRTG to ssh to the server and run a script. The output of the script will determine if the GETH process is running or not and will PRTG will alert me if the GETH process stops.
This is a very simple script I added to the EVM node. Ignore the next PRTG specific information but the script format will remain similar to what I have set below.
PRTG can use a simple Linux SSH sensor to execute a script on a monitored node. The sensor looks for a script in the /var/prtg/scripts/ directory. I created the directory and then added a script called checkproc.sh. Make sure you have permissions to execute the script from PRTG or any remote monitoring tool, and secondly make sure the script and the directory naming format is what PRTG requires.
The script is as follows –
dtembe@labevmnode01:/var/prtg/scripts$ more checkproc.sh
#!/bin/bash
# Check if geth is running
# -x flag only match processes whose name (or command line if -f is
# specified) exactly match the pattern.
if pgrep -x “geth” > /dev/null
then
echo “0:200:GETH is running.”
else
echo “1:404:GETH is not running.”
fi
All I am doing here is to run pgrep and check to see if GETH exists and if it does then echo process is running, in a format PRTG can understand. The echo statement can be changed to something else like snmptrapd or logger to send the alert to a remote system via snmp trap or log to a local/ remote syslog server.
On the PRTG side, I added a sensor to run a script (you can set the time, I chose 5 minutes to rerun). If you don’t have a tool like PRTG, you can execute this script via crontab at the interval you choose.
The output is above on what you will see when the PRTG server polls the EVM node and queries for the GETH process. If the process is not running then the sensor will go into warning state. All this can be customized as needed (as in the sensor can go in critical state instead of warning).
Next step is to ensure that we are collecting all the data from the log file that GETH is generating. The command I use to start GETH is
dtembe@labevmnode01:~$ geth — testnet — datadir “~/.ethereum-testnet” — mine console 2>> ~/.ethereum-testnet.log
So in my home directory there is a log file called “.ethereum-testnet.log” that we can tail for all the blockchain status information we need. After observing the data in the GETH log-file, I decided to send it all to a syslog server where I can then manipulate the data with filters and regex to then enrich it for my “hypothetical” NOC or SOC.
- Load a Syslog sensor on the PRTG server so I can push the Ethereum testnet log to it via syslog. You can run ELK Stack or WAZUH, Graylog or Splunk to collect this data.
- Change the Ubuntu syslog process to watch the GETH log-file and send all new lines to a remote server.
On a Ubuntu server you will make the following change to the /etc/rsyslog.conf file. At the very bottom of the file add the following snippet of code, change the highlighted to suit your specific environment. The last 2 lines with IP address are the remote syslog servers. The @ before the IP address denotes UDP and @@ will denote TCP as the protocol to send to the remote syslog.
# Adding in code to send logs to remote server
#/etc/rsyslog.conf
$ModLoad imfile
$InputFileName /home/dtembe/.ethereum-testnet.log
$InputFileTag ethereum-testnet
$InputFileStateFile stat-ethereum-testnet
$InputFileSeverity error
$InputFileFacility local3
$InputRunFileMonitor
*.* @http://192.168.1.230
*.* @http://192.168.1.210
Once you have changed the rsyslog.conf then you will restart syslog service on the EVM node server.
By now you should start seeing all the EVM node GETH messages in your syslog. This shows the PRTG syslog sensor collecting all the syslog messages. I also have a server running ELK (WAZUH) for PCI DSS compliance reporting which is also getting the same data so I can report on PCI DSS related reports.
Monitoring Blockchain State, Mining Hash rate & Peers –
During my search for monitoring an EVM node, I came across a post by Mr. Luis Daniel Lucio Quiroz. Here are the links to his blog, his GITHUB repo and his post on configuring the monitoring.
http://inside-out.xyz/technology/configure-nagios-to-monitor-your-ethereum-node.html
https://github.com/daniel-lucio/ethereum-nagios-plugins
https://github.com/daniel-lucio/ethereum-nagios-plugins/releases
I used the scripts that were developed by Mr. Quiroz, and followed the steps provided in his blog, referenced above. Make sure that you install the “jq” package on your EVM node. I edited the scripts to make them PRTG friendly. Using the changes made, I was able to implement the following –
1) PRTG Custom Script sensor to monitor blockchain status.
2) PRTG Custom Script sensor to monitor mining status.
3) PRTG Custom Script sensor to monitor mining status.
All Sensors are set to alert based on the same parameters set forth by the script author. I only changed the format of the echo statement to match what is needed by the PRTG ssh Exe/Shell sensor.
For next steps, I plan to add a second action to forward these alerts to syslog for PCI analysis. This can be done via the script or via the sensor actions settings.
At this stage, we have all the key health indicators of the EVM node, mining KPI and the GETH specific process and logs coming into our monitoring tool.
Smart Contract Monitoring -
Monitoring of smart contracts is a bit vague because there are at least 2 ways I would approach this, but mostly because I am still learning and a novice at the actual format of the Smart Contract and tools/capabilities that are currently available to do this elegantly.
The best practices in writing a smart contract state that, there should be logging of all events related to status of the smart contract as it executes on the blockchain.
This is key to any company that wants to ensure seamless execution of their smart contracts. Depending on which blockchain (public or private) you are executing your smart contract, and your tools capabilities, you might have different or better ways to do what I am doing. Monitoring of smart contracts and pushing the data into compliance software for regulatory auditing is critical.
For me the best methods available for monitoring a smart contract via a remote monitoring tool are listed below. In your case, you could be executing your contract on the ETH public chain, so you could just write a query on to a site like Etherscan (https://etherscan.io) in curl or via authenticated API, and post the response back to your monitoring tool.
Just remember a key item, the developers who are building the smart contract are aware of your requirements for audit capabilities and monitoring.
For monitoring smart contracts, I thought of the following 2 methods.
- You could ensure that your smart contract developers follow best practices, code in events for all states of the smart contract and then write these to log files. You can use the same process we used earlier (to write these logs to syslog and then filter/enrich these at the syslog aggregator side.
- You could also script the process to run GETH or your tools of choice to query the blockchain periodically via crontab or a tool, then once logged in, query for events, capture them on the monitoring tool and log out.
Since I am neither an expert (or a novice 😊) in smart contracts, I copied over and deployed a smart contract to the testnet blockchain while writing all events (results and errors) to the console.log.
Then using the same process, I used to push the GETH events to syslog, I edited the /etc/rsyslog.conf to include console.log as an additional input file.
#/etc/rsyslog.conf
$ModLoad imfile
$InputFileName /home/dtembe/console.log
$InputFileTag ethereum-smartcontract
$InputFileStateFile stat-ethereum-smartcontract
$InputFileSeverity error
$InputFileFacility local3
$InputRunFileMonitor
*.* @http://192.168.1.230
*.* @http://192.168.1.210
This allowed me to push all the smart contract events that were getting written to the console.log file, to my PRTG syslog server and also to my ELK/WAZUH server.
At this point, if I had a NOC or a SOC, that was interested in getting these alerts (state of the smart contract), I can filter them, write rules to enrich the event, display them real time or even push these into ITSM process workflow.
I can take the enriched events and push them into a SIEM for PCI/HIPAA compliance just as easily, since we have a usable format for the alerts. In fact, the same alerts in PRTG syslog sensor were also sent to ELK Stack running WAZUH so minor configuration changes and we have PCI compliance related reports. This could be a major help for most corporations using smart contracts.
Final Thoughts -
I am sure as I learn more about deploying smart contracts, I will make changes to the monitoring methodologies. Maybe as there are more command line tools available I will stop using Syslog and move on to API or some other monitoring facility to accomplish this elegantly.
For now, this is how the Node dashboard looks in my PRTG setup. Again, many thanks for Mr. Quiroz, for his Nagios monitoring scripts which I was able to edit and get working in PRTG.
Something high up on my list is to learn smart contracts deployment and best practices so I can continue to adapt to monitoring this environment better.
Disclaimer -
The opinions expressed here are my own & not those of my employer. As I research and learn different technologies my thoughts and opinions will change. As technology changes, so will my opinions.
Thanks for taking the time to read.
Dan
