MASSA NODE MONITORING

In the next article, we will review the most common commands to track and monitor CPU/RAM/network activity and other system functions in a node. With these tools we will be able to more effectively monitor the performance of our Massa nodes and help us identify problems that may occur and affect said performance.

We will use the Linux system, and Ubuntu Server distribution.

Some material has been taken from the document: Monitoring scripts and commands, and another tools have been added.

https://github.com/massalabs/massa/wiki/Monitoring-scripts-and-commands

Index

• Operating system monitoring
• Internet monitoring
• System log control

Operating system monitoring

Monitor processes

top command displays total number of processes and resources consumed on the system.

Command: top

First line on the header shows the average system load and uptime:

• 17:13:36 Current time (local time where the server is located).
• up 24 days. How long the system has been working.
• 1 user: Number of active users.
• load average: 0,02, 0,01, 0,00. Average load in intervals, indicates the processor workload, with the average of the last minute, the last 5 and the last 15 minutes respectively.

Show values from 0 (no workload) to 1 (working at 100%), more than 1, it means that the processor can’t attend all the processes and there will be an input queue.

Second line of header shows the total of tasks, processes, and the status of this processes.

• Task: 120 total. Total processes.
• 1 running. Total running processes (in execution). It is the number of processes currently running or ready to run.
• 119 sleeping. Total number of sleeping processes. It is the number of sleeping processes waiting for something to happen (depends on the process) to be executed.
• 0 stopped. Total number Stopped processes.
• 0 zombie. Total Zombie processes. The processes that are not being executed. These processes are left in this state when the parent process that started them dies.

Third line shows us the percentages of processor use according to the number of cores.

Last two lines give us information about read only memory and swap respectively.

At the panl body we will see the list of processes with the following information:

• PID (process ID) the number that identifies each process, for example if the Massa node, has the PID 688, we can view it individually:

Command: top –p 688

• PR: Is the priority number given to the process. With RT it is running in real time. The available priority range is -20 to 20, -20 is the highest priority and 20 being the lowest.
• NI: Assign priority. If it has a low value (up to -20) it means that it has more priority than another with a high value (up to 20).
• VIRT: The amount of virtual memory used by the process.
• RES: The amount of physical RAM used by the process.
• %CPU: The percentage of CPU used since the last update.
• %MEM: The percentage of physical memory used by the process since the last update.
• TIME+: The total CPU time that the process has used since it started.
• S (ESTADO): Indicates the status of the process.
• COMMAND: The command used to start the process.

To exit the top command, click on CTRL+C

We have a more visual version of this tool through the command htop.

To install the command:

Command: apt install htop

Run the command:

Command: htop

Some of the shortcuts that we can use with the command:

• M key: sort processes by memory consumption
• P key: sorts processes by processor consumption
• T key: sort processes by execution time
• t key: displays the process tree
• F key: visually marks a process to follow
• K key: hide Kernel processes
• H key: hide user processes
• k key: kill a process
• q key: exit htop

Comandos de texto del sistema

-free: shows the amount of free and used memory that the system has, it also allows you to view the cache and buffer memory consumed by the kernel.

du: shows the amount of space used by the files/archives on the hard drive, as well as their path.

du -h: shows file sizes in a more simple format (GB, MB, KB).

df: shows information related to hard disk usage, available partitions, and mounted disk drives in the system, on readable drives in the system.

Command: df –h

Internet monitoring

Test Speed

speedtest: you can test internet speed in Ubuntu.

To install the command:

Command: sudo apt install speedtest-cli

With the — bytes attribute shows the unit of measurement of the data speed:

Command: speedtest-cli --bytes

We can display the data in a simplified way with:

Command: speedtest-cli --simple

To monitor network traffic we will use nethogs, which shows the bandwidth used by individual processes and sorts the list by putting the most intensive processes at the top.

To install the command:

Command: sudo apt-get install nethogs

To show processes order by bandwidth:

Command: sudo nethogs

Test system ports

netstat: displays network statistics, such as network connections (incoming and outgoing), routing tables, and a number of network interface statistics.

To install the command:

Command: sudo apt install net-tools

To check open ports:

Command: sudo netstat -lp --inet

An open port does not necessarily mean that it can be accessed from the Internet.

• Ports preceded by localhost mean that they can only be accessed from the host.
• 0.0.0.0.:ssh the SSH service port is only accessible from your local network.

The following example shows details about the connections established through the ports of Massa:

sudo apt install net-tools netstat -ntlp | grep massa-node

How to identify physical and logical network interface names

ifconfig shows all network interfaces (up and down), with physical names and Linux names:

Command: ifconfig -a

Test full duplex

One of the problems that we can find in our servers running Massa nodes is that the connection to the network is not full-duplex:

FAQ · massalabs/massa Wiki

Once we know the network interface using the previous command (in our example veth73a09b1), we can view the technical information of the interface.

Command: ethtool veth73a09b1

System log control

System logs are essential to know everything that is happening in the operating system, prevent attacks, resolve malfunctions and improve the capabilities of applications.

To access the main Linux system logs folder:

Command: cd /var/log/

In the log directory there are many files that record all type of activities:

• The main system log file is syslog.
• The auth.log file records all user access to the system.
• The dpkg.log file contains the log of the packages that were installed and uninstalled on the system..

To search for a text in a file we use the grep command:

grep –w [text to search] [file]

Some utilities:

Search Massa incidents:

Command: grep -w massa syslog

Search system alerts:

Command: grep -w WARN syslog

Search important system faults:

Command: grep -w panicked syslog

Information about when the node started:

Command: grep -w Successful bootstrap syslog

Information about when the node stopped:

Command: grep -w Stopping Massa syslog

We can also search by directories:

grep –ir [text to search][directory]

We look for all the warnings of all the logs:

Command: grep -ir warning /var/log/*

The tail command displays the last 10 lines of a file:

Command: tail syslog

Useful links about Massa:

Testnet explorer: massa.net/testnet
Tutorials: https://massa.readthedocs.io/
Source code: github.com/massalabs/massa
Discord: discord.com/invite/TnsJQzXkRN
Reddit: reddit.com/r/massa/
Youtube: youtube.com/channel/UChVfdvYpn0eFk4B-T7TGmOg
All releases and change logs for all versions