Troubleshooting applications in Linux
I have developed several applications in my career and by heart, I am a Software developer and coding is my livelihood. I have learned a few Linux commands which I find very useful to troubleshooting applications.
Troubleshooting applications starts by knowing the status:
lsof: lists open files
When you want to know the list of the files opened by the various processes.
$ sudo lsof
COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME
httpd 28689 root 9w REG 202,1 0 21582 /var/log/httpd/ssl_error_logThis is useful to resolve errors like “Too many open files”
When you want to know the process ID by the port number
$ sudo lsof -i :80 # (PORT)
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
httpd 23872 apache 4u IPv6 6431188 0t0 TCP *:http (LISTEN)This is useful to resolve errors like “Bind failed error: Address already in use”. If you get this error then you need to use this command:
kill -9 <PID>to the kill the process which is already using desired port.
When you want to know the port numbers of a process ID
$ sudo lsof -P -i|grep 23872 # (PID: process ID)
httpd 23872 apache 4u IPv6 6431188 0t0 TCP *:80 (LISTEN)
httpd 23872 apache 6u IPv6 6431198 0t0 TCP *:443 (LISTEN)ps: Process status
Gives the details about the process by using the name or ID
# $ ps aux|grep apache # (By Name)
apache 23872 0.2 0.3 516232 30760 ? Sl 17:13 0:49 /usr/sbin/httpd -DFOREGROUND$ ps aux | grep 3378 # (By Process ID)
root 3378 0.0 0.1 694888 13624 ? Ssl 2019 1:28 /usr/bin/amazon-ssm-agent
service: Services status
Some applications such as web servers run like services so we can use the service command to know their statuses.
Use the service command to know the status of a particular service
$ service httpd status # here service name is httpd
Redirecting to /bin/systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-11-25 19:35:03 UTC; 1 months 8 days agoTroubleshooting applications using logs
Checking the logs is the next important process in troubleshooting applications. Almost all the applications emit some kind of logs and by looking at them we can know about the health and stability of the applications.
grep: prints matching lines to a particular pattern
It may be not an exaggeration to say that we can’t work on the Linux machine without using grep at least once a day.
Search access logs by a date string
$ grep "01/Jan/2020" access_log
172.16.2.36 - - [01/Jan/2020:00:00:00 +0000] 448 "GET /health HTTP/1.1" 200 7 "-" "ELB-HealthChecker/2.0"Search access logs by regular expression
# Get access logs for 5xx errors
$ grep -E " HTTP/1.1\"\s5[0-9]{2}\s" access_log
100.100.100.2 - - [04/Jan/2020:07:16:35 +0000] 3015173 "GET https://www.tensult.com/cloud-reports/ HTTP/1.1" 500 38677 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)"# Get access logs which doesn't have 2xx status
$ grep -v -E " HTTP/1.1\"\s2[0-9]{2}\s" access_log
100.100.100.2 - - [29/Dec/2019:03:34:29 +0000] 472 "GET /tensult-azure HTTP/1.1" 304 - "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
Get the matching access logs with context
The following command prints 5 lines before and after the matching file so this helps to get more pre and post context about the error.
$ grep -A 5 -B 5 error error_log[09-Oct-2019 14:38:41 UTC] PHP Notice: Undefined index: HTTP_USER_AGENT in /home3/wordpress/public_html/tensult-blogs/wp-content/plugins/mojo-marketplace-wp-plugin/vendor/aws/endurance-wp-module-spam-prevention/spam-prevention.php on line 146
[09-Oct-2019 14:38:44 UTC] PHP Notice: Undefined index: HTTP_USER_AGENT in /home3/wordpress/public_html/tensult-blogs/wp-content/plugins/mojo-marketplace-wp-plugin/vendor/aws/endurance-wp-module-spam-prevention/spam-prevention.php on line 146
[09-Oct-2019 14:38:45 UTC] PHP Notice: Undefined index: HTTP_USER_AGENT in /home3/wordpress/public_html/tensult-blogs/wp-content/plugins/mojo-marketplace-wp-plugin/vendor/aws/endurance-wp-module-spam-prevention/spam-prevention.php on line 146
[09-Oct-2019 14:38:47 UTC] PHP Notice: Undefined index: HTTP_USER_AGENT in /home3/wordpress/public_html/tensult-blogs/wp-content/plugins/mojo-marketplace-wp-plugin/vendor/aws/endurance-wp-module-spam-prevention/spam-prevention.php on line 146
[10-Oct-2019 12:46:54 UTC] PHP Notice: Undefined offset: 0 in /home3/wordpress/public_html/tensult-blogs/wp-includes/class-wp-query.php on line 3244
[22-Oct-2019 07:17:12 UTC] PHP Parse error: syntax error, unexpected '<', expecting ')' in /home3/wordpress/public_html/tensult-blogs/wp-content/themes/melos/admin/main/options/04.footer.php on line 240
[22-Oct-2019 07:17:25 UTC] PHP Parse error: syntax error, unexpected '<', expecting ')' in /home3/wordpress/public_html/tensult-blogs/wp-content/themes/melos/admin/main/options/04.footer.php on line 240
[28-Oct-2019 08:36:46 UTC] PHP Warning: stripslashes() expects parameter 1 to be string, array given in /home3/wordpress/public_html/tensult-blogs/wp-content/plugins/newsletter/includes/module.php on line 2172
[28-Oct-2019 08:36:46 UTC] PHP Warning: stripslashes() expects parameter 1 to be string, array given in /home3/wordpress/public_html/tensult-blogs/wp-content/plugins/newsletter/includes/module.php on line 2172
[28-Oct-2019 08:37:12 UTC] PHP Warning: stripslashes() expects parameter 1 to be string, array given in /home3/wordpress/public_html/tensult-blogs/wp-content/plugins/newsletter/includes/module.php on line 2172
[28-Oct-2019 08:37:12 UTC] PHP Warning: stripslashes() expects parameter 1 to be string, array given in /home3/wordpress/public_html/tensult-blogs/wp-content/plugins/newsletter/includes/module.php on line 2172
[28-Oct-2019 17:48:00 UTC] PHP Warning: stripslashes() expects parameter 1 to be string, array given in /home3/wordpress/public_html/tensult-blogs/wp-content/plugins/newsletter/includes/module.php on line 2172
awk: pattern scanning and processing language
A useful tool to get statics from structured data such as access logs or any tabular data.
# Get IP list from apache access logs
$ awk '{print $1}' access_log# Get request counts by IPs from apache access logs
$ awk '{print $1}' access_log | uniq -c |sort -nr# Get 5xx errors from apache access logs
$ awk '{if($10 >= 500) {print }}' access_log# Get 5xx error counts by IPs from apache access logs
$ awk '{if($10 >=500) {print $1}}' access_log | uniq -c |sort -nr
Troubleshooting application’s performance
List the processes which are consuming more memory
$ ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head
PID PPID CMD %MEM %CPU
3502 2873 codedeploy-agent: InstanceA 3.5 0.0
2003 1 /usr/lib/systemd/systemd-jo 1.6 0.0
3380 1 /usr/sbin/rsyslogd -n 1.5 0.0
11788 1 /usr/bin/python2 -s /usr/bi 0.6 0.6
17063 28689 /usr/sbin/httpd -DFOREGROUN 0.3 0.4List the processes which are consuming more CPU
# List the process which are consuming more cpu
$ ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu | head
PID PPID CMD %MEM %CPU
11788 1 /usr/bin/python2 -s /usr/bi 0.6 0.6
18221 28689 /usr/sbin/httpd -DFOREGROUN 0.3 0.6Check the disk size: if the available disk size is less then the application’s performance may get affected.
$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 432K 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/xvda1 20G 13G 7.7G 62% /
tmpfs 798M 0 798M 0% /run/user/1000Check the log file size: as the size of the log file grows the application’s performance degrades.
$ ls -lh *log
-rw-r--r-- 1 root root 164M Jan 4 10:50 access_log
-rw-r--r-- 1 root root 25K Jan 4 08:26 error_log
-rw-r--r-- 1 root root 0 Nov 20 03:34 ssl_access_log
-rw-r--r-- 1 root root 0 Aug 28 2018 ssl_error_log
-rw-r--r-- 1 root root 0 Nov 20 03:34 ssl_request_logDelete generated files: We need to delete the application generated files from the server on a regular basis using cronjobs.
# The following deletes the files older than 30 days
$ find /path/to/files* -mtime +30 -exec rm {} \;Make sure to backup the important files to a Cloud storage such a AWS S3 before deleting from the server.
Conclusion
I hope you learned a few tools that are useful in troubleshooting applications in a Linux environment. Let me know if you have any questions or suggestions in the comments section. Also, please subscribe to our newsletter for more interesting updates.
Originally published at https://blogs.tensult.com on January 04, 2020.
