Diagnose an Ubuntu Server With These Five Quick & Helpful Commands
You can investigate a faulty server numerous ways, but not every option will get you the answer you’re looking for. Investigating and identifying symptoms of an issue can begin to lead you to the root cause of the problem; knowing where to look is an important half of a battle. Depending on where the issue is, different commands will help you determine and debug the issue.
Here are five basic commands to diagnose an ailing Linux server.
Please note that we’re assuming you have connectivity to your server with root access. If you do not but are using Linode, LISH can help you get in.
netstat -plntu
What do you do when you know the software is running, but can’t connect to it? Netstat is one of the first tools to use to troubleshoot this common problem.
Results of running netstat will provide information on the connections going in and out of your server. Running the command with the flags ‘-plntu’ will show a nicely formatted output with what service is making use of your network, what port it’s on, its process identifier (PID), and more.
Server issues revealed by netstat -plntu typically include services using the wrong ports, or not listening at all. Resolving such issues warrants going to that service’s configuration and making a change to the right port.
iptables -nvL
Iptables are a set of rules to deal with incoming and outgoing server traffic — they’re firewall rules. This command will list your iptables, showing you any rules that could block connectivity to a previously accessible site or server, and the options added at the end will make it easier to read for most people.
One of the most common connectivity-blocking culprits is having an unexpected rule, or lack of exception, in the iptables. Perhaps a change wasn’t saved? Or maybe the domain (as opposed to IP address) input is not resolving and causing issues? Checking the iptables rules will shed light on that.
Steps to resolve issues found here can vary and will include actions such as making new rules or flushing unwanted ones. This guide can help.
free -m
This command shows information about your RAM usage. If you’re seeing a slow server, the lack of RAM could be the issue. Run this command to diagnose if that is the case.
It’s best to use this command when your server is exhibiting issues — perhaps a slow site. Any other time won’t give the full picture.
Allocating more RAM is the obvious immediate fix, but there’s more to it than that. Properly configuring whatever is exhausting that RAM will take you far toward a solution, especially if it’s affecting your site hosted on the server (proper configuration could more than double the amount of users in the server). Improved configuration depends on what’s going on in the server, but application-specific tools like Apache2Buddy and MYSQLTuner can do the diagnosis and suggest the configuration, making a solution easier for you.
htop
Top will list all running processes on your server, its PID, and the resources used for said processes. Htop will do all this, but show it on an interactive output with easier to follow bars and color-coded segments. This is not shipped on any distros by default, but a quick installation (apt-get install htop) will solve that matter.
The suite of information shown by this command can help with many issues, but I mostly use it to see where performance bottlenecks are taking place. Is the server slow? I’ll just run htop and see if the server load is high and if there’s an unexpected culprit.
Finding an issue using htop typically means you’ll need to go to the respective service’s configuration and/or logs to see if something unexpected is going on. If everything is working or configured fine but the load is still high, you may need more resources overall.
Sometimes, stranger issues, such as a stuck service that just won’t kill itself when it should or maybe a memory leak, can cause the problem. These occasions might necessitate a more immediate action, such as running a kill command accompanied by the PID of the process needing to be killed.
tail /var/log/exampleservice/example.log
This command (altered to your specific directory) gives the end (most recent) portion of the log for the identified service being checked. You can also change the number of output lines by appending the command with ‘-n 40’ or whatever number you desire.
Checking logs is critical when an issue occurs; as long as a service is setup to output to logs for errors, you will get a good indicator of what went wrong.
If the logs show your server ran out of memory (OOM) then you should check the service that caused the OOM. If it says a term is not recognized, go to the corresponding configuration and see if you’ve mistyped something. So on and so forth for each different kind of issue found.
These five commands can do plenty for you in the world of server troubleshooting. Following are five examples where each command might be used to identify a problem.
Website isn’t working (ex. database error or 5xx error)
Check whether the services related to your website is running (ex. Apache, Nginx, MySQL). Bring the affected services online if they’re down, but it’s important to figure out why it went down in the first place. You will want to check your logs to help with that.
If MySQL was the service that went down unexpectedly, I’d recommend running a ‘df -h’ to see if your disk is out of space as well. MySQL requires some free disk space in order to properly function.
Tailing the log will help by giving the more relevant log info. If you need more, increase the number of output lines by adding ‘-n’ or check the entire log for what could’ve gone wrong.
Site or content hangs when trying to access it
Run a free -m and/or htop to see if you’re running out of resources.
Check the site’s logs, sometimes files being requested are larger than the limit specified in your settings.
Emails aren’t sending out
Check netstat -plntu for listened ports, sometimes the wrong port is assigned. If that’s not the issue, checking the logs should provide much more information as to what happened when the server attempted to send the email out.
If everything seems to be working, you can also check your iptables to see if you’re blocking port 25 (or any other essential ports).
Can’t log into the server (but logged in with LISH or any other out-of-band method)
Check netstat -plntu to verify SSH is listening on the expected port. Some servers will have SSH set up to listen on a port other than 22.
If netstat looks good, run iptables -nvL to see if there’s a rule blocking SSH connection on the indicated port.
Server is incredibly slow
Check to see if you have enough resources, meaning the free -m and/or htop command will come in handy here. It would also be worth running a quick ‘df -h’ command, showing your total disk usage. The output for those commands should show if the load is high, your disk is full, or if something is hogging all the resources.
Many more similarly effective and efficient commands are available, but these five serve in a pinch. These commands should be known by system administrators, new and old, as they address frequent and common troubleshooting issues. Each alone may not present the immediate, correct course of action, but using them in combination will show the right way forward.
Have any nightmare sysadmin situations that you resolved and want to share? Put it down in the comments or send a message to me on Twitter @feelingsohsoh.
