Baby’s First Post Mortem

Kristen Loyd
Aug 28, 2017 · 3 min read

AKA: My first Incident Report

Issue Summary

Holberton School provided an Ubuntu container running Nginx which was not listening to port 80 on the server’s active IPv4 IPs. Due to this, 100% of my attempts to locally curl to port 80 were unsuccessful from 21:22 PM to 21:36 PM Pacific Daylight Time. Any HTTP requests made during that time would also fail if attempted. The root cause of this issue was that Nginx was configured to listen on port 8080 for all IPv4 IPs.

Timeline

  • 21:23 PM— ssh in and attempt to curl 0:80 , but find that curl isn’t installed.
  • 21:24 PM— Ran sudo apt-get update . Ran sudo apt-get install curl
  • 21:24 PM — Ran curl 0:80 and got a “Connection Refused” message.
  • 21:26 PM — Ran netstat -lpn to see what port Nginx is listening on. I see it’s 8080 .
  • 21:27 PM — Looked at /etc/nginx/nginx.conf . Nothing has been set to listen in this configuration.
  • 21:29 PM — Looked at /etc/nginx/sites-available/default content. Server block is set to listen to port 80 as default.
  • 21:30 PM — Looked at /etc/nginx/sites-enabled/default content. Server block sets listen to port 8080 as the default. This is the root cause of the problem
  • 21:31 PM — Inspected /etc/nginx/sites-enabled/default file. It was an ACSII text file instead of a symbolic link to the /etc/nginx/sites-enabled/default file.
  • 21:32 PM — Deleted the /etc/nginx/sites-enabled/default file.
  • 21:33 PM — Attempted to run man ln to ensure I was creating the symbolic link correctly. man was not installed.
  • 21:34 PM — Installed man using sudo apt-get update and sudo apt-get install man .
  • 21:35 PM — Created the symbolic link needed by running ln -s /etc/nginx/sites-available/default /etc/nginx/sites-enabled/default .
  • 21:36 PM — Started Nginx by running sudo servide nginx start and ran curl 0:80. The curl request was properly handled and therefore the problem was resolved.

Root cause and resolution

This issue was caused because the file /etc/nginx/sites-enabled/default was not configured to listen at port 80. Instead, it was listening at port 8080.

The issue was fixed by making the file a symbolic link to the file with the correct configuration, /etc/nginx/sites-available/default. To do this, I deleted the incorrect file in /etc/nginx/sites-enabled and made the symbolic link. Then I started Nginx.

Corrective and preventative measures

In the future, there are many things that could be improved on this Ubuntu container. For one, it could be set up with the tools needed to inspect the issue or use as references, such as curl and man.

Directly relating to this issue, Holberton could have simply installed Nginx via apt-get, as the default installation configures the servers to listen at port 80 and the symbolic relationship between the two default files is set up correctly.

If Holberton would still like to issue a Ubuntu container with Nginx configured incorrectly, then this Bash script from my GitHub may prove helpful. The file must be downloaded and made executable, then it may be run in the container to automate the changes that need to take place to correct this issue.


Kristen Loyd is a Full Stack Software Engineering student at Holberton School, located in San Francisco, CA.

As a software engineer by day, and improvisation master by night, she enjoys the creative limitations programming can impose.

Ms. Loyd looks forward to joining the growing industry of Education Technology in the near future.
Examples of her code can be found at her GitHub.
Please feel free to reach her via LinkedIn.

)
Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade