Baby’s First Post Mortem
AKA: My first Incident Report
Issue Summary
Holberton School provided an Ubuntu container running Nginx which was not listening to port 80 on the server’s active IPv4 IPs. Due to this, 100% of my attempts to locally curl to port 80 were unsuccessful from 21:22 PM to 21:36 PM Pacific Daylight Time. Any HTTP requests made during that time would also fail if attempted. The root cause of this issue was that Nginx was configured to listen on port 8080 for all IPv4 IPs.
Timeline
- 21:23 PM— ssh in and attempt to
curl 0:80, but find that curl isn’t installed. - 21:24 PM— Ran
sudo apt-get update. Ransudo apt-get install curl - 21:24 PM — Ran
curl 0:80and got a “Connection Refused” message. - 21:26 PM — Ran
netstat -lpnto see what port Nginx is listening on. I see it’s8080. - 21:27 PM — Looked at
/etc/nginx/nginx.conf. Nothing has been set to listen in this configuration. - 21:29 PM — Looked at
/etc/nginx/sites-available/defaultcontent. Server block is set to listen to port80as default. - 21:30 PM — Looked at
/etc/nginx/sites-enabled/defaultcontent. Server block sets listen to port8080as the default. This is the root cause of the problem - 21:31 PM — Inspected
/etc/nginx/sites-enabled/defaultfile. It was an ACSII text file instead of a symbolic link to the/etc/nginx/sites-enabled/defaultfile. - 21:32 PM — Deleted the
/etc/nginx/sites-enabled/defaultfile. - 21:33 PM — Attempted to run
man lnto ensure I was creating the symbolic link correctly.manwas not installed. - 21:34 PM — Installed
manusingsudo apt-get updateandsudo apt-get install man. - 21:35 PM — Created the symbolic link needed by running
ln -s /etc/nginx/sites-available/default /etc/nginx/sites-enabled/default. - 21:36 PM — Started Nginx by running
sudo servide nginx startand rancurl 0:80. The curl request was properly handled and therefore the problem was resolved.
Root cause and resolution
This issue was caused because the file /etc/nginx/sites-enabled/default was not configured to listen at port 80. Instead, it was listening at port 8080.
The issue was fixed by making the file a symbolic link to the file with the correct configuration, /etc/nginx/sites-available/default. To do this, I deleted the incorrect file in /etc/nginx/sites-enabled and made the symbolic link. Then I started Nginx.
Corrective and preventative measures
In the future, there are many things that could be improved on this Ubuntu container. For one, it could be set up with the tools needed to inspect the issue or use as references, such as curl and man.
Directly relating to this issue, Holberton could have simply installed Nginx via apt-get, as the default installation configures the servers to listen at port 80 and the symbolic relationship between the two default files is set up correctly.
If Holberton would still like to issue a Ubuntu container with Nginx configured incorrectly, then this Bash script from my GitHub may prove helpful. The file must be downloaded and made executable, then it may be run in the container to automate the changes that need to take place to correct this issue.
Kristen Loyd is a Full Stack Software Engineering student at Holberton School, located in San Francisco, CA.
As a software engineer by day, and improvisation master by night, she enjoys the creative limitations programming can impose.
Ms. Loyd looks forward to joining the growing industry of Education Technology in the near future.
Examples of her code can be found at her GitHub.
Please feel free to reach her via LinkedIn.
