We locked our production server so that even we couldn’t have access !

This is real true story about one of our production servers at TreeScale.

During server maintenance, when everything working very well, and all available software working without crashing themselves, and even log collector daemons working without an issue, we decided to make our frontend production server more secure.

So we have Nginx as a frontend servers and few execution servers which are just reverse proxy servers for Nginx and they don’t have any global access. Only our frontend servers with Nginx have internet availability, which is the main structure for our infrastructure.

We started shooting our leg!

Because of the main role keeping infrastructure without global internet availability, making super secure server structure, and we have only frontend servers as a hole to our infrastructure. And obviously because of that on Frontend servers we had only 80(HTTP), 443(HTTPS) and 22(SSH) ports open and nothing else.

So we thought ok lets close SSH port 22 and make it available only from local network, so that we can make a VPN connection to our server network and make ssh over VPN. And it seems super Secure and Fancy! :)

After making all setup done, using standard available VPN solution for Ubuntu, we made it!

What went wrong ?

VPN -> SSH combination is not a new thing and a lot of companies doing in that way, but we forgot to configure it with SystemD (we have some custom version of that). So when we configured and few days later decided to restart frontend servers, we have been locked ! It turns out that VPN service didn’t started and frontend servers not in a infrastructure network and even we cant have an SSH access to them.

But thanks to Microsoft Azure for allowing download virtual machine disks with super simple command

az storage blob download -c <contauner> -n <filename> -f <local file>

So we downloaded hard drives created new local virtual machines with Virtual Box and reconfigured them using local Virtual Box console and then uploaded them back!

Lesson we learned

Configuration is really simple and intuitive, but after having that kind of experience we decided to keep central point for configurations and integrate it with monitoring tool for making sure that all configurations went well and they have successfully tied with existing softwares. So automating server management gave us a lot of benefits and more performance for doing new things.