Guide to using OpenVPN to access servers in your VPC

An OpenVPN server can provide secure access to your cloud servers that are isolated from the Internet

In a previous series of articles I showed you how to create a Kubernetes cluster using Terraform and Ansible on a set of Binary Lane servers. This article looks at how to make this more secure using an OpenVPN server.

In the original series, all the servers in the cluster were accessible to the Internet. This is useful when demonstrating tools such as Ansible as you have direct access to every server. In reality, this is a bad practice and all your non-Internet facing servers should not be connected directly to the Internet.

This raises the question that, if my Private Server in the diagram above is not connected to the Internet, how can we set it up and manage it?

In this Article we will introduce another server and set up OpenVPN on it. This will form the connection between the Internet and the Private Server. You will connect to the Virtual Private Network (VPN) using a Public Key Infrastructure or PKI-based security solution. This will secure the access to your Private Server.

To do this we will follow these steps:

1. Set up a Certificate Authority (CA) to create your certificates and keys
2. Create a Virtual Private Cloud (VPC) to hold your private servers and your OpenVPN server
3. Create an OpenVPN server
4. Connect to the OpenVPN server from your development machine
5. Create a private server
6. Connect to that private server

First we will do this manually so we understand what is happening and then we will automate the creation using Terraform and Ansible.

Binary Lane

If you have read my other articles, you will know that I use the Australian Binary Lane cloud provider for my cloud services.

Binary Lane is a reliable, economic, no-frills cloud provider that allows me to create cloud solutions from the ground level without using pre-defined managed services such as databases and Kubernetes clusters. This allows me to learn how to create solutions without relying on these types of specific service provider products (and should avoid me becoming locked in to any single supplier).

The solutions I discuss in this article have been designed for and tested out on Binary Lane. Note that all Binary Lane services incur fees, for which you are responsible.

Step 1: Setting up your own CA

Your OpenVPN clients need to be able to trust that your OpenVPN is who it says it is. For this you need a certificate from your OpenVPN server that is signed in a way that your client can validate it.

In this article, we are going to use a self-signed certificate but you could choose to purchase one from a trusted supplier.

If you want to use your own CA and don’t have one, you can follow this article which tells you how to set up your own on a Debian server. Once you have set up your CA, you can proceed to the next step.

Step 2: Creating a Virtual Private Cloud (VPC)

Virtual Private Clouds

At its heart, a VPC is a private subnet. You can read more about subnets here but put simply, a private subnet is a segment of the network that connects two or more servers and is not directly accessible directly from the Internet.

Typically, your cloud supplier provides a gateway to the Internet to allow your Virtual Private Servers (VPSs) in your VPC to access the Internet but not the other way round. Your cloud supplier will also allow you to create a server that has two interface connections, one to the Internet and one to your VPC, allowing it to act as a gateway between the two.

Creating a VPC on Binary Lane

Binary Lane allows you to create VPCs to segregate your projects and to secure services that should not be directly Internet facing.

While we understand how openVPN can be used with our VPC, we will create things manually. Later on we will then use Terraform and Ansible to create the solution automatically using Infrastructure as Code (IaC) techniques.

Creating a VPC is easy. You give it a unique name and then give it a subnet address range. By default, this is 10.240.0.0/16.

Once created, you can now create Virtual Private Servers within this VPC. You can even move in any existing Binary Lane VPS you have into the VPC (note that this will reboot the VPS).

Assuming you have a Binary Lane account, create a new VPC called My VPC and leave it with the default 10.240.0.0/16 range.

Step 3: Creating an OpenVPN server

Once you have created your VPN, you should now create a small VPS, such as:

• Debian 11
• 1 Standard VCPU
• 1 GB RAM
• 20GB HDD

Now update and upgrade with:

sudo apt update
sudo apt upgrade -y

Creating a server certificate

You will need to create a certificate for your OpenVPN server (note the change of user):

sudo apt install easy-rsa -y
sudo adduser server-admin
sudo usermod -aG sudo server-admin

(login as server-admin)

mkdir ~/my-server
chmod 700 ~/my-server
ln -s /usr/share/easy-rsa/* ~/my-server/
cd ~/my-server
./easyrsa init-pki
./easyrsa gen-req server nopass

Enter the common name for your OpenVPN Server

At this point you will now have a private key (pki/private/server.key) and a CSR (pki/reqs/server.req). The server.req filename is used regardless of your common name.

You now need to transfer the server.req file to your CA server and sign it. You will then get back a signed certificate, eg: openvpn.crt and the ca.crt, ca certificate Place them both in your home folder.

Installing OpenVPN and adding server keys and certificates

The next thing to do is to install OpenVPN and copy in your server certificates:

sudo apt install openvpn -y
sudo cp ~/openvpn.crt /etc/openvpn/server.crt
sudo cp ~/ca.crt /etc/openvpn
sudo cp ~/my-server/pki/private/server.key /etc/openvpn/

Note that the name of the server certificate is changed to server.crt when it is copied to openVPN. This is the default name expected by openVPN. You can choose to change the name by configuring it in the openVPN configuration later in this article.

OpenVPN will use the Diffie-Hellman key exchange protocol to set up the communications link. This key exchange requires us to create a key for this exchange.

This generates a dh.pem key file in the pki folder and then copies it to openVPN:

cd ~/my-server
./easyrsa gen-dh
sudo cp ~/my-server/pki/dh.pem /etc/openvpn/

Finally, we need to create the keys required for the HMAC (key-hash message authentication code) to authenticate and validate our messages. This is created by openVPN but then needs to be copied back in to openVPN:

sudo openvpn --genkey secret ta.key
sudo chown server-admin:server-admin ta.key
sudo cp ~/my-server/ta.key /etc/openvpn/

Our openVPN server-side certificates and keys have now been created. Next is our client side certificates.

Setting up openVPN

OpenVPN gives you a starting point for configuration of your openVPN service. We will start with this:

sudo cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf /etc/openvpn/
sudo nano /etc/openvpn/server.conf

Make sure that your HMAC TLS secret is enabled by uncommenting this line by removing the ; at the start of the line (it is likely already uncommented):

tls-auth ta.key 0 # This file is secret

Now let’s set the encryption cipher to use for the communications once a connection has been established, ensure this line is uncommented and add in the line beneath it:

cipher AES-256-CBC
auth SHA256

Next, we ensure that the Diffie-Hellman key exchange is set up correctly. This line (starting with dh) may need to be changed to this:

dh dh.pem

To keep our server secure, we now run the openVPN service as the nobody linux user/group.

Ensure these lines are uncommented:

user nobody
group nogroup

Depending on your situation, you may want to restrict what the user can access from their computer when the VPN tunnel is open (eg: not allowing access to the Internet etc). These optional configurations are beyond the scope of this article.

One important configuration is the default certificate and key names. You may remember that we changed the name of the certificate file when we copied it to openVPN. If you want to use names other than server.crt and server.key, you will need to change these two lines accordingly:

cert server.crt
key server.key

Save and close the file.

Network configuration

For openVPN to work, it needs IPv4 forwarding to be enabled. Edit the following file:

sudo nano /etc/sysctl.conf

And ensure this line is in the file and uncommented (ie: the remove any # at the start of the line):

net.ipv4.ip_forward=1

Save the file and restart the service:

sudo sysctl -p

Firewall rules

When creating a Binary Lane Debian VPS, no firewall is installed. We first need to install one:

sudo apt install ufw -y

You can see the available application profiles with:

sudo ufw app list

This should list OpenSSH. This needs to be enabled or else you will not be able to SSH into your openVPN server if you log out:

sudo ufw allow OpenSSH

Now enable the firewall and check its status:

sudo ufw enable
sudo ufw status

You should now see that the only traffic being allowed by the firewall is your SSH traffic. Before dropping from this session, make sure you can still log in to the server using ssh on another command line instance.

If you created your Binary Lane VPS with a default SSH key from the computer you are using, you should be automatically logged in. If not, you will need to provide the password.

You should probably consider updating your sshd configuration to prevent root login and to only allow your server-admin user to log in and only then with SSH keys.

Now we have our firewall in place, we need to enable openVPN traffic through it. First you need to know the network interface for your public connection.

ip route | grep default

This will show something like eth0 and this is your network interface that leads to the Internet. For now, I will assume this is your network interface.

Now lets add the firewall configuration for openVPN. Whilst rules are generally added using the ufw command, we need to add some rules that are placed before the ufw command line rules.

sudo nano /etc/ufw/before.rules

First we will add routing rules to route Internet requests from the client to the public network interface (change eth0 as required). Find the section (or add near the top of the file) and add these lines (note the inclusion of the VPC subnet, in my case 10.240.0):

#
# rules.before
#
# Rules that should be run before the ufw command line added rules. Custom
# rules should be added to one of these chains:
#   ufw-before-input
#   ufw-before-output
#   ufw-before-forward
#

# START OPENVPN RULES
# NAT table rules
*nat
:POSTROUTING ACCEPT [0:0] 
# Allow traffic from OpenVPN client to eth0 (change to the interface you discovered!)
-A POSTROUTING -s 10.8.0.0/8 -o eth0 -j MASQUERADE
-A POSTROUTING -s 10.240.0.0/16 -o eth0 -j MASQUERADE
COMMIT
# END OPENVPN RULES

# Don't delete these required lines, otherwise there will be errors

Note that this is also assuming that you are using 10.8.0.0/8 for the internet range of your clients. This is the default used by openVPN.

Save the file.

Next, enable default forwarding of packets by editing this file:

sudo nano /etc/default/ufw

And add/change these lines:

DEFAULT_FORWARD_POLICY="ACCEPT"

Save the file.

Assuming you did not change the port and protocol defaults in the openVPN configuration file, you must now enable openVPN connections through the firewall:

sudo ufw allow 1194/udp

Now read in the new configuration:

sudo ufw disable
sudo ufw enable

You should now have set up your firewall to allow openVPN connections. If you have any problems, you may want to consider disabling the firewall briefly to see if it is a firewall problem but remember to re-enable it!

Starting openVPN

We are now ready to start the openVPN service:

sudo systemctl start openvpn@server

The server name on the end tells it to use the /etc/openvpn/server.conf file. By creating new configuration files, you can start multiple openVPN services.

Check the status:

sudo systemctl status openvpn@server

Check the status shows (running). If it is not running, look at the logs with:

sudo journalctl -xe

You can also check you have a new tunnel interface:

ip addr show tun0

If all is ok, you can now enable the service to start automatically:

sudo systemctl enable openvpn@server

You are now ready to connect your development machine to your VPN.

Step 4. Connecting to your OpenVPN server from your development machine

To connect your development machine to your openVPN server, you will need to first create the configuration files on the server to use on your machine.

Creating the client-side configuration

If you have ever been asked to set up a VPN client, you may have been given a set of files to install into the application.

The method presented here reduces this to a single file that packages up all the information required by the client to connect to your server.

This must be done for each different client or user that is going to be allowed to use this VPN service.

Let’s create a folder to put our client files in and lock down access:

mkdir -p ~/my-clients/keys
mkdir -p ~/my-clients/files
chmod -R 700 ~/my-clients

Now generate a CSR for the client (in this case I am calling it myclient:

cd ~/my-server
./easyrsa gen-req myclient nopass

Confirm the common name and then copy the created key into our client folder:

cp pki/private/myclient.key ~/my-clients/keys/

Now create a certificate on your CA server from the pki/reqs/myclient.req file. Once the certificate has been created (say, myclient.crt), copy it back to your openVPN server into your server-admin home folder.

Copy this to our client keys folder along with our ca.crt and our ta.key:

cp ~/myclient.crt ~/my-clients/keys
cp ~/my-server/ta.key ~/my-clients/keys/
cp ~/ca.crt ~/my-clients/keys/

We now have all the files we need in our client keys folder.

Like the server configuration file, openVPN also gives us an example client configuration file.

cp /usr/share/doc/openvpn/examples/sample-config-files/client.conf ~/my-clients/base.conf

Now let’s update it.

nano ~/my-clients/base.conf

First add a line in at the top of the file to tell your VPN client to route requests to your VPC through the VPN tunnel. In my example, my VPC subnet has a defined CIDR (Classless Inter-Domain Routing) block of 10.240.0.0/16. This need to be added as a netmask as follows:

route 10.240.0.0 255.255.0.0 

Make sure that these lines align to any changes you made to the port and protocol to be used by openVPN:

. . .
# The hostname/IP and port of the server.
# You can have multiple remote entries
# to load balance between the servers.
remote your_server_ip 1194
. . .
proto udp

Uncomment the user and group directives (or add them):

# Downgrade privileges after initialization (non-Windows only)
user nobody
group nogroup

Comment out (by adding a # at the start of the line) the following as we will be adding in the required files into this file.

# SSL/TLS parms.
# See the server config file for more
# description.  It's best to use
# a separate .crt/.key file pair
# for each client.  A single ca
# file can be used for all clients.
# SHA256
ca ca.crt
# cert client.crt
# key client.key

. . .
# If a tls-auth key is used on the server
# then every client must also have the key.
# tls-auth ta.key 1

Ensure the cipher and auth setting match what we set for the server earlier:

cipher AES-256-CBC
auth SHA256

Add in the key-direction directive:

key-direction 1

If you happen to have linux clients, you will need to uncomment these out for those clients only:

# script-security 2
# up /etc/openvpn/update-resolv-conf
# down /etc/openvpn/update-resolv-conf

Save this file.

Now create a script that will create a single client configuration file from all the separate files you have created.

nano ~/my-clients/make_config.sh

Add in these contents, changing the username (server-admin) as you require:

#!/bin/bash
# First argument: Client identifier
KEY_DIR=/home/server-admin/my-clients/keys
OUTPUT_DIR=/home/server-admin/my-clients/files
BASE_CONFIG=/home/server-admin/my-clients/base.conf
cat ${BASE_CONFIG} \
    <(echo -e '<ca>') \
    ${KEY_DIR}/ca.crt \
    <(echo -e '</ca>\n<cert>') \
    ${KEY_DIR}/${1}.crt \
    <(echo -e '</cert>\n<key>') \
    ${KEY_DIR}/${1}.key \
    <(echo -e '</key>\n<tls-auth>') \
    ${KEY_DIR}/ta.key \
    <(echo -e '</tls-auth>') \
    > ${OUTPUT_DIR}/${1}.ovpn

Save the file and then run it (after making it executable):

chmod 700 ~/my-clients/make_config.sh
~/my-clients/make_config.sh myclient

This will create a myclient.ovpn file in your ~/my-clients/files folder.

Note that this only automates the last stage of the process. You will need to create new keys and certificates for each of your clients.

Setting up your machine

You will now need to follow the instructions associated with your openVPN client on your development machine to install this configuration file. I will describe how it is done for a Mac, as this is what I use!

Tunnelblik VPN client

Tunnelblik is a free, VPN client for macOS. You can download it here.

Once downloaded and installed, it will ask you if you have a configuration file. Select the ‘I have configuration files’ option. It will then tell you how to install the client configuration.

Copy the myclient.ovpn from your openVPN server to your development machine. Using Finder, drag the file over the Tunnelblik icon in the top menu bar.

If it is successful, you will be asked if the configuration is to be installed for all users or just yourself. I suggest selecting Only Me. You will then be asked to log in to prove you are you. You should now have your client configured.

Click on the Tunnelblik icon and select VPN Details … as this will help debug problems. Select your VPN on the left (myclient). Click Connect.

All being well you will a load of log lines spin up the screen and finally reporting that you are connected. Congratulations!

If not, check that you have copied all your keys and certificates correctly (it is easy to miss off a ‘-’ if you are copying as text!). Google the errors you come across and you should find pointers as to what is wrong.

Remember if you change anything, you may need to recreate and reload your client configuration.

Note: If you are connecting via a public WiFi network, you may receive warnings about your public IP address. This is because the network is a private subnet in their own right and have a Network Address Translator (NAT) gateway which means your device will not have its own IP address.

Now we are connected to our openVPN server, you should be able to log in to your openVPN server:

ssh root@10.8.0.1

This works as 10.8.0.0/8 is the subnet for your connection to your openVPN server. This is important to understand as you have created a new network over your openVPN connection that is different to your local development machine and any Internet connection and your VPC subnet.

Let’s see how we can use your new connection to access a private server on your VPC.

Creating a private server

We now need to create a new server in our VPC but this time we will not give it a public IP address. Now, you might realise that this will give us a problem. How do we connect to a server that is not connected to the Internet?

That is really the point of this article.

Go to your Binary Lane account and create a VPS under your VPN. It can be a minimal configuration for this demonstration. For this demonstration, select a Debian 11 image. Call it my-private-server. I would advise that, rather than using root passwords, you opt to register your development machine’s SSH key with Binary Lane as a default for any VPS you create.

If you are using the new Binary Lane user interface (recommended), you will need to click View All. This will show you more options.

Under IP Addresses, select No External IP Address 0.

Add the server. As it has no Internet access, you probably want to capture the root address that you are given as you may need to access your server via the web terminal interface.

Connecting to your private server

Now we need to connect to our new VPS without using a direct Internet connection.

The first step is to prove we can connect to it. SSH into your openVPN server. You may be tempted to then hop into your new VPS but, on Binary Lane, you will find that you cannot do this as outgoing SSH connections are blocked. To unblock this, go to the configuration of your openVPN server and find Port Blocking. Select this and then disable port blocking and save.

Now try logging in to your openVPN

ssh root@10.8.0.1

Then, from the openVPN server, try to SSH into your VPS. Using 10.240.0.169 as an example VPS VPC address:

ssh root@10.240.0.169

Note that you may not be able to log in if logging in as root using a password is not allowed. So long as you get to the login prompt all is good.

Before you can connect to your VPS from your development machine there is one more configuration you need to make.

Enable VPC routing

We need to tell Binary Lane that our openVPN server is a gateway to the openVPN subnet. To do this, go to your VPC and settings. Select routing.

Now add a route.

• Destination: 10.8.0.0
• Target: 10.240.0.0
• Description: OpenVPN subnet

Connect to your VPS

All being well, you should now be able to directly SSH into your VPS from your development machine, eg:

ssh root@10.240.0.169

This will now work as you are logging in from your development machine and (if you set up your VPS correctly), your SSH keys will be on that machine.

You now have access to any VPS that you create within your VPC, even though they do not have direct access to the Internet.

Preventing a client from connecting

Should your client configuration be leaked, may be through loss of your computer or through a cyber event, you would want your client configuration to be revoked.

In my article on creating your own CA, I tell you how to revoke a certificate and add it to the Certificate Revocation List (CRL). I also explain how to then sign that CRL and it is this file that you will need to create, typically called crl.pem. Place this in your server-admin home folder.

sudo cp ~/crl.pem /etc/openvpn

Edit the configuration file:

sudo nano /etc/openvpn/server.conf

Add the following line to the end of the file:

crl-verify crl.pem

This will mean that openVPN will check the revocation list each tim ea client attempts to connect.

Save the file and restart the service:

sudo systemctl restart openvpn@server

The client you revoked will no longer be able to connect to your openVPN service.

Summary

In this article we manually set up an openVPN server that allowed us to connect to our VPS that is connected within a VPC.

We then connected our development machine to our VPS using the openVPN server.

You can now read this article to see how to create an OpenVPN automatically.

Finally, we looked at how you might revoke a client’s access to our VPN.

If you found this article of interest, please give me a clap as that helps me identify what people find useful and what future articles I should write. If you have any suggestions, please add them in the comments section.