How to expose a local development server to the Internet

Our ambition at Botfuel is to provide the chatbot industry with the best chatbot development tools imaginable. We found no better way to battle-test our products than being our own client by developing chatbots ourselves. As a bot developer, I wanted to share some hard-earned lessons to improve my productivity.

For any developer remotely involved with networking, it is quite common to need to make a local server reachable from the outside. This is particularly the case for chatbot development. If you are developing a chatbot on Facebook Messenger, you probably want to configure the messenger endpoint to point to your local workstation.

Unfortunately, we work most of the time from private IP networks, be that at the workplace, at home or at the coffeeshop. The router(s) that stands between our workstation and the internet makes it harder to expose a local socket to the outside. Most of the time, this is preferable for security. Be sure to understand the risks before allowing anyone to access your local sockets!

Powerful tools exist to expose local services. However it is sometimes hard to find relevant documentation and examples. This article explains what you need to know to cope with most of the situations.

Why isn’t my IP public?

A typical home network is composed of a router and a number of devices connected to the router. The router is assigned a public IP address by the internet provider. Every device in the home network, including the router itself, is assigned a private IP address, usually in the form 192.168.x.x. The router acts as a gateway between the home devices and the internet. This gateway performs network address translation (NAT) between the public and the private networks. While this allows a local device to reach a public IP address, it is usually not possible to initiate a communication from the outside.

For most of the internet users, this is overall a great thing. It mitigates the IPv4 address exhaustion and it provides a way to share an internet connection. Furthermore, it protects the private networks from distant attacks. For example, the Network File System protocol is designed explicitly to be used in private networks and exposing it to the outside would lead to high security risks.

Configure your router

A solution is to change the configuration of the gateway so that it allows communication to be initiated from outside.

For example, consider the case where:

• the gateway is reachable from the internet by the IP address 5.6.7.8,
• from the home network, the gateway uses the IP address 192.168.1.1,
• your laptop is connected to your home network with the IP address 192.168.1.10 and you want to expose a server that listens to port 8889.

For most routers, the socket 192.168.1.1:12345 can be permanently linked to 192.168.1.10:8889. Of course you can replace 12345 with any port number that is not already used by your router.

When a remote client sends a request to 192.168.1.1:12345, the gateway knows that it is legitimate traffic that should be forwarded to 192.168.1.10:8889.

In some cases however, you cannot edit the configuration of the router. This is the case when you use the WiFi at a shared office or at a coffee.

Tunneling services

Some services offer you to set up a tunnel between their servers and your local workstation. These services can be very easy and convenient to use and provide advanced features such as monitoring and statistics, domain name and HTTPS support. However, these services are either charged or subject to limits and you may want to avoid a third-party accessing your data.

ngrok is one of those services, with a freemium model. I tested localtunnel which is free but I didn’t have the chance to see it work.

Self-hosted remote port forwarding

If you do not need advanced monitoring features, setting up a tunnel is straightforward using OpenSSH and a publicly accessible server that you control. In short, a SSH tunnel is initiated by the local workstation and it is kept active between your local workstation and the server. This tunnel is used to forward every incoming connection. The NAT router only sees one long-standing connection and it has no way to tell which party initiated each communication. All the traffic is encrypted using symmetric encryption, meaning that no one between the public server and the local workstation can read your data (though Deep_packet_inspection and timing analysis can reveal some information such as which protocol is used behind SSH).

Make sure that the following options are set in the /etc/ssh/sshd_config of the remote server and reload the SSH server if needed:

AllowTcpForwarding yes
GatewayPorts yes

Set up remote port forwarding (the tunnel) from the local workstation:

ssh -nN -R 8888:localhost:8889 remoteuser@1.2.3.4

Here :

• 1.2.3.4 is the public IP address of the remote server
• 8888 is the port the server is listening to
• 8889 is the port of your workstation that you want to expose
• remoteuser is the name of a user that has the right to connect to the server using ssh
• -n prevents reading from stdin, because you don’t want to use the tunnel from the command line
• -N means that you do not want to execute remote commands, just do port forwarding
• -R (as Reverse or Remote port forwarding) means that the connections are forwarded from the remote server to your local workstation, instead of port forwarding where the end that initiates the tunnel is also the one that initiates the communications across the tunnel.
• optionally, you can use a specific ssh key instead of the default ~/.ssh/id_rsa : -i ~/.ssh/id_rsa_2

Test

Listen on the destination port of the workstation :

netcat -l -p 8889

Send message from anywhere in the world :

echo “abc” | nc -v remoteserver 8888

You should receive “abc” in your workstation’s terminal.

Troubleshooting

If the message is not properly conveyed, the verbose option of the ssh client (-v) is of great use. You can also verify that everyone is listening as expected using netstat:

netstat -pln

Summary

Let’s summarise how the tunnel is setup. On your remote server, the OpenSSH server listens on port 22, awaiting for SSH commands. From your local workstation, the SSH client initiates a longstanding tunnel with the SSH server that can persist for days.

Now this is how the tunnel is used: when a HTTP request comes to the remote server (port 8888), the remote server cannot reach directly the client for the reasons exposed before. However, it can use the longstanding SSH connection to transfer all the information to the OpenSSH client. The OpenSSH client forwards the original HTTP request without any modification to localhost:8889.

For the one who created the HTTP request and for the development server, it’s exactly as if they where talking with each other directly. For the servers isolating your local workstation from the outside, the SSH tunnel is just a TCP connection like any others. Your sysadmin could find strange that you keep hours-long connections between your personal computer and a remote server, though.

Remote port forwarding (very short) summary

HTTPS support

Facebook Messenger requires to use HTTPS. Additionaly, it does not accept self-signed certificates. Then, if you want to expose a bot endpoint to Facebook Messenger, this endpoint should have a domain name with a proper certificate and it should support HTTPS. Anyway, it’s always a good idea to use HTTPS, especially if you deal with personal or sensitive data.

Set up DNS

Usually, the provider of your remote server provides also a domain name service. Use the default domain name of your server or change it to a more human friendly name.

Consequently, the remote server must have a DNS set up. The server provider may provide a default DNS but it may be not very human friendly. You can purchase it for about 10$ a year.

Set up HTTPS certificate

Setting up HTTPS used to be a pain. That time seems long over now that Let’s Encrypt and Certbot exist.

The process listening locally can handle HTTPS messages, provided it is properly configured and it has access to the private part of the server certificate. Here is a sample nodejs server configuration:

Remote port forwarding summary (HTTPS)

const fs = require(‘fs’);
const https = require(‘https’);
https.createServer({
    key: fs.readFileSync(‘privkey.pem’),
    cert: fs.readFileSync(‘fullchain.pem’),
    hostname: ‘remote-server-domain-name.com’
}, app).listen(8889);

Set up reverse HTTPS proxy

Sometimes it is more convenient to support HTTPS upstream, for example when you do not want to complexify your listening process. The task of decrypting HTTPS to HTTP can be done by a web server installed on the remote server. nginx excels at that. Here is a configuration that tells nginx to listen HTTPS on port 8888 and forward everything, unencrypted on HTTP, to the local port 8080:

server {
    listen 8888 ssl;
    ssl_certificate /etc/letsencrypt/live/remote-server-domain-name.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/remote-server-domain-name.com/privkey.pem;
    server_name remote-server-domain-name.com;
    location / {
        proxy_pass http://127.0.0.1:8080/;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_max_temp_file_size 0;
        client_max_body_size 10m;
        client_body_buffer_size 128k;
        proxy_connect_timeout 90;
        proxy_send_timeout 90;
        proxy_read_timeout 90;
        proxy_buffer_size 4k;
        proxy_buffers 4 32k;
        proxy_busy_buffers_size 64k;
        proxy_temp_file_write_size 64k;
    }
}

Then on the local workstation:

ssh -nN -R 8080:localhost:8889 remoteuser@remote-server-domain-name.com

Remote port forwarding with a reverse HTTPS proxy

Note that since the listening is performed by nginx now, you can disable the following options in the OpenSSH configuration:

AllowTcpForwarding no
GatewayPorts no

The 90s are still rocking

The protocol SSH was invented in 1995 and it’s standard implementation OpenSSH dates back to 1999. Yet they are still the best tools available to handle some common situations of our modern developer life. Sharing about these powerful tools is crucial to avoid our profession being trapped in perpetual infancy.