Building a HTTP Tunnel with WebSocket and Node.JS
When we develop some apps or bots which integrate with third party service, we need to make our local development server to be exposed into Internet to get Webhook message. To do that, we need HTTP tunnel for our local server. How does HTTP tunnel work? In this article, I will show you how I build a HTTP tunnel tool based on WebSocket and Node.js stream, learn how to use Node.js stream to transfer big data.
Why we need to deploy our own HTTP tunnel service
There are lots of awesome online services for HTTP tunnel. For example, we can use ngrok to get paid fixed public domain to connect your local server. It also has free package. But for free package, you can’t get a fixed domain. Once you restart the client, you will get a new random domain. It is inconvenient when you need to save the domain in third party service.
To get a fix domain, we can deploy HTTP tunnel in our own server. ngrok also provides an open source version for server side deployment. But it is old 1.x version and not recommended to deploy at production with some serious reliability issues.
With our own server, it can also keep data security.
Introduction about Lite HTTP Tunnel
Lite HTTP Tunnel is what I build recently for self-host HTTP tunnel service. You can deploy it with Heroku button in the Github repository to get a free fixed Heroku domain quickly.
It is built based on Express.js and Socket.io with just few code. It uses WebSocket to stream HTTP/HTTPS request from public server into your local server.
How I implement it
Step 1: build a WebSocket connection between server and client
Support WebSocket connection at server side with socket.io:
Connect WebSocket at Client side:
Step2: Use JWT token to protect WebSocket Connection
In server side, we use socket.io middleware to reject invalid connection:
Step 3: Data Stream Transmission
In Node.js, both HTTP Request and Response are streams. On the server side, Request is a Readable stream, while Response is a Writable stream.
The normal data stream transmission in a node.js web server is shown in the following diagram:
And now that our Web Server is inside the local firewall, we use a public server to forward the Request and Response through. Therefore, the user’s HTTP data first goes to the Tunnel server, which sends the Request to the Tunnel client, and then the Tunnel client sends the Request to the Local web server to get the Response, which is finally returned to the Tunnel server for transmission to the Client side.
To transmit the Request and Response streams between the Tunnel server and Tunnel client, we implement a TunnelRequest writable stream class and TunnelResponse readable stream class on the Tunnel server side, based on WebSocket, and a TunnelRequest readable stream class and TunnelResponse writable stream class on the Tunnel client side.
Tunnel server side：
Tunel client side:
To learn more about Node.js stream, you can refer to the official documentation. For the implementation of TunnelRequest and TunnelResponse, you can visit https://github.com/web-tunnel/lite-http-tunnel/blob/main/lib.js.
After completing all of the above steps, we now support streaming HTTP requests to a local computer and sending responses from the local server back to the original request. This is a lightweight solution, but it is highly stable and easy to deploy in any Node.js environment.
Step 4: Deploy HTTP Tunnel service
We can deploy the HTTP tunnel service to a cloud provider such as Heroku/Render. The project Lite HTTP Tunnel contains a Heroku/Render button in the Github repository, which allows you to deploy the service to Heroku/Render quickly.
More
So we have introduced about how to transfer HTTP requests based on WebSocket and Node.js Writable and Readable stream. In latest version of Lite HTTP Tunnel, we refactor the project with Duplex stream to support requests from WebSocket. You can check that from source code.
If you just want to find a HTTP tunnel service with a free fixed domain, you can try to deploy Lite HTTP Tunnel project with deploy button in the Github README. Hope you can learn something from this article.
