How to Solve RustDesk Relayed and Unencrypted Connection Issues

7 min readAug 27, 2022

Accessing Computers Remotely

In my business as an MSP/IT Consultant, having remote access to clients’ computers is a necessity. I’ve used a product that first started out as LogicNOW, was eventually sold to Solarwinds, and then the company changed their name to N-able. The product offering of N-able is pretty extensive, but I’m a customer of their RMM product. Out of the RMM product, the tool that I use the most is their remote access product, Take Control. You also have the option of using TeamViewer, but they really push for you to use Take Control as there are probably licensing issues or costs when using TeamViewer (and to be perfectly honest, when attempting to use TeamViewer, I would get caught by the 5-minute limitation because I wasn’t a paid user of TeamViewer).

A few years into using Solarwinds/N-able RMM, I discovered AnyDesk, which is a competing product of TeamViewer, and one that has less draconian rules for using it. Before paying for AnyDesk, I never ran into an issue where I would be subjected to a time limit, but even paying for their bottom tier of service provides you with some extras, like an address book.

A Free Alternative to TeamViewer and AnyDesk

And then comes along a little open-source project called RustDesk. Now, I’m not a programmer (I leave that to my sons), but this is an implementation of remote desktop, written in Rust, in the same vein as TeamViewer and AnyDesk with a big attractive feature. It’s free.

You can download the client for your favorite operating system at RustDesk | Open source remote desktop software | The Open Source TeamViewer Alternative

Without doing anything other than installing the software on two or more PCs, you can remotely access the desktops of Windows, macOS, and Linux PCs. RustDesk runs a public “relay server” which allows a client to connect to another PC which is behind a firewall without having to open ports on your choice of security gateway technology. The downside — the connection might be a bit slow AND it will be unencrypted.

Both of these issues can be easily solved by running your own RustDesk Relay Server or, as they call it, Self-Hosting. The software and a tutorial can be found at Deploy your own server (rustdesk.com)

I’ll provide links at the bottom of this article on a great how-to on setting up RustDesk Relay Server on a Digital Ocean instance using Docker (which I have to admit was the very first time I’d ever used Docker). The setup is pretty straight forward and once it’s up and running, the speed of accessing your RustDesk clients using your own relay server is really good. But I ran into an issue that even the tutorial and the RustDesk documentation didn’t fully explain how to overcome.

Relayed and Unencrypted Connection

Once you have your own self-hosted relay server up and running, you’ll want to secure it so not everyone can use your server without your knowledge. Though it doesn’t use a lot of bandwidth, unsanctioned use of your relay server could potentially cost you more money than you counted on. The tutorial does explain how to secure your relay server so only the clients that have your key in them can use the relay server. I’m not going to fully explain how this is setup here as it’s in the tutorial I’ll link to below.

I had this working fine, but I started noticing an issue. None of my connections, whether they were direct connections (i.e., connecting from and to PCs on my local LAN) were encrypted. Connecting to clients that were outside my local LAN would show as “Relayed and Unencrypted Connection “which isn’t something you want when connecting to a PC on the public Internet (especially if you’re going to use this to access clients’ computers that have sensitive information that could be snooped over an unencrypted connection).

The tutorial I followed can be found at RustDesk (opensourceisawesome.com) Brian does a great job of explaining how to set it all up and provides his own scripts for installing the necessary components for a Docker installation of RustDesk.

In his write-up, he has a section called “How do I keep others from using my RustDesk server?” which explains where the public key is located on the server, and what changes to make in the server configuration — the docker-compose.yml file.

There are two lines that have to be changed. The first one deals with the hbbr, which is the relay server service. Here’s the example that is provided:

And of the hbbs service, his example shows:

His hbbs example is using the hostname of where his relay server is running (i.e., rustdesk.routemehome.org). This can be an IP address of your Digital Ocean instance, for example, or you can setup a host name for it as he did.

That’s great, right? Now only RustDesk clients that have the correct ID/Relay server hostname/IP address AND the key in their configuration can use your relay server. But wait — the connection still isn’t encrypted. What gives? This is what caused me a bit of grief until I finally found the answer at the github repository for RustDesk

Why Doesn’t It Work?

First, the discrepancy I found between Brian’s how-to, and the official documentation is the following:

Brian shows that you put “ hbbr -k <your public key value here>” on one line and then the “ hbbs -r <hostname or IP address>:21117 -k <your public key value here> “on the other. However, the official documentation says you change both of those lines to read as such:

“ hbbr -k _” and “ hbbs -r <hostname or IP address>:21117 -k _” — at no point are you supposed to put your public key on those lines. I made those changes, rebuilt my server using “docker-compose up -d “and then attempted to connect.

I’d either get one of two errors — “connection reset by peer” or “key mismatch “. What was the issue?

The public key is stored in a file named id_ed25519.pub which itself is located in /docker/rustdesk/hbbs in a docker installation of RustDesk. However, when you rebuild the server using the “-k _” parameter on those two lines, the system creates two new files called id_ed25519.pub and id_ed25519 (the private key), but in the /docker/rustdesk/hbbr folder. The id_ed25519.pub/ id_ed25519 files are still located in the /docker/rustdesk/hbbs folder as well. As you can guess, the public key that now sits in the “ hbbr” folder isn’t the same as the public key that sits in the “ hbbs “folder, which is why the connection gets reset.

The Solution

I have to give credit for the github link I shared earlier in this article and to “open-trade “on the Reddit post I created when I was attempting to find an answer to this situation. It seems the answer is pretty simple.

Copy your id_ed25519.pub and id_ed25519 files from your /docker/rustdesk/hbbs folder to your /docker/rustdesk/hbbr folder. If you wish, rename your existing id_ed25519.pub and files in the hbbr folder before you copy the ones from the hbbs folder (though it won’t matter because they wont’ be used). Once the copy is made, restart the service and test your connection. You’ll find you now have a relay server that’s only accessible by those that know the key AND the connection will be encrypted.

The green shield indicates your connection is encrypted

Summary

RustDesk is a simple yet elegant open-source software solution for remote desktop access that’s both fast and secure, when setup properly, that is. At the time of this writing, it is lacking in some feature sets, mainly the one I could use being an address book. It looks like they’re working on it as it has the option, but the backend hasn’t been written yet (I guess) as it asks you for a login which I’m going to presume is where your address book will be stored online. I don’t necessarily need it stored online, but then again that will make it easier if you need to use RustDesk as the client on more than one PC (which I currently do). As a comparison to the way AnyDesk does its address book, whichever client has the license you’ve purchased is the one that will have access to the address book, so I’m going to presume AnyDesk also stores the address book online.

You can add connections to favorites AND you can create shortcuts on your desktop, which I have done and then I rename them and place them into specific folders for each client PC I use it on. I simply double-click that shortcut, and it launches RustDesk and connects.

Overall, I’m glad I came across this project and I wish them continued success, and my thanks to all those that have and will continue to contribute to it.