WRITE-UP: Adding A Web-Based Remote-Shell to a PaaS

Published in

CONNECT platform

10 min readJun 5, 2019

DISCLAIMER: This is not a guide. The scope of the problem and the technical challenges and solutions involved might feel pretty specific to a lot of programmers and thus seem irrelevant. However, I personally enjoy such in-depth technical stories and would enjoy more prominent existence of such write-ups, which is why I’m writing this. If you feel the same, also join the band and write and share similar stories of fun technical journeys of yours.

What I Wanted to Do

Background

CONNECT-platform is a project that I’ve been working on for some time now, the core is a free and open-source NodeJs framework built on top of Express that enables visual programming without limiting capabilities of NodeJs (you can read here more about the intentions behind it here). We then expanded this core to a Platform as a Service (PaaS), where you can get ready made instances of NodeJs running CONNECT-platform on the fly, in your browser (fun fact is that we built the whole PaaS using the framework itself to ensure that it is pretty good even in really complicated scenarios, and it took us less than a week to put the first version of the PaaS on AWS this way).

The way that works is simple:

The default NPM package hosts the framework and the Visual Code Editor,
There is a boilerplate project on github that runs the framework with basic necessary configuration (also reading necessary config from environment variables),
There is the docker image that clones that boilerplate project and does all the setup necessary,
The PaaS itself runs a container from that image for each project, feeding necessary environment variables to it,
Finally, an OpenResty server distributes requests to each instance.

Goal

In the setup described above, the Visual Code Editor is basically the only interface one will have for controlling their projects on the PaaS. As a developer this felt pretty sub-optimal for me, as besides a code-editor, the rest of the main stuff that I do with a coding project involve a shell. For example, I conduct version control of my projects with shell, I do log reading with shell, etc.

The solution? Obviously web-based remote shell access to any instance on CONNECT-platform. And by shell access I do not mean simply running bash instructions on the instance (to be honest that was already available as the Visual Editor would give me full JS-powered control over the Node VM running on each instance), but to run a full terminal emulator capable of interactive commands (such as using Vim to edit files).

Security-wise: someone with access to the panel (Visual Editor) of each instance basically already could do whatever shell access could do, so the corner-stone for the security design of this remote shell should have been to fully reduce its security problem to the security problem of the panel itself, i.e. access should be granted to the shell only through the panel, and the shell should have exactly the same access-level in the container as the Node process.

How It Went

Part 1: The Remote Shell Daemon

Initially I was not fully aware of the difference between full-blown interactive shell emulation and just sending bash commands and receiving the results on the front-end, so I started with looking for an easy-to-integrate front-end solution that I could attach to the existing Visual Editor (the panel). A quick google search got me to Xterm.js, which looks like the de-facto front-end tool for that. I checked out some of the projects using Xterm.js and quickly realized that what I needed was complete terminal emulation, so I basically had to have a process on each instance doing that job for me as well, ideally working nicely with Xterm.js on its own too.

The first candidate for that was WeTTy, simply because it seemed like I could just run it on my existing Node process in the container. However the API doc (or the README) were really not helping on how to do attach a WeTTy server on an existing Express server, and further-more a simple test proved that it couldn’t be done seamlessly without some minimum level of configuration (you can check it out on this repl.it piece). As a super-lazy developer, this caused me to simple drop WeTTy as a candidate here, with the prospect of not having a separate process for the remote-shell besides the Node process.

I found other options, most prominently GoTTy and ttyd. Both seemed really similar and capable of getting the job done, and both also provide the Shell-Client as well (so no need to wrestle with Xterm.js on client-side). However not only ttyd has instructions on how to install on linux systems, it has a proper Dockerfile alongside one specifically for alpine images, which is the base image for CONNECT-platform containers as well. Hence, it was the obvious winner.

Part 2: Security Layer

So to have a web-based remote-shell, I would be running a ttyd service alongside each instance of CONNECT platform. ttyd would listen on some port and offer the whatever I was looking for. However, the security concerns mentioned above would not be satisfied if I was to just run the ttyd within each container and expose it to the internet. To recap, these are the security criteria I wanted to meet:

The remote-shell should have identical access-level to the Node VM, i.e. something can be done by the shell if and only if it can be done by the NodeVM
Access to the remote-shell is identical to access to the panel, i.e. someone can access the remote-shell if and only if they can access the panel.

The first criteria could easily be solved by ensuring that the NodeVM spawns the ttyd process as a child process:

const { exec } = require(‘child_process’);const run = () => {
  let proc = exec('ttyd bash');
  process.on('exit', () => proc.kill());
}

the extra-line here is to ensure that when the Node process is dead (for whatever reason), so would be the ttyd process.

I could have alternatively just ensured that the process for the shell is executed by a user (or user group) with identical access level. However, that would mean that anyone making subsequent changes to the access level of the Node process should also be careful to update the access level of the remote-shell, which would increase the chance of error and hence unnoticed security flaws.

As for the second constraint, I decided to seal-off the ttyd server within the container and only give the Node VM access to it. Since the panel has a secure line to the Node VM, the Node VM could then authenticate and authorize requests to access the shell originating from the panel, similar to how it would authenticate and authorize other requests from the panel, and if authorized, proxy the requests to the ttyd process. As with the first constraint, I could keep the two services completely separate and ensure identical authorization mechanism on a higher-level, however again that would mean that any subsequent changes to auth process of one of the services would need to follow corresponding changes to auth process of the other, again, increasing possibility of error.

Part 3: Proxying ttyd

So the security design meant that there should be an authentication mechanism known only to ttyd and the Node process in the first place. ttyd supports basic HTTP authentication, so it would suffice to create a random user-name and password on each execution and feed it to the process:

const run = (credentials) => {
  let proc = exec(`ttyd -c ${credentials} bash`);
  process.on('exit', () => proc.kill();
}

and in the main guy supposed to run this whole charade:

let credentials = randomToken() + ':' + randomToken();
run(credentials);

Now I needed to track what are the requests that the ttyd client makes to its server, which of those are authenticated. For these requests then I would need to:

Authenticate the incoming request through Panel’s auth process,
Modify the proxied request to add the basic authentication HTTP header according to the generated credentials,
Proxy the request

As it turned out upon further inspection, the requests in need of authentication are the first original request to get the shell, and another request for a file named /auth_token.js , which seems to be used to establish and authenticate a WebSocket with the ttyd server.

As a lazy developer, this meant that I would need a simple proxy tool working nice with Express also properly capable of handling WebSockets. As it turned out with another quick googling, http-proxy-middleware is just that. So the proxying code would look something like this:

const proxy = require('http-proxy-middleware');const SHELL_URL = '<local address for ttyd>';//
// this will be the url that the shell will be accessible on 
// publicly, e.g. if the shell is to be accessed via:
// https://whatever.connect-platform.com/shell
// then TTY_PATH would be '/shell'
//
const TTY_PATH = '<some url sub-path>';const doproxy = credentials => {
  return proxy([TTY_PATH, '/auth_token.js'],
    {
      target: SHELL_URL,  // so we send everything to ttyd,
      ws: true,           // so we proxy WebSockets,
      changeOrigin: true, // so that ttyd is not aware of proxying
      pathRewrite: path => {
        //
        // well, ttyd expects requests to reach it on '/', so
        // we should properly trim them.
        //
        if (path.startsWith(TTY_PATH))
          return path.substr(TTY_PATH.length);
        else return path;
      },
      onProxyReq: (proxyReq, req) => {
        if (auth(req)) { // check if the request is authenticated,
          //
          // well we need to set 'Authorization'
          // header on requests, with credentials
          // in base64 format.
          //
          proxyReq.setHeader(
            'Authorization',
            'Basic ' + new Buffer(credentials).toString('base64')
          );
        }
      }
    }
  );
}

With the main code being modified to look like this:

const main = app => {
  let credentials = randomToken() + ':' + randomToken();
  run(credentials);
  app.use(doproxy(credentials));
}

As for authenticating the requests, since the request to the root path of the shell ( TTY_PATH ) originate from panel itself, and since panel’s authorization is done with verifying the proper access token, I could just inject the token to the URL on panel’s side, and read and verify it here. In other words, the initial request to access the shell would look like this:

https://whatever.connect-platform.com/<TTY_PATH>?token=<the_token>

so it could be easily checked like this:

const auth = req => {
  if (req.originalUrl.startsWith(TTY_PATH))
    return req.query.token && verify(req.query.token);
  return false;
}

Note that the verify() function here can be any form of token verification. On CONNECT-platform, we use JWT (which also has a pretty nice Node package).

However, the other request to /auth_token.js is made by the ttyd client itself (which I suspect is possible to modify, but again, lazy developer here). However, the referrer on that request must still be the original request to TTY_PATH , which means the token must reside in the referrer and can be verified similarly. Which means the auth() function would turn into this:

const { URL } = require('url');const auth = req => {
  if (req.originalUrl.startsWith(TTY_PATH)) {
    return req.query.token && verify(req.query.token);
  }
  else if (req.originalUrl == '/auth_token.js') {    const referrer = new URL(req.get('Referer'));
    return referrer.searchParams.has('token') &&
           verify(referrer.searchParams.get('token'));  }  return false;
}

Part 4: Configuring OpenResty

So now we have a ttyd running on the container, providing the Shell-Client as well, while also being fully concealed by Node VM and secured (or at least, exactly as much as the Node VM). The task seemed to be mostly done. I made the whole remote-shell (and its access path) configurable, in case that ttyd is not even installed in the environment that the platform is running in (because remember, the core of the platform is a NPM package that can be run anywhere). I also added a bit more tracing of the ttyd process to the JS code, so that if it would detect errors in executing the ttyd process, it would not share the link to the remote-shell with the client.

Subsequently, I added the proper piece of code to the boilerplate project to read configuration of the remote-shell from the environment variables, and added the proper environment variables to the Dockerfile. This is also where the alpine Dockerfile of ttyd came in REAL handy, as I just added the minimal content to the Dockerfile of the platform itself.

Now everything seemed nicely working, so I published the new version of the NPM package, pushed to the boilerplate repo’s master and re-built and pushed the latest version of the Docker image. I asked my colleague to update the Docker image used on the PaaS, and sat back thinking the task was done.

Obviously though, it wasn’t. As it turned out, It seemed like the WebSocket could not conduct a proper handshake. As it turned out, the handshake process is initially a simple HTTP(S) request bearing an Upgrade and a Connection header with values indicating that this request should be “Upgraded” to an open-tunnel for the WebSocket. Apparently, NGINX (and by extension, OpenResty) does not set these headers by default, so you should ask it explicitly to do so, simply by adding the following to its config:

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";

The http version is just to ensure that the proper version of the protocol is used for establishing a WebSocket connection.

The End

So in the end, CONNECT-platform was equipped with a pretty dope and relatively safe remote-shell. I was actually surprised by how relatively fast and smooth the whole process went (it took me 1.5 days in the end), considering that I had really vague idea of many concepts involved (from the shell itself, to the proper proxying mechanism or how WebSockets work precisely).

All the code changes mentioned in this article can be found in the repos that are linked. The bulk of the code changes on the platform can be found here, in case you are curious to inspect it more closely.

Anyways, this whole technical write-up thing is something new that I am experimenting with. This is something that I would personally love to find and read more online, thats why I thought of sharing my own fun technical experiences. I would be really glad to hear what you think, both the actual technical challenge or .Also special bonus for mentioning sources and blogs bearing similar tech write-ups.