Local echo & xterm.js

An HP 2647A graphics display terminal connected to a HP 1000 E-Series minicomputer

The past week I have been working on an exciting new feature for a product we are preparing in Mesosphere. Me bing a classical version of myself, I started pre-emptively optimising stuff and a few hours later I dug myself into a very intriguing rabbit hole. Getting out of there was an interesting journey, that I wanted to share with you.

My experience can be described in one sentence : doing proper local terminal echo is harder that you could ever imagine! But let me elaborate on this …

This new thingie that I was building involves — among other things — an interactive shell that is presented to the user in a browser window. I was very confident that this is something very simple to do. I previously had experience with xterm.js and I have already written most of the server-side components. So I just wired them together and enjoyed the satisfying experience of typing “vi” in the browser, and interactively writing some garbage on a text editor over a web console. Yeyyy!!

Assuming that 99% of the project is done, I glimpsed back on the requirements sheet, getting my pen ready for striking out the next item. I had a small brain freeze when I read the next item:

The user input should be validated and only valid commands should be forwarded on the server

I quickly broke the problem down in my mind: 1) The user should enter a command on the browser window, 2) that command has to be submitted to the server, 3) and only if the command is valid, the server should open a sub-process and pipe the input/output from/to the browser terminal… Yes! That sounds right! And with this newly-gained confidence let’s dig into the implementation…

Part 1 — The Beginning

I started my refactoring by disconnecting the xterm.js instance from the websocket back-end. My thinking is that I should first capture the user input. But how could I read the user input from the terminal? I need a local echo, without sending anything to the server.

I am a visual guy, and I really work by example. So let’s see what happens when I blindly redirect the user input back to the terminal?

term.write("~$ ");
term.on("data", (data) => {
  term.write(data);
});

For a few split seconds I thought that this would be an easy journey! What a fool…

To the ones of you that haven’t realised the problem already, let me clarify something : a terminal is not just an input field. It’s a complex system that provides advanced formatting and interactivity with the user, over a plain character stream.

A terminal was the user-facing part of the mainframe computers of the 70’s. Through it, a user could read the output and feed input to a dedicated process in the mainframe. We may have changed the way we do computing nowadays, but surprisingly enough, we are still using the same protocols.

A terminal uses the lower 32 characters of the ASCII table (also known as “non-printable” or “control” characters) as commands to, or from the mainframe. (Does “\n” (line feed) or “\r” (carriage return) sound familiar? Yup, they are control characters!)

So, coming back to our task, we just care about the visible characters, so we are going to blindly ignore every control character that we receive:

term.write("~$ ");
term.on("data", (data) => {
  if (data.charCodeAt(0) < 32) return;
  term.write(data);
});

Now we are getting somewhere! (By the way, that weird single-character deletion you see is just me forgetting to exclude the DEL (127) character from the echo code)

This means that we just have to stack the characters in a buffer, wait for the ENTER key and then process the input. The character we am looking for is the Carriage Return (13), so let’s try this now:

var input = "";
term.on("data", (data) => {
  const code     ​= data.charCodeAt(0);
  if (code == 13) { // CR
    term.write("\r\nYou typed: '" + input + "'\r\n");
    term.write("~$ ");
    input = "";
  } else if (code < 32 || code == 127) { // Control
    return;
  } else { // Visible
    term.write(data);
    input += data;
  }
});

I stand up, I go to the coffee machine and make myself a cup of coffee. I just re-invented a write-only input field… man we have a long way to go…

Part 2 — From the Stone to the Iron Age

So far, we ignored almost all of the ASCII control characters, we prohibited character deletion and obviously we can’t interactively edit our input. If computer history was the history of mankind, we are right now somewhere around stone age. So let’s start fast-forwarding ourselves into the present.

Some time around 1970s, the terminal manufacturers came together and agreed on the ECMA-48 standard. This is nowadays known as the “ANSI escape sequences”, and describes all the fancy things that you see in the modern terminals : colours, text formatting, cursor control and even mouse pointer events. Now we are talking!

So how does an ANSII escape sequence look like? As the name prescribes : it’s the ESC (27) character, followed by a list of printable characters that describe the operation to perform.

To quote a few (you can see the full list here):

I roll up my sleeves, I drink a sip of coffee and I start writing my fancy case statement to handle all these cases.

You can imagine my steps from this point onwards : I used a string buffer to keep track of the input that the user has entered, and a number to keep track of the location of the cursor. This time, the user input is not append on the input buffer, but it’s injecting characters on the respective location:

var input = "";
var cursor = 0;

term.on("data", (data) => {
  const code     ​= data.charCodeAt(0);
  if (code == 27) {
    switch (data.substr(1)) {
      ...
      case '[C': // Right arrow
        if (cursor < input.length) {
          cursor += 1;
          term.write(data);
        }
        break;
      case '[D': // Left arrow
        if (cursor > 0) {
          cursor -= 1;
          term.write(data);
        }

        break;
      ...

    }
  } else if (code == 13) { // CR
    ...
  } else {
    input = input.substr(0, cursor) + 
            data + 
            input.substr(cursor);
  }
});

It’s quite awesome watching this guy coming to life. Starting from the silly, write-only input field, we now have something that resembles much more what a terminal should look like:

Someone could say that this is something that can be shipped to a customer. But in my PoV it lacks many things. The only moment that I would be satisfied is when this terminal feels like bash. My dearest coffee mug, you deserve a refill…

Part 3 — Making it shiny

To cut this short, I spent the next [put a big number here] hours replicating what bash is awesome on doing the past 30 years : interactively reading the user input…

My first stop was the Alt+Arrow navigation (next word / previous word) and Alt+Backspace (full word delete). Having implemented a proper cursor navigation algorithm, finding the word boundaries was easy:

Then, I decided to have a look on the multi-line bash commands. If you don’t know what I am talking about, try typing “ls &&” in your terminal and hitting ENTER. Instead of accepting the command bash will prompt you with a nice-looking arrow, asking you to continue your command!

Man that was crazy, but I realised that I only had to figure our the conditions on when to break. Breaking to a new line shouldn’t be that hard. Right? … yup, it was that hard!

But on the positive side, the algorithm that I came up with, allowed me to do something even more awesome: edit all the lines and not only the last one! (eat my dust bash!)

Another feature that I love in bash is the history . That’s easy to do. Just keep track of what the user has typed in a ring buffer, keeping only the last N commands:

And finally, I really wanted to add auto-completion! Even though you don’t know much on the client side about the files that exist on the server, you could still auto-complete some obvious commands and make the user experience even better!

Conclusion — Say hi to “local-echo”

After all this experience, and after the pain that I had to go through, I am quite proud of what I managed to achieve. And since this is something that many people are asking for on the community, I decided to make this part of the code public domain.

You can find the project on Github, on wavesoft/local-echo and the usage is quite straightforward. Just create a LocalEchoController instance, and call .read to interactively read a line from the user:

// Start an xterm.js instance
const term = new Terminal();
term.open(document.getElementById('terminal'));

// Create a local echo controller
const localEcho = new LocalEchoController(term);

// Read a single line from the user
localEcho.read("~$ ")
    .then(input => alert(`User entered: ${input}`))
    .catch(error => alert(`Error reading: ${error}`));

I hope you enjoyed reading this as much as I enjoyed writing it, and if you like the project, share it with your friends and colleagues!