Terminal escape codes are awesome, here’s why

Update: This post is also available on my blog, here.

This might just be my longest post to date. It all started when I began working as a DevOps engineer at Tikal Knowledge. I’ve seen people 7 times more professional than me and after attending an internal talk by Rafael Bodill, seeing his NeoVim configuration, I have decided to try switching to Vim as my main tool for writing code.

I found out that by putting set -o vi in my .bash_profile file, I could use vi-like keybindings in Bash, and move between vi-command and vi-insert modes, just like in vi/vim. There was one problem though – I could never really know which mode I’m currently in. I wanted to look for a way to visually tell the mode I’m in, and so, of course, I took it to Google.

Seeing a bunch of old(ish) Stack Overflow questions about the subject, for example, this, this and this, felt reassuring to know that I’m not alone in this requirement.
Now, these questions did have answers on what needs to be done, but I was astonished by the lack of proper documentation and explanation of what the helm all these characters mean, that I decided to hit the books. Good thing I did because man are those documents scattered across the internet. A detailed list of sources is provided at the end of the post.

Before we start I would like you to know that this post is split into 2 parts, the first part is where I’ll show you what I put in my dotfiles to solve my issue, and the second part will be a deeper dive into the mechanisms in play. Let’s start.

Part One: Bashing it out

Let’s start from the end — customizing the shell to show us what mode we’re currently in.

You need to edit 2 file, add the following line to your ~/.bashrc\~/.bash_profile:

set -o vi

Add the following to your ~/.inputrc (you can skip the comments, obviously):

# Enable vi keybindings in bash
set editing-mode vi
set show-mode-in-prompt on
# Breakdown of vi-ins-mode-string:
# Recurring characters:
# \1 — Start zero-width character (\001)
# \2 — End zero-width character (\002)
# \e — By itself simply means “an escape character”
# \e[ — Signal CSI (Control Sequence Introducer)
# \e] — Signal OSC (Operating System Command)
# Note that the multiple \1s and \2s aren’t necessary and are only specified
# for easier understanding of the lines
# Actual breakdown:
# \1\e[38;5;015m\2 — Set foreground color to 8-bit color 015 (white)
# \1\e[0m\2 — Reset color settings
# \1\e[38;5;064m\2 — Set foreground color to 24-bit color #bffe5d
# (R=191,G=254,B=93) — Green color
# (ins) — Print “(ins)”, literal text
# \1\e[0m\2 — Reset color settings
# \1\e[5 q\2 — Change cursor shape to blinking bar
# \1\e]1337;SetColors=curbg=7300f2\e\\\2 \
# This is a proprietary iTerm2 OSC ,Let’s break it down:
# OSC 1337 is equal to xterm’s OSC 50
# ;SetColors=[key]=[value]
# Our key is the cursor background (If you ask me it should be called the
# foreground but who am I to say)
# Our value is an RRGGBB string
# This needs to end with a literal backslash and therefore we specify \e\\ at the end before finally closing with a \2
set vi-ins-mode-string \1\e[38;5;015m\2uu\1\e[0m\2\1\e[38;2;191;254;33m\2(ins)‚u\1\e[0m\2\1\e[5 q\2\1\e]1337;SetColors=curbg=bffe21\e\\\2 set vi-cmd-mode-string \1\e[38;5;015m\2uu\1\e[0m\2\1\e[38;2;191;121;33m\2(cmd)u\1\e[0m\2\1\e[2 q\2\1\e]1337;SetColors=curbg=bf7921\e\\\

I have some unicode characters that won’t transfer properly here so I swapped them for “u” in the above snippet.

Another thing I want to point out is that I use Oh My Posh for my custom prompt, which proved to be a bit of a challenge. What I ended up doing what making a multiline prompt that ends in a newline, that way the entire last line was up for grabs, because unfortunately these mode strings only show up at the beginning of the last line of the prompt. I did find a guy by the name of Dylan Cali who made a forked version of Bash that supports placing these strings elsewhere in the prompt.

Here’s the result:

Part Two: Making sense of the visual noise

First Things First

In this part I will try to explain what goes where and why.

Important notice on readability before we start: During this part of the post I will use the same convention used in the documentation, and here’s the gist of it.

ESC – Escape character
SP – Space
ST – String terminator
Ps – A single numeric parameter
Pm – Multiple numeric parameters, separated by ;
Pt – A text parameter
Note that literal spaces are to be ignored and lowercase characters are to be taken literally.

Also, throughout this post I will be using Bash 5.1.16 with iTerm2 on MacOS Monterey, and the terminal type xterm-256color selected. The terminal type tells iTerm2 which terminal to emulate and use. You can find your own terminal type by typing echo $TERM into the console.

This is important because different OSs and terminal emulators implement some escape codes while omitting others. An example is OSC 12, which isn’t supported on iTerm2 despite MacOS being a unix-like OS. If you find a sequence that doesn’t work for you, this may just be the problem.

Terminals & Terminal Emulation

First of all, let’s talk about Terminals. nowadays, a terminal is synonymous with a terminal emulator (Such as Windows Terminal, iTerm2, and many more), but back in the day, terminals were pretty dumb devices, a keyboard with a screen (or even a printer instead of a screen if you go back enough), that was used to communicate with a remote server.

These terminals would receive bytes of data, but without any way to distinguish the different types of data it receives there wouldn’t be much of a way to control what goes where. That’s where escape sequences come in.

If you’ve been around long enough, you might be familiar with ANSI, the American National Standards Institute. They have created a standard for escaping characters in terminals; superseded by the ISO’s ISO 6429 standard (I wish I could tell you more about it but standards apparently cost(?!) a hundreds of US dollars);
And as we all know, standards make the world better, because when implemented, even partially, they make things more universal and cross-compatible. ANSI also left a handful of bytes for vendor-specific escape codes.

In our case (Bash), this is all implemented by the GNU Readline library, which makes pretty much the entire concept and experience of a terminal possible.

Escape sequences

Denoting an escape sequence

In order to tell the terminal we want to use an escape sequence, and not print out a piece of text verbatim, we have a special escape character, in unix-like systems the escape character is usually \e or \033, after which another character follows, usually terminated by a BEL (\007) or ST (Usually ESC \ [\e\\ in our case]). A lot of escape sequences are terminated using various other characters.

In order for the terminal to not mess up formatting, line wraps and spacing, we also need to tell the terminal when a character is zero-width, meaning it shouldn’t take it into account when outputting text to the console, for this purpose we can prefix our sequences when \001 to denote “Hey, this is the start of a zero-width portion”, and suffix it with \002 to denote “Hey, we’re done with the zero-width portion”. Some systems can also handle \1 and \2, which is what we’re going to use throughout this post, for the sake of truncation.

CSI & OSC sequences

Now that we know about the most important characters, the ones that enable us to denote a sequence, let’s take a look at CSI and OSC sequences.

CSI stands for Control Sequence Introducer, and is invoked by using ESC [, as stated in the beginning of this post, this is the same notation used in the official documents and what we will need to pass to our terminal is actually \e[. A CSI followed by a number of bytes (remember, 0 is also a number), and each sequence has its own set of rules and conventions on what it does and which arguments it expects. We’ll encounter some cool CSIs later in this post.

OSC stands for Operating System Command, and has been implemented mostly by Xterm, if we look at Xterm’s documentation, we’ll see the following text:

Control Sequences and Keyboard

The Xterm Control Sequences document lists the control sequences which an application can send xterm to make it perform various operations. Most of these operations are standardized, from either the DEC or Tektronix terminals, or from more widely used standards such as ISO 6429.

So yeah, Xterm doesn’t really re-invent the wheel but rather implements some standards, which is, again, great for us. We can use OSC sequences to, for example, change the cursor’s color. OSC sequences can be OS-specific, and some of the ones Xterm uses won’t work on other emulation systems; Which is why, again, the terminal type you choose within your terminal emulator is important.

An OSC can be invoked by using ESC ], we will of course use \e]. OSC sequences are also followed by a number of bytes, and terminated by a predefined string. I think now is a good time to explore some of the awesome CSI and OSC sequences.

Manipulating text

CSI 0 m – Resets settings

CSI 38 ; Ps ; Pt m – Can be used to change the foreground color of text. Needs to be preceded by either a 5 (To denote 8-bit color values) or a 2 (to denote 24-bit [RGB] color values).
Example:

# Steps:
# Change foreground color to RGB colors: Red = 35; Green = 194; Blue = 40
# Print "Hello"
# Reset settings
# Print " there!"
# Newlines to space out my prompts
echo -e '\n\e[38;2;35;194;40mHello\e[0m there!\n'

Result:

CSI 48 ; Ps ; Pt m – Pretty much the same as CSI 38, but this one is used to change the background color of text, rather than the foreground.

Example:

# Steps:
# Change foreground color to RGB values: Red = 35; Green = 194; Blue = 40
# Change background color to RGB values: Red = 255; Green = 0; Blue = 0
# Print "Hello"
# Reset settings
# Print " there!"
# Newlines to space out my prompts
echo -e '\n\e[38;2;35;194;40m\e[48;2;255;0;0mHello\e[0m there!\n'

Result:

CSI 7 m – Invert text colors. This one’s a neat trick. If we are lazy and want to switch our background and foreground colors, we can use it.

Let’s say I want to leave our “Hello” as is but have ” there!” be the exact opposite color-wise.

# Steps:
# Change foreground color to RGB values: Red = 35; Green = 194; Blue = 40
# Change background color to RGB values: Red = 255; Green = 0; Blue = 0
# Print "Hello"
# Invert settings
# Print " there!"
# Reset settings
# Newlines to space out my prompts
echo -e '\n\e[38;2;35;194;40m\e[48;2;255;0;0mHello\e[7m there!\e[0m\n'

Result:

Defining the cursor

CSI Ps SP q – Changes the cursor. Ps can be any integer in the range 1-6. Here’s a breakdown:

| Digit | Result                 |
|-------|------------------------|
| 1     | Blinking block         |
| 2     | Steady block (default) |
| 3     | Blinking underscore    |
| 4     | Steady underscore      |
| 5     | Blinking bar           |
| 6     | Steady bar             |

OSC Ps ; Pt ST – Somewhat unsurprisingly, despite my best efforts, I couldn’t get these codes to work. I did see other people use them but I personally couldn’t get them to work. I’d actually like to share my trial and error process. Looking at the Xterm documentation section for OSC sequences, I came across this text:

OSC Ps ; Pt ST
Set Text Parameters.
Some control sequences return information:
o For colors and font, if Pt is a “?”, the control sequence elicits a response which consists of the control sequence which would set the corresponding value.
o The dtterm control sequences allow you to determine the icon name and window title.
…
The 10 colors (below) which may be set or queried using 1 0 through 1 9 are denoted dynamic colors, since the corresponding control sequences were the first means for setting xterm’s colors dynamically, i.e., after it was started. They are not the same as the ANSI colors (however, te dynamic text foreground and background colors are used when ANSI colors are reset using SGR 3 9 and 4 9 , respectively).
These controls may be disabled using the allowColorOps resource.
At least one parameter is expected for Pt.
Each successive parameter changes the next color in the list.
The value of Ps tells the starting point in the list.
The colors are specified by name or RGB specification as per XParseColor.
If a “?” is given rather than a name or RGB specification, xterm replies with a control sequence of the same form which can be used to set the corresponding dynamic color.
Because more than one pair of color number and specification can be given in one control sequence, xterm can make more than one reply.
Ps = 1 0 –  Change VT100 text foreground color to Pt.
Ps = 1 1 –  Change VT100 text background color to Pt.
Ps = 1 2 –  Change text cursor color to Pt.
Ps = 1 3 –  Change pointer foreground color to Pt.
Ps = 1 4 –  Change pointer background color to Pt.
Ps = 1 5 –  Change Tektronix foreground color to Pt.
Ps = 1 6 –  Change Tektronix background color to Pt.
Ps = 1 7 –  Change highlight background color to Pt.
Ps = 1 8 –  Change Tektronix cursor color to Pt.
Ps = 1 9 –  Change highlight foreground color to Pt.
Ps = 2 2 –  Change pointer cursor to Pt.

If you look at the very beginning, you can see that we can specify a question mark as a parameter for the OSC sequence and we will get a response telling us how our sequence should be structured.

Only OSC 10 and OSC 11 returned my calls, but here’s a look at the output I got:

echo -e '\e]10;?\e\\'
OUTPUT: ^[]10;rgb:edd6/e736/e736^G

If we look at the Wikipedia page for ANSI escape codes we can see that ^[ means ESC and ^G means BEL (for historic reasons, Xterm also supports BEL being used as ST), and if we take another look at that documentation snippet above, we can see that color OSC sequences parse input using the XParseColor function. Looking at XParseColor’s documentation, we can see it expects input in this format: rgb:rrr/ggg/bbb, and a bunch of other options. So if you ask me, any of the following should work to, for example, change the cursor’s color:

echo -e '\e]12;rgb:255/0/0\e\\'
echo -e '\e]12;red\e'

Alas, they don’t. Having no more insight into this, I decided to just use iTerm’s own proprietary escape codes for my shenanigans.
Now let’s move to the second part of this post, where I’ll show you how you can leverage these sequences to create a better terminal experience for yourself.

Bonus content

Here’s an example for a percentage counter in Bash, adapted from an example on this helpful resource

for i in $(seq 0 100); do echo -en "\e[200D$i%" && sleep 0.05; done; echo -e "\n"

What this does is loop through a sequence of numbers from 1 to 100, prints a number, waits 0.05 seconds, and repeats, until 100 percent is reached. Give it a try!
This sequence is CSI Ps D which tells the cursor to move Ps lines to the left, in my case I went overkill and put the value 200 (In my defense, the other guy put 1000).

Get [more] out of your shell

Well then, this seems to be the end of this post for now. I had a blast researching this topic and learned a ton in the process. One of the issues I mentioned facing at the beginning of this post is that the information I needed was scattered far and wide and wasn’t as straight-forward, readable, and/or accessible as I would’ve liked it to be.

To keep you from having to repeat the same painstaking process of scouring the web for clues, I’ve compiled a list of recommended resources that have helped me make sense of all of this (eventually…):

• Thomas E. Dickey’s Xterm manpage
• GNU Readline Init file documentation
• Proprietary iTerm2 escape codes
• Linux’s console_codes manpage
• The XFree86 Project’s documentation
• The Wikipedia page for ANSI escape codes
• The Wikipedia page for ISO 6429

Honorable mentions:

• Dylan Cali’s custom Bash fork
• An Article on Linux Gazette
• A GitHub Gist on ANSI escape sequences
• Google’s documentation on escape sequences
• Fabian Loison’s Bash tips on colors and formatting
• TLDP’s HOWTO
• X.org’s documentation

For any comments, requests, inquiries and the likes, you are always welcome to e-mail me at orel@fichman.co.il