Writing a Code Editor with Layers
Why write a new editor?
This is a really good question. VS Code, Sublime, Atom, or the venerable vim and emacs, there seem to be plenty of great editors to choose from. Well, I understand things much better if I pull them apart and try and build my own. So let’s go with this being a way to get better at text editing… I have played with this on and off for years, learning new bits as I go along. I have built nearly fully functional editors only to rip them up and keep a few bits.
Some things I’ve learned
Code editors are much harder and much easier than I had imagined.
The easier
With the speed of modern computers fancy algorithms for raw text storage and manipulation are not really required. Simple immutable strings, replaced by new text, though inefficient, is fast enough.
When stripped down to just text editing it is simply inserts and deletes on a group of lines.
Goal 1: Make the text storage with inserts and deletes independent so that different versions utilizing better algorithms can be easily swapped in.
The harder
My first big lesson, editing text is easy, it’s everything else that’s hard. If the act of editing text is simple than the hard parts are the things that make doing so efficient for humans. So let’s look at the things in editors that people are drawn to.
- Syntax coloring: Allows visual assessment of code and it’s structure
- Contextual completions: Lowers the human memory required to remember function and variable names as well as requiring less typing.
- Fast navigation: Most editing occurs on code already written. Finding that code and quickly making changes increases the productivity of the human using it.
- Keyboard shortcuts: Copy / Paste, adding and deleting lines, inserting snippets, jumping to a header file, or to the next function are examples of what most coders expect from a code editor.
Goal 2: Focus on configuration over code to accomplish tasks
Layers
I bought a Ctrl mechanical keyboard from Drop. I love this keyboard, one of its great party tricks is the QMK firmware configurator, https://drop.com/mechanical-keyboards/configurator/preset/ctrl--default. It uses layers, by pressing specific keys it changes the layer and when keys are pressed they mean different things depending on which layer you activate. So cool.
Hey wait, what if I designed an editor like that? By pressing key combinations it changed how the editor interpreted key presses! Hey wait, part 2, isn’t that what vi, vim, and neovim do with modal editing? Why yes it is. Hmmm, why do they predefine the available modes? The basic difference between modes is grouping functionality around a task, colon commands, visual selection, normal mode navigation, insert for typing, etc. But if the behavior is defined for starting and ending a mode and how to handle key matches, partial matches, and non-matches, then we could define the mode by configuration.
Ta-da Layers.
Goal 3: Allow layers (modes) to be defined and the behavior when entering, leaving and interacting with to be specified.
Text Editing as a Language
Uh-Oh, a vimer. If you haven’t been exposed to it do a Google search on the language of text editing. Starting with vi and continuing through to the most famous of the vi’s, vim.
Vim is a clone, with additions, of Bill Joy’s vi text editor program for Unix. Vim’s author, Bram Moolenaar, based it upon the source code for a port of the Stevie editor to the Amiga and released a version to the public in 1991. — Wikipedia
I like millions of others who have drunk the cool-aid, swear by text editing as a language. At it’s most basic level it’s using a verb-noun format to do amazingly powerful things.
Press ‘d’, the verb for delete, in vim Normal mode followed by a target… delete a character, delete a word, delete a line, delete till the next occurrence of some character, the current selection, etc, etc.
ok, sidetracked, this isn’t a vim fanboy post. I encourage you to go out and read about the philosophy of vim and the art of text editing.
Goal 4: Support the concept of using a syntactical style similar to vim
Text Objects
Another of the great ideas from the vim world. If we want to do some work on something we need to define it. A word, line, sentence, beginning of the document, first non-blank character in a line and on and on. If we want to be able to say delete the next word by typing “dw” we’ll need to know what exactly a “word” is. Text objects to the rescue. A text object is defined by a set of rules like what is the beginning of the object what is the end of the object, does it span multiple lines and complex examples like HTML tags where we may want to manipulate the things between the tags or we may want to edit the tags themselves.
Goal 5: Determine all the things that can define a group of characters as a distinctive thing and allow them to be configured and then manipulated with our basic functionality of insert, replace, delete.
User Interface
Windows, OSX, Gnome, QT, or a terminal what is the right UI to build around? The obvious answer is all and none.
Goal 6: Do not tie editor functionality to any visual output but instead create functionality that makes it easy to implement in any current or future output target.
RPC
Another of the great learning experiences came from this talk by Raph Levie about the xi-editor https://youtu.be/4FbQre9VQLI. He has lots of innovative ideas but the one that really fired up my imagination was the complete separation of the editing core from not only the visual presentation but the user interaction as well. Keyboard and mouse parsing, terminfo, OS quirks all avoided by requiring commands to be sent to the core in a standardized JSON format. The results are then returned, in xi’s case as a render of the changed data.
Goal 7: Do not tie editor functionality to an input source but instead create functionality that makes it easier to implement in any current or future input source.
What shall we Build
A toolbox of functionality that can be combined to create an editor. The first question being what language should I write it in. Early work was done in C++ but that is a bit cumbersome for something I know will get thrown away and rewritten hundreds of times. I settled on Go. It’s fast, it’s easy and I’m pretty good at it. For anyone familiar with Go it has a nice interface implementation, specifically around the concept of Readers and Writers. Let’s use that to start defining our goals.
Text Storage
How we store the text seems like a good place to start.
// TextStorer is a generalized text storage interface
type TextStorer interface {
Reset(s string)
NumLines() int
LineLen(line int) int
Len() int
LineString(pos int) (string, error)
LineRangeString(line, cnt int) ([]string, error)
NewLine(s string, line int) (int, error)
DeleteLine(line int) (string, error)
ResetLine(s string, line int) (string, error)
String() string // adds delimeters
ReadRuneAt(line, col int) (rune, int, error)
LineAt(line int) (Liner, error)
SetLineDelim(str string)
GetLineDelim() string
}For editing individual lines we can use Go-style readers and writers
// Liner is a generalized text storage interface
// for a single line of text
type Liner interface {
LineReader
LineWriter
}The reader
// LineReader provides advanced reading for lines
type LineReader interface {
Len() int
Size() int64
Read(b []byte) (n int, err error)
ReadAt(b []byte, off int64) (n int, err error)
ReadRuneAt(offset int64) (rune, int, error)
ReadByte() (byte, error)
UnreadByte() error
ReadRune() (ch rune, size int, err error)
UnreadRune() error
Seek(offset int64, whence int) (int64, error)
WriteTo(w io.Writer) (n int64, err error)
fmt.Stringer
}The writer, which includes undoing functionality that we will cover later
// LineWriter provides advanced editing for lines
type LineWriter interface {
Flush() undo.Entry
Len() int
String() string
Seek(offset int64, whence int) (int64, error)
Reset(s string)
Write(p []byte) (int, error)
Replace(p []byte, cnt int64) (int, error)
Insert(p []byte) (int, error)
WriteByte(b byte) error
WriteRune(r rune) (int, error)
WriteString(s string) (int, error)
WriteRuneAt(r rune, offset int64) (int, error)
InsertRuneAt(r rune, offset int64) (int, error)
WriteAt(p []byte, offset int64) (int, error)
InsertAt(p []byte, offset int64) (int, error)
ReplaceAt(p []byte, offset, cnt int64) (int, error)
}I think this is where I will end part 1. We have laid out some general goals of what a text editor toolbox might look like and we have begun defining the interfaces for those things. In Part 2 we’ll look at defining layers and text objects.
Cheers
Part 2 Undoing, Text Objects, Cursor
Part 3 Syntax Detection
