One rendering tree
A peek behind the graphical stack of Glamorous Toolkit
When one looks at Glamorous Toolkit, it’s the shiny interfaces that capture the attention. It’s less apparent that we chose to build our own graphical stack. Building a graphical stack is not a straightforward decision. It is rather costly, especially for a small team, and it comes with high risks and uncertain payoffs. We went this route by going backwards: we imagined a few interfaces, asked ourselves how we’d implement that and the answer was that we need a new stack. So, we built one. Looking back, it was a good decision.
Given that we only had a rough idea of where we would like to go, we chose to guide our development through a set of principles. The most important of these principles is that we wanted our graphical scenes to rely on one single rendering tree at all times. There were others, such as no full modality, but the one single rendering tree proved to be the essential one.
A short history. The graphical stack is called Bloc. The project was initially started by Alain Plantec and Stéphane Ducasse. Glenn Cavarle was also a major contributor until 2017. From our team at feenk, Aliaksei Syrel and Tudor Girba joined the project around 2015, and later on other people from our team also started to contribute.
During its 5+ years history, Bloc was completely rewritten multiple times. Bloc’s original goal was to replace Morphic, the classic graphical engine from Squeak and Pharo, with a vector-based one. Since 2015, the goal evolved to support a broader ambition of offering a live environment that would enable new kind of user interfaces. In particular, our team’s goal was to invest and make it the future infrastructure of Glamorous Toolkit.
In this article, we talk about the current Bloc incarnation that enables Glamorous Toolkit.
What does one rendering tree mean?
Take a look at this picture.
We see multiple widgets. There is highlighted code. The larger panes are connected via thick lines and are arranged as a graph. There are lines that start from inside the text editor. Some panes hold example code, and others hold inspectors on the resulting objects. And they are all live. We can scroll, type, change views. This is possible because all visual elements are part of one large scene that is internally organized as a tree. A single rendering tree.
To reach one rendering tree at all times, we had to overcome two — well, there were more, but let’s just focus on these — distinct design problems:
- make graph layouts be regular layouts, and
- make a scalable text editor out of small elements.
Graph layouts as regular layouts
Glamorous Toolkit is a development environment. Nowadays, the typical environment is made of panes and widgets. Yet, a significant set of problems in software development are representable through visualizations. As this is an important class of representations, it follows that it should be directly supported in the environment. After all, the I in IDE stands for integrated.
Granted, we are not the first to think of adding visualizations in the development environment. However, typically these visualizations are confined in a canvas with a world of their own. What’s the problem with this? Well, the user interface framework has a tree, and somewhere in that tree, there is a canvas, and in that canvas … well, there is another world, with another tree. That these are different worlds, and different worlds means we cannot combine them easily. We do not want different worlds. We want just one. A seamless one.
So, the question is how can we support both typical widgets and visualizations within the same graphical infrastructure?
As much of software is representable as a graph, supporting graph visualizations was on top of our list. Graphs are formed of nodes and edges. In particular, to lay a graph out we need to traverse its nodes following edges. To this end, the nodes should know about the connected edges. Yet, we wanted to make any widget be able to be a node in a graph scene. So, how should we do this without hardcoding the knowledge about edges in all elements?
Let’s look at an example. In the code below, we display all classes from the SequenceableCollection hierarchy as text elements arranged in a grid layout.
root := BlElement new constraintsDo: [:c |
c horizontal fitContent.
c vertical fitContent].
SequenceableCollection
withDeep: #subclasses
do: [ :class |
root addChild: (BlTextElement new
text: class name asRopedText glamorousRegularFont;
background: Color white;
look: BrShadowLook new;
padding: (BlInsets all: 5);
margin: (BlInsets all: 5)) ].
root layout: (BlGridLayout new columnCount: 5).
root children do: [ :element |
element constraints grid horizontal alignCenter ].
root asPannableElementThe result looks like this:
SequenceableCollection hierarchy shown as text elements aligned through a grid layout.It’s nothing fancy. Just a set of elements arranged with a layout. An interesting part is that for each children we specify element constraints grid horizontal alignCenter, a constraint specific for the GridLayout that instructs the layout to center-align the elements horizontally. In fact, every layout can react to dedicated constraints defined on a per-element basis. Nothing new here.
So, what happens if we want to arrange the same elements in a tree layout? The tree layout needs to know about the edges between our elements, so we simply represent these edges as constraints. Take a look:
root := BlElement new constraintsDo: [:c |
c horizontal fitContent.
c vertical fitContent].
classToElement := Dictionary new.
ArrayedCollection
withDeep: #subclasses
do: [ :class |
element := BlTextElement new
text: class name asRopedText glamorousRegularFont;
background: Color white;
look: BrShadowLook new;
padding: (BlInsets all: 5);
margin: (BlInsets all: 5).
root addChild: element.
classToElement at: class put: element ]
relationDo: [ :superclass :subclass |
| edge |
edge := GtGraphEdge new
from: (classToElement at: superclass)
to: (classToElement at: subclass).
(classToElement at: superclass)
constraints graph addConnectedEdge: edge.
(classToElement at: subclass)
constraints graph addConnectedEdge: edge ].
root layout: GtGraphHorizontalTreeLayout new.
root asPannableElementAnd the result is:
The magic happens in how we add GtGraphEdge to the corresponding elements. Certainly, this example is rather manual, but it depicts the idea that the tree layout is represented in the same way as the grid layout, or any other layout for that matter. That is reasonably interesting.
Ok, so graph layouts are regular layouts. So what? Well, this means that the graphical engine is already a visualization engine, too. We do need dedicated layouts, but all other pieces are reusable, and more importantly, combinable. Contrast this with a typical HTML + SVG type of solution. While we can show visualizations in SVG, we cannot easily combine and connect them with the surrounding HTML world.
A moldable editor made of elements
The picture below shows two inspectors opened on the same editor object: on the left, we see the contents of the editor from a user perspective; on the right, we see the internal structure of the elements tree.
There are a few of things to notice here. Each word and each space are elements. The red rectangle is part of the same tree. And, the cursor itself is part of the same tree, as well. As the entire editor is made of elements, the arrangement of these elements is expressed by a layout. A flow layout, in our case.
This organization offers a high degree of flexibility. One immediate application is that of adornments, dynamic artifacts that can appear in the text. In our example, the word “moldable” is colored in blue and uses a monospace font. Nowadays, these are typical abilities in a text editor, and they are supported through text attributes. In our case, the code looks like this:
(text from: 5 to: 12)
attributes: {
BlFontFamilyAttribute named: 'Source Code Pro'.
BlTextForegroundAttribute paint: Color blue }Less obvious is that our editor treats the red rectangle as a text attribute, too:
(text from: 39 to: 39)
attributes: {
BrTextAdornmentDynamicAttribute new
beAppend;
stencil: [
BlElement new
size: 70 @ 50;
margin: (BlInsets all: 5);
background: Color red ] }We call these adornments, and they are just a specific kind of text attributes.
Armed with this, we can start playing with the text styler. A styler relies on a little process that goes through the text and augments it with various attributes. As adornments are attributes, the styler can augment the text with arbitrary visual elements, too. Let’s take a look.
Here we see a Coder editor opened on a Pharo method. The interesting thing about this is that all messages sent to self get a little triangle next to them. These triangles are added by the syntax highlighting, dynamically, as you type. When expanded, another editor appears inline showing the corresponding method.
New kind of programming interfaces
This infrastructure makes possible all sorts of new interfaces.
Let’s start with the unfolding code editor from above. This is interesting in a couple of ways. First, we now get an interface that allows us to read code as a contiguous narrative. This means that we no longer have to choose between short methods and more “readable” code (This debate typically becomes apparent when people familiar with C-like programming languages start learning another language and get confused by the myriad of tiny methods. No more.).
But, it gets even more interesting when we combine this ability with knowledge about specific parts of the system. For example, below you can see a Coder opened on the baseline for Glamorous Toolkit itself. What’s a baseline? It’s a configuration describing what a project is made of and what its dependencies are, along the lines of Maven or npm. The baseline is written through a fluent interface provided by the Metacello library. As with any configuration, it relies on strings for pointing to the contained packages and the dependent projects. In our case, if the dependent project is already available in the image, we show a triangle that, when expanded, reveals the baseline of the respective project. This is a simple thing, but it simplifies creating these baselines significantly without requiring one to learn a new tool.
The expansion adornments also paved the way to reinventing documentation. For example, below we see a snippet from the Glamorous Toolkit documentation. The document contains text and a graph showing all dependencies of Glamorous Toolkit extracted automatically from the baseline. The more interesting part is that the graph is embedded in the document through a markup instruction. Furthermore, the markup appears only when the cursor gets closer to it. Essentially, we get a new class of editors that is in between editor+view and a WYSWIG (what you see is what you get) interface. We get the power of programmability through markups while still retaining direct manipulation.
One more example is an interface for supporting explanations. In this case, the editor on the left has terms that are marked with special attributes through which we get elements that are inserted in the text and that are linked with the markers on the right when hovering over them. For this to work, the explanation word, the target anchor and the line must reside in the same rendering tree.
The same mechanism can be used to explain snippets of code. Here, the method on the right famously utilizes the entire syntax of Pharo. On the left, we have an explanation that links to the various parts. Interesting in this case is that the adornments in the code snippet are added through syntax highlighting.
The annotated tree
One rendering tree allows us to combine arbitrary elements in a graphical scene. But, there is one more thing. A typical approach to graphical user interfaces is to rely on higher level models that generate the actual rendering tree. Often these higher level models can be complicated and come as a layer on top of the base tree. This then hinders the creation of fluid interfaces because the higher level of abstractions usually can do less than the lower level ones.
We wanted one single space. One rendering tree to rule them all. Now, of course, we also want to be able to compose elements at higher level of abstractions. To accommodate this, we turned the problem around: we made the basic elements annotatable with higher level constructs.
For example, below we see on the left a scene with an inspector embedding a visualization. On the right, we see the rendering tree of that scene. Hovering over elements shows that they are all part of the same tree.
The tree on the right also reveals some nodes as labelled. In our case, the labels are phlow and graph. These are annotations. Phlow is the engine for defining inspector views and workflows, while the graph is … well, a graph. So, what can we do with this?
For example, given an element, we can ask it for element graph connectedEdges. graph is a tiny object that knows the element and essentially extends the interface of the element with higher level predicates. Similarly, phlow offers predicates distinguishing whether an element defines things like a view title or a view content.
Following this pattern, we always rely on one single tree and we make it the responsibility of various libraries to enrich that tree. The outcome is that we can combine arbitrary abstractions rather seamlessly. For example, in our case, the root of the visualization is annotated with both graph and phlow it is also the content of the view.
Before we part
In the age of 3D games, AR/VR and smart physical environments, it is perhaps strange to talk about a 2D graphical engine. Still, our audience is fairly rooted in sitting in front of a desktop and working with rather antiquated interfaces. Our audience are programmers, and our goal is to reimagine the development experience.
Indeed, in this day and age, the typical development experience is dominated by text. In the rare cases when text is augmented with visuals, that augmentation happens on the outskirts. That’s because the editor is too special a box. To break this, we set to make the text be made of tiny regular elements. Along the same line, we also wanted visualizations to be first class citizens. So, we had to accommodate those too, and naively perhaps, we formulated the one rendering tree principle that we wanted to stick with at all times.
Of course, having a rendering tree is not new. Of course, there are text editors that are made of nice small elements. Of course, extensive visualization engines do exist. Still, having them all together in one cohesive and reasonably performant infrastructure is new. At least to us.
But, we think the real challenge is only yet to come. We have built a vehicle, but we believe the challenge lies in the uncharted territory.
Just consider this example again. The visualization is not confined to a compartment in a bentobox-like interface. Instead, it is the visualization that defines the frame for the widgets. And while there are visual boundaries around the editors and inspectors, they remain exactly that: visual. The logical connections are not affected and can cross those boundaries to create a seamless scene.
This is not a small change. This is an entirely new kind of interface. And this is but one example. We are still learning how to think about designing interfaces in this world, but the adventure does appear to be exciting.
