Increasing RxJS Insight by Writing a TypeScript Transformer

Asynchronous code is hard to grasp and using RxJS is no exception. Having multiple parallel processes and plenty of data flowing through your application, it’s often difficult to understand what’s happening when, why and how. The desire to increase the insight in this everchanging state of these programs raised the question of how to extract knowledge from a running program without manually logging every line of code.

TypeScript transformers might be the solution for this and other problems alike. Using TypeScript Transformers one can extend the behaviour of certain programs in an automated manner. In this article, I will describe how to write such a TypeScript transformer.

Prerequisites

• This article will dive into writing a TypeScript transformer but won’t cover implementing a TypeScript transformer in your project, read this article by David Kingma to learn more about injecting a TypeScript transformer into an Angular project.
• The abstract syntax tree(AST) being such a central concept in transforming TypeScript programs it would help to understand this article having a basic understanding of either parse trees or AST’s. A great introduction by Vaidehi Joshi can be read here.
• A basic understanding of RxJS will help to understand the code snippets but is not required.

Introduction

In this article, I will describe a way to write a TypeScript transformer to replace RxJS nodes in a program without altering the original behaviour. The purpose of this transformer is to collect and send metadata to a GUI application, enhancing the realtime insight for a given developer. This article will focus on the writing aspects of a TypeScript transformer, and not on RxJS itself. The simplified structure of the transformer is shown below:

The general structure of the TypeScript transformer.

The topics covered in this article are:

• AST Traversal
• Node Classification
• Generating Metadata
• Replacing Nodes
• Replacement Functions

AST Traversal

The input for a TypeScript transformer is a program consisting of multiple source files. Each source file comes in the form of an AST, populated with nodes.

The first step in the process of transforming a TypeScript file is deciding which parts of the programs require to be mutated, thus which nodes of the AST should be affected.

The first step in this process is traversing over every node and passing it to a dispatch function:

The visitSourceFile function for traversing every node in an AST.

Inside the dispatch function, each node is first classified. Depending on the classification, a node is transformed by the appropriate function or returned unchanged.

Dispatch function to match classifiers and mutators.

Node Classification

To classify a node, and to make sure it is a certain part of an RxJS statement a collection of classifier functions are used. Each classifier will look at both the structure and the contents of a node.

In the code snippet below, the isObjectOrSubjectConstructor function is shown. This function classifies RxJS object and subject construction statements; e.g.

new Observable<number>();

Without looking at the structure of a node, invalid classifications will be made. For example, the following comment should not be transformed.

//new Observable<number>();

isObjectOrSubjectConstructor classifier function.

A great tool for assistance with writing the classifier functions is AST Explorer. This tool helps by creating an AST representation for given TypeScript code.

AST Explorer from astexplorer.net

Unit Testing

Another great tool of assistance during the development of a TypeScript transformer are unit tests. Using unit tests prevents having to compile a complete project using the transformer saving a lot of time in the long run.

To make the unit testing of functions acting upon TypeScript nodes less of a struggle I’ve created some helper functions. The createNode function creates an AST from a given code snippet in string format and the printNode function converts a node back into a string.

Helper functions for unit testing.

Using the createNode function enables the ease of unit testing as shown below, greatly aiding development.

isObjectOrSubjectConstructor unit test.

Generating Metadata

Once a node is classified as part of an RxJS statement the metadata for this part of the expression must be generated. The extraction of metadata from a node is for the biggest part fairly straightforward.

In my case, I’ve collected the file, line and position of the node, generated a UUID and collected the UUID’s of connected observables, pipe operators and subscribers. This was enough data to visualise a graph of the observable structure in the GUI.

For things like fetching the identifier belonging to a given observable, it’s key to take a look at the AST Explorer tool yet again. Based on the structure of an AST for a given RxJS statement, a recursive algorithm can be constructed to traverse to the identifier node starting from the observable node.

fetchIdentifier recursive traversal function.

It’s important to remember that the transformer performs its magic during compilation, while the metadata must be sent to the GUI application during run-time. The reason being the desired result is a dynamic analysis of a program, not a static analysis.

To reach this desire the metadata is stored in the AST. Because a normal object literal can not be stored in the TypeScript AST, a TypeScript object literal must be created for it.

The creation of the TypeScript object literal node.

Replacing Nodes

Now the nodes are classified and the according metadata is generated, the nodes must be replaced. The goal of this part of the transformer is to replace classified nodes containing parts of the RxJS statements in a fashion where the original behaviour as intended by the programmer is unaltered, but the generated metadata will be collected and sent to the GUI application.

This will be done by replacing RxJS statements with a wrapped version, for example:

new Observable<number>();

will be replaced with:

wrap(metadata)(new Observable<number>());

Upon execution, the curryable wrap function will send the metadata and return the observable construction to the program. In this way, the normal execution of the program will continue as if nothing has been changed by the transformer.

The following code snippet creates a nested call expression node, resulting in the call of a curried function call as seen in the previous code snippet. The original node, together with the metadata and a specific sendMessage identifier function, is passed along as arguments.

The function returning the replacement node.

As seen in the above example the touch function is called upon the original node. This is required because of the way the AST is traversed during transformation. The original node is replaced with a wrap node, containing the original node as a child node. Without the touch function, this original node will be replaced by a wrap node on every repeating occurrence, triggering an infinite loop. By marking a node as touched, the dispatcher will have the knowledge this node is already transformed preventing it from sending it off to be transformed again.

Tree data structure top-down traversal.

Importing Functions

The replaced node is a function call to a function originally not existing in the program. The problem this raises is its reference is unknown. To solve this issue the transformer must add the required import statements to the transformed files.

As you might have seen inside the AST Traversal code snippet earlier, a Set of RxJSParts is created during the traversal of a source file.

const imports = new Set<RxJSPart>();

Upon each dispatch of a node, the classification is added to this set, utilizing the property of a set only to store unique values. After finishing visiting every node of the AST, this set of classifications is handed over to the importer. The importer creates the appropriate set of import declaration nodes and adds them to the AST recursively.

Import Function Responsible for Adding the Appropriate Import Declarations.

Replacement Functions

The last step in writing a TypeScript transformer is creating the replacement functions. In some cases this as simple as sending the metadata and returning the original code.

Simple Replacement Function.

Final Result

As can be seen in the final result below, a lot of metadata has been injected in the code. All this metadata plus some metadata generated during run-time is sent to the application containing the GUI for graphical analysis.

Left: Original Code, Right: Transformed Code.

The application consuming all this metadata will be covered in a future article. I hope this article presented some insight into the difficulties surrounding the writing of a TypeScript Transformer.