Talking to JavaScript: A Story of Types and Code Generation
Developing an application is rarely a job for a single programming language. Be it building a script, using a lower-level API, or debugging an obscure bug, they all require the developer to know several programming languages and tools, and how they’re integrated.
In web applications this is the bread and butter. The developer not only has to know several programming languages, but he also has to consider different execution contexts: the client and the server. For this, the developer needs to maintain clear and compatible APIs, but also synchronize relevant data in a disciplined way between both sides.
The Web is Everywhere Now
Nowadays, the Web is everywhere. With the rise in popularity of technologies like Node.js, JavaScript has invaded the server and is now widely used on both sides. Actually, it is the most popular language in both the client and the server, according to StackOverflow’s 2017 survey. Programming in JavaScript has also changed a lot in the last years: it is the target of several new languages like Microsoft’s TypeScript or Google’s Dart, which extend it with new features, types and cleaner syntax.
So one might ask: why not solve the problem using JavaScript? A common language enables greater reuse, and static types allow for sane, compile-time checking of APIs. But we can’t really write the world in JavaScript, can we? We might still want to integrate with some other system, written in C#, Java, or a plethora of other languages. Interoperability between languages and maintaining API contracts are still issues we have to deal with.
Interoperability Between JS and C#
That’s precisely the case of this incredibly awesome IDE. Although it is mostly built on top of C# and WPF, several dialogs and editors are built using standard web technologies like HTML, CSS and JavaScript (actually TypeScript). The Web Screen’s WYSIWYG editor was perhaps the first example of this, but other examples include the Aggregates’ editor, the Entity Diagram viewer or the Consume REST configuration. They run inside embedded browser controls in the overall WPF UI, communicating with C# through well-defined APIs and protocols.
Keeping the APIs between JavaScript and C# synchronized is a repetitive and error-prone process, aggravated by the lack of static typing in some steps. Not only that, but we also need to ensure changes to the state of the model (C#) correctly affect the state of the view (JS). Often, this requires specialized knowledge that is neither present in all our teams, nor should it be required to build a new editor from scratch.
Early on, we adopted TypeScript instead of writing plain JavaScript code, using types as our contract between both contexts. Although it reduced the amount of errors in the JS-side, we still had to go through the same error-prone process of keeping both sides in sync, often stumbling upon new fields that were not serialized to JS, or JS events that were not triggering their handler in C#. There had to be a better way!
Throw Code Generation At It!
At OutSystems we have a culture of using code generation to solve big issues — from the very core of our product, which generates C# or Java code from visual models, to a myriad of tools we use in our internal development processes. It allows us to generate several artifacts from a single definition, increasing our productivity, freeing us from writing repetitive code and ensuring no off-by-one errors are inserted in the process. Besides, it is fun!
Considering our pain was about repetitive and error-prone tasks, we decided it was time to throw code generation at the problem once again. We already had the view APIs and model clearly defined as TypeScript interfaces and modules, so we started building a tool, TS2CS, that allowed us to parse that information and generate the repetitive C# glue-code needed for everything to work. At first, the APIs were “blindly” generated using C#’s Code Dom library, but as the architecture evolved we moved to the use of templates (using T4) for added flexibility and (very important!) legibility.
Now, we didn’t want to translate all the definitions in a TS file to C#. Our editors are written in TS, so those definitions are much more than an interoperability API specification. They may contain APIs that are only relevant for the browser, or private functions, or transient state of the editor. We had to define which definitions would actually be translated to C#.
Our first approach was to exclude definitions that were not exported or public, but this semantic overload had an obvious problem: often, we wanted something to be exported to other TS modules, but we didn’t really need it to be exported to C#. We needed a non-intrusive way to specify those exceptions, so we turned to the use of annotations.
TypeScript didn’t support annotations at the time, so we rolled our own syntax using comments. We started using an ignore() annotation to solve our problem, but soon we started using annotations to mark classes as abstract, or members as read-only, and other template-specific use cases.
TS2CS to Glue It All Together
Since its creation, TS2CS has been used in the development of several new editors, created by distinct teams, sometimes with minimal context about the glue-code that is being generated and that makes everything work together. All of it with very few adaptations to the templates we use, and even fewer to the tool itself. With just 7 editors using it, the tool already generates 9k LOC of glue-code that supports the APIs of 40+ model objects and 120+ view events. It has paid off the initial investment and has greatly improved our productivity and confidence on the quality of what we build.
That’s not to say there’s no room for improvement.
The need for TS2CS started back when TypeScript was in version 0.8, so we implemented a custom parser that mainly only cared about definitions (Modules, Interfaces, etc.) and not the whole language. Since then, the language has been extended with annotations, abstract classes and many other constructs we don’t support or take advantage of.
Luckily, direct integration with the official TypeScript compiler is much easier now, and it’s definitely in our next steps towards the evolution of this tool and its open source release.
