W Key Developers & The Compiler

W Keying is a term borrowed from behemoth zeitgeist, Fortnite.

It describes those players who relentlessly and uber aggressively press forward in all-out-attack mode, spamming bullets as they go.

Yet, it could also easily describe the loudest voices in the world of web development over the last 10 years.

Relentlessly pressing forward with endless new tools and frameworks (Ember, Backbone, Angular, React, Vue etc.), coercing impassioned developers to follow suit and hop from their last church to this new one… only to find once again, false promises and falser prophets.

The thing that has worn me down most over the years is just how long it takes to get from an idea to MVP. And I’ve started to wonder why?

This is in part due to my own failings to stay true to the kernel of an idea, but it’s also because development is synonymous with the choreography of myriad tools and scripts just to be in a position to build, deploy or run an application.

These tools do not always play well together, and the error messages don’t always tell you why.

Worst of all, the complexity scales with the number of artefacts you need (eg. mobile client, web app, server, database etc.).

It all adds up to days lost, momentum lost, and the fire for an idea being irreparably dampened.

Here is just a taster of a common web development toolchain:

• NPM/Yarn — package management from a project manifest file
• NPX — Package runner
• Babel — Javascript transpiler to ensure modern JavaScript constructs are compatible with older environments
• TSC — TypeScript compiler that transpiles TypeScript code to JavaScript
• ESLint/TSLint — Code linting per defined rules
• Jasmine/Mocha/Jest — Testing frameworks
• Gulp /Grunt— Build configurations
• Watchman — Triggers based on file system changes
• Webpack/Browserfy — Transform Node code to be compatible with browser environments

The list goes on, and whilst all do a useful job, the fact that they exist as separate tools you have to burn days of time and energy wrangling is a sad indictment of where we are right now.

On top of this, I’ve witnessed even the smallest development teams insist on a separate repository for each and every artefact they need — a problem further exacerbated by the microservices revolution.

So now you have issues like:

Issue: How do we share common type definitions or interfaces across artefacts/repositories/codebases?
Answer: Yet another(!) repository/codebase that all artefacts can depend on!

Issue: What if artefacts execute require different languages?
Answer: Duplicated effort to implement the same logic or data structures in multiple languages (which is prone to bugs from bad ports)

Issue: But if we use one big repository/codebase how can we deploy separate artefacts?
Answer: By decoupling repositories/codebases from the actual artefacts. They do not need to be intrinsically linked!

So combining the issues of complex (and often fragile — ‘it worked on my machine!?’) toolchains, with gratuitous repositories, and it becomes very easy from an objective standpoint to see how the joy can soon ebb out of an idea, like the sands of time it cost you to get to the sullen point of dejection.

But note, this is in no way a criticism of hungry, passionate and eager engineers, for I too am one. It is a realisation that my time should always be spent on crafting my app, not configuring my dev environment.

And currently that is not the case.

Fixing Friction

With just a programming language and a compiler we actually already have the building blocks we need to fix this problem.

You already know your chosen programming language like second nature. The syntax is rich and expressive, helping you articulate any concept you can think of — concepts that could range from business logic, to yes(!), build logic!

The trouble is that compilers are currently a blackbox. And too often considered a magic blackbox.

We rely on them for objective optimisations, but they don’t know enough about our business context to perform subjective optimisations. Nor do they give us the keys to do it ourselves.

Objective Optimisations

I define objective optimisations to mean transformations that the compiler can perform without any knowledge of the problem you are trying to solve.

Examples include:

• Dead Code Elimination
  Removing chunks of code that are not referenced/utilised by the application
• Baking Procedure Arguments
  Hardcoding parameters or polymorphic types in procedure definitions to lower runtime overhead
• Loop Unrolling
  Replacing loops by a sequence of statements
• Function Inlining
  Avoiding the overhead of a function call by replacing the call site with the function body (at the expense of binary size)

Subjective Optimisations

I define subjective optimisations to mean transformations that you can inject into the compiler in order to suit your business use case.

Examples include:

• Codegen
  Generating hardcoded app code chunks dynamically (like boilerplate or computing expensive data sets)
• Introspection
  Analyse the metadata and structure of your program, including the symbols and types used, and where
• Modification
  Omitting code you don’t need for a particular configuration, or transpiling the code for older environments
• Injection
  Pulling other source files or plugins into the build (emergent manifest)
• Assertions
  Failing the build if certain business conditions about the code are not met (and even generating these from telemetry data about your app in the wild! No more meaningless unit tests!)
• Linting
  Emitting code that is compliant with the configured linter rules
• Semantic Analysis
  Being prescriptive about the semantics you want to validate your source text against, not having this predetermined by the compiler (eg. compiling JavaScript syntax against iOS semantics for transparent, native-out transpilation to Swift/iOS)
• Multiple Artefacts
  Generate artefacts in different languages and different deployment environments, that can implicitly share commonalities between them like type definitions or interfaces (eg. your server and mobile client needing to know the shape of a User object)

Build Logic

I define build logic as the encompassing set of all objective and subjective optimisations.

As the name implies, all this happens at build time. Not run time. With the resulting artefact optimised for your use case:

• No more having to make the user wait with a loading spinner whilst you perform some set up logic that could have been computed and cached during compilation
• Get rid of checks like Platform.OS == “ios” being evaluated every time a code block executes. You can decide that kind of thing when you build an artefact, and strip out code that will never fire
• Native code emissions without having to learn a language compatible with the target platform, or use a virtual machine
• Smaller binary/payload size

Generally you might class some of these transformations as metaprogramming, which is not a new idea in Computer Science by any means.

But with ever increasing commonality, developers are opting to build their applications from a web inspired foundation, and hence are currently starved of metaprogramming options to support and streamline their efforts.

Sempiler

The original Sempiler sales pitch was:

• Write code in a programming language syntax you are comfortable with (eg. TypeScript)
• Apply analysis using the semantics of your target platform (eg. iOS)
• Emit valid, functionally equivalent code in a native format for said target platform (eg. Swift)

Which means:

• You do not have to learn new programming language syntax
• You do not have to sacrifice performance by abstracting away the target platform (blackboxes, frameworks, virtual machines etc)

You’ll notice this workflow is just a subset of the subjective optimisations discussed above.

The goal of the project is to support all of the build logic scenarios I described.

Building Out Build Logic

The problem space of building out build logic can be split up into two distinct parts, Articulation and Evaluation.

Articulation

We need to devise a way for source authors to articulate that a piece of code must execute during build time.

We need to consider the following properties:

• Unambiguous
  Build logic sits alongside the business logic, so this articulation must be unambiguous — ie. the compiler should know for sure that the author wishes to execute a particular piece of code at build time
• Reusable
  Build logic may also be business logic — eg. a function you wish to call at build time to compute something may also be invoked at run time
• Appropriate
  The way build logic is expressed will vary according to the particular source language used, the syntactic constructs that parser supports, and how it feels in-situ

For example, with TypeScript we already use a modified parser, and so extend it further to allow use of # symbol to articulate directives — a lexical construct inspired by JAI.

Evaluation

We need to devise a way to evaluate build logic to obtain a result and feed it back into the program AST.

We need to consider the following properties:

• Order
  Given nested directives, we should evaluate the innermost first — much like parenthesised expressions
• Isolation
  Ensuring we only execute the build logic code at compile time, and suppress the actual app entrypoint
• Context
  Running the build logic in the same context as the target context for the app (eg. Java/JVM)
• Mutations
  Consuming mutations that occur as a result of evaluating each directive, and updating the program AST

Implementation

Imagine our target platform is Java, and our source language is TypeScript.

We want to:

• Take some TypeScript source with directives in it as input
• Emit a compile time version of the Java code to evaluate those directives
• Execute the compile time Java code, and update the program AST with the results (mutations)

Here is our source program, with the lines of interest being:

• Line 33, we use a directive to tell the compiler we want to invoke explosive() at build time
• Line 10, inside explosive()we have a nested directive to invoke ha() and mutate the AST to store the result of doing so— ie. the resulting artefact will have const i : int = 23;
• Line 16, 21, 26, tell the compiler to evaluate this if/else statement at build time and mutate the AST so only the relevant code remains — ie. the resulting artefact will have:
  System.out.println("Case 3!");
return 23;

Our simple TypeScript input with Directives

Now we need to convert this original source intent into a program we can execute at compile time to evaluate the directives.

Helpfully there is prior art in the Sempiler codebase to help us with emitting code in a different programming language, and passing the sempiled output to another process.

But Sempiler is written in C# with .NET Core. So how can we consume the mutations from the Java process back into the program AST?

At first I considered emitting the AST inside the Java code, having the Java code mutate it in-situ, and then serialise the AST back before it exits.

This is useful because you do not need to keep crossing the C# to Java boundary, but it bloats the Java code and means any other concurrent process orchestrated by the compiler would not see the changes to the AST as they happen.

Additionally, how would you do things like asking the compiler to compile new sources dynamically if you have no mechanism by which to communicate with it?

So I then considered some sort of interop solution between the CLR and JVM like the IKVM project — but this project is not under active development, does not scale to all target contexts (eg. Swift, C++), and just felt like overkill.

If you have been following my articles you might remember I wrote previously about Microsoft’s CodeDomProvider. I have since ruled this out because of it’s limited language support (C#, VB, JScript) and concerns over how accurately it would simulate the actual target context.

Lastly I considered a duplex socket implementation, in which the C# compiler acts as the server end responding to messages from the Java client end.

This works synchronously because the Java process will block whilst waiting for the compiler host to consume a message and respond, which alleviates race conditions.

The Java process does not end up with a potentially out of sync copy of the data because the build logic is run in a single thread (even though Sempiler itself is multithreaded).

This felt like an elegant and flexible solution that did not bloat the Sempiler project, and was not limited to a particular target context. Winner!

The API that source authors use to communicate with the compiler is still in design, but I am thinking it will feel like the DOM because that is a familiar model to Sempiler’s initial target demographic of web developers.

Regardless of the outward facing facade, under the hood the sockets are exchanging marshalled commands such as:

• ReplaceNodeWithNodes(nodeID, replacementNodeIDs)
  For when we want to remove a node from the AST, and put the specified replacement nodes in the vacated location — eg. replacing an #if directive, with the set of statements it evaluates to
• ReplaceNodeWithValue(nodeID, type, value)
  Similar to the above, but removing a node from the AST, and replacing it with a new node representing the given type and value — eg. replacing a#run directive by the value it evaluates to
• DeleteNode(nodeID)
  Removing a node from the AST — eg. removing a void #run directive once it has been evaluated

This is just a very small subset for demonstration purposes. In reality the API will expose all manner of queries and mutations, and injecting source files or plugins dynamically.

Now to look at some code.

We can see the marshalled commands present in the compile time Java program that Sempiler has generated, with the interesting facets being:

• The unique and intentionally noisy ID (ie. __SomeLegalCTIDHere123456__) that is used throughout the file in order to hide compiler generated code amongst user code without conflicts
• Line 1, Sempiler has wrapped everything in a root class which it uses as a container for global mechanisms that the nested code can refer to
• Line 2–24, the client socket that is used to communicate with the compiler
• Line 25, 28, 32, the marshalled command implementations
• Line 36, source files are emitted as nested classes
• Line 38–50, an #if directive is computed once and cached
• Line 43, 45, the AST mutation as a result of evaluating the #if directive
• The pretty printing is for demo purposes, and in reality this code would be minified

Sempiler generated Compile Time Java Program for Directives

And here is the generated program entrypoint:

• This main() method will take precedence over any user defined main()method in nested scope, which is crucial to ensure only the build logic executes and not the actual app (ie. the aforementioned Isolation property)!
• Line 100, establishes the socket connection
• Line 101, calls into a nested class to ensure the JVM loads and executes the static logic (this lazy class evaluation by the JVM was a gotcha!!)
• Line 102, ends the socket connection
• Line 105, unexpected exceptions will result in a non zero exit code which Sempiler detects to signal a failed build

The Entrypoint for the Compile Time Java Program

Note that whilst this solution adapts the input to adhere to Java semantics, the actual principles are agnostic and could easily be reworked for other contexts.

Let’s switch back to the compiler now, and look at the code that executes the compile time Java program.

Here are the interesting points:

• It does not need to know or care what the source language was
• It does not need to know or care about any previous or subsequent transformations that mutate the program AST
• Line 72, it will only bother executing a compile time representation of your code if you’ve used build logic
• Line 74, we write the compile time version of the source text to disk
• Line 78, a delegate that will parse and interpret messages received from the Java client process
• Lines 85, 89, 94 are where we mutate the AST and send the result back to the Java client process (note, placeholder response code pictured!)
• Line 106, invalid messages will result in us killing the Java client process (and propagating a failed build error eventually)
• Line 113, we use a CommandLineConsumer to invoke the java executable (the same solution we use for passing sempiled results to tools downstream in the Consumption phase)
• Line 129, if the CancellationToken we were passed is cancelled, we will also cancel the Java client process
• After consumption we also perform clean up like stopping the server, and deleting the files we wrote to disk (not pictured)

Executing the Compile Time Java Program

Finally, now let’s run the program and examine the output as a sanity check:

Output from Executing the Compile Time Java Program

One thing to note is that we only see the messages that were sent using the socket stream, and not the messages written to System.out (the standard output stream in Java).

This is so that the compiler can be sure that any message sent on the socket are intended as marshalled commands.

Hence, if the compiler fails to parse one of these messages it indicates the message is malformed, and unambiguously a fatal error.

Moreover, the underlying cause of the error would most likely be a compiler implementation bug, seeing as the compiler generates the compile time program (unless the user code happened to find and use the socket!).

Contrast this approach with sharing a stream such as System.out. The compiler could not know for definite that a message was a malformed marshalled command. After all, it may just be user code printing some data!

End Goal

For getting this far in the article you deserve to be rewarded by knowing how all this would make your life more pleasant as a developer.

This is best illustrated by describing what your workflow could be…

Imagine, you sit down with a burning new idea, your code editor open and the Sempiler executable.

As you’re writing the code for your masterpiece you think, ‘Oh I need to pull in that file’, so you just write the build logic simply and easily inline for doing that:

function includeX()
{
    Sempiler.addSources("foo/*.ts");
}
#run includeX();
// ... your awesome app code ...

Because build logic is just code like the business logic.

It can be one or the other, or both.

You need a plugin? Same again — plugins are just source code too.

You want to transform some part of the program based on a config value? You have the power in your code to do it.

What about QA? With the #assert directive you can ensure you never produce an invalid build for your business use case ever again.

Even if your team is just using a single codebase/repository you can configure and build as many different artefacts as you need, and share common code between them as appropriate.

Transformers, emitters and consumers are fully scriptable, right there in your source files. No need to write separate DLLs, maintain manifest files or configure complex toolchains.

You can write subjective optimisations bespoke to your app, that you have full control over thanks to democratised compilation.

Collaborators can hit the ground running with just the repository cloned, and Sempiler downloaded. No more days lost reading stale onboarding docs, struggling to get the project working on a new machine.

Put simply, you are never taken out of the flow, and that is the most crucial part —much more of your time and energy is invested in writing your app.

So imagine, being completely focused on your code from the first moment, rapidly iterating, experimenting and tweaking without interruption.

It’s about time we cracked open the compiler, and made that the reality.

Sempiler is coming Summer 2019. 💎