The risks and benefits of building software with cutting edge technology

About three years ago I was approached by Giorgio Serra and David Stone, the two founders of Exit Live, an online platform that allows artists to sell their live recordings to fans. The project was nothing more than an idea — albeit a very vivid one. They had found me through my open source projects, notably Dropzone, and thought that my skills would be a great fit for the company, so they offered me a non-binding invitation to come and work out the foundation on which Exit Live should be built technologically. I’m sure they knew that their enthusiasm about the project was infectious, and that eventually I would become a core member of the team, as lead developer and CTO.

At that time, I was primarily programming in JavaScript; both on the backend with Node.js and the frontend with various bigger frameworks (like Ember.js or AngularJS) or smaller libraries and tools, and there was one thing I had learned: never to start a big project with JavaScript again.

There are many things to love about JavaScript: it’s omnipresent, easy to dive into, well supported by practically every Editor and IDE, has a huge user base, and libraries for everything you can think of. But there are also many things to dislike about it and I knew right from the start that I didn’t want to fall into the same trap as before: rapid initial development at the cost of messy code that is hard to maintain.

The server languages we considered were:

• Old-school and battle-tested Java (Spring, for example)
• Ruby (with Ruby on Rails as a prime candidate)
• Go (a solid language but a bit outlandish)
• Dart (the new kid on the block, with no prime examples)

The client languages were:

• JavaScript (the only native language in the browser)
• TypeScript (Microsoft’s JavaScript superset)
• Dart

CoffeeScript was not really considered because I had developed a hatred for it (after writing many lines of code in it).

Why we chose Dart

My preference at that time was Dart. Not only was I personally involved in, and attached to, the language (my brother Florian Loitsch was working on the compiler at that time and has since been promoted to the library manager) but it seemed to have a lot going for it:

• It was typed, but not as heavy as Java, since it introduced a fairly novel concept of optional types. This alone solved many issues I had with JavaScript.
• It came with it’s own runtime, meaning that we could use it directly on the server as well (and not compiled to JavaScript running on node.js).
• It came with many great core libraries, making it very comfortable to work with the DOM without needing third party libraries.
• It had its own package manager (unlike JavaScript where you have to choose one of the many package managers).
• It had an opinionated style guide that we could follow, instead of defining and writing our own guides.
• It was the only language that had the possibility to make it as a native browser language with its own VM (which, unfortunately, never became reality).
• It was potentially faster than JavaScript and Ruby since the developers who created the language came from the JavaScript V8 engine and designed the language to be more optimisable.
• It is very familiar to, and easy to learn for Java and/or JavaScript developers.

All of these pointed to Dart being a great candidate! We would have the same server and client language — a language I would enjoy writing in (unlike Java or JavaScript), that I was familiar with (unlike Go) and where it would be easy to get new developers due to its familiarity with JavaScript and Java.

There were obvious downsides as well of course. Dart was in its infancy, and there was no way of knowing whether all the promises made by the Dart team would be fulfilled, whether we would have all frameworks necessary to finish the project or if the development of Dart would simply be canceled one day.

We discussed all these pros and cons and finally came to the conclusion: let’s take the risk, and build something with tools we’d love, not tools that are safe.

It was a bold move, but Exit Live was all about being bold.

Our first year with a language nobody knew

The first months were full of ups and downs, and we needed to roll with the punches. There were times at which I wasn’t sure whether everything was going to collapse, but in the end we made it through.

The programming aspect

At that time, the best way to work with Dart, was with their dedicated and now discontinued IDE DartEditor, which was a customised Eclipse build. Although I had never liked working with Eclipse before that, they did a great job reducing all the unnecessary clutter and optimising the experience for Dart. Working with their editor, and using their workflows made sure that everybody in the team got the full “Dart experience”: code completion, type suggestion, navigate to definition, etc… I think that this saved us a lot of time during our setup phase, because we didn’t need to start configuring tools and writing guidelines, which is especially difficult with a new language.

Apart from minor annoyances and issues here and there, the development process was quite enjoyable. What wasn’t enjoyable was the growing concern about their capability to deliver a server solution in time…

Choosing our server infrastructure

Before I started working at Exit Live, I was working on a very ambitious shopping platform. As was modern at that time, we built our whole infrastructure on “microservices”. We had a file server, a template renderer, a backend, a frontend renderer, etc… all either Node.js or Java instances communicating with each other through JSON APIs. Although the concept is nice, it was a nightmare to deploy and debug, especially since we were running our own servers with our own scaling tools.

I am not a “server guy” and I don’t enjoy fiddling with server configs, so with Exit Live I wanted to go the complete opposite route: use Google’s App Engine solution — a completely hosted, autoscaling server infrastructure with load balancing and health checks.

The Dart team told us, that it was not long, before Dart was one of the supported languages for App Engine, so we decided to wait for it and start building the app blindly until we get all the specification on how to run a server on App Engine.

Unfortunately, at the same time in 2014, Google was working on a complete overhaul of their App Engine and Compute Engine services with the intent of introducing a hybrid: an autoscaling, managed server environment (like App Engine) but with full control and flexibility over what runs on these servers (like Compute Engine) which is now known as Google Flexible Environment.

This led to the App Engine team not wanting to add Dart as a supported language, with the argument, that soon it would be possible to run Dart servers with the new flexible environment. There were no specific timelines for these changes, so we had to hope and wait and see.

Because our project was quite ambitious and one of the biggest ones written in Dart, we received an invitation to do a presentation at the Google I/O 2014 — ironically the same Google I/O where they would announce their new Google App Engine solutions. So here we were, setting up our infrastructure on hacked together servers to be able to present our work at the same conference where a crucial and missing part of our platform was being presented and released.

We ended up showcasing an RPC library that we had written, that allowed transparent communication between our server and client, with classes that could be used on both sides. Since there are types in Dart, you would create your message in the client, with full autocompletion, send it to the server, and know what to expect on the server, instead of sending arbitrary JSON data. It was the perfect example to showcase Dart’s strengths.

Our Presentation at the Google I/O 2014

Apart from the lack of a proper server infrastructure, there was also a painful lack of stable and tested server frameworks to work with. So we built something on our own, knowing that we will have most of our presentation logic in the clients.

Choosing our client framework

The more we advanced, the more it became crucial to find a good client framework that we can build on. Dart has excellent libraries to manipulate the DOM built in, but building an app, the size of Exit Live, requires a solid framework to build on, with proper routing, dependency injection, etc…

We had our eyes on polymer, yet another incredibly young and unknown project at that time. We went for it because there was a team dedicated to allow interoperability with Dart, and we wanted to use the frameworks the Dart team endorsed.

Polymer is not an application framework though. It lacks routing or dependency injection, it’s merely a very elegant way to encapsulate and reuse elements on a website; which was good enough for us at that time.

But working with Polymer wasn’t pleasant. The Dart version was just a wrapper for the JavaScript library underneath, so a lot of strange bugs and behaviours needed to be debugged on a daily basis.

It was later announced, that Angular was going to support a native Dart version, and as soon as it was released, we started porting the client to angular. For about 6 months, we were running Angular and Polymer simultaneously until we got rid of Polymer completely.

All the rest

Writing a platform like this, with a language that was about a year old and a tiny community proved to be very challenging in many other areas. Trivial tasks became so incredibly complicated that it was nearly comical.

We decided to go with the Google Datastore — a decision that only seemed logical given that we wanted to use all of Google’s server infrastructure and that we had made the decision to accept we would be locked into their services to allow faster development. There was just one problem… there was no Dart library to communicate with the Datastore! So we had to start implementing our own library (and of course, later on, exchange it with the official library published by Google itself).

We got an alpha version of the Google Flexible Environment (which at that time was called “Google App Engine Managed VMs”), so we could start developing with it, but deploying or running a Dart app on it proved to be a lot more complicated than we had hoped. We would run into regular issues of “Too many files”, because the deployment process tried to copy all dependencies recursively due to the way Dart programs were structured at that time. All in all, the refresh cycle to test changes became really slow. But it got gradually better and better, and the Dart team did a great job in solving all issues we encountered.

It was the wild west… libraries appeared and disappeared. People created projects and abandoned them. It was hard to decide which ones to use. It is never easy and the risk of an open source project dying exists in every language, but it was intensified due to Dart being such a new language.

Some of my lolcommits during the first year

Bringing everything together — the good parts

In the end, we got it all to work. We are running on Google Flexible Environment, with Continuous Delivery, a solid code base and an excellent development environment.

We are now reaping the benefits of choosing Dart and all the chaos and roadblocks we had met are a distant memory — funny stories we remember from time to time and can’t believe we have experienced.

I started most of my previous projects with enthusiasm, but as the projects grew so did my dissatisfaction and indifference towards it. Something that is very unique to this project, and that I had never experienced anywhere else, is the feeling that the bigger our project gets, the happier I am with our codebase and the way our project can be maintained. Some of this is due to my experience and what I have learned about project management in the past, but a lot of this must be attributed to Dart.

Dart formatter and style guide

Two seemingly minor utils in the Dart development process proved to be huge time savers: the excellent Dart Formatter (written by Bob Nystrom) and Dart’s extensive style guide. I have spent hours (probably days) discussing formatting issues in merge requests. Should you use tabs or spaces? Is it better to add a newline after an if (condition) or should the curly brace be on the same line? I have my preferences about all these questions but in reality I simply want them to be uniform! I want a decision to be made, and ideally I am not the authority on it, because I will second guess my decisions and start to refactor my code every few months.

We have setup our tests in a way, that the first test we run checks whether our codebase is compliant with the Dart Formatter. This means that it’s impossible for a merge request to introduce code that is not perfectly formatted. No more discussions; no more ”You forgot a space here” review comments.

We do the same thing with the dartanalyzer, and reject all merge requests where the dartanalyzer produces a Warning. The dartanalyzer can be configured to follow nearly all Style Guides, so introducing code that doesn’t follow the official guides is practically impossible.

Tree shaking

One of the biggest undervalued features of Dart, is its tree shaking when compiling the client code.

Dart automatically removes all unused code from your project when generating the resulting JavaScript code that will run in the browser. This means that you don’t need to think about whether you’re importing an unnecessary large library in your project.

In the world of JavaScript this is a very common problem. Libraries pride themselves on being as small as possible — in fact, it’s one of the first metrics you often see on their website.

You might want to use Moment.js but is it really worth the 16kB just to display a date? Some jQuery functions are nice, but do you want all the unnecessary baggage you get with it? This issue is the root of many JavaScript package managers that try to address it by splitting up libraries into the smallest parts possible to avoid shipping unnecessary code.

These problems don’t exist in the Dart universe. You can import the biggest libraries you want, and only use one function if you wanted to — all unused lines will be removed during the compilation process.

Refactoring

Compared to JavaScript, refactoring is a breeze. I’m sure every JavaScript developer knows the anxiety you feel when you rename an attribute, and hope that you got all occurrences. Whether you’re using an IDE that offers a renaming function or you’re writing your own Regular Expression I have never felt confident that my JavaScript project will still run after I did some big refactoring.

This is not an issue in Dart, so you can make sure that your code base stays clean. When renaming a class, variable or attribute becomes a risk, you tend not to do it, and so your code tends to become messy.

Core libraries and tools

Ever had to decide which Promise/Future library to use in JavaScript? Which package manager are you using? Ever heard of Grunt, Gulp, Webpack, Bower, etc… You can’t ignore them, because out of the box, JavaScript by itself is terrible, but the choices are endless, and every year there’s a new shiny hot tool that everybody jumps on board with.

I was part of this crowd, and I know all to well the excitement I got when a new JavaScript library or tool was announced that would finally make something easier or solve another problem.

Dart comes with everything you need, and is opinionated about the tools you should use, and that is so incredibly helpful that it is probably the number one reason why I like using it so much.

You want to install packages? Use Pub. Want to automate workflows? Use Grinder. Want to use a Future/Promise library? Use Dart’s native libraries and async/await keywords. Want good DOM manipulation functions? Dart’s got you covered. Code Formatting, Debugging, Analysis, Testing and mocking frameworks? Yup. All there. And not 10 solutions to choose from, but in most cases one that is the standard.

Now, are all of these libraries and utilities perfect? No. But they are good and getting better, and many of them are near perfect.

Our first drawing board

Would I choose Dart again?

I am now incredibly happy with the way our project is built. I still enjoy using Dart as my primary programming language, and it is getting better every year. Our code is stable, fast to debug, well suited to collaborate and easy to read and understand. So I am very pleased that we decided to use Dart three years ago. I would use it again for my next project without a doubt.

If I had the possibility to go back in time, and tell myself what to expect in the first year of development? Well… luckily I can’t.