Ideation of an IDE: The .NET and ReactJS Love Affair
Some challenges are bigger than others. In early 2019, we began creating a cross-platform version of our integrated development environment (IDE), Service Studio. We completed the project two and a half years later, with the launch of our new IDE in late July 2021.
The bold move was driven by the need to give macOS users a version of Service Studio they could natively run on their Macs. From a small market share in 2010 to an increasingly relevant slice of users, developers running on macOS were not enjoying the best experience OutSystems is known for. It was time to change that.
Most developers are aware of how difficult migration projects can be, and some might also be familiar with the additional challenges posed by developing for multiple platforms. The first of a series of articles on this fiercely complex challenge, this post will explain how we reused code, introduced new features in Service Studio, and migrated the existing user interfaces from Windows Presentation Foundation (WPF) to ReactJS without doing two implementations, one for Windows and another for macOS.
We will also provide ready-to-use NuGet packages for .NET C# projects and links to our Git repositories.
But first, some context.
Bye, WPF! Hello, macOS!
In 2007, we rewrote the OutSystems IDE, Service Studio, from scratch using .NET and WPF. The latter had just been released some months before, in 2006.
At the time, the team chose WPF because it was the next generation of frameworks for building desktop apps, made by Microsoft, that would eventually replace Windows Forms.
So back then, it was the natural choice for building .NET desktop apps. But the WPF power came with a cost: a steep learning curve.
Over the years, Service Studio evolved into a much more complex piece of software than it was upon its release, and that evolution meant legacy code. On top of that, we also struggled to find people with WPF skills to join the team.
When we launched the first .NET version of the Service Studio three years later, in 2010, the macOS market share in the development space was almost insignificant. But with the years, it became increasingly relevant, especially in a few larger tech markets, like the US. With that, the demand for a cross-platform IDE grew.
After .NET became a multi-platform framework, the dependency on WPF meant our product was still Windows-bound, thus preventing us from releasing a macOS compatible version.
With the rising demand, we moved forward and decided to ditch WPF, replacing it with other user interface (UI) technologies that could run on multiple platforms (starting with Windows and macOS), were easier to learn, and had great communities around them.
Being an enormous product, we could not simply throw away everything and start from scratch. We had to find ways to reuse a lot of the existing .NET code. Migrating the existing code from .NET Framework to .NET Standard compliant was the first big challenge faced by the team.
Picking a WPF Replacement
Reusing the existing .NET code was critical because we wanted to have a single source of truth (to reduce the development and maintenance costs) and reuse a lot of the (good) work done in the past.
Replacing WPF while maintaining the .NET app foundations was not trivial, and there were very few options when we started the move. We could either find similar alternatives to WPF, like AvaloniaUI or Xamarin.Forms (.NET MAUI or Blazor didn’t exist at the time), but that would not solve some of our WPF problems, such as the learning challenges and lack of people with skills. Or we could opt for other less likely natural choices for a desktop application: web technologies.
Many popular desktop apps nowadays use web technologies to display their user interface: Visual Studio Code, Slack, Figma, Spotify. Some of these apps use Electron, a popular framework for building desktop applications using JavaScript, HTML, and CSS. That is possible due to the use of a browser embedded in the application that displays the interface.
Since we had a lot of .NET code to reuse, Electron did not suit our needs, although its idea was very interesting. There wasn’t a good Electron implementation for .NET apart from Electron.Net, which uses an ASP.Net server embedded in the app — not great in terms of latency and memory consumption.
Using a similar approach, we could create a cross-platform app, sharing the same code among platforms. Also, by using web technologies, we would find more people with skills and eventually build user interfaces for other products of the OutSystems Platform that run on the browser.
The next question was, which web UI framework to choose? Not going into too much detail in this article, we opted for ReactJS as it was (and still is) one of the most popular frameworks. It has a great community around it, and we already had some skilled people in-house.
A Cross-Platform Web-Based UI for .NET
Replacing WPF for ReactJS on a .NET desktop app was not a trivial task. To run ReactJS, we need a browser. Luckily, we already used a browser embedded inside the IDE to display web content.
That browser implementation — CefSharp — is a .NET wrapper that allows you to embed Chromium (the open-source browser engine at the heart of Chrome) in .NET applications. Since CefSharp is not portable and only runs on Windows, we had to find other alternatives.
CefGlue appeared to be the only viable alternative because it was the only .NET portable Chromium wrapper available. Contrary to CefSharp, CefGlue lacked maturity and some of the key functionalities that CefSharp provided: interoperability, JavaScript evaluation, browser lifetime events, .NET Standard compliance, ability to display the browser in-place without using off-screen techniques, etc.
We introduced these features and substantial stability improvements to make CefGlue a viable solution on the feature front. Today, CefGlue is on par with CefSharp but runs on Windows and macOS.
As the CefGlue API is very much tied to the underlying Chromium Embedded Framework (CEF) API, we developed a wrapper with a much more straightforward — yet powerful and pleasant — application programming interface. Any WPF or Avalonia apps can use this component to embed a browser.
However, the browser won’t stand without a host app container technology responsible for creating and managing the application windows.
So, when we started this project, we had two approaches for the app hosting container:
- Use one of the existing multi-platform frameworks — AvaloniaUI or Xamarin.Forms.
- Implement our own app container layer.
We avoided the latter because it would imply knowledge of the underlying native supporting technologies, namely Win32 and AppKit, and consequently two different implementations, one for each platform. We also evaluated Xamarin.Forms but had trouble making it play well with the browser. Additionally, it also needed platform-specific code.
We decided to go with the AvaloniaUI framework because it was the only one that allowed us to comply with the WORE — write once, run everywhere — paradigm. Although it was in the beta stage, we had few dependencies on it and “only” needed to use its windowing features and interoperability capabilities to display the browser inside a window. We ended up establishing close contact with the Avalonia Development Team and, in the meanwhile, they have worked on several improvements we identified in their framework.
Seamless ReactJS Integration
The next big problem on our list was how to make ReactJS and JavaScript play well with .NET code. How do we keep the JavaScript APIs in sync with .NET? How do we pass data to and from each context in a seamless way?
To communicate with JS from C#, we can send a text snippet to the browser containing the JS code to be evaluated. The opposite can be accomplished (JS to C#) by registering special objects on the JS window context that serve as proxies to C# objects.
But doing this communication manually can be error-prone because APIs can evolve, quickly resulting in runtime errors. We searched for existing tools (generators), but none suited our needs, so we created our own tooling.
Using TypeScript is mandatory and crucial for us. Not only is it an excellent language for safely writing JavaScript, but it also allows us to take advantage of its typing information to create the code generators to improve the development process.
We developed a toolset that kicks off at compile time and generates the component bindings (the interop APIs) for C#, making it available as a standard C# class. Bonus points: we can plug it into any Avalonia or WPF application.
The toolset ensures that every time the developer changes the ReactJS component API, the C# bindings are updated, and any errors are caught at build time. The framework is also responsible for generating all the data structures and marshaling between .NET and JavaScript, and vice-versa. This approach allowed us to reuse a lot of the existing C# code with some modifications.
Bootstrapping ReactJS
Setting up ReactJS can take some effort (without extra tooling). One needs to download the ReactJS packages and properly load them into a web page context. Loading styles and additional resources may also require some effort.
To introduce ReactJS and seamlessly integrate it with the rest of our codebase, we created a framework capable of loading a React component along with its stylings and feeding it with data. The best part is that we encapsulated all of this logic in a generic and open-source framework.
With a very minimal setup and after installing the framework’s NuGet packages, developers can start developing their ReactJS components using TypeScript and instantiate and interact with them in the C# code of the app, pretty much as they would do with Windows Forms or WPF controls. ReactJS is also packed in the framework, which in turn will load it, so there’s no need to include it — developers just need to focus on the UI development.
Serving Content
Despite using a browser, all the resources (JavaScript files, CSS files, images, etc.) are embedded in the application DLLs and served by the browser without intervention from the network layer, which is possible due to the usage of custom internal browser protocols. We can serve content faster from the application DLLs and minimize latency introduced by the network stack.
It’s a well-known fact that JS applications tend to explode in the number of modules, and therefore the number of files. That’s okay when you want to favor small pieces for composition and reusability but introduces a terrible overhead when loading all this code in the browser.
Adding to that, several JS module systems don’t play well together, creating problems when we want to import third-party components with different module systems. To solve those issues, we included Webpack — the popular bundle system for web apps.
Webpack analyzes all the dependencies (JavaScript, CSS, and other resources) at compile time and creates an optimized bundle with all the resources. If there’s something broken, such as a missing resource or wrong path, Webpack can trigger build errors.
This is due to Webpack’s powerful plug-in system, which we used and extended, integrating it into the .NET app build pipeline (MSBuild). In the end, the developer can import resources (CSS, images, and others) like they would import JS modules. What’s best? The framework takes care of all the loading resource magic.
For those using external resources, as in external to the app package, it’s also possible to use standard URLs.
Going Asynchronous
Due to the asynchronous browser nature, all the interactions between JavaScript and .NET are asynchronous. This posed a significant challenge because, in the previous model, all non-views logic interacted with the views synchronously.
Due to this paradigm change, we had to take care of concurrency, as it was now possible and frequent to have multiple threads reading and writing over the same data, while as before we used the UI thread for most of the work. Although less than ideal, we used that approach primarily to avoid having to deal with concurrency issues.
To prevent regressions and deal with such issues, we introduced a dispatcher, which handled all the communication between JS and .NET. The dispatcher serializes all tasks to make sure no concurrent reads and writes are made.
While this introduced a bottleneck, it is a much better solution because now we can have multiple browsers on our app, each using its own dispatcher. Instead of having a single bottleneck, each browser now has its own bottleneck, meaning that two parallel and non-dependent tasks can co-occur. It’s like having multiple UI threads that don’t conflict — they don’t read or write the same data.
That Annoying Blank Screen
When the first complex user interfaces gained shape, we began witnessing behavior we didn’t observe with WPF. Whenever we launched a new window with a new browser, the browser would display a blank canvas for some milliseconds until it finally rendered the first content. This is perceived as usual on the web, but on desktop applications, that’s not acceptable.
After some investigation, we found that bootstrapping the browser and loading ReactJS and the user interface took some precious milliseconds, leading to that initial blank state.
To minimize this issue, we created a cache to store one browser instance with previously loaded ReactJS. When a browser is instantiated, we take the one from the cache and load the user interface on it. Then, we create another browser instance and place it in the cache to be ready to use the next time.
While this mitigated the problem, the blank canvas was still noticeable. We also had to create a cache with user interfaces to improve that further, which stores the HTML of an interface’s first render. The cache is used to speed up the interface’s first render, while React renders the component. The cache only stores the HTML static parts to avoid displaying potentially outdated content. With these improvements in place, the blank canvas became almost unnoticeable.
Again, the best part is that these behaviors are built into the framework.
UI Composition
Don’t mess with my CSS
Mixing different components can be a problem because they may have conflicting styles, resulting in broken user interfaces. Because of the nature and complexity of our IDE, several parts of the UI are dynamic, and we can have thousands of possible component (each rendering a UI fragment) combinations, raising the likelihood of conflict.
Creating modular CSS is one way to overcome that issue, and you can use several techniques: some require developer intervention while others need tools to automate the process. Another possible approach is to use DOM isolation, either with IFrames or Shadow DOM.
Although IFrames provide complete isolation (DOM, JS, and CSS wise), they have one massive disadvantage: moving an IFrame inside the DOM tree will cause it to lose its state, making it unviable for our case.
Shadow DOM allows for less isolation (only DOM and CSS), but it was still enough for our needs. CSS styles inside the Shadow root don’t leak to the outside and vice-versa. To use Shadow DOM with ReactJS, we have to use the createPortal API from React. Fortunately, that was also something we abstracted in the framework — making its usage very simple.
Sh** happens
We all know that sometimes things don’t work as expected, so we have to be prepared if something bad happens and have ways to recover from errors. React has the concept of Error Boundaries, which is very useful to create logging and recovery mechanisms.
Mixing these with the Shadow DOM usage, we built a mechanism into the framework that can recover parts of the UI when a serious rendering error occurs. So, when something bad happens, we can reload that part of the UI without touching the other parts.
The Final Picture
In the end, we released a .NET application that runs on Windows and macOS and uses a browser to display 99% of its user interface.
The most fulfilling part is that we managed to reuse the majority of .NET’s code, even if there are still a few specific lines to each platform. They exist mainly to deal with some of the Operating Systems’ differences which sometimes are not abstracted by the underlying technologies.
Open Sourcing
We had to create several tools and components along this journey and open-sourced most of them.
If you are curious, you can check the Git repositories and the NuGet packages, which are ready to include in a .NET C# project.
Each project has a sample application, which allows you to see how they work.
Changing the Tires While the Car Is Running
Thousands of OutSystems developers use the Windows version of the IDE, and we wanted to keep offering them not only bug fixes but also new features.
Because our source codebase is huge, we took advantage of the existing architecture and created a new product development line that shares much of the code but will end up replacing the current Windows version. We’ll deep dive into this in future articles.
This approach allowed us to introduce new features and migrate the user interfaces from WPF to ReactJS without doubling the efforts of having two distinct implementations, one for each product line. By doing this and some feature toggling mechanisms, we can deliver value to the customer and get more feedback sooner while at the same time improving and evolving our codebase.
What a Journey
In the beginning, this project was not easy to predict and estimate due to the amount of uncertainty. We knew we wanted to reuse a significant part of the existing code base because time was running against us — we couldn’t just simply throw away all we had and rewrite it — but the rest was defined as we went along.
To minimize the risks, we did small experiments and proofs of concept, although we had to keep adjusting and fine-tuning along the way.
This was not a typical project in the sense that despite the features’ implementation, we also had to build many infrastructural pieces to support the desired functionalities (cross-platform application) and meet the established requirements (share code among platforms and reuse existing code).
In the end, it was an incredible journey with lots of unknowns and challenges we had to overcome. Many of the roadblocks we bumped into were due to the lack of maturity of some of the technologies we used, while others were due to our inexperience, especially in the macOS development front.
Many of the developers on the team never had used macOS in the development context and had to switch from Windows to macOS along the way. For a great period of the project, we faced instability and application crashes, which proved to be harder to diagnose on macOS than on Windows. When something terrible happens on macOS, getting details (especially a readable stack trace) proved more complicated than on Windows. We were in a new world where we ended up learning a lot of exciting things!
Two different technologies, .NET and ReactJS, can marry and play along very well. What was, at first sight, an impossible relationship proved out to be a successful bond.
Originally published at https://www.outsystems.com.
