Writing a front-end WebAssembly framework in Rust: lessons learned

Introduction

Over the past few months, I’ve been writing Smithy, a very work-in-progress front-end WebAssembly framework written in Rust.

My goal for Smithy is to enable you to use idiomatic Rust to write front-end code. This has costs: for example worrying about lifetimes and using Rc<RefCell<State>> to share state. But this also has the potential to give you the safety guarantees that the Rust compiler provides when writing browser code!

Smithy is still in progress, but you can peek at the code at: https://github.com/rbalicki2/smithy, https://github.com/rbalicki2/jsx_compiler and https://github.com/rbalicki2/wasm_website_frontend.

You can see a deployed example of a Smithy app here! https://todolist.robertbalicki.com

One can write a Todo List in Smithy

Writing Smithy has involved many dead-ends and rewrites (and will certainly entail many more.) It’s been a great learning opportunity, and in this post, I want to detail what I’ve learned about Rust to WebAssembly programming in general and about Smithy in particular.

Note: This post gets a bit technical, and goes into a few design decisions that I hope will be helpful to others writing front-end frameworks 🙂.

Rust might not like self-referential data structures, but Medium has no problem with them

Table of Contents

• Quick overview of Smithy
• WebAssembly is moving quickly
• jsx_macro relies on unstable features (procedural macros) whose API is changing quickly
• Getting things right is hard: the compiler is a cruel master
• Micro foundations for macro ergonomics
• Smithy’s unsafe original sin
• Reconciliation and HtmlToken vs BareHtmlToken
• Long road ahead

Quick overview of Smithy

Smithy can be thought of as a “React in Rust”-ish framework. High-level, it works as follows:

• There is a jsx! macro that turns <div /> into HtmlToken::DomElement(DomElement { ... }).
• A component is created and stored in a thread_local variable in WebAssembly. The event loop begins:
• The Javascript code asks the WebAssembly module for information on how to render the component. It renders it.
• On every event, a message is passed back to WebAssembly (a serialized version of the event and a path to the component that triggered it). If an event handler is found, it is executed, updating the component state, and the Javascript code schedules a re-render.

WebAssembly is moving quickly

WebAssembly is moving very, very quickly. Exciting new tools (I’m looking at you, js-sys and web-sys!) are coming out every week. This means that some of the choices I make today may be prudent now, but won't be the right choices when Smithy becomes stable in the future.

For example, a notable drawback of how Smithy is currently written is that there is a non-trivial amount of Javascript framework code that lives in the crate that the web developer creates. This would be akin to React requiring you to include a special file, and not allowing you to just write import 'react/special-utils';.

This is mostly a hack, and likely could be worked around with a webpack configuration. However, js-sys (which I first heard of a few days ago) and web-sys (not out yet?) both provide great ways to execute external js that are even better that using webpack!

jsx_macro relies on unstable features (procedural macros) whose API is changing quickly

Sometime between when I wrote the first draft of Smithy and now, the proc_macro API changed dramatically. I had learned a lot, so this was a great opportunity for a rewrite!

But still, caveat developer: the ground might shift substantially beneath your feet.

Getting things right is hard: the compiler is a cruel master

The compiler is exacting and unforgiving. You can spin your wheels for long periods of time attempting to placate the borrow checker. And if you are uncertain about the relationship between different parts of your program, the experience is less like “being guided by an infinitely patient friend to the right solution” and more like “trying different combinations until they work”. (The reader is left to infer which of the two better described my experience.)

Furthermore, web development is complicated (involving app state, sub-components, an HTML representation and many callbacks, each of which can mutate the app state.) Rust does not like self-referential data structures (e.g. this), and it certainly will not allow multiple mutable references to the app state!

Case in point: this is how I would currently code an on_input callback that mutated the component's state.

#[derive(Clone)]
struct InputComponent {
  my_string: String,
}
impl<'a> Component<'a, ()> for InputComponent {
  fn render(&'a mut self, _props: ()) -> HtmlToken<'a> {
    // We need to clone the app state in order to use it
    // again in the final jsx!, because it is moved into the
    // state_cell below.
    let self_copy = self.clone();
    // Because there is only callback, we don't need to move
    // it into a Rc<RefCell<InputState>>.
    // If there were multiple, we would have had to.
    let state_cell = Rc::new(RefCell::new(self));
    let set_my_string: Box<events::InputEventHandler<'a>>
      = Box::new(
        move |e: &events::InputEvent| {
          let mut state = state_cell.borrow_mut();
          // not all InputEvents have a value
          if let Some(ref val) = e.value {
            state.my_string = val.to_string();
          }
        }
      );
    // We're using my_string twice, so the first time,
    // we have to clone it!
    jsx!(<div>
      You entered: {self_copy.my_string.clone()}
      <input value={self_copy.my_string} on_input={set_my_string} />
    </div>)
  }
}

Whew! We have Rc, RefCell, Box and lifetime parameters on structs! But change any of it up, and the Rust compiler will stop you.

Micro foundations for macro ergonomics

If we draw any conclusion from the previous example, it’s that the Rusty solution for web-development can be quite involved. (Also, I doubt that the above solution is ideal. Open to suggestions!)

The goal of Smithy at this stage to “get it right at a low level” and take as few shortcuts as possible. Then, when a stable set of practices have emerged, layer on macros on top that allow us to get the same compile-time guarantees with improved ergonomics!

Q: Why do Component<'a, Props> and HtmlToken<'a> require a lifetime parameter?

A: Because HtmlToken's can contain callbacks which can modify the component state. The mutable reference to the component state has a lifetime ('a), and the callbacks cannot outlive the component state.

Smithy’s unsafe original sin

Those who give up a essential safety to obtain temporary liberty deserve neither… -Bizarro Ben Franklin

If there’s anything Rustaceans don’t like, it’s unsafety! And there’s a very big, glaring reason why Smithy needed unsafe blocks: wasm-bindgen's FromWasmAbi and IntoWasmAbi aren't implemented on arbitrary structs.

Since we can’t pass the app state to JS-land, we must store the app state in WebAssembly. And if it’s not going to disappear after a single render, it needs to be stored in a std::thread::LocalKey (created by thread_local!).

At first, I thought that this required that all lifetime parameters be 'static. This caused all sorts of headaches, and required me to use unsafe blocks to call std::mem::transmute to convince the compiler that the signature of the root component is Box<(dyn for<'a> jsx_types::Component<'a, ()> + 'static)>, and not Box<dyn Component<'static, ()>>.

However, as it turns out, thread_local! accepts the for <'a> syntax, which allowed me to clean up all of that unsafe code!

Reconciliation and HtmlToken vs BareHtmlToken

On every update, Smithy compares the last rendered HtmlToken to the currently rendered value and emits a series of updates to the DOM. The goal is to avoid replacing the DOM node entirely.

However, HtmlToken<'a> is created from a call to render(&mut self, props: Props). Because there can only be one mutable reference at a time, we cannot render an HtmlToken<'a>, keep it around, update the state, re-render, and compare. Furthermore, we can't clone an HtmlToken<'a> because it contains FnMut closures, which don't implement Clone.

So, we needed a separate data structure, BareHtmlToken, which is an HtmlToken minus the event handlers. Because we handle all of the event handlers at the top-level, we never need to worry about them when emitting reconciliation instructions!

Long road ahead

It is still very early in the Rust to WebAssembly story, and there is still a lot to code. Some of things that are required to make Rust to WebAssembly a reality for actual users include:

• Macros to make everything easier: In Smithy, the jsx! macro takes a invalid Rust code like <div foo="bar" />, and turns it into HtmlToken::DomElement(DomElement { node_type: "div" ... }). It uses the nomparser combinator library.

Procedural macros in Rust are tough to test. Under the hood, they take a TokenStream and return another TokenStream. So this means that if I incorrectly coded my macro, jsx! could return the wrong type, and we'd only find out when someone used the macro!

Thus, we’d find out after shipping the macro that it was broken, and the type system would not save us.

Thus, test driven development is extremely important when it comes to procedural macros.

• Reconciliation: under the hood, Smithy uses a very rudimentary set of reconciliation instructions (insert, set attributes, delete, replace) to keep the JS-land HTML in sync with the representation in WebAssembly-land.
• The web API is weird. For example, you can update most attributes by doing node[attr] = val. But not if the attribute is class! Then you have to do node.classList = val. Le sigh
• CLI Tool: I want to build a CLI tool that bootstraps a Smithy app.
• CSS: This is a whole can of worms I haven’t touched. There is a lot of potential here! For example, Rust’s type system could conceivably enforce that if a parent component has display: flex, then all of its children have the appropriate flex properties set.
• Deploy process: There isn’t any to speak of, yet.

Conclusion

Smithy is still very alpha. It’s got a lot of rough edges, namely around developer ergonomics. Many things are unimplemented, and many things are probably implemented poorly.

But I’m excited about it and hope you are too! Thanks for reading 🙂