Of umbrellas, transducers, reactive streams & mushrooms (Pt.1)

In which we start delving deeper into the land of thi.ng/umbrella

This is the first chapter of an ongoing blog series, other parts available:

  • Part 2 — HOFs, Transducers, Reducers, ES6 iterables
  • Part 3 — Convolution, 1D/2D Cellular Automata
  • Part 4 — Disjoint Sets, Graph analysis, Signed Distance Fields

Once upon a time, in the cold February of the year 2018, this narrator sat by his fireplace and told of a small software device to conjure so-called user interfaces within the glowing rectangles of people’s networked screens, with nothing but the native data structures provided by any of the recent incarnations of JavaScript. Though (arguably) poignant, that story more or less ended in an anti-climax with an open ending, and this author didn’t make it clear, it was nothing but a chapter in a much larger undertaking, one which has been brewing & evolving for several years (since late 2015)…

This larger undertaking/labor of love is:

thi.ng/umbrella

  • Data structures for various use cases (graphs, sets, trees, heaps, lists etc.)
  • Binary data manipulation, encoders/decoders
  • Functional programming
  • ES6 iterators & transducers (a primary topic)
  • Reactive dataflow streams & graphs
  • Immutable data / state containers & updates, time travel
  • Extensible value-based (not pointer-based) equality, diffing
  • Interceptor-based, composable event handling
  • 2D geometry, color, gradients, visualization
  • Higher-order string formatting
  • Finite-state machines
  • Declarative SPA routing
  • Structured, multi-target data logging
  • HTML / SVG / XML / CSS / Markdown generation & parsing
  • Strided memory pooling/mapping/ allocation (WebGL/WASM interop)
  • nD vector & matrix operations (dense & sparse)
  • Oscillators/wave generators (not only for audio)
  • Seeded PRNGs w/ uniform API
  • Forth-like, point-free stack DSL
  • …and, of course, above-mentioned component based UI generation

In any application, this last topic (UI update) is just a piece of the bigger picture, and in most cases not part of the “business logic”. Yet, the architectural design choices of this UI layer can have a huge impact on the rest. Unfortunately, the previous article from a year ago didn’t go into any detail about possible workflows and real-world usage patterns supported and enabled by thi.ng/hdom (the library discussed in that article), especially the much more important aspects of state handling and deriving UI components from other values (app state). This was partly because these issues are outside the scope of hdom, but also was due to the already lengthy nature of the post. Additionally, even after 3+ years of working with some of these tools on an almost daily basis, I’m still regularly discovering, adding and experimenting with new approaches.

Primarily for this last reason, i.e. experimentation with new approaches, thi.ng/umbrella is more of a toolkit and explicitly not a framework. Instead, it aims to support a wide variety of use cases with very disparate architectural needs and approaches. It largely provides fundamental constructs to enable a “pick & mix” approach for different aspects of app development. It’s designed to avoid lock-in and most customization points only expect either raw, JS native data or certain interfaces rather than specific implementations. Some packages offer higher levels of abstraction or more context-specific functionality as separate opt-in layers. It’s maybe also worth pointing out that most packages in this project heavily rely on various ES6 syntax features and are not explicitly considering legacy ES5 compatibility nor do they provide polyfills of any form. However, all packages are distributed in these three formats: ES6 modules (individual files) and bundled CommonJS & UMD, incl. type declarations and source maps.

Several of these packages are partial, updated ports of some of my earlier Clojure / ClojureScript (and Go) projects. Others are at least heavily influenced by the Clojure approach to (plain) “Data First”. As I said many times before, I’m heavily indebted to all that this language and community have taught me, even though my focus now has shifted to other languages, better suited for my endeavors…

Image for post
Image for post
Long tails everywhere — Package file sizes of all current thi.ng/umbrella projects

Since the publication of the previous article, there have been 2250+ new commits (excluding merges), hundreds of test cases, dozens new packages, 40+ documented example projects (totaling a few thousand lines & many showing how various packages can be used in combination), plus dozens of smaller example snippets in various long-form readme files and docstrings in the source code. So much for the rough overview. There’s still a lot more to do, particularly in terms of documentation & testing, but the full scope of topics covered by the whole project, by now makes it impossible for me to cover meaningfully in a single article or two. Therefore, I will apply some dose of Divide & Conquer and aim to publish more targeted, shorter posts, each dealing with some (IMHO) interesting features the different packages have to offer and explain how these various parts can be combined in the wild.

A small selection of example projects included in the repo:

Image for post
Image for post
Dataflow graph example — Live demo / Source code (needs updating)
Image for post
Image for post
Pump & dump in action — Crypto-currency SVG live chart - Live demo / Source code
Image for post
Image for post
thi.ng/hdom benchmark — Live demo / Source code
Image for post
Image for post
Incremental Voronoi / Delaunay via Quad-edge data structure of 20k Poisson-disk sampled points, accelerated using kD-Tree spatial indexing
Image for post
Image for post
Transducer based Figlet style bitmap font transformer — Live demo / Source code
Image for post
Image for post
Recursive polygon tessellation & thi.ng/hdom-canvas visualization — Live demo / Source code
Image for post
Image for post
D0 & stochastic L-Systems made w/ thi.ng/lsys
Image for post
Image for post
Minimal Markdown parser / hiccup converter — Live demo / Source code
Image for post
Image for post
Sine-plasma effect w/ realtime multi-contour extraction — Live demo / Source code
Image for post
Image for post
hdom virtual scroller component with fuzzy search — Live demo / Source code
Image for post
Image for post
Worker based, interactive Mandelbrot explorer with cosine gradient colorization — Live demo / Source code
Image for post
Image for post
Interactive, additive wave synthesis SVG visualization — Live demo / Source code

In addition to these bundled examples, most packages have also been used in several private production projects, some of which already for several years.

Where to go from here…

Welcome to Shroomania!

Image for post
Image for post
Screenshot of the current state of Shroomania — Play it @ demo.thi.ng/shroomania.

You can play the current prototype at: demo.thi.ng/shroomania

Aim: Collect all edible mushrooms, avoid the toxic ones. FYI. There’s no gratifying element waiting for you if you manage to collect them all…

Controls: WASD or cursor keys. Sorry, there’s no mobile version.

To set expectations, and before we start digging in, let’s briefly explain the topics covered by this example project:

Image for post
Image for post
Cellular automata example used as starting point for the game — Live demo / Source code
  1. Procedural generation: Apart from the cute pixel art base tile set, obtained from opengameart.org (plus a few additions made by myself), all other aspects of the game world generation are 100% procedural, created with 2D Cellular Automata (CA) and signed distance fields (SDF). Btw. the entire game world is based on one of the configuration presets of the above CA demo.
  2. Graph analysis: Since the game world is created via a semi-random process, not all results produce a good playable environment. We will cover some basic (undirected) graph theory, algorithms & data structures (e.g. Disjoint Sets) to analyze the generated raw environment, verify playability and, if successful, extract more meaningful data for successive map generation steps.
  3. Functional programming with transducers: Almost all data processing steps in terms of terrain generation and game state handling are done via transducers (and compositions of such), one of my favorite (and IMHO one of the most powerful) functional programming concepts, able to maximize DRY in many usage scenarios. These (transducers) in turn heavily rely on ES6 iterables, which we will cover in some detail as well.
  4. Reactive dataflow: Zooming out slightly, the aforementioned transducers are all connected into a dataflow graph of reactive values making up the game state & processing logic. Each reactive value is essentially of a stream-based nature and we will use a number of thi.ng/rstream constructs and operators to connect them into a compute graph, with injected transducers doing the actual work (or stopping downstream traffic).
  5. Declarative, data-driven, on-demand canvas drawing: Excluding some minor pre-processing steps for the minimap overlay, where we use the “low-level” HTML canvas API, the main game render step only updates the UI when needed and uses thi.ng/hdom and thi.ng/hdom-canvas components (i.e. JS arrays & iterators).

Epilogue

Now go on and read:

  • Part 2 — Transducers
  • Part 3 — Cellular automata
  • Part 4 — Disjoint Sets, Graph analysis, Signed Distance Fields

Lastly, if you have any questions or feedback about any of the projects mentioned, please do get in touch via Twitter, the GH issue tracker and/or join our Discord channel.

Until then, Happy hacking! :)

Written by

Computational design, data, TypeScript, Clojure/script, C

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store