Why I’m building Conductor

tl;dr I am building a modern JavaScript utility library called conductor. Here is a visual sum-up.

conductor in one picture for the visual learners

Purpose 👾

Some context

Today, whether you’re making HTTP requests to an API, reading from or writing to the file system on Node.js, or creating animations in the browser, you’re using code which runs asynchronously.

Heck, you’re probably using the async/await pattern right now, or chaining Promises like a pro while distractingly reading this article!

Enter the time travel machine

But let’s go back in time just a little. The term AJAX was first coined in 2005, by Jesse James Garrett. Node.js’ first version, heavily relying on the callback pattern, was published in May 2009. ES6's final spec, including the long awaited (no pun intented) Promises, was published in June 2015. ES8's final spec, which brought the async/await keywords, was published in June 2017, less than a year ago! So even by the standards of our m̶a̶d̶d̶e̶n̶i̶n̶g̶ beloved JavaScript world where 1 second ≃ 3 new frameworks, a new superset of JavaScript and 6 months of reading Medium or HackerNews to catch up, using asynchronous operations everywhere isn’t that old.

However, most of JavaScript’s utility functions such as:

• Array.prototype.map
• Array.prototype.filter
• Array.prototype.reduce

have been around for quite some time. Obviously, they weren’t designed to work with all kinds of asynchronous operations (try using a function which returns a Promise with Array.prototype.filter), or interoperate with Promises, RxJS’ Observables and so forth. Also, they were introduced at a time Object Oriented Programming (OOP) was the holy grail of programming paradigms (not that it’s a good or bad thing. Today we’re moving towards a more functional programming paradigm, who knows what it will be tomorrow?). These utilities are getting harder to use every day, because they did not evolve alongside the language.

With a little help from my friends

Libraries such as Lodash or Underscore did a great job at normalizing these ‹utility functions and providing us developers with a consistent utility belt. Ramda helped us embrace the purity (pun intended) of functional programming. Async.js and RxJS allowed us to have a more fine-grained concurrency model.

However, it feels like you have to install too many different things to build simple data pipelines. Async.js helps a lot with controlling in which order things execute, but it also has its shortcomings: composing Async.js utility functions with other functions can be tricky, and it simply can’t be used with synchronous functions. So you might add Lodash or Ramda to the mix to get everything covered, but they all have different APIs, best practices and pitfalls. Or you could use libraries such as RxJS which provides a more than complete set of tools. But interop with non-RxJS code is tricky. Of course, there are ways to do it (toArray, toPromise), but having parts of your codebase both with & without RxJS can quickly become frustrating.

AKA one of your colleagues converted half of the codebase to RxJS’ Observables overnight

So that’s why I am building conductor: to provide you with a modern set of tools you already know to help you write simple code. “But wait”, you’re thinking, “Doth mine eyes deceive me? Is he complaining about having to install too many libraries only to write yet another JavaScript library? Hmprff” (*grumbles*) “Where is it?” (*head scratching*) “Ha! I found it, let me just past…”. Stop! Let me save you the trouble. Here it is:

Standards (credits: XKCD, https://xkcd.com/927/)

Guilty as charged! I see conductor as an experiment. Maybe it will be a step in the good direction, maybe it won’t. But sometimes, to make progress, you have to start off with a clean slate. My only defense will be that I tried to make the library as simple and intuitive as possible, reusing names, patterns, functions you already know.

OK, tell me more

Because of the fast evolution of paradigms & uses in JavaScript, I believe the code we write today is unnecessarily complex: the intent gets lost in the implementation details.

Example #1

Let’s take a quick example. Suppose you have this very simple object:

I’d like you to sum this object’s values. Take a few seconds to think about it.

Got it?

So, “easy peasy”, says you:

Nice! So you figured a good way to sum the values would be to use reduce, since we’re “reducing” a collection to a single value. Good thinking. But why did you add that extra Object.values call at the beginning? You probably went through the following thought process: “Okay, I need to use reduce. Oooh but Objects don’t have a reduce method. Let’s convert it to an Array first”.

That’s fine, but why did you have to think about it in the first place? You have some data, and you want to reduce it. That’s all one should see when reading your code.

Example #2

The same goes for dealing with asynchronous operations. Suppose we have an array of numbers, and an asynchronous predicate (meaning the filtering function returns a Promise):

How would you filter this array using this asynchronous predicate?

Take a few seconds to think it through.

You should come up with something like that:

Ok, so that wasn’t so complicated. But do you think you, or one of your colleagues, would guess the intent of this code if they were skimming it?

Since Array.prototype.filter can not handle an asynchronous predicate, you first have to store the result of running the predicate on every item in an array using Array.prototype.map, wait until every Promise is resolved using Promise.all, and then use Array.prototype.filter to filter the items synchronously. It’s very easy to make a mistake and confuse numbers and results. Once again, you need to waste extra time to circumvent the limitations of JavaScript for a use case you have on a daily basis.

Example #3

If you’re not convinced yet, fine. Let’s all pull out all the stops! Let’s combine the first example with the second one. Let’s say I want to filter an object using an asynchronous predicate. How ‘bout that?

So we have this object and our usual predicate:

How would you do that?

Figured it out yet?

Here’s what I came up with:

That’s the best I could do. Perhaps you’ll find a more elegant solution that saves one or two characters (don’t hesitate to comment if you do), but the problem remains. Why the extra code? Why the obfuscation of what’s really going on? The intent gets completely drowned among implementation details.

How can we do better? 🌟

Expressiveness

conductor wants to make your code (and you!) as expressive as possible: it should allow you to focus on what’s going on logic-wise rather than waste time on implementation details. Let’s go back to our first example:

Why couldn’t you write something like this instead?

This is only a few characters shorter, but we only left what was strictly necessary to understand what’s going on. We are reducing an object to a single value. Nothing more, nothing less.

Let’s revisit our second example now:

What if we could write something like this instead?

Again, it’s not that much shorter, but it’s much nicer, isn’t it? You can immediately understand the intent, and there’s no need to use little hacks.

Finally, let’s see how we could improve the third example. We wanted to filter an object, using an asynchronous predicate:

Now don’t tell me you can understand what this code is doing at first glance. The operation (filtering) is completely hidden in implementation details introduced because of the limitations of vanilla JavaScript operators.

What do you think of the following?

I think it’s much better, because it keeps the focus on the data transformation itself, not on how the transformation is implemented.

Of course, every language has its little quirks and there’s always a way around them. You might (rightly) feel you don’t need a library to save a few characters. Real men & women write vanilla JavaScript, right?

Point taken. But I think our code should be as expressive (business logic-wise), concise, and meaningful as possible. Think about the people who will read this code: your coworkers, GitHub collaborators, or even you in a few weeks. What should immediately pop to the eye is the intent. What does this do? Where is it going?

You have a data structure and you know how you want to transform it. That should be the end of it. Instead, we often find ourselves with code littered with little “hacks”. In contrast, conductor aims at being the Nike of code:

➤︎ Data. Operations. Just write it. (or why I did not end up doing marketing)

Modularity

Since we’re striving for expressiveness, building small, concise and neat code blocks will definitely help. Ideally, functions should be responsible for doing one thing & one thing only. conductor will help you write modular code, because it has a great set of helper functions which are all pure functions. This means they won’t have unexpected side effects, modify their input parameters or only work in a specific context.

Writing very simple functions can seem counterintuitive, “juniorish”, simplistic, cumbersome or even (insert your favorite condescending adjective here), but remember:

“Simplicity is the ultimate sophistication”
- attributed to Leonardo Da Vinci, but apparently he never wrote it

Or if you want it from a more universally agreed-upon source, check out this great answer by Sindre Sorhus about writing one-line modules. He currently has 1123 published and actively maintained packages on npm.

Modularity has three humongous upsides:

• Readability: it’s easier to read (& understand) 100 one-liner functions than 1 hundred-line-long function. Each of them has a very specific & limited job which you should be able to infer from the function’s name. Remember, developers actually spend most of their time reading code, not writing it. So do your future self & your colleagues a favor, do not fear writing simple code. In the long run, it will never end up as simple as you’d like to be.
• Code reuse: how many times have I found myself copy/pasting little bits of code from one function to another? I lost count. By making sure every little logic operation, data transformation is contained in a neat function, you’ll find it very easy to reuse them and avoid unnecessary code duplication.
• Testability: it’s incredibly more easy to test a function which simply does exactly one thing than a “very smart” function, which accepts a wide variety of inputs and updates 3 databases while making you a coffee. It’s actually a virtuous circle: you write a one-line function, then you may think “Why not write a unit test? It will only take 30s”, and then you’ll know for sure this code’s reliable. Now, you can reuse this function in the next one you write: again a simple one. So why not test it? And before you know it, you’ve got yourself a pretty sweet code coverage. Something needs to be refactored? No worries, the tests will prevent you from breaking the Internet.
  Write a simple function → Test it → Reuse it in another one → Test the new one → Repeat 🔁
  (Dear TDD people, do not hit me too hard, I ❤️ you too. I know ideally tests should be written beforehand. What can I say? Sometimes they don’t, and I much prefer having to write tests for small & contained functions than for book-length ones.)

Build data pipelines

Since most of conductor’s functions are data transforming ones, it’s easier to think of the modular functions you will write as (parts of) data pipelines: they take data as input, and they output (transformed) data.

Let’s take an example. Suppose we want to know how many Star Wars characters are from each planet. We’d like to get something like this:

We’ll start with an array containing the characters’ ids:

We’ll be using the awesome Star Wars API to retrieve the data we need. First, let’s define some helper functions:

So we created two utility functions:

• fetchJSON, which makes an HTTP request to a URL and returns the result as JSON
• fetchCharacter, which uses fetchJSON to retrieve a character’s data from its id

Now let’s create a function which will return a character homeworld’s name:

As you can see, it’s quite easy to compose conductor utilities and our own helper functions. Now, let’s create a function which will “count” an item:

Let’s use the previous function to create a function which count homeworlds:

Now, let’s create a function which merge every homeworld into an accumulator:

We can reuse this function to define our reducer, which will use the different functions we just created:

Finally, we can reduce our array to get our expected output:

Putting everything together, we get:

As you can see, we created very simple functions which all do exactly one job and were able to easily compose them to create a pipeline where the data simply flows. We can reuse these functions, test them and it’s very easy to know what they are doing. 3 greens! ✈️

So what’s in the box, really? 🎁

Here’s a quick tour of conductor’s main features

Automatic async operations handling ✅

Stop wasting time on handling flow control & concurrency.

Do you need to map over a collection using an asynchronous mapper? Easy: map + await.

No need to wonder how you should combine await and Promise.all.

The same goes for any data-transforming operations. Do you want to filter a collection using an asynchronous predicate?

Create your predicate and rock on 🤘! filter + await = 👌.

➤ Every function in conductor which takes a function as parameter will automatically return a Promise if the parameter function is asynchronous.

This includes compose, which can compose synchronous and asynchronous operations in a breeze 💨, as you can see in the example above.

It works like magic, and boy have we p̶a̶t̶e̶n̶t̶e̶d̶ promoted it 🍏.

However, if the function you pass to these operators is synchronous, they will return a result synchronously! The operator is exactly the same: you won’t find a filter & a filterSync in conductor as you might do in Node.js.

One operator to rule them [types] all ✅

Nearly all data-transforming functions in conductor work not only on Arrays, but also on Objects , Sets and Maps. No need to wonder how to extract the values of an Object or a Set, or how to use reduce when you’re simply trying to filter an object.

filter + Object = 👌.

Functional by design ✅

Automatic currying

All functions in conductor are automatically curried.

This feature (borrowed from Ramda) and partial application allow you to create reusable functions using a point-free style: no need to use free variables (whose names are often hard to find) when defining your functions.

The mapByAdding1 function can be reused in different places of your codebase.

Data as last parameter

Again borrowing from Ramda, the data you want to transform is always passed as the last parameter of the function you are using. Combined with automatic currying, this enables you to build data pipelines by composing very simple partially-applied functions.

In the example above, you don’t need to define a single free variable: you can really focus on making your intent as clear as possible.

Interop with vanilla JavaScript ✅

Because conductor does not introduce new types or store data in classes (such as RxJS’ Observable), you can easily mix conductor and non-conductor code, use it only in some parts of your codebase, or use only the functions you like. If your input is an Array, so will most likely your output (well, it depends on which function you use). The same does not apply to RxJS. Once you’re in a RxJS function chain, you kinda have to stay into it.

Coda 🎶

conductor vs X

So that’s it for the main features’ tour! If you want to try it, the adoption should be pretty easy as I tried to reuse names from vanilla JS, or Ramda/Lodash. If you’re wondering whether it can be replace any of the libraries mentioned in this article, here’s my take:

• The main goal, as stated earlier, is to use conductor as a standard operations library
• However, you will always find a use case where one of this library is more suited than conductor
• Specifically, I think you can fairly easily replace Lodash or Ramda with conductor. Async.js too, if you are not using the more complex methods. RxJS serves a different purpose, so depending on your use, you might not be able to replace with conductor. The idea is rather to avoid adding it to a new project if you just need basic operations on collections of Promises.

Do you want in?

• GitHub repository 🐙🐈
• npm package 📦
• Documentation 📖
• Want to help? Plenty of things to do! 🤗

WaldoJeffers/conductor

I would love to hear (or read?) your thoughts (good or bad) about this project, so grab your keyboard! ⌨️

Thanks for reading!