Build Your Own Versions of the Software Libraries You Already Use
Learn new skills and deepen your knowledge with this exercise
Read Through Software Projects You Use
Over the last few weeks, I’ve been spending some time picking out some of the software projects I use and attempting to write small, simple versions of them. It’s been a pretty neat way to learn how and why certain things work the way they do.
For instance, I’ve used Express professionally for years. I’ve used Express for practically any web project I’ve built with Node, and chances are you’ve used it too if you’re a back-end node developer.
Maybe sometimes you question how the whole middleware pattern even works or why you have to use a callback function instead of returning a Promise to advance to the next middleware function.
Maybe you’ve been stepping through your web application and you find yourself in
node_modules/express/lib/router/index.js, and you can see the matrix (and somehow you now know kung fu). Ending up in a
node_modules file might be kind of intimidating, but if you look a little further, you might realize that scary module code is easy enough or small enough for you to grasp if you take a little time out of your day to read it.
For example, Express only has 12 files for all of the request, response, routing, view rendering, nesting, and middleware chaining functionality it has. It is totally possible to read through Express in a weekend. You probably won’t understand all of it right off the bat, but you’ll learn some neat implementation details, and you can always take notes of parts that you don’t quite understand to go back to later.
The first time I was reading through the router logic, I came across this kind of weird detail where if you register a middleware function that has more than three arguments, Express just silently skips it and moves on to the next middleware function.
They don’t write a debug message if you’re in development mode or even call the function knowing that the fourth argument will just have to be
Would you have done that differently if you were writing an application library? Up until that point, I was reading through the source and looking at it like, "Yeah I guess that makes sense" or "Cool, I wouldn't have thought of that," but that was the first instance when I thought that I'd rather have done something differently. Which leads me to my next point ...
Rewrite Software Projects You Use (aka Make Your Own Versions of Them)
I believe that reading through software projects and libraries you use is helpful in better understanding how the whole application works, but I think the next step in that journey is to pick some small libraries and make really simple versions of them.
Continuing on with the Express example, you can make an Express clone that only implements
app.use. Doing just that gives you an end product that can handle all of the routing and rendering outside of the framework.
In order to get to that point, you need to create an application object that can have middleware registered. You’ll also need to decide how you want to store middleware, how to chain each of the functions together, and how to pass in a next function that will eventually call the next middleware.
Then you’ll need to figure out how to run an HTTP server to get the Request and Response objects/streams that you'll pass to your registered middleware functions.
Finally, you might add some default middleware functions like Express does to handle query string parsing and body parsing. Do you want to build in functionality for automatically parsing JSON request bodies, or do you want to leave that up to the user? It's totally your call. Maybe you can parse the query string, and let the user access it as a SearchParams instance instead of a plain old object.
Something you can also aim for is replicating the library's interface and then dropping it into a project that uses the reference implementation. You can get a lot of insight from drop-in testing it — learning which parts break or aren't implemented yet.
I've been trying this out with Rapid, an Express clone. Is it going to replace Express? Hell no. Am I going to use it instead of Express for all of my personal projects? Probably not. But it's been fun, so far and there's plenty of more functionality to add or recreate. Also if you feel like making a PR on Rapid, they seem more than welcome.
Don’t just rewrite the projects verbatim, though. Copy the interface, but try the implementation out for yourself — maybe you’ll get the opportunity to arrive at the same decision crossroads the original implementors did. You could end up making a different choice or at least appreciating the different ways you can tackle the problem. You can always go check your work against the reference afterward and improve it if the reference has a better solution. It probably will if it’s a large project like Express. Big projects can be encumbered with complexity from maintaining backward compatibility for features you might not need or care about.
Another cool project to recreate is Data Loader. Data Loader is a tool to batch load and cache resources. You make an instance of it by constructing it with a batch function that takes some keys and returns a Promise that resolves to the values that those keys relate to. The batch function could run a SQL query or hit a REST API or some other data source.
The user interacts with the instance by calling a
load function with a single key and that returns a Promise which will resolve to the key's value when the batch function successfully loads the data.
The cool part is that Data Loader schedules the batch function to run after the current frame of execution by using some Node asynchronous primitives and caches the results. If you ask for the same resource multiple times throughout your web request lifecycle, Data Loader will only load it once. Also, you can ask for individual resources throughout the request lifecycle and Data Loader will batch load them at the end of each frame of execution.
Data Loader’s implementation is only one file (excluding tests), and there’s even a YouTube video from one of the creators that covers the entire source.
Writing your own implementation of that means you’ll have to cover concepts like creating promises, caching, using things like
process.nextTick for asynchronous operations, and deciding how to handle errors from invalid user entries like bad batch functions.
Hopefully, you have some projects in mind that you use often but don’t really understand how they work under the hood.