The Evolution of Electrode Archetype

Introduction

W ith the Electrode Platform from @WalmartLabs we introduced a new concept called archetypes that play an instrumental role in the platform. For people who have been trying out Electrode you are probably familiar with archetypes. Archetype was the result of a need to improve developer productivity in an organization with hundreds of developers. In this post I will discuss their evolution from the inception, proof of concept, past iterations, to current version, and the future.

The Inception and PoC

What is an Electrode Archetype? It is a pair of npm modules that contain the typical standard stuff like configurations, dependencies, and tools and scripts for developing an application or module. The application consumes the archetypes as dependencies. Please see our docs here for further details.

The initial problem we wanted to solve was to rapidly onboard and support teams to our platform. Typically a generator would create a bare bone app with a standard setup, but then how would we propagate updates and fixes to established apps?

We started an experiment by creating a tool called bolt with a set of common tasks for linting, transpiling, bundling, and testing. The repo is deprecated now, but it’s still available here. The result was encouraging and helped the inception of npm scripts based archetype.

The Builder Iteration

We love using npm scripts for builds. They are simple and readily available. We want to utilize them for our builds, so we based our initial archetype on them.

A tool named builder was created for us by Formidable. It loads the npm scripts from the archetype and the application and allows you to invoke them with builder run.

We hit the ground running with builder archetypes. As we solved issues encountered, the tool improved and matured along with the archetype concept. There were many small things, but the biggest issues were related to node_modules dependencies.

dependencies vs devDependencies

We want dependencies like babel, webpack, karma, etc to be a devDependencies for the app, so it should consume our archetype as such.

Well, at the same time, there are some things that we want to make available to the app in production so we also need the archetype to be in the app’s dependencies.

So we split the archetype into two modules. A main one and another with a -dev suffix, which only serve to pull in dependencies so it mainly contains just a package.json file. The dual modules are consumed as follows:

Loading the dual archetypes in our getting-started sample app.

The node_modules dependency tree

The biggest problems we ran into were mostly related to the indeterministic nesting of modules installed by npm. We couldn’t get it to work with npm@2. Fortunately, npm@3 with module flattening was already available and we switched to it.

npm@3’s flattening promotes module as high in the dependency tree as possible, but our archetype’s dependencies may not get promoted if another module depends on a different version of the same thing.

As an example, if our archetype needs foo@6 but the app needs bar that pulls in foo@5, then we may get foo@5 at the top level of node_modules and foo@6 nested inside the dev archetype. If we require foo from the main archetype, then it gets foo@5— not exactly what we wanted.

The package.json of a test-app that pulls in bar which pulls in foo@5 to the top level.

And foo@5 being promoted to the top level, while foo@6 got nested inside the dev archetype.

The solution was simple. Since NodeJS creates an instance of require for each module, we can pass that around and it would always be requiring from the original module’s location.

So we create a simple file in the dev archetype that simply exports its require. In the main archetype we use that whenever necessary and it worked nicely.

module.exports = require;

It still doesn’t solve the problem completely because some tools internally require modules from its own context. Fortunately this situation is rare and so far have all been easy to work around.

On a side note, base on this solution I created a small module require-at that lets you call require pretending you are at any directory you wish.

Promoting the dev archetype

Eventually we realized that most of the stuff in our archetypes are development related tasks only and they should really be in the dev archetype. So we moved all the config settings and scripts from the main archetype into the dev archetype, which got a nice promotion from being an empty module with only the package.json file.

The gulp Iteration

In time, the npm scripts in our archetype became very large and difficult to maintain. We had more than 50 shell scripts each in a JSON string. Another problem was that many of them likely would break on Windows.

While we implement more complex stuff in JavaScript and invoke them from the shell script, it was still hard to maintain. At this point, npm scripts were not as appealing anymore.

A peek of how it’s like with all shell scripts inside package.json (wordwrap set at column 200)

I wanted to find a way to get out of maintaining these shell scripts in JSON strings. The obvious answer was JavaScript.

I found that gulp can execute tasks in JavaScript, but only concurrently. Luckily, there is a plugin run-sequence that can execute tasks serially.

Next I need an easy way to transfer the shell scripts to JavaScript. So I wrote a simple loader to get them into gulp and execute them with shelljs. The result was surprisingly easy but effective, and we switched to gulp based archetypes.

The xclap Iteration

Occasionally some users want to inject their own code to do setup or other work before an archetype task is invoked. However, if they add a task with the same name to gulp, then the archetype’s task is lost. For this, something like namespace for tasks would be useful.

Implementing task namespace for gulp was not trivial and any changes in gulp was unlikely. So I created a drop-in replacement task executor xclap with more features, one of which obviously is namespace for tasks.

With the support of namespace from xclap, users can define tasks with the same name from the archetype but still can invoke the original tasks.

Additonally, xclap can load and execute npm scripts, serially or concurrently, and properly nesting the execution hierarchy. It has an event driven execution lifecycle so it’s possible to develop different visual progress reporters. It also offers better integration for the tasks, giving them more flexibility to influence the execution of other tasks.

xclap showing tasks from package.json and our archetype electrode-archetype-njs-module-dev.

The Future

Occasionally we still run into issues with module dependencies getting mixed up and applications have to explicitly depend on certain modules to get them installed at the top level. Currently there is really no easy way to get around this and we will continue to search for a better solution.

Further, we are continuously improving and updating our existing archetypes to increase developer productivity, and if we identify a new class of project, then we may develop new archetypes to make their development simpler.

Conclusion

Electrode Archetype was the result of a need to improve developer productivity. Over the course of their development, we solved issues related to Node modules and different approaches to implementing tasks, and developed different tools to meet those needs. We are continually searching for better tools and approaches to take the repetitive chores out of our developers’ hands and allow them develop their applications faster and easier.

Acknowledgements

Formidable is instrumental in the development and evolution of the archetypes. Their developers working for @WalmartLabs made a lot of contributions to help bring Electrode and the archetypes to where they are today.