The Evolution of Electrode Archetype
W ith the Electrode Platform from @WalmartLabs we introduced a new concept called archetypes that play an instrumental role in the platform. For people who have been trying out Electrode you are probably familiar with archetypes. Archetype was the result of a need to improve developer productivity in an organization with hundreds of developers. In this post I will discuss their evolution from the inception, proof of concept, past iterations, to current version, and the future.
The Inception and PoC
What is an Electrode Archetype? It is a pair of npm modules that contain the typical standard stuff like configurations, dependencies, and tools and scripts for developing an application or module. The application consumes the archetypes as dependencies. Please see our docs here for further details.
The initial problem we wanted to solve was to rapidly onboard and support teams to our platform. Typically a generator would create a bare bone app with a standard setup, but then how would we propagate updates and fixes to established apps?
We started an experiment by creating a tool called bolt with a set of common tasks for
testing. The repo is deprecated now, but it’s still available here. The result was encouraging and helped the inception of
npm scripts based archetype.
The Builder Iteration
We love using
npm scripts for builds. They are simple and readily available. We want to utilize them for our builds, so we based our initial archetype on them.
We hit the ground running with builder archetypes. As we solved issues encountered, the tool improved and matured along with the archetype concept. There were many small things, but the biggest issues were related to
dependencies vs devDependencies
We want dependencies like
karma, etc to be a
devDependencies for the app, so it should consume our archetype as such.
Well, at the same time, there are some things that we want to make available to the app in production so we also need the archetype to be in the app’s
So we split the archetype into two modules. A main one and another with a
-dev suffix, which only serve to pull in dependencies so it mainly contains just a
package.json file. The dual modules are consumed as follows:
The node_modules dependency tree
The biggest problems we ran into were mostly related to the indeterministic nesting of modules installed by
npm. We couldn’t get it to work with
npm@3 with module flattening was already available and we switched to it.
npm@3’s flattening promotes module as high in the dependency tree as possible, but our archetype’s dependencies may not get promoted if another module depends on a different version of the same thing.
As an example, if our archetype needs
foo@6 but the app needs
bar that pulls in
foo@5, then we may get
foo@5 at the top level of
foo@6 nested inside the dev archetype. If we
foo from the main archetype, then it gets
foo@5— not exactly what we wanted.
The solution was simple. Since NodeJS creates an instance of
require for each module, we can pass that around and it would always be requiring from the original module’s location.
So we create a simple file in the dev archetype that simply exports its
require. In the main archetype we use that whenever necessary and it worked nicely.
module.exports = require;
It still doesn’t solve the problem completely because some tools internally
require modules from its own context. Fortunately this situation is rare and so far have all been easy to work around.
On a side note, base on this solution I created a small module require-at that lets you call
require pretending you are at any directory you wish.
Promoting the dev archetype
Eventually we realized that most of the stuff in our archetypes are development related tasks only and they should really be in the dev archetype. So we moved all the config settings and scripts from the main archetype into the dev archetype, which got a nice promotion from being an empty module with only the
The gulp Iteration
In time, the
npm scripts in our archetype became very large and difficult to maintain. We had more than 50 shell scripts each in a
JSON string. Another problem was that many of them likely would break on Windows.
npm scripts were not as appealing anymore.
I wanted to find a way to get out of maintaining these shell scripts in
I found that
gulp and execute them with
shelljs. The result was surprisingly easy but effective, and we switched to
gulp based archetypes.
The xclap Iteration
Occasionally some users want to inject their own code to do setup or other work before an archetype task is invoked. However, if they add a task with the same name to gulp, then the archetype’s task is lost. For this, something like namespace for tasks would be useful.
Implementing task namespace for gulp was not trivial and any changes in gulp was unlikely. So I created a drop-in replacement task executor
xclap with more features, one of which obviously is namespace for tasks.
With the support of namespace from
xclap, users can define tasks with the same name from the archetype but still can invoke the original tasks.
xclap can load and execute
npm scripts, serially or concurrently, and properly nesting the execution hierarchy. It has an event driven execution lifecycle so it’s possible to develop different visual progress reporters. It also offers better integration for the tasks, giving them more flexibility to influence the execution of other tasks.
Occasionally we still run into issues with module dependencies getting mixed up and applications have to explicitly depend on certain modules to get them installed at the top level. Currently there is really no easy way to get around this and we will continue to search for a better solution.
Further, we are continuously improving and updating our existing archetypes to increase developer productivity, and if we identify a new class of project, then we may develop new archetypes to make their development simpler.
Electrode Archetype was the result of a need to improve developer productivity. Over the course of their development, we solved issues related to Node modules and different approaches to implementing tasks, and developed different tools to meet those needs. We are continually searching for better tools and approaches to take the repetitive chores out of our developers’ hands and allow them develop their applications faster and easier.
Formidable is instrumental in the development and evolution of the archetypes. Their developers working for @WalmartLabs made a lot of contributions to help bring Electrode and the archetypes to where they are today.