Converting a MeteorJs project to a package based architecture

Stefano Cudini
May 23, 2019 · 22 min read

This article is a clone of the original offline:
http://experimentsinmeteor.com/package-based-architecture/
Author:
Nick Riebeek
Furthermore it was used for the design of the framework KeplerJs and inspiration to write the article:
Software Architecture for large-scale NodeJS applications

I recently viewed a great screen cast by Sacha Grief where he explains Telescope’s package based architecture.

We’re going to take an existing application and apply the principles laid out in Sacha’s screen cast to convert the application to a package based architecture. We’ll be replicating a lot of the conventions used in Telescope such as utilizing a lib and core package.

Why packages?

So before we get going, what’s the point of using a package based architecture?
Well, there are a number of reasons that Sacha covers in his screen cast:

  • File load order. The load order of files is explicit when using packages. This means you can avoid any load order issues that might occur with a traditional structure.
  • Updates. Updates on your application can be handled via Meteor update.
  • Customizations are easier. If someone alters the source to add features, this can be done via a package, thus ensuring the core code of the application is not compromised.
  • Code organization. With packages, organization of your code is more modular and coupling between components is reduced or eliminated completely.
  • Typically each feature has it’s own package.
  • Each feature is independent, you can remove a package and the application still works.

Why not packages?

The only real drawback to using a package based architecture is that it adds some minor complexity.

For smaller projects, a package approach might be overkill. However, I’m very impressed with the package approach and feel even for small projects it could be worthwhile despite the added overhead.

What we’ll build

Image for post
Image for post

For those of you who have gone through the paging and sorting tutorial, this is going to look familiar. Sorry about the recycled application, but taking an existing application seemed like a good way to demonstrate the differences between a standard and package based architecture.

Note we won’t be doing much explanation or examination of the actual implementation code, mostly we’ll be moving code around and dealing with package.js files. If you want an explanation of the code, check out the paging and sorting tutorial.

Creating the app

If you’ve gone through the paging and sorting tutorial you can use the existing code you built out from that. If not or if you want to start from a fresh code base you can clone the starting point for this post from GitHub.

Clone the Repo

Note, if you aren’t familiar with Git and / or don’t have it installed you can download a zip of the code here.

A quick over-view of where we’re starting from

Open up the code in your text editor of choice and you’ll see a pretty standard Meteor file structure.

Image for post
Image for post

By the time we’re done, our project structure will look radically different:

Image for post
Image for post

Start up the app

OK, let’s see where we’re starting from.

You should now see the starting point for our application when you navigate your browser to http://localhost:3000.

Image for post
Image for post

An over-view of the packages we’ll be creating

Our goal is to break up the code into a package based structure.

Now our application is a contrived example and as a result we’re going to go a little package crazy with it, we’ll be splitting up the “list”, “add”, and “newest customer” functionality.

This is for illustration purposes, with a real application, likely all the customer functionality would be contained in a single package, i.e. customertracker-customer.

However, for our purposes, we’ll be creating the following packages:

  • customertracker-lib
  • customertracker-core
  • customertracker-customer
  • customertracker-list
  • customertracker-add
  • customertracker-newest

We’ll be taking an iterative approach so our application will continue be usable throughout the conversion process.

Let’s get started!

Creating the lib package

We’ll be following the same convention as Telescope and creating a lib package. The purpose of this package is to contain all our core Meteor and 3rd party references. This is also where we set up the global namespace for our application.

Implementation

First off we need to create some directories and files.

Then we’ll set up the global namespace. We’ll call our application ‘Customer Tracker’ so will go with that for the namespace.

All we’re doing here is setting up a namespace and setting a version attribute on the namespace. The point of a global namespace is to ensure we don’t end up with namespace collisions with 3rd party components or other javascript libraries.

Now let’s move onto the package.js file.

The main purpose of the lib package is to reference our 3rd party packages, so let’s see what packages our application currently uses:

Image for post
Image for post

Awesome, with the above information we can now fill in package.js.

We’ll quickly go over each section of package.js. For an excellent and very detailed explanation of Meteor packages, check out writing a package by the Meteor Chef.

The describe section

This section of the file just sets some basic attributes for the package. The name attribute is important as that is what we’ll use to refer to the package when we add it to our application, i.e. when we issue the meteor add command.

The version is also a critical piece of information as it is used by meteor update to indicate when a new version of the package is available.

Specifying compatible Meteor versions

OK, next we have the package.onUse block which is basically the definition of the package. The api.versionsFrom line indicates the version of Meteor that our package requires. We’ve specified 1.0, so we’re saying our package can be used with any version of Meteor 1.0 or higher.

Linking to 3rd party packages and Meteor core packages

Next we specify the packages our custom package uses.

Here’s where the information we gleaned from meteor list comes into play. We set up a packages array to contain our list of packages. Note for third party packages the version number is required.

api.use indicates which packages are required by our custom package and makes use of the package array we put together.

api.imply exposes the internal packages of the lib package to any packages that in turn use lib. This means for instance that if you use the lib package in a second package, say secondpackage, then secondpackage has access to iron:router etc. without having to explicitly include it in it’s own package.js file.

This is very useful as it means you only need to specify a dependency once and don’t need to worry about conflicting version numbers for the same 3rd party package creeping into different parts of your application’s package.js files. If you’re a little confused, don’t worry, when we implement the core package we’ll see a concrete example of imply which will help to better illustrate why imply is so useful.

Specifying the files to load

Next we have the addFiles section.

Unlike a traditionally structured Meteor application you need to explicitly specify which files Meteor should load. For lib, we only have the one file, core.js and it should be available on both the client and server, thus the ['client', 'server']array.

Exporting items available outside the package

Finally we need to export any variables we want to have access to outside of our package.

We want access to the CustomerTracker namespace variable, so we’ve added an export entry for it.

Usage

Now comes the exciting part, using the package we created. First thing we’ll do is blank out our existing package file.

contents of: .meteor/packages.js

Bam, as expected our app is now crashing hard!

Image for post
Image for post

Time for our new package to come to the rescue.

And we’re back working!

Image for post
Image for post

Also we now have access to our namespace variable.

Image for post
Image for post

Our main package file now contains the single custom package we created:

contents of: .meteor/packages.js

This is pretty cool, we’re now referencing all our third party and Meteor core packages through the lib package we created.

Another great thing about the way packages work that we’ve illustrated is that if you’re converting an existing application to use packages, you can do it in steps instead of all at once. This means you can continue to add features instead of needing to stop all new development until everything is converted.

Creating the core package

Next up is customertracker:core. This package will contain the bare skeleton of our application. In our case we’ll include our general layout pages, our router configuration and our custom parameter checking code.

Implementation

OK, let’s get started by creating some directories and files.

Terminal

Now let’s move some of our existing code into our new package.

Terminal

In the case of the custom check code we’ll update it to use the namespace we created in the lib package.

contents of: /packages/customertracker-core/lib/server/custom-checks.js

OK, all we’ve done is update the code to use our new namespace.

We’ll need to change our publications to reflect this namespace change.

contents of: /server/publications.js

Simple, we’ve just updated CustomChecks... to CustomerTracker.checks....

Next let’s update our stylesheet in both our existing application and the package.

contents of: /client/stylesheets/styles.css

contents of: /packages/customertracker-core/lib/client/stylesheets/styles.css

All we’re doing is moving the “core” styles into our core package. Since we don’t have much styling going on, it’s only the bodystyle element that we need to move.

Now that we’ve moved some of our layout and router code out of our application and updated our stylesheets, we’ll see that the UI isn’t looking too hot.

Image for post
Image for post

Our styling is gone and our publication for listing customers is failing due to the custom check code not being available.

Let’s update our package file so that we can get our original look and feel back and get the publication working again.

contents of: /packages/customertracker-core/package.js

OK, so very similar to the package file we put together for customertracker-lib. The main point of difference being that we have a single package we’re using, our lib package. By including it, we are also including all the 3rd party packages from libvia the imply line included in lib that we discussed earlier.

Note once again we are using imply with the core package as we want packages that use core to have access to the packages referenced within core, i.e. all the packages included in lib.

A diagram will give a better illustration of what we’re aiming for:

Image for post
Image for post

Usage

Let’s update our application to use the new package.

Terminal

And there we go, our layout and styles are now back in place!

Image for post
Image for post

Creating the customer package

Now we’ll start getting into the packages that deal with our customer specific functionality. The customertracker:customerpackage is going to contain our collection and fixture code, as well as our routes for the customer functionality.

Implementation

Once again, let’s start off by creating some directories and files.

Terminal

Now we’ll move our schema and fixture file out of our existing application and copy over our collection file.

Terminal

As expected this is going to cause all kinds of trouble for our application.

Image for post
Image for post

Before getting started on our package file, we need to make an update to our collection in both the package and existing application.

/packages/customertracker-customer/lib/collections.js

/lib/colletions/customers.js

We’ve moved the collection into our package, while maintaining the insertCustomer method outside of the package.

So let’s get our package file updated so we can get our app back up and working.

/packages/customertracker-customer/package.js

Pretty simple, we’re using a single package, the core package we created earlier. Then as before specifying the files in our package and exporting the Customers collection so it can be used outside of our package.

Usage

Now to get things working we just need to make use of our new package.

Terminal

Load order example

Let’s go on a quick diversion and see an example of load order in action. We can manipulate the load order of the customerpackage and trigger an error via altering the “client / server” addFiles call. If we reverse the load order the application will crash.

/packages/customertracker-customer/package.js

Image for post
Image for post

As the console log suggests, the problem is that if we attempt to load our schema file before the collection file, we’ll be attempting to attach our schema onto a customers collection that is not yet defined.

So this illustrates how you can control the load order of a package, the files get loaded in the order they are specified in the addFiles array. This can be a much more elegant solution to load order issues compared to a traditional Meteor application where you would need to create nested folders or rename files.

Creating the newest customer package

OK, time to package up some of our UI elements. The customertracker-newest package will be responsible for the “most recently acquired” section of our UI, i.e.

Image for post
Image for post

Implementation

As usual we’ll start off by creating some files and directories.

Terminal

And now let’s move some code.

Terminal

With these changes our application is once again crashing. This is because we are now missing the newestCustomertemplate.

Image for post
Image for post

We’re going to come up with a better solution for handling this but for now we’ll make a change to the listCustomers template to get rid of our error.

/client/templates/customers/list-customers.js

All we’re doing here is adding a new helper function that returns a boolean value indicating whether the newestCustomertemplate exists or not.

We can then use this helper in our HTML file.

/client/templates/customers/list-customers.html

With that in place our app is back working without the new customer portion of the UI.

Before updating our package file we need to add the publication for our newest customer functionality.

So first let’s update our existing publication file.

/server/publications.js

Nothing complicated going on here, we’ve just commented out the publication we’ll be moving.

Now let’s add it to our package.

Terminal

And now we can update the file.

/packages/customertracker-newest/lib/server/publications.js

Simple, we’ve just moved the code we commented out into our package.

Now let’s update package.js.

/packages/customertracker-newest/package.js

Pretty simple, we’re just adding the necessary files and referencing our core package.

Usage

Time to get our newest customer showing up again.

Terminal

And with that our newest customer component is once again showing up in the UI.

Image for post
Image for post

Creating the list package

Next let’s move the list functionality into a package.

Implementation

This is getting redundant right? But that’s a good thing, we’re getting the hang of creating these packages and so far it’s turning out to be pretty easy. Let’s get some files and directories created.

Terminal

And now we can move some files.

Terminal

Again, we’re just moving the relevant files to our package, and as expected this causes our application to crash.

Image for post
Image for post

With our list component we have a style element to move, so let’s update our existing style file.

/client/styles/styles.css

We’re slowly getting rid of our styles, we’ve commented out the th style and we’ll move this into our package.

/packages/customertracker-list/lib/client/stylesheets/styles.css

Now it’s time to update package.js.

/packages/customertracker-list/package.js

Once again we’re using core and adding the necessary files.

Usage

OK, let’s get our app back working.

Terminal

Sweet, that was easy, onto the next package!

Creating the add package

OK, our last package… once we get this sucker finished off, we’ll be completely package based.

As you can probably guess from the name, this package will handle our add functionality.

Implementation

Yup, that’s right, time to create some directories and files.

Terminal

Now let’s move our files.

Terminal

Now our add customer functionality is broken.

Image for post
Image for post

We can move our final piece of styling out of our original application.

/client/stylesheets/styles.css

Select

Everything in the original file has been commented out, let’s add the button style to our add package.

/packages/customertracker-add/lib/client/stylesheets/styles.css

Select

Now let’s update the package file.

/packages/customertracker-add/package.js

Select

No explanation needed, we’ve seen this all before!

Usage

OK, let’s get our add functionality back working.

Terminal

Select

And there we go, we’re package complete! Our application is working as before but everything is running from packages.

Some clean-up

So now that we have everything wrapped up in packages we can do some clean up on our directory structure.

Terminal

Select

I was recently made aware of the trash command via this excellent post by Sam Corcos. As per the trash homepage, trash can be easily installed via Homebrew with brew install trash.

You can also go with rm -r client etc. or manually remove the folders if you don’t want to install trash.

In any case, however you delete your folders, you should now see that everything is completely within package directories.

Image for post
Image for post

Updating our packages to be more modular

There are a few artifacts of the original implementation which limit the modularity of our package based implementation. Ideally we want our ‘feature’ packages, i.e. the ‘list’, ‘add’, and ‘new’ packages to work independently.

This isn’t currently the case however, if we remove our list functionality the application crashes, i.e.

/.meteor/packages

We’ve commented out customertracker:list, which results in:

Image for post
Image for post

Likewise if we remove our add package, we still have an add button on our main list page.

/.meteor/packages

Image for post
Image for post

Making our three feature packages modular.

To enhance modularity we will create a main index page for our customer functionality. We’ll also remove the add button from the listCustomer template and put it in it’s own template.

Let’s start off by creating the index page. A good place for this file will be in the customer package; it is specific to our customer functionality but not associated with the “list”, “new”, or “add” functionality.

Terminal

Select

/packages/customertracker-customer/lib/client/templates/customer-index.html

Select

OK, so all we’re doing here is laying out the components we want on our main index page. What’s up with that renderfunction? Well we want to only render components that exist so we’ll create a helper to handle the rendering of our optional components.

We’ll throw this into our core package as it’s something that could be useful outside of the customers functionality. For instance if we enhanced our application to include orders, we might want access to the helper in the orders packages. These packages might not reference customertracker:customer but would certainly reference core.

Terminal

Select

/packages/customertracker-core/lib/client/spacebar-helpers.js

Select

Pretty simple we’re just checking if a template exists, if it does we return it, otherwise we return null. This will prevent attempts to render non existent templates.

So let’s update our package.js files to take into account our new items.

/packages/customertracker-customer/package.js

Select

/packages/customertracker-core/package.js

Select

Now we need to create our add button template and remove the button from the listCustomers template.

Terminal

Select

/packages/customertracker-add/lib/client/templates/add-customer.html

Select

/packages/customertracker-add/lib/client/templates/add-customer-button.js

Select

Simple, we’ve just copied the HTML and JS code for the add button into our newly created files.

Let’s update our package file.

/packages/customertracker-add/package.js

Select

Now we need to remove the add button and newest customer code from our list package.

/packages/customertracker-list/lib/client/templates/list-customers.html

Select

So we’ve removed the rendering of both the newestCustomer template and the add button code.

We can now remove the add button handler.

/packages/customertracker-list/lib/client/templates/list-customers.js

Select

And also the helper method we added earlier to check for the existence of the newest customer template.

/packages/customertracker-list/lib/client/templates/list-customers.js

Select

Next we need to update our routes since we no longer want our listCustomers template to act as our root route.

/packages/customertracker-customer/lib/routes.js

Select

We want our root route to hit our new customerIndex template instead of directly hitting the listCustomers template.

With the router changes we need to update code that refers to the old listCustomers route to instead refer to the customerIndex route.

/packages/customertracker-list/lib/client/templates/list-customers.js

Select

So here we’ve changed instances of Router.routes.listCustomers.path to the new route of Router.routes.customerIndex.path. We’ve also replaced instances of Router.go('listCustomers') with Router.go('customerIndex).

We need to make a similar change in add-customer.js.

/packages/customertracker-add/lib/client/templates/add-customer.js

Select

Again we’ve replaced instances of Router.go('listCustomers') with Router.go('customerIndex).

And with that we can now use any combination of the 3 packages.

For instance removing the list package results in the following.

Image for post
Image for post

Pretty cool, our application still works, we can add customers and view the latest customer that was added, but our list functionality is no longer present.

Summary

Although very much a contrived example, hopefully this post gives some insight into what’s involved in a package based architecture.

Is it pretty silly to use a package based architecture on a small example application? I’d say yes without a doubt! However, I am very impressed with this approach to building Meteor applications and I think it merits consideration when creating a non-trivial Meteor application.

Using packages has many advantages, primarily around code organization and keeping your code modular and decoupled. The ability to add and remove features via meteor add/remove is pretty sweet as well. I can see many situations where this would be useful, such as demonstrating beta features to customers.

The fact that a project can be converted in an iterative manner means an initially small application that looks like it will grow can be upgraded to use packages without completely stalling new feature development, this is great! Getting buy in on a refactor that will completely halt new feature development is much more difficult than diverting partial effort towards a refactor.

There are certainly some drawbacks to a package based architecture, it is a little more complicated to develop applications in this manner. You could also run into complexities with package dependencies, for instance circular package dependencies where package A needs package B which needs package A. In someways this is probably a good thing however, as a circular dependency likely points to a fault with your application structure / design and indicates a refactor is in order.

Next steps

A great next step if you’re interested in further exploring package based architecture is the Telescope source on GitHub. There’s nothing like looking through a real world application.

For some other thoughts on various options regarding Meteor project structures, there’s an interesting Meteor Casts episodethat is worth a watch.

References

A couple of great resources that helped with putting together this post include:

Thanks for reading and I hope you enjoyed this post!

This article it was used for the design of the framework KeplerJs and inspiration to write the article:
Software Architecture for large-scale NodeJS applications

The open source full-stack geo-social network platform

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store