Embracing Microservices
This is a second in a series of posts which share how we built the FlyAnywhere platform. In this post, I’ll describe how we embraced microservices, and why we built Hydra: a light-weight library for use with distributed computing projects.
Migration to Microservices
Before joining Flywheel Sports I learned of their existing monolith written in PHP and Drupal. To its credit, the monolith supported the growth of Flywheel from a one fitness studio business to 42 studios across the US and a membership of over half a million customers. Indeed, the opportunities our engineering team at Flywheel enjoy today stands on the shoulders of that giant.
Notwithstanding, a problem with all monoliths is that they have a tendency to continue to grow over time. And that’s exactly what happened at Flywheel during the prior five year period.
Just over two years ago we started thinking about embracing microservices. The catalyst for this was our CTO Mike Burlando who was recruited from a major fitness company where they built microservices. At his prior company, Mike had lots of resources to draw upon. However, when he joined Flywheel the development team consisted of only a few web developers and most of Flywheel’s systems were built by external contractors.
Mike had his work cut out for himself. Within a month of his arrival at Flywheel, he started recruiting engineers. Mike and I worked together in a prior effort and he convinced me to join his team at Flywheel. While I wasn’t initially pursued, I recognized that Flywheel’s use of IoT (before the term became mainstream) was particularly intriguing and even then there were internal aspirations of building a nation-wide live video streaming service.
During the first two months of my time at Flywheel, it was all hands on deck in support of our then new rebranding efforts. That meant that every engineer was focused on building a new ReactJS based site with a NodeJS backend. The complexity of that effort lied in the fact that our website was actually a web application which allows customers to book classes and view their historical stats. Due to time constraints, our efforts at the time relied heavily on the pre-existing monolithic backend API layer.
After the successful relaunch of the company site, Mike and I discussed building microservices in an effort to chisel away at the existing monolith. We understood that the microservices architecture embraces both focused modularity and distributed computing in an effort to build a collection of loosely coupled systems. That was particularly appealing because subsystems could be delegated to individual developers and external teams.
An emerging key driver was that we were preparing to build our FlyAnywhere platform which would allow our customers to ride at-home using our newly designed at home bike and our yet to be built, nation-wide live video streaming service. We were clear that such a service would require the agility that well-built microservices could offer.
Initial challenges
So we decided to embrace Microservices, after all, how hard could it be?
If you’re like the rest of us — some of these terms are still on your checklist. Now, it’s true that you don’t need to know all this to build microservices. But they are important tools and concepts, so navigating these concepts can help prevent complete loss time due to rework such as rewrites.
So, clearly there was a lot to consider, and unfortunately, we were also a small team with limited resources. That’s three strikes if you’re counting! With microservices, there’s a lot more to consider, such as how services will work together, be monitored, DEBUGGED! and how their data sources will be organized.
Experts will tell you that small teams should not embrace microservices. One reason is that the microservices approach adds to the busy workload that developers already have. That was certainly the case at Flywheel where we had an extensive backlog of IT related projects across every department.
A core challenge was that there’s much to learn — which amounts to a lot of study time for an already busy team. Also, given that Flywheel is a business and not an R&D group, we were naturally expected to deliver business features which didn’t necessarily include embracing microservices.
Still, we were clear that embracing microservices was the right thing to do in the short term and that the merits would have long-standing benefits.
One of the many challenges we faced was that we were too small a team to have our few developers running off in different directions — each implementing their version of a microservice. So we were leaning heavily on the use of a framework that would allow the team to build similar microservices while avoiding constructing the same general boilerplate code to address common concerns.
Our team at the time had varying degrees of Node and ExpressJS experience and so we wanted to keep close to that base while extending those skills with microservice features.
We were also growing as a team and we initially felt it would be faster to find ExpressJS developers than to find some with HapiJS, Koa or KrackenJS experience. In hindsight that probably wasn’t really a big deal.
Over-engineering and importance of right-sizing
As seasoned developers, we understood that a common pitfall that befalls teams of all sizes involves biting off more than they can chew. Whereby, over-committing and under delivering.
So a reality check was in order. We realized that we’re not Netflix, and as you read this, chances are you’re not either. Sure, if you are Netflix or any number of extremely large companies then you do need large scale solutions with fleets of supporting staff. We were clear that it was better to ship products than to spends months preparing to do so — by learning to use technology that exceeded our needs.
We were also keenly aware of the great tools that larger companies have contributed to open source. However, even those gifts from the gods come with steep learning curves.
Given our size, development budget and real business priorities we needed to “right size” a solution. We needed something small and robust enough to quickly get us started while affording us a runway whereby we could later embrace other solutions — as needed.
One of our most important goals was that an ideal framework had to be simple to use by less experienced developers. Ideally, a module implemented as a single package import. Our initial goal was to augment our Node and ExpressJS experience with tools that would allow us to build true microservices. I’d like to place emphasis on the phrase true microservices. To paraphrase Yoda (not that I’m not a fan of Starwars — as I don’t like fuzzy creatures in my SciFi — Star Trek Trebles notwithstanding…)
A simple ExpressJS app does not a microservice make.
We knew enough to know that building Node services with isolated concerns and matching RESTFul APIs falls short of what one could achieve with microservices. Granted, that probably would have in itself been a great start.
But where to begin?
In his book, The 7 habits of highly successful people, self-help author Steven Covey, wrote in habit number two: “Begin with the end in mind”. That is — begin by understanding the outcome you’re after. Then work towards that outcome.
So, we started by assembling a checklist of features we felt our microservices needed.
* Presence
* Service discovery
* Messaging
* Queuing
* Load balancing
* Routing
* Health
* Logging
* Configuration Management
If you already work with Microservices then this checklist is quite familiar.
At the time we started by investigating existing NodeJS modules for building microservices via NPM. In particular, we looked at SenecaJS— a mature toolkit for building microservices. There was a lot to like about Seneca. Its message first paradigm is spot-on for distributed computing. Couple that with its message pattern matching facility and you have an extremely powerful starting point. The one thing that didn’t resonate was that a SenecaJS-based microservice is markedly different from an ExpressJS app. The toolkit requires that developers embrace an entirely different mindset.
FeathersJS is another mature framework we considered. With FeathersJS, the mindset required is a closer match for developers with ExpressJS experience. Still, we didn’t feel that it was entirely what we had in mind.
One of our most important goals was that an ideal package had to be simple to use by less experienced developers. The reason was that we wanted to keep our expenditures in check by limiting the need to hire extremely senior engineers.
In the packages that I investigated, it seemed that each varied in their support for microservice features and relied on external infrastructure dependencies.
So I’ve been working with Redis for some time, and along the way, as I considered our checklist, I decided to take a closer look at Redis. Most of us were exposed to Redis as a cache earlier in our careers. We like to say that we came for the cache and stayed for the data structures.
Along the way, I started seeing Redis as a sort of distributed computing glue.
It looked like Redis could to be used to bind and unify microservices.
* Each microservice binds to a shared Redis cluster
* Presence is maintained by key/expiration
* Inter-process communication via Pub / Sub
* Routes placed in set stores
But was it possible? Microservice features using a single infrastructure dependency?
Creating Hydra
We thought so! So we set out to build an internal library called Hydra.
The name Hydra seemed ideal because a service type often has multiple instances working together as a larger beast.
We felt that if Hydra could address our feature checklist then it would be a good starting point for our efforts. Again, fully realizing that we were free to swap out functionality in favor of other technologies should the need arise.
How about Docker, Kubernetes, and Nginx?
Now, we’re routinely asked why we built Hydra instead of just using Docker or Kubernetes or even go retro with Nginx?
After all — aren’t we just duplicating existing functionality? It’s true that some of what Hydra offers is already present in Docker — but what’s nice about Hydra is that it works WITH or WITHOUT Docker, Kubernetes or Nginx.
It’s not an either-or proposition — but rather a matter of what you’re trying to achieve and how quickly you can get started. This was an important benefit for us as we were busy building products while learning about other technologies.
Today, we use Docker Swarm clusters on AWS in production without modifying Hydra.
Powering FlyAnywhere
So Hydra, with its single module import and a fairly simple class methods, allowed us to chisel away at our monoliths and start building our FlyAnywhere platform as a collection of microservices powered by Redis.
Open Sourcing Hydra
Fast forward nine months. FlyAnywhere was already well underway and Hydra was working really well for us. And we felt it would be useful to others. So, on November 7th, 2016 at the Smithsonian Museum of the American Indian in downtown Manhattan we open sourced Hydra on stage at the Empire Node conference.
I demo’ed Hydra running in a cluster of Raspberry Pi’s. My point in building the cluster was to reinforce the notion of a light-weight microservice library. In fact, light enough to run on $5 dollar Pi Zeros.
Here’s a close-up view of the cluster. It consists of eight computers. Seven of which are Raspberry Pi’s and one of which is an Arduino-based microcontroller.
Each of the Pi’s were running Hydra-enabled microservices using Node 6.5. In total, the cluster contained 16 CPU cores.
Since its open source debut, Hydra has been embraced by both smaller and larger companies than Flywheel Sports. A small but growing community has emerged and open source contributions are helping to further improve Hydra.
Is Hydra right for your organization? One way to check is to try it on a prototype or proof of concept project.
We suspect it’s particularly well suited for startups which have the freedom to choose their tools and who are rightfully concerned about quickly proving the market viability of their ideas. Flywheel Sports is far from a startup and we’re delighted in how well it’s served us. That said, we wouldn’t recommend Hydra for use by massive companies as their scale exceeds the practicality of using Hydra. For everyone else, Hydra may just be the sane way to build microservices.
Using other programming languages
We’ve built Hydra as an NPM package for NodeJS users. However, there’s nothing preventing anyone from doing the same with other languages. And because Hydra is open sourced it can serve as a reference platform for such efforts.
Along those lines, we’re pleased to announce that we have a Hydra-inspired Golang version in development, which we hope to open source soon. We’re also considering a Java-based version.
If you’re interested in building a Hydra package for your preferred platform, please feel free to reach out — we’d love to help!
Launching FlyAnywhere
So that’s why we built Hydra, and how we were able to both build and launch our FlyAnywhere platform, which is delighting our customers, nationwide!
Here is a photo of our Bluetooth-enabled at-home bike and metrics display.
A key callout here is that our microservices are dealing with real-time communication. Our Bikes emit real-time data which devices package into messages for transport via web sockets.
We broadcast live from our production studio in New York City. Our studio utilizes state-of-the-art broadcasting equipment similar to what you’d find in a News station. However, perhaps, unlike other studios, ours run a collection of microservices which support the automation of each live broadcast.
So microservices power our production studio and serve customers via our AWS infrastructure. And we do all this using light-weight NodeJS microservices powered by Redis.
To learn more about Hydra visit NPM.org.
In the next part of this series, we explore exactly how Hydra leverages Redis to build light-weight microservices.
