So why did Snagajob choose MongoDB
Snagajob is America’s largest network for workers and employers in the hourly employment space. By now we have more than 60 million registered user accounts. Average site response times according to Alexa is 1.9 secs as of today. Though we still have some work do, response times have dramatically improved over the last couple of years what brings me to the topic for this post.
Let’s roll back time by 5 years. Snagajob had less than half the number of user accounts, half the number of products, but average response times were more than twice as high. Secondly and more importantly, headroom on the database tier was virtually non existent. So to support our growth we had to find a solution to pivot the database tier (which was relational) from scaling up to scaling out.
Due to the nature of our main data structures (Postings & Applications) we quickly homed in on document databases. Our initial requirements for the new platform included:
- Support of complex documents structure, i.e. sub-documents, arrays, different data-types)
- Secondary Indexes
- Horizontal scalability
- High Availability
- Strong operational stability
- Easy-to-Use adhoc client
- Support for multi-datacenter Active/Active architecture
In 2011, MongoDB was the only system we had found, that checked almost all requirements out-of-the-box. We decided to spin up a POC and the results were very encouraging. Performance and stability were far better than we had expected. Secondly, we found that our code could be dramatic simplified. In one case we were able to consolidate 11 calls to the relational db tier with 1 call on the MongoDB side.
With positive results from our POC we decided to move forward and initiate a project to migrate one of our 2 major LOBs from the relational model to MongoDB. The best way to describe this is to replace the engine in your car while driving down the highway at 70 mph. Sounds like fun, doesn’t it?
Key to solving this challenge was to increase the capabilities of our existing data abstraction layer (nHibernate) and make it much more robust. We needed to insulate the API from the complexity whether data was stored relationally or in MongoDB. Our new DAL, code name Tucson, did exactly that. It generates code handling all database operations while providing a standard POCO interface. The complexity for knowing where the data “lives” is hidden in the Tucson layer.
So what are the results for Snagajob? Not only did we solve the original problem of building a horizontally scalable platform, and to improve performance dramatically, we also eliminated the need for database deployments completely. This resulted in improved time-to-marked as well as developer efficiency. Uptime is nothing short than amazing and all operational processes (patching/upgrade, index builds, database maintenance) can now be executed during regular business hours without impacting availability or performance at all. Not a big deal, you say? Just ask your DBA’s.
Though our experience using MongoDB is excellent I’d like to add one word of caution. If you have a hammer, everything looks like a nail, right? Though MongoDB is a nice hammer for a wide variety of nails, you might be out of luck if your use-case calls for transaction management. In MongoDB, modifications to individual documents are atomic but if your use case requires transactions in the sense of modifying multiple documents as a unit, then MongoDB might not be your solution. Secondly, there is a learning curve associated with MongoDB for people used to SQL as their query language to get familiar with the MongoDB query language. Nothing that can’t be overcome but certainly something to keep in mind. Best way to get started is to take one of the excellent and totally free classes on MongoDB University.
Originally published at engineering.snagajob.com on May 6, 2016.
