Parse Server Review — Migration down to the penny and performance comparison
Sadly, Parse will close down — thanks for that, Facebook <sarcasm>. That decision forces several entrepreneurs from startups all over the world to decide what will be the future of their businesses. A few options are available so that businesses can keep in black and this article focuses on the Parse Server alternative. So, no switching to Firebase, CloudKit, …, it is Parse once again.
-Ok. It’s Parse again. Now what?
Glad you asked! Parse’s Github suggests a few solutions for either setting up your own server or using a Backend as a Service (BaaS from now on). The point is: which one is better? And why? I have tested some of the solutions listed there, and the results were impressive. Check out below!
The tool used to evaluate the performance was the ApacheBench tool — a simple enough tool so that the experiments can be reproduced and the results be compared and contrasted to other boxes’ results around the world.
The ApacheBench is present in most Linux distributions and also on Mac and is easy enough to use so that you will have no difficulty in trying it yourself and checking out what solution fits you the most.
The testing method was to make several requests via the REST API of each Parse Server / Parse Server Provider and checkout the metrics that the very same tool reports. This is a really platform agnostic test, hence there are no platform-related issues to invalidate the tests. The concurrence level chosen was 20 (-c option) and the number of total requests was 1800 (-n option) for all of the options when testing the response time. Burst tests were made only at PaaS solutions and the concurrency level used was 2500 (-c option) and the number of requests was 25000 (-n option) — note that to perform such test you have to tune your box.
Also, a few more metrics have been added that are related to cost, geographic location and how easy it is to use the tool, with a complete list of metrics below:
- Cost in dollars;
- Maximum requests per second allowed — how many concurrent requests can the server handle;
- Total time to handle all the requests — this measure includes the connection latency as well;
- Average time to handle a request — as the total time, it includes the connection latency;
- Standard deviation of the time to handle the request — how volatile is the server when a load is applied;
- Maximum time to handle a request — sees how wide is the response time range to that server;
- Location of the servers — where in the world are the servers;
- Ease of migration — how easy it is to migrate from Parse.com to this tool;
- Ease of use — how simple it is to use the tool.
These metrics tried to cover several of the problems that most DevOps teams find when using some Backend solution and is thought to be adequate enough to give a sufficiently broad view of the Parse Migration process. Other features, such as database size, storage size, requests per month have been discriminated at every test. I tried to make them as near to the others as I could, so that they are not an issue when choosing one or another.
A few services have been tested, which are listed in the Parse’s Github, listed below:
- Heroku + mLab — Heroku’s PaaS with a nice mongo Database as a Service, mLab;
- NodeChef — A PaaS solution that has DB integrated and is very simple to use and scale;
- Back4App — BaaS that looks a lot like Parse.com and offers a nice migration solution;
- SashiDo — A BaaS solution that has a really cool and intuitive interface.
At every experiment that is run, some expectation is created regarding the results, and this one is no different. It is expected to see:
- Higher maximum requests per second and higher burst tolerance on PaaS solutions over the BaaS ones due to the fact that BaaS are a single endpoint to several apps;
- Lower average time to handle a request, smaller volatility and a narrower response time range in a PaaS solution than a BaaS solution;
- PaaS should scale easily, as this is controlled by the user, whereas in the BaaS you shouldn’t worry with this, as they scale in an on-demand basis;
- Every BaaS should have similar performance, as the base code that is used is the Parse’s repository one;
- PaaS should have a smaller cost per maximum number of requests ratio when compared to BaaS;
- BaaS should be easier to use than PaaS solutions;
- Design should be fine in every solution.
With the right tool, right metrics, targets and expectations, we can proceed to the amazing results.
Starting out with the BaaS solutions, more specifically the Back4App provider, the following results have been observed:
Back4App’s performance:
- Cost: $4.99 for the basic plan — which was the one I tested
- Maximum requests per second: 30 req/s, advertised on the site and measured
- Time to handle all requests: 60.2 s
- Average time to handle a request: 664 ms
- Maximum time to handle a request: 1656 ms
- Standard deviation/Volatility: 38.4 ms
- Location of the cloud: AWS US East
In terms of ease of use, I’d say it is a an ok interface, as it is very similar to the Parse.com. Its performance handling the connections, on the other hand, look rather good. I will first check out other solutions to give a verdict.
Also, in terms of migration, Back4App offers a tool to migrate the app from Parse.com to their services, and has an article explaining that. Looks really simple to migrate.
Its location in the AWS US East places its servers in a nice central location, which absolutely won’t jeopardize any app that is US-centered, but might add latency to apps on Europe, Asia and South America.
Back4App limits the Database Storage to 1.5GB, 10GB File Storage, 100k requests per month and 1 Cloud Code Job for the $4.99 plan. These numbers look pretty nice limits for a small startup to create an app and manage their businesses.
Next on the list is the SashiDo provider. Here are the results that I observed:
SashiDo’s performance:
- Cost: $4.95 for the smallest plan there
- Maximum requests per second: 19.5 req/s, measured; site advertises that there are no limits
- Time to handle all requests: 92.4 s
- Average time to handle a request: 1020 ms
- Maximum time to handle a request: 4571 ms
- Standard deviation/Volatility: 208.7 ms
- Location of the cloud: AWS — probably Europe
SashiDo’s has a simple, beautiful and intuitive interface with a lot of small things that work properly, such as the duo Tab/Shift+Tab to go back and forth through the fields.
The migration looks like a pretty painless process, in which you go to the migration page, press the migrate parse app button and follow the flow of the page. Simple and clean.
I am, however, a little bit disappointed with their promises of performance that are not really met: they promised no limits in requests per second, which were not met by the measurements that have been made. Actually, trying the very same test made on Back4App at SashiDo’s API led to a much inferior request per second rate. It looks like they invested everything they had on their interface and forgot to do their homework on the backend.
I would not say such things from a startup that is investing in their frontend and delaying optimizations to the backend, however a core product of their business is the backend and it should be at least polished.
SashiDo limits also the database to 1 GB of storage, 40 GB of File Storage, 500 GB of Data Transfer and 1 background job. They look like promising numbers.
Comparing both, I’d certainly go for Back4App. Their performance is much superior and the gap of ease of use is not big. I really liked SashiDo’s interface, truly, from the unfathomable depths of my hearth, but their performance is far behind Back4App’s one, and this would really hurt the final user of every and any startup.
Now to the PaaS solutions, starting with the Heroku+mLab:
Heroku + mLab performance:
- Cost: $7.00 for Heroku’s Hobby dyno + $15.00 for mLab 1GB Shared hosting
- Maximum requests per second: 270 req/s, measured
- Time to handle all requests: 59.914 s
- Average time to handle a request: 662 ms
- Maximum time to handle a request: 1362 ms
- Standard deviation/Volatility: 69.5 ms
- Location of the cloud: AWS US East
Heroku’s interface is nice to see, but it looks a little bit complicated to use if compared to other interfaces. There’s just too many things to look at and some things are not where I expected them to be. E.g., to view the logs of the app, which is a really common task, you have to press a “more” button that is not really at focus, it is just a button on the upper right corner. It is a pretty interface, but with things not really in the place you expect it to be.
The migration of an app to both Heroku+mLab and NodeChef require to setup a new Parse Server and upload it to the cloud, which might not be the easiest or fastest thing to do. Some configurations have to be inserted in code, which might give the admin some trouble.
When talking about performance, it is clear that the performance on Heroku is absurdly good. I can only make compliments about the results, as they look really nice for the price you pay and will of course give the user a nice and smooth run.
Limits on this PaaS provider are 512MB of RAM for the Hobby dyno — a custom, kind of fancy, name for instance — and 1GB database storage for the mLab plan.
Now, to the NodeChef:
NodeChef’s performance:
- Cost: $28.00 per month for the instance that has 512 MB of RAM, as in the site
- Maximum requests per second: 180 req/s, measured
- Time to handle all requests: 66.1 s
- Average time to handle a request: 729 ms
- Maximum time to handle a request: 1792 ms
- Standard deviation/Volatility: 90.8 ms
- Location of the cloud: probably US
NodeChef’s interface is also nice, with everything a DevOps team needs in the place they need to be. Looking the logs, comparing to Heroku, is also in a dropdown button, but it is not in some corner: it is at focus and you are led to the button naturally.
My comment on migration is pretty much the same I did for Heroku: might cost you time and effort that could be invested somewhere else, perhaps deploying features and increasing the value of your product.
Performance is ok in the NodeChef: you won’t have problems with performance, as the instance is dedicated for you, however it really is far behind Heroku.
NodeChef gives you also 100MB of DB memory, 1GB of storage and an app with 512MB of RAM, for the $28 plan.
Bottom line: I’d choose Heroku+mLab, as it is cheaper and performs better. I could live with the interface, as it is just a matter of getting used to it. Also, the price is lower: and that certainly makes a difference, as your business might scale and the $6 difference might turn into $60, which could be invested somewhere else — e.g., advertising.
So, from the BaaS solutions turns out Back4App shines as in the PaaS solutions Heroku+mLab dominates. But… which one to choose? Back4App or Heroku+mLab?
Say you have a small business and does not require too much processing, or you can’t afford to have a dedicated team looking at CPU usage, RAM leaks, parameters tuning, … — problems that are not really focused on building and deploying features. Or say you just want to get on the business kind of now. I’d definitely go for the BaaS solutions, and by looking at the numbers, I’d go for Back4App. An ok interface, with a promise of performance that can actually be achieved and no need to keep looking at how many instances will I need for the next black friday, that might be a nice deal for the additional dollar per request that is paid.
Heroku+mLab on the other hand is certainly the cheapest performance solution you can have by looking at their amazing request per second rate and their prices: roughly 12.3 requests per dollar, which would be $2.45 for a hypothetical service with the same specs of Back4App’s one — less than half of the price. So, let’s say you can have a team just for that — you started on some garage and <boom> in a couple of years you have a whole 10-floor building just for your business —in this case, maybe using a PaaS might save you a nice amount of money that could pay the whole team that will be operating the PaaS instances and also could be invested somewhere else — e.g., happy hour every week at your business. Looking at the numbers I’d certainly go for the Heroku+mLab in that case.
As in price, it is clear that starting with BaaS is cheaper in total cost, but might get a little annoying on the long run, whereas running an instance on Heroku+mLab starts with nice performance and the price gets marginal as the size of the business grows — but might stagger the business growth as well. It really is a choice that should be made in a per case basis, which is left for each entrepreneur to do it.
Looking back at the expectations, the PaaS really is cost effective, with the cost per request being way smaller than the BaaS solutions. Also it has better performance with a surprise that the NodeChef having an average time to finish a request roughly the same as the Back4App’s one — which might be a location related problem.
Burst tests conducted on Heroku and NodeChef point to Heroku having a larger burst tolerance, for some reason yet to be known — I received a lot of SSL handshake failure from NodeChef’s endpoint. Also, maximum requests per second are certainly higher on PaaS solutions — at least 6 times the best of BaaS. Volatility measured is really variable and depends a lot on the experiment run — with results changing a lot from experiment to experiment — which is not an issue for every solution tested here.
BaaS solutions did not show any mechanism to scale up your solution — I guess you have to call them and ask? — whereas PaaS have shown a simple way to scale solutions and keep load in acceptable ranges.
Surprises appear when comparing BaaS solutions: SashiDo’s approach is to invest a lot in a pretty interface and delay polishing the server whereas Back4App invests heavily in the server, leaving the interface in a state that is usable. I thought they would have similar approaches, which is a wrong assumption.
PaaS solutions are also trickier to use, which require some basic knowledge of operating an instance and a monitoring team to avoid bad behaviour of the servers. This is not transparent at all at every BaaS solution, meaning either: a) No reason to worry, b) All reasons to worry. I believe these solutions are mature enough so that I would take a) and just trust the BaaS solutions work.
The design of every solution tested is ok, with NodeChef and SashiDo having the best interfaces, which does not mean that Heroku’s and Back4App’s is bad and impossible to use.
And that ends my view of a few of the solutions to the Parse.com shutdown problem. Thanks for reading and I hope you give me your feedback on what I wrote.
