Cloud Code with kue (Migrate from Parse.com to a hosted Parse server)
This note was supposed to be posted months ago. I think I wrote this someday in January but since then totally forgot about that- and the file was in my backup directory. People who forget easily, like me, shouldn’t put things off. (Aug 15. 2016)
Parse.com will be retired
As Parse.com announced, they’re closing their service next year which is a pretty bad news to the people who has been using it for a long time. The announcement to a lazy coder like myself means, I have to develop a complete backend solution for the existing mobile app. However, it’s too early to sweat. They left behind the lagacy, open source version of the Parse backend, parse-server.
parse-server is highly compatible to the Parse.com backend but it has been written completely new so some functionalities may not working the same. For example, Parse.Cloud.httpRequest didn’t take params two weeks ago. And the Dashboard user interface came out recently. Even there’s no Job Scheduler yet. Although there are potential bugs and such, the open source community has been working very hard to fill up the change log.
Build a new hosted Parse server
- Open sourced parse-server is not a silver bullet. It only provides backend functionality. You need to prepare a DB and a job handler if you use Cloud Code from parse.
- If you’re not confident of DB indexing, using a hosted DB service that provides automated indexing would be a far better choice than setting up a dedicated on-your-own DB server. It could be a piece of cake if the DB is not so sophisticated but usually dealing with slow queries for the performance will give you a headache. (How Parse.com does indexing)
- After migrating DB from Parse.com, you can sync updated data for only 24 hours. And you have to open the “Migration” page every time you want to sync. I was planning to keep the data in sync until our new mobile app(pointing the new Parse server) comes out but now I have to start migrating with a clean slate again when the app is released. If I had a reverse proxy kind of server that can point which server to connect, it could have been easier.
- Once you finalize the migration, you can’t revert ever. (Prepare your migration wisely!)
Que es ‘kue’?
Again, building a hosted Parse server is a big deal but not so difficult. But running Cloud Code is a different story. Since Native Cloud Code module is not available in parse-server(Even though there’s a Cloud Code directory), you will have to find and setup a replacement all by yourself.
kue is an excellent option. It’s a job queue backed by redis. If you setup kue on Express.js application, you will be able to see a simple GUI like this (I have it tested with a simple logic):
Setup Cloud Code with kue
Basically kue is a producer-consumer structure. It creates a job queue and processes jobs.(as a consumer) All you need to do is to create a job and put it in the queue.(as a producer)
Hey dude, you’re leaking.
It turned out the parse server had a memory leak. Pretty massive one. It could be my code or setup that occurs so I debugged with these options.
node --gc_global --optimize_for_size --max_old_space_size=460 --use_idle_notification --always_compact --max_executable_size=64 --gc_interval=100 --expose_gc index.js
I was able to find the issue caused by the legacy code. However, I found another issue. Once client requests burst, memory usage hits the limit as well and it recovers very slowly even though I turned all the debug parameters off.
PM2 works as a process manager for Node applications. It allows multiple processes of a user Node application works as a cluster. The purpose of using this was very simple and plain stupid. Basically there are primary and secondary processes working the same job and one of their memory usage hits a certain amount, kill the process(primary) immediately and the other one(secondary) works as primary. While it’s happening, another process is being created and starts working as secondary. It’s not the perfect solution but it worked better than waiting the fat process cuts down its weight.
Done done done.
Except for the memory leak, anything else was totally doable even though I’m not super familiar with Node and Parse.com. I think kue works very well with parse-server as well as PM2. If I have a Node project that requires multiple task handling in the future, I’d love to put it together again.