Implementing recurring todo’s with Habitica and Google Cloud
As I explained in a previous post, I like to make todo lists and monthly goals to organise my life. I store my todo lists in Habitica, and used to have a system where at certain intervals, new todo items would be automatically added for recurring events such as “Update monthly goals” or “Clean bathroom”.
(I recently discovered that Habitica itself supports recurring tasks, but the system isn’t very expressive; it doesn’t let you do things like once-a-year tasks very easily, and only works for “dailies” rather than “todo’s”.)
My previous setup worked by having a cron job make a request to the Habitica API whenever it was time to create a new todo item. It was a super-simple approach, that mostly worked, most of the time. However, it had two flaws:
- When the HTTP request to Habitica failed, I wouldn’t know because I didn’t have any feedback mechanism set up (though this would have been relatively easy to fix with a few lines of bash).
- The cronjob was running on a Digital Ocean server which was hosting my website. In the interests of saving money, but also to reduce the security and environmental footprint of having a server permanently running, I switched my website to Google Cloud recently. This meant that I no-longer had a machine that was reliably turned on at the scheduled times to create new todo items.
I considered three options for a new system with similar features:
- Having a permanently plugged in Raspberry Pi would provide exactly the same functionality as before, except I wouldn’t have to pay Digital Ocean. However, the environmental (power usage) and security issues (getting hacked) remain, and so I considered other options.
- IFTTT is a service that lets you configure automatic actions to happen in response to predefined triggers. In my case, I could configure a HTTP request to be sent on specific dates (using the ‘Date and Time’ trigger, and the ‘Maker’ action). However, upon playing around with the system, I couldn’t work out how to make it work with the authentication that the Habitica API required, and so I dropped this approach.
- The final option I considered (and implemented) was to implement a solution to run on Google Cloud. I saw that tasks can be scheduled with Cron in Google Cloud (configured with XML rather than crontab), where you define HTTP endpoints in your web service that should be called at the specified interval. This is perfect for me, since I already have a web service running on Google Cloud, and the system implements things like automatic retries (based on the response code returned from the endpoint), logging and error handling for free.
As mentioned above, while writing a bash script to make a HTTP request, retry if it fails, and log any errors can likely be implemented in a single line of code, the same can’t be said of a robust Java implementation. The stack I use for my website is rather boilerplate heavy (mainly as a consequence of everything being dependency injected, and of, well, using Java), so I decided to write a design doc so that I could properly plan my changes and avoid bad design decisions early on.
I wanted to support having multiple recurring todo items, and for each to have different recurrence frequencies. There were implementation approaches that I considered:
- Having one cron entry that repeatedly calls the same endpoint of my website, and for the website to work out what action to be taken on each call. This is sub-par, since the cron job would need to occur often so that the resolution of the timer was low enough to support any (reasonable) job, say, a few minutes, which would mean that the instance of my website would always be running (as it would always receive requests), which isn’t good financially or environmentally. Also, it would disregard the automatic retry feature built into the Cron system, since the logic for working out if a specific todo had failed would have to live inside the Java implementation.
- With one cron entry per recurring task, the system is more aligned with how the web was originally envisioned; the server is stateless in the sense that it can receive a request at an endpoint corresponding to a single recurring todo, and send a success/failure response that directly corresponds to whether the action to add a todo worked or not. The server logic is then far more simple than having one cron entry, and the retry system will work out of the box. The only issue, is that since cron is configured in an XML file that is stored in Google Cloud, editing the recurring tasks requires pushing a configuration change, but since this is a relatively rare event, I don’t think it’s an issue.
Each cron entry looks something like this:
Notice that each entry only specifies the recurrence schedule, and the id of the todo. The id corresponds to an entry in Datastore (Cloud’s NoSQL database) that specifies the exact details of the todo to add in Habitica (it’s name and associated notes, e.g. a link to the spreadsheet I keep my goals in).
In the backend, when a HTTP request is sent to /cron/tasks/, it is handled by a servlet that checks whether the request really did come from Cron (by looking at the headers in the request to prevent people calling the endpoint and DDOS’ing my Habitica account), loads the details of the todo corresponding to the id given as a GET parameter, and then calls the Habitica API appropriately.
Once I’d written the code, I wrote some unit tests to ensure that everything was wired up properly, aiming to exercise all of the different cases for a given todo item (e.g. the id being wrong, Habitica failing, the api credentials being wrong, success, etc). Though the tests may look verbose, using dependency injection combined with Mockito makes it relatively easy to thoroughly test the code; I managed to get to 100% line coverage of the cron servlet.
However, dependency injection is notorious for being easy to mess up in production (for example, by installing a module in unit tests that isn’t installed in production code). Since I don’t have an integration testing stack set up for my website (though I think it’d be fairly easy with Google Cloud’s versioning features), it was important to manually test the system in production before declaring it ‘finished’.
- As stated, since each todo item has a corresponding cron XML entry, adding, updating or deleting them requires a configuration change which is a little annoying; it’d be great to find a way to do that without having to have access to the command line.
- Having an integration test that can talk to a test account in Habitica and automatically verify that todo items were indeed created, and that the dependency injection was configured correctly would be a neat improvement in terms of having more confidence in the system.
- The cron system as I’ve built it runs in the same binary as my website, though it doesn’t share much (if any) code. Separating the cron system from the website would split the codebase into more manageable chunks, potentially improve the gathered statistics, and contain the impact of events such failed deployments.
- Finally, the whole thing is massively over-engineered. Since it’s relatively simple, mostly stateless and well tested, I expect that maintenance will be needed relatively infrequently and will be largely painless. I guess the scale of the project relative to its goals shows how the extra effort you have to go to to make things well tested, scalable, type safe etc all add up to a non-insignificant amount.