Overcoming the pitfalls of Google App Engine Cron

Harshit Dwivedi
Oct 22 · 4 min read

Having used GCP heavily for our data pipeline at https://roobits.com, we were looking for an effective way of scheduling certain tasks to be run at regular intervals

For example, from the earlier blog, you might remember that we allow businesses to collect events from their websites/apps on a per second basis and store them into BigQuery for a later retrieval/analysis.

While we primarily aim to do this in real time, some of our customers demanded a batched option instead at a cheaper cost.

To achieve this, we can save the incoming requests into a text file in our App Engine instance and run a cron job to load that file into BigQuery () at every few minutes.

Thankfully App Engine has a built in cron service that allows you to write a simple containing the time at which you want the job to run and which endpoint it should hit and that’s it!

App Engine will ensure that the cron is executed at the time which you have specified.

Here’s a sample that hits the endpoint in our AppEngine deployment at every 2 minutes.


However…

While it sounds good on paper, the solution provided by App Engine failed miserably for our use-case.
Why, you might ask!

The simple reason being, that the cron job will only run on a of your App Engine deployment!

This was a deal breaker for us, because we had over 12 App Engine instances running, each having a text file containing the user data for the past 2 minutes.

Since the cron job targets a single instance, , which is totally unacceptable!

Not only this, but with cron job you are also making A/B testing impossible for us.

Cron job is also limited to the fact that it can only make a request at the endpoint you specify in , thereby leaving your server vulnerable to crawlers and other attackers.


Alternatives

Since using a cron job was out of question, we started looking at alternatives, the first one being .

While Cloud Scheduler had everything we wanted, including option to target specific version, instances, services and also sending a POST request to the endpoint that handles running the logic behind the cron functionality.

While the cron provided by AppEngine was free, .

Assuming that we have to run a job every 2 minutes, the total cost we will have to bear if we are using Cloud Scheduler is .

It’s also important to note that the first 3 jobs you create using Cloud Scheduler are free, so if you have a single script that is supposed to run multiple times per month, you don’t have to pay anything!

Kudos to vineet garg for pointing out that Cloud Scheduler is priced per job and not per execution!


The second solution that we stumbled upon was using our own custom cron that runs on our server.

While it requires us to write a basic script to keep track of the time that has passed, it’s free and doesn’t have any limitations which App Engine’s cron had.

We ended up writing the following snippet in our server which was deployed on App Engine.

The job we created ran once every 2 minutes and called the method which loads the files into BigQuery.

This also runs on every instance and since it isn’t hitting an exposed APIs, it eliminates all the issues we had earlier with regards to practicality and security of the solution.


Conclusion

If you are someone who is in a similar shoes as us I think that it’s worthwhile to invest time into writing your custom cron job instead of relying on prebuilt services.

While it requires us to write some code, but as you can see it’s only a few lines of code and the benefits that we get from it are worth the 15 minutes of time investment that we had to provide.

Thanks for reading! If you enjoyed this story, please 👏 to help others find it! Feel free to leave a comment 💬 below.

Have feedback? Let’s connect on Twitter.

Google Developers Experts

Experts on various Google products talking tech.

Harshit Dwivedi

Written by

Android Developer, has an *approximate* knowledge of many things. harshithd.com

Google Developers Experts

Experts on various Google products talking tech.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade