Google App Engine: Data Export

Google App Engine (GAE) is a PaaS that I used as the backend for an old project. GAE is very easy to use, scales with reasonable guarantees, and allows developers to focus on building their applications.

In accordance with Lean Startup Principles, I wanted to run analytics on the data stored in my app. I was immediately presented with two problems:

  1. GAE did not allow you to connect directly to your (hosted) database.
  2. App Engine uses the Google Query Language (GQL), which is slightly different from the ubiquitous Structured Query Language (SQL).

The above two roadblocks pushed me to write a Data Export tool for Google App Engine, which I have open-sourced on Github. This tool helps you to decode the db files that you download from App Engine using the gsutil tool provided by Google. Downloading the data via this tool is a prerequisite to running GAEDataExport (I should have named the tool more appropriately, I guess).

The primary goal of my project was to convert the downloaded database into CSV files, which are easily readable and universally understood. Once you have your data in CSV files, then you can upload it into any database of your choice, since most of them can read CSV files. I will add a blog post soon about another one of my OSS projects that allows you to import these (or any other) CSV files into your Postgres database.

I have been using this GAEDataExport tool regularly for such data exports, and it has been working very well for me so far. I do have some open issues that could use some fixing and if you are interested in pitching in, I’d much appreciate the help! If not, then I hope you can find the tool as useful as I have!