Cloud Console for Google CloudSQL and AlloyDB — Compute
I really like Google Cloud UI Console. Yes, I know, to be cool we are supposed to like and do everything in a shell (like in movies), but I have to admit that I like that I can open Google Cloud Console, go to BigQuery, for example, see my datasets, tables, and run a query in a nice UI.
But there is one place where I’ve been missing it (and had to use Cloud Shell) — CloudSQL. There are a lot of things I can see about my DB in Google Cloud Console, but to actually run any query — I have to use Cloud Shell. (At this point, if you completely disagree with me — you can just skip the rest of the article :). And to be fair, I really like Cloud Shell too for many other use cases.).
So, we thought — why not build a simple Cloud UI Console for CloudSQL ourselves? And while we are at it — let’s make sure it can also be used for the new and cool AlloyDB. And of course, do it open-source — so that anyone who shares our thoughts can use it too.
And more importantly, as Ralph Waldo Emerson said — “It’s not the destination, it’s the journey.” Planning to do such a project, we realized that we have to make a list of architectural decisions similar to decisions we have to make for many other projects. So why not convert this process into posts to share our thoughts (and maybe even learn where we were wrong)?
This and the following posts will cover this “journey” and hopefully, arrive at the destination. Note: Since this is a Google Cloud series, we will not talk about the UI part and will only focus on the backend.
If you are here for the destination (meaning that you actually like the idea of SQL UI Console and would like to use one but don’t need details on how it is built) — here is the link: Google Cloud SQL Console. Note: this is still a work in progress :)
The journey starts with requirements
As for any other project, we will have to make the following decisions:
- Compute. Which computing service would we use to run our app?
- Storage. We’ll most likely need to store some data (Users, Credentials, etc.). Where will we store them?
- Security. Obviously, we would need to protect our app. How will we do it?
Before we can make any decisions, we need to discuss what we actually want. Here are our requirements:
- Easy to use. We want our UI Console to be very easy to use (if it is harder than embedded Cloud Shell — what is the point?).
- Easy to install. Since it won’t be a standard part of the Google Cloud Console, we need to build it in a way that it would be easy to install for a user.
- Scaling to 0 (or almost 0). We assume that this app will only be used occasionally. It means that the infrastructure cost to run it should be very small when used and even smaller (ideally zero) when idle.
- Secure. That is the obvious part.
- Not limiting. To fulfill requirements of “Easy to install” and “Secure” we will need to deploy this app within the Google Cloud project where CloudSQL is used. Otherwise, our options will be to either open CloudSQL to the public (which definitely violates “Secure”) or ask the user to make some complicated network configurations (which violates “Easy to install”). If we plan to install our app into the user’s project — we need to make sure that it does not interfere with anything else running there (or maybe be running in the future).
With our main requirements established, let’s start to make our decisions. First — compute.
Compute
Google Cloud gives a lot of options:
- Google Compute Engine
- Google Kubernetes Engine
- Cloud Run
- Google App Engine Flex
- Google App Engine Standard
- Cloud Functions
Any of them can actually be used to run the application we want to build. But let’s see which fits best and let’s start by looking into what does not fit.
Google Compute Engine
GCE is a great service and a very easy entry point for those used to pre-cloud VMs. But I don’t think it is the best fit for our project because:
- If we let the GCE machine with our app run all the time, that would violate our “Scale to 0” requirement.
- If we ask the User to start and stop the GCE instance each time the user wants to use our app, it violates the “Easy to use” requirement.
- We could have built a special service on Cloud Functions, for example, that automatically starts/stops the VM, but that would probably violate the “Easy to install” requirement.
Let’s see what our other options are.
Cloud Functions
Jumping to the other part of the specter — Cloud Functions.
Again, this is possible but Cloud Functions were designed for even-driven workflows while the user maintains a session with our app (it won’t be nice to ask for credentials each time the user runs a query). Let’s skip this for now.
Google Kubernetes Engine
We love GKE and it is used in all our projects and if there’s a GKE cluster running in a project already, adding one more pod with SQL Console would be a great option.
However, if GKE is not used in a project, this approach won’t work because:
- Starting and managing GKE cluster for a user that never used it can’t be considered “easy to install”.
- Due to the management fee, it won’t “scale to 0”. (Technically, there is a free tier that provides $74.40 in monthly credits per billing account that applies to zonal and Autopilot clusters. But that only covers one cluster.)
Because of that, we should skip GKE for now but add one more nice-to-have requirement: be able to deploy SQL Console into the GKE cluster (this would be used by users who have GKE clusters that are already running). We can call it “Containerization”.
Now we are down to three contestants…
Google App Engine Standard
Google App Engine Standard was the first (and only at that time) service available on Google Cloud and basically started Google Cloud as a platform. It was designed to allow building scalable web apps quick and easy and it still serves the same purpose.
Note: GAE did not gain popularity at its time, but in my personal viewpoint, that was due to the only storage available at this time — Datastore. Datastore had eventual consistency, which limited the set of apps that could be built using it and required new approaches.
However, Google App Engine Standard comes with the following limitations:
- You have to build an app specifically for GAE Standard, which means it won’t be possible to take the same app and later deploy it to GKE.
- The list of supported languages is limited. In our case, we wanted to use .NET, which is not supported.
So, now we are down to two and we are at the final round!
Google App Engine Flex and Cloud Run
GAE Flex and Cloud Run actually have more in common than differences. Both allow to deploy and run the container-based app while taking care of scalability, availability, redundancy, underlying infrastructure, etc.
Note: The underlying technology and history of those services are actually rather different. App Engine Flex was initially built as an extension of App Engine Standard to solve for limitation it had, while Cloud Run was built based on an open-source knative project.
Since our app will be container-based, it also means that we can later move the same app to GKE without changing the code.
Since both services require building the app exactly the same way — we actually do not need to make the final choice — we can build our app and let the user decide. Our preference, however, would be Cloud Run.
BTW, App Engine Flex does not support scalability to 0 and always keeps at least one instance running. Cloud Run, however, will take down all pods if the service is not used. 1–0 for Cloud Run!
Let’s declare Cloud Run a winner!
Let’s see how Cloud Run matches our requirements:
- Easy to use. Using Cloud Run would allow us to access our app using the browser.
- Easy to install. Deploying an app in Cloud Run is actually rather easy (as long as you know how to build a container, which is a “must have” in 21st century…). However, we would need to ensure that our CloudRun-based app can connect to the CloudSQL special network configuration. Luckily it is well documented.
- Scaling to 0 (or almost 0). All OK here — Cloud Run can scale to 0.
- Secure. Cloud Run gives us enough control over security (more on that later).
- Not limiting. Users can have multiple Cloud Run services within the same Google Cloud project, which means our SQL Console won’t conflict with other services.
- Containerization. Cloud Run allows deploying containerized apps, so everything is fine here.
Looks like everything checks our and we can move to our next problem — storage.
