JavaScript, but deployed as a cloudfunction
Forewords
The emergence of the cloud and managed services in the early 2000s has been a revolution in the software development industry. With IaaS (Infrastructure-as-a-Service), it is no longer necessary to manage physical servers and virtualization: the provider takes care of everything.
Other products such as PaaS, CaaS or KaaS offer managed services that take responsibility for even more tasks, in order to abstract as much infrastructure provisioning related complexity as possible to the developers, so they can focus on their actual capital gain: the business.
The current trend follows the so-called Serverless products, which offer 100% managed services capable of scaling down to zero. This is the ultimate “pay as you go” product: you only have to pay for requests executions. As long as the service does not process any requests, you won’t be charged for anything.
In this article, we will see how at HiPay, we made the most of these Serverless products, especially cloudfunctions, to deploy a whole system on Google Cloud Platform (GCP) at low cost.
Context 📖
Payment reconciliation is part of HiPay’s payment process (see nwack’s article). It consists of establishing links between the resources provided by our acquirers and the bank statements transmitted by our bank. These resources are supplied daily at fixed times: it is a punctual and (almost) predictable event.
In the legacy project, resources were collected by PHP batches triggered by CRONs, which then trigger PHP daemon processes through PostgresSQL queues. Both batches and daemons were executed on HiPay OnPremise infrastructure.
Although this infrastructure isn’t exclusive to the payment reconciliation process, it is important to note that servers must be provisioned every single day of the year to be able to launch these processes, even if they only last a few hours.
Did you talk about Cloud? ☁️
Cloud migration is a good opportunity to optimize the deployment of these processes. With serverless products, we can look at significant infrastructure savings by running CRONs at specific times.
On GCP, there are three serverless products available:
- App Engine allows you to deploy an application without having to worry about the runtime (#PaaS).
- Cloud Run allows you to deploy self-managed containers (#CaaS)
- Cloudfunctions allow you to deploy code without having to worry about the runtime (#FaaS). The product shares similarities with App Engine, but cloudfunctions is designed for simpler and stand-alone needs.
I don’t have any preferences, which one would you recommend?
All three products are able to execute serverless processes. From our experience, the choice of one product over the others has nothing to do with their performances or their technical specificities, but is more related to their philosophy and their compatibility with the GCP ecosystem.
Nowadays App Engine has increasingly been put aside in favor of Cloud Run and cloudfunctions. Cloudfunctions uses Cloud Run behind the scenes, so use whichever product suits you best, it will likely be the same. We decided to redesign our system based on cloudfunctions, because its functioning is closer to a system that can be split into interconnected modules than a large application, and we did not explicitly need containerization. You can learn more about it on Google Cloud’s blog!
Choose the right runtime🎲
Hey, can I come up with this freshly created new revolutionary framework?
Nowadays, you can choose between multiple runtimes, including Python, Java, Node.js, Go, Ruby, PHP and .NET. But even if you want to execute Rust or whichever fancy language on the cloud, you can, by using containerized products like Cloud Run!
Historically, payment reconciliation processes have been written in PHP. When the cloud migration began in 2021, the supported versions for PHP were not compatible for a “lift and shift” migration strategy.
Instead, JavaScript, and more specifically TypeScript, has been pushed enterprise-wide as the primary development language. It’s an easy-to-learn language, and its support has widely been adopted by cloud products.
Developing a cloudfunction 🧱
You were waiting for it: let’s talk about cloudfunctions developpement. As we said before, cloudfunctions has been designed to meet simple and event-driven needs.
What you want to deploy as a cloudfunction isn’t your regular JavaScript application. They have their pros and cons, features and limitations. For example, their execution duration is limited in time: 9 minutes for the first generations and up to 60 minutes for the new generations (cf. Quotas — Time Limit). Knowing that, you don’t want to deploy your regular statefull Java application as a cloudfunction (nor on any other serverless product for that matter).
Wow, that’s quite the limitation… Why would I ever use this?
Cloudfunctions rather stands out when you talk about event-driven processes. Think about a traditional SQL trigger: you can use them to invoke stored procedure whenever a special event in the database occurs. That’s the idea, but at cloud scale!
Historically, there were two ways of triggering cloudfunctions: HTTP requests and PubSub messages. Since the introduction of EventArc, it is even possible to trigger a cloudfunction from the deposit of new files on Cloud Storage, or more generally, from almost any event on GCP.
Okay, you have me convinced! I’m in! Where do I start?
Depending on the trigger, the cloudfunction entrypoint has different arguments at its disposal:
/**
* HTTP
* https://cloud.google.com/functions/docs/writing/write-http-functions
*/
export function entrypoint(request: Request, response: Response): void {
console.log('This is an HTTP function!');
}
/**
* PubSub
* https://cloud.google.com/functions/docs/writing/write-event-driven-functions#background-functions
*/
export async function entrypoint(message: PubSubMessageNative, context: PubSubContext): void {
console.log('This is a pubSub function!');
}In the case of an HTTP request, the function takes as arguments an HTTP request and response, in the same format as Express middlewares.
In the case of a PubSub function, the message contains a payload containing base64 encoded data, and some message attributes. This is a personal choice, but we decided to encapsulate all our PubSub cloudfunctions in a High-Order function, in order to abstract and factorize the payload data decoding process.
function pubSubWrapper<T>(pubSubFunction: PubSubFunction<T>): PubSubFunctionNative {
return async (message: PubSubMessageNative, context: PubSubContext): Promise<void> => {
try {
const data = JSON.parse(Buffer.from(message.data, 'base64').toString('utf-8'));
await pubSubFunction({...message, data}, context);
} catch (error) {
Logger.analytics(CLOUD_FUNCTION.ERROR, {message: error.message, error});
}
};
}
export const entrypoint = pubSubWrapper(async (message: PubSubMessageNative, context: PubSubContext) => {
console.log('This is a pubSub function with decoded message!');
});It is possible to perform any kind of processing within the cloudfunction. Keep in mind that if your function is performing an asynchronous processing, you have to wait for all your Promises settlement before terminating the function (see Tips & Tricks), otherwise unexpected errors may occur.
Done! What’s coming up?
Congrats! We’re almost done, all you need now is to deploy your function in production!
Deploying a cloudfunction 🚀
Wait, deployment? What is that? I usually push my commits and go home.
Don’t worry, Cloudfunctions deployment is accessible to everyone! You don’t need to build a Docker image, nor open an SSH connection to your production server. All you have to do is to drop off your JavaScript source files and your package*.json files on GCP. There are a bunch of tutorials on how to deploy your cloudfunction through GCP Console interface or using gcloud CLI, but if you want a bit of a challenge, let’s take one step ahead and write our infrastructure as code with Terraform!
For those who don’t know about it (or just want a reminder), Terraform lets you describe your desired infrastructure using HCL language. You can provision computing resources as well as databases, storage resources, networking resources, and so on.
Once you’re done, Terraform compares your desired state with your current state, i.e. the resources that have already been created by Terraform. Then it performs a sequence of creation, replacement and destruction operations through your cloud provider API to achieve your desired state.
That looks complicated. I just want to deploy my function, why would I bother using such a tool?
Sure, I never said it would be an easy thing. Actually maybe I did, but let’s take a look at the pros before focusing on the cons! The benefits of using such a tool is that you can version your infrastructure (that means you can easily collaborate with your favorite colleague, track your changes, automate changes, etc.) and make it variable in order to deploy it on several environments (no more excuses to take down the production). To learn more about Terraform principles and policy, have a look at their documentation!
Okay, that’s fancy. Now in practice, how does it work?
Now we’ve introduced Terraform, let’s come back to what brings us here: how can we deploy our cloudfunction?
To deploy a cloudfunction, you need to deploy various types of resources, at least the following ones:
- A service account, a resource necessary for any non-human identity
- A bucket and a Cloud Storage Object to store your freshly coded source code
- A cloudfunction, because that’s what we want, huh?
You can describe and deploy each of these resources individually, or directly use the Terraform module which does it for you! You’re free to create your own module, but for this one time, let’s take a premade module.
resource "google_storage_bucket" "bucket" {
name = "test-bucket-cloudfunction"
}
module "cloudfunction" {
source = "github.com/google-terraform-modules/terraform-google-cloudfunctions-http"
bucket_name = "${google_storage_bucket.bucket.name}"
bucket_archive_name = "helloHttp.zip"
local_path = "${path.module}/code/helloHttp.zip"
function_name = "helloHttp"
function_entry_point = "entrypoint"
}All that remains to be done is to execute the terraform init and terraform apply commands to provision the cloudfunction. You can perform these actions locally, but of course, these commands are aimed to be included in your deployment pipelines.
Is it this simple? Really? I’m glad I got this far!
Actually, no, but you got the idea and that’s the matter! For this example to work, you would have to declare your provider, your project and a few more things in your Terraform configuration. Sorry for the disappointment, but it would take more than an article to explain how to set up Terraform. But don’t worry! As a compensation, you will find a complete sample here, with step-by-step explanations on how to run it!
Conclusion 🙇
We’ve come a long way to deploy our first “Hello World” as a serverless cloudfunction. But you know how it is? Once you’re done with the “Hello World”, you’re almost an expert! Almost…
Serverless products like FaaS (Function as a Service) and CaaS (Container as a Service) bring an unprecedented level of flexibility! You can run almost anything on the cloud, and only pay for what you really use!
They obviously come up with downsides: it would be inappropriate to host statefull applications in serverless products (but not impossible), as much as how cold start might be inappropriate for applications with high performance and reactivity needs.
Thank you for reading me! I’ll be glad if my article helped you in any way. Feel free to ask any questions or leave feedback in the comments!
