I have a personal project that needs to take screenshots of approximately 100,000 websites per month. I could use an existing website screenshot API service for this — there are plenty of those — but it would be way more fun to build one myself.
In this blog post, I describe the steps I took to set up this API, let’s dive in!
I can use this package to spin up a headless chrome instance, navigate to a website and take a screenshot.
To start I’m going to create a local node project and install the
npm install puppeteer
Next, I create an
index.js file and write the code to take a screenshot.
Note that I am making the
async. This way I can use the
await keyword in the function to wait for all the promises.
After running the code I get the following screenshot! 🎉
Google Cloud Functions
So I now have a local script that I can call to take a screenshot, but I want to build an API. The next logical step is to put this script on a server somewhere.
I don’t want to worry about my server running out of memory, so I’m going to put it on Google Cloud Functions. This way it can handle a huge number of requests without me having to worry about buying more RAM memory.
Once I have the cloud function running, I can call it with an HTTP request — meaning that I will have a working screenshot API 🚀
Let’s port the previous code to the Google Cloud Function format. The cloud function I created is
async and called
Because I’m calling this cloud function with an HTTP request it is going to send a JSON response to the caller. If successful a
200 status code is sent back along with a happy message. If unsuccessful the error is sent back.
On line 23 the function will go to the URL located in the request object. This means that I can pass the URL as a
POST parameter to the HTTP request 🤖
For example, I would call the cloud function like this to take a screenshot of github.com
Finally, I should update the
package.json file in the google cloud function to include the
After saving the function I’m using Postman to call the cloud function.
Unfortunately, there is an error. Google Cloud Functions don’t allow writing files to the disk — so I cannot store the screenshot in the
There are 2 ways to solve this.
- Don’t store the screenshot in a
screenshot.pngfile but return the raw image data to the caller.
- Upload the screenshot to Google Storage and return the URL to the Google Storage file.
Method 1 — Return the raw image
By not specifying a path in the
page.screenshot() function it will return a buffer instead of saving the screenshot to a file.
I update the
takeScreenshot() function to return the buffer.
The next step is to update the
run() function to send the buffer data back to the user.
I set the
image/png to notify the caller that I’m going to send an image. And then I simply pass the buffer to the
The result — Github’s homepage showing up in my Postman client 🎉
Method 2 — Uploading to Google Storage
So far I have a working screenshot API. But I’m going to extend it by uploading the screenshots directly to Google Storage.
I’m going to use the
@google-cloud/storage npm package for this.
Note that I have created a Google Cloud Storage bucket called
screenshot-api checkout this page for how to set up a storage bucket.
And after adding it to the
package.json I require the Storage object in the code.
I create a new function
uploadToGoogleCloud() that takes
filename as arguments.
I also created 2 constants
BUCKET_NAME holding those values.
uploadBuffer() function is simply a wrapper around Google’s
file.save() function to transform it into a promise.
The final thing to do is to return the Google Storage URL to the caller.
The new result — My postman client is showing the URL to the screenshot 🚀
Note that in the code above each screenshot is saved as
screenshot.png on Google Storage. In the real world, you would need to generate a random id for each image.
This was a fun project to do. You can find the source code of the completed Google Cloud function here. Also, I went a bit overboard and build an entire SaaS application around this, check it out here if you’re interested.
Thanks for reading!