Building a Website Screenshot API

I have a personal project that needs to take screenshots of approximately 100,000 websites per month. I could use an existing website screenshot API service for this — there are plenty of those — but it would be way more fun to build one myself.

In this blog post, I describe the steps I took to set up this API, let’s dive in!

Puppeteer

Puppeteer is a node package that allows you to control a headless chrome browser using Javascript. A headless chrome browser is just a browser without a window.

I can use this package to spin up a headless chrome instance, navigate to a website and take a screenshot.

To start I’m going to create a local node project and install the puppeteer package.

npm init
npm install puppeteer

Next, I create an index.js file and write the code to take a screenshot.

Note that I am making the takeScreenshot() function async. This way I can use the await keyword in the function to wait for all the promises.

After running the code I get the following screenshot! 🎉

Google Cloud Functions

So I now have a local script that I can call to take a screenshot, but I want to build an API. The next logical step is to put this script on a server somewhere.

I don’t want to worry about my server running out of memory, so I’m going to put it on Google Cloud Functions. This way it can handle a huge number of requests without me having to worry about buying more RAM memory.

Once I have the cloud function running, I can call it with an HTTP request — meaning that I will have a working screenshot API 🚀

Let’s port the previous code to the Google Cloud Function format. The cloud function I created is async and called run().

Because I’m calling this cloud function with an HTTP request it is going to send a JSON response to the caller. If successful a 200 status code is sent back along with a happy message. If unsuccessful the error is sent back.

On line 23 the function will go to the URL located in the request object. This means that I can pass the URL as a POST parameter to the HTTP request 🤖

For example, I would call the cloud function like this to take a screenshot of github.com

POST https://google-cloud-endpoint/my-function
{
    "url": "https://github.com"
}

Finally, I should update the package.json file in the google cloud function to include the puppeteer dependency.

After saving the function I’m using Postman to call the cloud function.

Unfortunately, there is an error. Google Cloud Functions don’t allow writing files to the disk — so I cannot store the screenshot in the screenshot.png file.

There are 2 ways to solve this.

• Don’t store the screenshot in a screenshot.png file but return the raw image data to the caller.
• Upload the screenshot to Google Storage and return the URL to the Google Storage file.

Method 1 — Return the raw image

By not specifying a path in the page.screenshot() function it will return a buffer instead of saving the screenshot to a file.

I update the takeScreenshot() function to return the buffer.

The next step is to update the run() function to send the buffer data back to the user.

I set the content-type to image/png to notify the caller that I’m going to send an image. And then I simply pass the buffer to the res object.

The result — Github’s homepage showing up in my Postman client 🎉

Method 2 — Uploading to Google Storage

So far I have a working screenshot API. But I’m going to extend it by uploading the screenshots directly to Google Storage.

I’m going to use the @google-cloud/storage npm package for this.

Note that I have created a Google Cloud Storage bucket called screenshot-api checkout this page for how to set up a storage bucket.

And after adding it to the package.json I require the Storage object in the code.

I create a new function uploadToGoogleCloud() that takes buffer and filename as arguments.

I also created 2 constants GOOGLE_CLOUD_PROJECT_ID and BUCKET_NAME holding those values.

The uploadBuffer() function is simply a wrapper around Google’s file.save() function to transform it into a promise.

The final thing to do is to return the Google Storage URL to the caller.

The new result — My postman client is showing the URL to the screenshot 🚀

Note that in the code above each screenshot is saved as screenshot.png on Google Storage. In the real world, you would need to generate a random id for each image.

Conclusion

This was a fun project to do. You can find the source code of the completed Google Cloud function here. Also, I went a bit overboard and build an entire SaaS application around this, check it out here if you’re interested.

Thanks for reading!