Exploring Cloud Vision Landmark API With Firebase Cloud Functions
Artificial intelligence is getting increasingly popular year by year. One thing fueling this trend are services such as Google Cloud Vision API, which make it straightforward to add AI capabilities to your apps.
In this article, we cover how to use Google Cloud Vision API’s AI with Firebase Cloud Functions. Even though you’ll need to enable billing with a credit/debit card during this guide, the project will be free to create and run. In the next article of this two-part guide, we’ll build on the foundation laid here by adding React Native to the mix. As a result, we’ll have a simple mobile app that uses Google’s AI.
What is the Cloud Vision API?
The Cloud Vision API offers pre-trained models that can be used via REST in a single API request to analyze images. We will focus on detecting landmarks, but the Vision API also supports other use cases. To mention some, the feature list includes detecting faces, logos, optical character recognition to extract text and much more.
What are Firebase Cloud Functions?
Similar to AWS Lambda, Firebase Cloud Functions allow running code without needing to worry about backend frameworks or servers. Combining Cloud Functions with the Vision API gives us a REST API that uses AI to validate if the images given to it contain landmarks.
Brief look at the architectural design:
Looks good? Let’s get started with setting everything up.
Setting up Firebase with Cloud storage and Cloud Functions
First, open https://console.firebase.google.com/ and create a new project and name it landmark. As you start typing the project name, a project id appears. Take note of it, you will need it later.
Once within the project click Develop > Storage > Get Started.
Accept the suggested basic rules, click Next and then select your cloud storage location. Any location works, but choosing a location close to the application user is a good idea. Doing so improves the user experience by lowering latency.
We want to add authentication as well. Click on Authentication in the side menu and then on the Set up sign-in method button. We will be using Anonymous authentication, so enable that and save.
We can now continue with the Cloud Functions. Open a command line window. Assuming you have Node.js installed, you can install the Firebase tools with
npm install -g firebase-tools
Then login to the same account used to create the Firebase project with
Permit all that Firebase is asking for and then you should get this screen:
Create project folder
Next, let’s just create the project folder with
mkdir landmarkcd landmark
Make sure that you are in the landmark root folder, then run
You can now choose what you want to add to the project. Let’s select Functions here. For the next questions, we’ll Use an existing project and select TypeScript as the language. Accept the defaults for the other questions. Firebase now generates a functions folder for you.
Setting up Google Cloud Vision API
To set up Google Cloud Vision, let’s open the GCP console and select the Firebase project we created. Link: https://console.cloud.google.com/projectselector2/home/dashboard
If you don’t see your project on the dashboard, search for landmark in the search bar and select it:
Next up, you need to enable billing by clicking Billing in the navigation menu:
From here, you can either Link a billing account or create a new one with Manage billing accounts.
Once we have the billing account setup we can enable the Vision API.
Go to https://console.cloud.google.com/flows/enableapi?apiid=vision.googleapis.com. Select the landmark project in the dropdown, then click Continue. You should now have the Vision API enabled for the project.
We still need to give our project access to this service locally and we are going to do so by creating a service account for the landmark project via https://console.cloud.google.com/projectselector2/iam-admin/serviceaccounts?supportedpurview=project
If the landmark project is not already selected, select it. Then click on Create service account and fill in the details
Click Create, then select the Project Owner role:
Then click Continue and Done. Now we need to create a key for the service account. Click the three dots under Actions next to the landmark account and choose Create key. Select JSON and click Create to download the key.
Once you’ve downloaded the key, set it as an environment variable. Replace [PATH] with the location of the downloaded JSON file.
We now have everything setup for Cloud Vision API to work both locally and online.
Coding the Cloud Function
Start by adding the only dependency we need to the package.json in the functions folder:
cd functionsnpm install @google-cloud/vision
npm install --save-dev @types/node
Firebase CLI created an index.ts file to the functions/src folder. Replace its contents with this:
Remember the Firebase project id you took note of when creating the project? If you don’t, just open the project in the Firebase console at https://console.firebase.google.com/, click the cog next to Project Overview and choose Project Settings. In any case, replace the projectId value landmark-bff66 with your own project id.
Make sure that you compile the code. While still in the functions folder run:
npm run build
This function uses the Cloud Vision API to validate that images contain landmarks. We start by adding the Cloud Vision dependency and creating a client with the constructor.
Then after that we are grabbing the photo id from the request body. We can now use these variables to tell the landmark client which photo to validate from our storage.
Finally we set the landmark array as a response. The vision client will return an array with all the landmarks from the photo, if there were no recognizable landmarks the array will be empty.
Testing the Cloud Function
Now that the cloud function is coded it’s time to make sure that it works! Start a test server with the following command in the landmark root folder:
Make sure that you have your service account JSON file set as an environmental variable otherwise it won’t work! Look at the end of the Setting up Google Cloud Vision API to see how it is done.
Let’s now test the function by using Postman. But first go over to the project in the Firebase console and click on Storage in the menu.
Create a folder called uploads here. Then click on that folder and click on the Upload File button.
Choose an image of a landmark, for example https://www.pexels.com/photo/eiffel-tower-under-blue-sky-3922718/ , and upload it here. Once you have your image uploaded within the uploads folder, go over to Postman.
In Postman, create a new POST request. The URL is of the form http://localhost:5001/YourProjectId/YourCloudFunctionLocation/validateLandmark and can be found in the output of firebase emulators:start.
We will also need to set a body for this post request, select raw and then JSON in the dropdown menu and then the JSON should look like this:
Swap the “photoId” value with the name of the image you uploaded. You can confirm the image name in the Firebase Cloud Storage.
This is what the request should look like now:
Go ahead and try to send the request! If everything went well, the response should look like this:
If you uploaded an image of something that the Vision AI doesn’t recognize as a landmark the array will be empty.
If everything is working you are now ready to deploy the function with
firebase deploy --only functions
Once you see Deploy complete!, open Postman and replace the request URL with the Function URL output by firebase deploy. You can also find it in the Firebase console by clicking Functions in the side menu.
Now test the function again. It should give the same response as before, but this time from the Firebase server instead of your local machine. Congratulations, you have now successfully completed your first experiment with the Google Cloud Vision API!
You now have a fully functional Cloud Function for identifying landmarks from images. In the next article, we will build a React Native mobile application that makes use of what we’ve done so far. Hopefully you’ve found this article to be useful and I’ll meet you again in the next one!
Mathias Rahikainen is a Full Stack Developer and Consultant at Acelvia, where we build web and mobile apps for our awesome customers. We’re always interested in taking on new interesting projects. Read more about us at https://www.acelvia.com/en or get in touch via firstname.lastname@example.org.