The “Quicker” Way to Integrate Google Firebase Landmark Recognition on React Native

As a recent software boot camp graduate and newly minted web developer, I recently had a moment in the midst of my studies to reflect on what I’ve really learned so far. When you make the personal commitment to answer a new calling in code, and you opt for the rigorous pace of a regimented, fast moving environment like a coding boot camp, you will become quickly accustomed to floods of information you are tasked with absorbing. Amidst this, you are also applying much of these new concepts in both long and short-term projects, often to help build what will become your portfolio. This can be fun and challenging, and it can also lead to many a frustrating moment when things don’t work exactly the way you’d hoped. Hours can turn into days of debugging, refactoring code to fix errors, rabbit holes of documentation, forum posts, and articles like this one.

It is because I feel your pain that I decided to write this article, which I hope can help other new-to-intermediate Javascript students save some time and get something working in a jiffy should they need to demonstrate some quick functionality on a project. On working on my own bootcamp Capstone, I found that documentation and article content for setting up Google Cloud and Firebase APIs was often verbose, lengthy, and did not always directly address what I was looking for: which is fine, but also lends itself to be a good topic that one can expand and add to in order to help others who may be searching for similar solutions and not finding exactly what they were looking for.

What we are going to do in this article is set up a simple and quick way for a React Native App to use Google Vision Kit APIS from the front-end, for easy integration of some powerful tools. It should be noted that first and foremost, this is not the recommended way to integrate these tools if you are building out a full-scale app which will be heavily based on these technologies: you should always aim to incorporate technology like Google Vision API to the back-end, on the server side of your application, and React Native allows you to do this via Express and Node JS. Depending on your comfort-level with Javascript and React Native, however, that process can be quite consuming and opens the door for lots of issues you will potentially need to address. This demo is to suggest a simpler alternative that can be used as a placeholder, demonstration, or addition on a pre-existing structure: it is not a substitute for more robust setups. This is also going to be kept in IOS land for the sake of not getting too caught up in the differences between porting to Android and Apple devices: suffice it to say if Android is your final destination, you will need to examine how that would work with the documentation I provide in these respective steps (and certainly some additional know-how on top of that). Furthermore, this article assumes you know how to run Xcode’s app simulator and have at least an entry level familiarity with Xcode, as you won’t be able to test your work unless this is the case (we will not be on Expo’s managed workflow).

With that said, if you’re on a time crunch, if that final project is due in a couple of days and you just need to get something working, or this is a purely secondary feature and your time is better served handling primary tiers of your project, there is a much simpler way to bring in Google Vision technologies like Landmark Recognition by using our good friend Firebase, and this is how we can do it:

Let’s get set up first, and for the purposes of showing a clean demo we’ll start a brand new React Native App from scratch. This article is not focused on setting up React Native, and if you need more help in doing that you should consult some of the documentation I provide below.

1.) Create a new project to start working using Expo CLI (note that because Firebase Tool kits do not work with Expo’s Managed Workflow style, we are going to convert this to a React Native Expo Bare project in the next step):

A. In your terminal, install and setup the new project:

npm install -g expo-cli
expo init FirebaseRecogDemo
cd FirebaseRecogDemo
npm start

As mentioned previously, if you also intend to start from a new project and run into any issues, or this is your first time setting up a React Native project, please see https://reactnative.dev/docs/environment-setup for the full, official rundown on getting your project going.

2.) We now need to convert this from Expo’s managed workflow to it’s bare workflow. You can read here (https://docs.expo.io/introduction/managed-vs-bare/?redirected) about these two formats in more depth, but suffice it to say that using React Native Bare is going to allow us the functionality with Firebase we need:

expo eject
// When prompted, enter an android and ios app name that follows the pattern “app.YOURAPPNAME.App”
cd ios
pod init
pod install
cd ..

3.) You will also need to have a registered and working Google Developer account and create a new project in Firebase for the purposes of this demo that has “Blaze Plan” access. While Blaze is a pay-as-you-go version that will suit our purposes for today (and we will stay well under Google’s free use threshold, so don’t worry about incurring costs), it will require you to enter a credit card and verify you are not a robot. That complete set up process can be found by following these documents here:

Essential Reading:

https://cloud.google.com/apis/docs/getting-started

https://firebase.google.com/docs/projects/learn-more (see “Setting Up a Firebase Project and Registering Apps”)

https://rnfirebase.io

https://firebase.google.com/docs/ios/setup (This is because we are deploying to IOS in this demo, even though we are using Javascript, we have to set this up as if we were building it in Xcode with Swift, and then treat our Node.js backend like we were setting up for a traditional Web-based app like in here: https://firebase.google.com/docs/web/setup)

Here is some additional, non-essential reading if you are interested:

https://rnfirebase.io

The steps below are an abbreviated overview of the setup process that these documents cover, but this set up can be intricate and information sensitive, so I HIGHLY recommend you follow the exact documentation to do so.

In a nutshell, this is what will you be doing:

A.) Set up a new IOS app project in your newly created Firebase project.

B.) Download the Google-Service-Info.plist file provided with the IOS App.

C.) Incorporate Firebase SDKs into your project by:

cd ios

Enter the following lines into your .Podfile folder:

# add pods for desired Firebase products
# https://firebase.google.com/docs/ios/setup#available-pods

Then a perform new pod update:

pod install
cd ..

D.) Open the .xcworkspace file in your ios directory in Xcode.

E.) Drag the Google-Service-Info.plist file into the root of the project.

npm install --save firebase
npm install expo-updates
cd ios
pod install
cd ..

F.) Open the FirebaseRecogDemo (or your project name) directory within your IOS directory, then open the Supporting directory. You will find a file name Expo.plist, add this just underneath <dict>:

<key>EXUpdatesURL</key>
<string>https://example.com</string>

G.) You should now be able to run your app by typing npm run ios in your root directory.

Note: Every time you add something new to this project, it is VERY important to remember to do a whole new pod install in the ios directory!

This part of the process is the most long-winded (unless you already have a Google developer account and experience setting up React-Native of course!), and the need to combine some aspects of the IOS and Web procedures from Firebase can get confusing, but rest assured: it will work!

4.) Let’s go ahead and create a starting View for our new App, where we’ll also demonstrate our API technology. Expo’s setup very conveniently automatically creates a working React-Native landing component for us in App.js:

import { StatusBar } from ‘expo-status-bar’;
import React from ‘react’;
import { StyleSheet, Text, View } from ‘react-native’;
export default function App() {
return (
<View style={styles.container}>
<Text>Open up App.js to start working on your app!</Text>
<StatusBar style=”auto” />
</View>
);
}
const styles = StyleSheet.create({
container: {
flex: 1,
backgroundColor: ‘#fff’,
alignItems: ‘center’,
justifyContent: ‘center’,
},
});

Let’s convert this into a React class component so we will have access to a local state:

import { StatusBar } from ‘expo-status-bar’;
import React from ‘react’;
import { StyleSheet, Text, View } from ‘react-native’;
export default class App extends React.Component {
constructor () {
super()
this.state = {
field: ‘’
}
}
render () {
return (
<View style={styles.container}>
<Text>Open up App.js to start working on your app!</Text>
<StatusBar style=”auto” />
</View>
);
}
}
const styles = StyleSheet.create({
container: {
flex: 1,
backgroundColor: ‘#fff’,
alignItems: ‘center’,
justifyContent: ‘center’,
},
});

Now let’s expand on that by adding an input field, alert, and submit button (along with styles and some form functionality) and changing up our text component so we will be able to test our landmark recognition once we’ve finished setting it up:

import { StatusBar } from ‘expo-status-bar’;
import React from ‘react’;
import { StyleSheet, Text, View, TextInput, Button, Alert } from ‘react-native’;
export default class App extends React.Component {
constructor () {
super()
this.state = {
field: ‘’
}
}
render () {
return (
<View style={styles.container}>
<Text>Insert image URL here:</Text>
<TextInput
style={{
height: 30,
backgroundColor: ‘#F5FFFA’,
borderWidth: 0.5,
borderRadius: 13,
padding: 5,
}}
placeholder=” Enter Image Here”
multiline={true}
onChangeText={(field) => this.setState({ …this.state, field: field })}
defaultValue={this.state.field} />
<Button title=”Submit”></Button>
<StatusBar style=”auto” />
</View>
);
}
}
const styles = StyleSheet.create({
container: {
flex: 1,
backgroundColor: ‘#fff’,
alignItems: ‘center’,
justifyContent: ‘center’,
},
});

When we run our app using npm run ios in our terminal (or we build within Xcode), we should now get something like this:

The Demo App’s home screen in IOS Simulator.

5.) Now it’s time for the fun part! We’ll first integrate Google’s ML Kit library, which is designed to set our app up to handle Firebase front-end requests:

A.) Before the nitty gritty, verify your app is set up and authorized properly via Google to utilize landmark recognition:

https://firebase.google.com/docs/ml/ios/recognize-landmarks

B.) Let’s install these dependencies first with an npm install in our terminal:

npm install @react-native-firebase/app
npm install @react-native-firebase/ml

C.) Now in the .Podfile in our IOS directory, we will add a new line with this:

pod 'Firebase/MLVision'

D.) Following this it is essential to run pod update before doing yet another fresh pod install so our Cocoapods matches our node libraries:

cd ios
pod update
pod install
cd ..

E.) Then, let’s go ahead and import the Firebase initialization we will need along with the ml object from the library at the top of our component:

import firebase from ‘@react-native-firebase/app’
import ml from ‘@react-native-firebase/ml’;

F.) We will now need to establish our configuration from our Firebase account in order to get proper authorization, and then initialize our connection. Create this new config object and enter your Google credentials in the appropriate key :

const config = {
apiKey: ‘<api-key>’,
authDomain: ‘<app-name>.firebaseapp.com’,
databaseURL: ‘https://<app-name>.firebaseio.com’,
projectId: ‘<project-id>’,
storageBucket: ‘<app-name>.appspot.com’,
messagingSenderId: ‘<messaging-sender-id>’,
appId: ‘<app-id>’
};
firebase.initializeApp(config);

Note: The API key must be a new credential created in your Google Cloud API section (see Creating An API Key in https://cloud.google.com/docs/authentication/api-keys).

MORE IMPORTANT NOTE: THIS IS NOT SECURE!!! PLEASE REMEMBER TO PROPERLY SECURE AND HIDE THIS INFORMATION BEFORE EVER LAUNCHING AN APP TO THE PUBLIC!!!! https://firebase.google.com/docs/projects/api-keys

6.) Next, let’s bring in some code that will act as a “call” for Firebase’s Landmark Recognition:

To insert after our React component in App.js:

async function processLandmarks(localPath) {
const landmarks = await ml().cloudLandmarkRecognizerProcessImage(localPath);
landmarks.forEach(landmark => {
console.log(‘Landmark name: ‘, landmark.landmark);
console.log(‘Landmark locations: ‘, block.locations);
console.log(‘Confidence score: ‘, block.confidence);
});
}

Let’s pause for a moment to understand what’s actually happening here: in the same way we’d make a server “POST” request if we were designing a web app, this function asynchronously sends out the address path of our image (localPath), waits for Landmark Recognition to do it’s magic, and then sends back a “landmarks” data package (which in this case, happens to be an array of objects). This function is designed to show you how to access the relevant data you need from that array by including a series of console.logs in order to point out different pieces of information you can tap into. That’s great: thank you Google! If we were to console.log our entire landmarks array, we would get something that looks like this:

[{“boundingBox”: [0, 0, 597, 330], “confidence”: 0.80711186, “entityId”: “/m/03d_7r”, “landmark”: “Millennium Park”, “locations”: [[Array]]}]

In this example (which uses Chicago’s Millennium Park) we see our array’s first object has a key pointing to the landmark name itself (landmark), a “confidence” level of the algorithm’s accuracy, and other identifying bits of data. If we were to log more of this, we would see subsequent objects in this array containing similar properties: Google landmark recognition gives us a list of most likely locations for the image we send it, sorted by confidence level. Pretty cool!

However, if we want to send that information out to be used in other parts of our app, we are going to need to comment out or delete those console.logs and do a good old simple return of our information so that it is now usable by the rest of this component:

async function processLandmarks(localPath) {
const landmarks = await ml().cloudLandmarkRecognizerProcessImage(localPath);
return landmarks;
}

We’ve got our function, we’ve got our data: now it’s time to use it!

All we have to do now is create a handleSubmit function to send it out once we click upload, and for our purposes we can create a quick alert to let us know Google landmark recognition has successfully returned a result:

import { StatusBar } from ‘expo-status-bar’;
import React from ‘react’;
import { StyleSheet, Text, View, TextInput, Button, Alert } from ‘react-native’;
import firebase from ‘@react-native-firebase/app’
import ml from ‘@react-native-firebase/ml’;
const config = {
apiKey: ‘<api-key>’,
authDomain: ‘<app-name>.firebaseapp.com’,
databaseURL: ‘https://<app-name>.firebaseio.com’,
projectId: ‘<project-id>’,
storageBucket: ‘<app-name>.appspot.com’,
messagingSenderId: ‘<messaging-sender-id>’,
appId: ‘<app-id>’
};
firebase.initializeApp(config);
export default class App extends React.Component {
constructor () {
super()
this.state = {
field: ‘’
}
this.handleSubmit = this.handleSubmit.bind(this)
}
async handleSubmit(field) {
let newLocations = await processLandmarks(field)
this.setState({…this.state, field: ‘’})
Alert.alert(`Your landmark is ${newLocations[0].landmark}`)
}
render () {
return (
<View style={styles.container}>
<Text>Insert image URL here:</Text>
<TextInput
style={{
height: 30,
backgroundColor: ‘#F5FFFA’,
borderWidth: 0.5,
borderRadius: 13,
padding: 5,
}}
placeholder=” Enter Image Here”
multiline={true}
onChangeText={(field) => this.setState({ …this.state, field: field })}
defaultValue={this.state.field} />
<Button onPress={() =>
this.handleSubmit(this.state.field)} title=”Submit”>
</Button>
<StatusBar style=”auto” />
</View>
);
}
}
async function processLandmarks(localPath) {
const landmarks = await ml().cloudLandmarkRecognizerProcessImage(localPath);
return landmarks;
}
const styles = StyleSheet.create({
container: {
flex: 1,
backgroundColor: ‘#fff’,
alignItems: ‘center’,
justifyContent: ‘center’,
},
});

If all is working correctly, we will receive an Alert Pop-up with a landmark that is the "landmark" key value of the first object in the array of possible locations that Google Landmark Recognition sends back to us: in this case the Trocadero Gardens of Paris, which is home to the Eiffel Tower.

The alert that pops up once you click submit with an image local file path pasted into the text input field (landmark recognition usually takes anywhere from 3–5 seconds on average to return a response that will trigger the pop up).

A few notes: to demo this in your upload form, you will need an exact local file path to insert into the text field (in the example above, I saved a picture of the Eiffel Tower from afar onto my hard drive, thus rendering a response of Trocadero Gardens). Additionally, while this is my “quick” speed version to set up landmark recognition, please keep in mind errors are not just possible, but highly likely, during this set up. React-native third party API integration is almost always filled with challenges: this is a reality of the technology. One thing that often does the trick: close the simulator, the terminals, and whatever else is running processes, and start fresh (Particularly during the steps towards the end of this walk-through). You’d be surprised what starts working with a fresh restart!

And there we have it! The possibilities, however, are truly endless, and this is only one, simple way we could use this. In my own project, we use RN-Camera to take pictures of locations as you are walking down the street, and have the camera send those newly made pictures directly over to Google landmark recognition, before they were even saved to a database or local file storage. You could also upload images with an upload page, or use other machine learning to plug in with this for a truly smart app. Furthermore, other technologies from the Google React-Native Firebase ML Kit give you a serious arsenal of technology that can now be easily incorporated to your front end (while each may have variations in it’s integration, you can assume at the very least there will be recurring patterns to the set up seen here).

While you should always aim to create the backbone of your app’s APIs with server-side functionality, which in this case would have hypothetically been back end Google Cloud Vision API server routing, Firebase offers a more limited form of Vision’s capabilities that this article has hopefully demonstrated can be a quicker and easier way to take advantage of these awesome technologies (especially if you’re in a pinch, fellow students), and hopefully save you some time and stress when you’re off solving the world’s problems with code. Happy developing!