Tello Drone Meets Google Home…(Part-1)

6 min readMar 7, 2020

Tello Drone Controlled By Google Assistant

We’ll be using actions on google , dialogflow and firebase to control tello drone from google assistant.

Lets Break it down into two parts as follows:

Part-1 — Building a Google Assistant Voice app to make changes in Firebase Database. (Actions on google — Dialogflow — Firebase Database)
Part-2 — Building a Python Program which controls the drone based on changes made in Firebase Database.

Part-1: Building a Google Assistant Voice app to make changes in Firebase Database.

The Goal of this part is we should be able to change values in database from Assistant.
Example: when we say “Go forward for 50 cms” then the database value will be updated with “forward-50–timestamp” as shown below

Now you’ve understood the goal we are trying to achieve in part-1 so before we dive in we need to understand few things about Firebase and actions on google.

Firebase Realtime Database is one of product of firebase & it is a cloud-hosted database. Data is stored as JSON and synchronised in realtime to every connected client. When you build cross-platform apps with our iOS, Android, and JavaScript SDKs, all of your clients share one Realtime Database instance and automatically receive updates with the newest data.
Actions on Google is a development platform for the Google Assistant. It allows the third-party development of “actions” — applets for the Google Assistant that provide extended functionality.

It is very much okay if you didn’t understand above two definitions because we will understand about it as we start working on them.

Let’s get started…

Go to Actions on Google Console here ,
Create a new project as “Tello-Drone” as shown below.

Now Click on “Conversational” under More options as shown below.

Enter a display name which is used to begin interacting with Voice app.
For instance , I’ve Kept it as Mr.Tello as shown

Go to Develop Tab -> Enter Display Name -> Save

1.Go to Account Linking.
2.Allow users to sign up with voice.
3.Select Linking Type as Google Sign In
4.Copy the client id in notepad.
5.Don’t forget to save

Time to Understand the important concepts in Actions on google.

Action
Dialogflow
Intent
Entity
Fulfilment

What is Action?

Actions on Google lets you extend the functionality of the Google Assistant with Actions. Actions let users get things done through a conversational interface that can range from a quick command to turn on some lights or a longer conversation, such as playing a trivia game.

What is Dialogflow ?

Dialogflow is a conversational platform that lets you design and build Actions by wrapping the functionality of the Actions SDK and providing additional features such as an easy-to-use IDE, natural language understanding (NLU), machine learning, and more.

What is Intent?

Intent is used to identify what user is trying to say or what user wants.
It is responsible to trigger certain action based on user input. In our case, “go forward for 50 cms” is one of training phrase in intent.

What is Entity?

Each intent parameter has a type, called the entity type, which dictates exactly how data from an end-user expression is extracted.

What is Fulfilment?

Fulfillment defines the conversational interface for your Action to obtain user input and the logic to process the input and eventually fulfill the Action.

Spend couple of minutes on this post to get clear idea about concepts used in Actions on google.

Spend some time on understanding basics of dialogflow at this Doc

Building Custom Intents Using Dialogflow

Go to Actions and Click on Build which will redirect us to dialogflow

Once you are into Dialogflow , Create an Agent as follows

It’s time to create an Entity

Now Let’s create Intents

Adding More Training Phases and linking them with Entities

Adding Responses and Enabling Web-hook Calls

Time to code

Before Diving into code, Let’s get the Firebase Database Url.

Open console.firebase.google.com and sign in to your account.

Now Open the Tello-Drone Project which was created automatically While creating Actions on google project.

Create a Realtime Database and start it in test mode as we don’t plan to implement Sign in Functionality in this post to keep it simple. (Will write about it in upcoming posts)

Save the Firebase database url in Notepad where client id was saved.

Adding Code into Dialogflow Inline Editor

We have two files namely index.js and package.json where the code in it needs to be pasted in inline editor as shown.

index.js

//'use strict';
process.env.DEBUG = 'actions-on-google:*';
//const {actionssdk} = require('actions-on-google');

const {
    dialogflow,
    SignIn
} = require('actions-on-google');

//const app = dialogflow();
const app = dialogflow({
    // REPLACE THE PLACEHOLDER WITH THE CLIENT_ID OF YOUR ACTIONS PROJECT
    clientId: "Replace_Client_id",
});

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp({
    credential: admin.credential.applicationDefault(),
    databaseURL: "Replace_database_Url"
});
const auth = admin.auth();
const db = admin.database();
const WORK_INTENT = 'movements';



app.intent('Default Welcome Intent', conv => {
    conv.ask('Welcome');
});

app.intent(WORK_INTENT, (conv, {
    direction,
    number
}) => {

    play(conv, direction, number);
});


exports.dialogflowFirebaseFulfillment = functions.https.onRequest(app);


function play(conv, dir, val) {
    console.log("direction is  ", dir);
    console.log(" value  ", val);
    return new Promise(function(resolve, reject) {
        if (val == "") {
            val = 20; //Default distance to move is 20 cms
        }
        devices_db = admin.database().ref('/tello_drone');
        devices_db.child('direction').set(dir + '-' + val + '-' + Date.now());

        return resolve;
    });
}

package.json

{
  "name": "dialogflowFirebaseFulfillment",
  "description": "This is the default fulfillment for a Dialogflow agents using Cloud Functions for Firebase",
  "version": "0.0.1",
  "private": true,
  "license": "Apache Version 2.0",
  "author": "Google Inc.",
  "engines": {
    "node": "8"
  },
  "scripts": {
    "start": "firebase serve --only functions:dialogflowFirebaseFulfillment",
    "deploy": "firebase deploy --only functions:dialogflowFirebaseFulfillment"
  },
  "dependencies": {
    "actions-on-google": "^2.2.0",
    "firebase-admin": "^5.13.1",
    "firebase-functions": "^2.0.2",
    "dialogflow": "^0.6.0",
    "dialogflow-fulfillment": "^0.5.0"
  }
}

Make sure to replace client id and database url with the one you saved in Notepad.

Deploying and testing in simulator

We need to Deploy the code which will take few seconds.
Once Deployed, We need to test it in simulator as shown and see if value changes in database.

Once it’s working as expected then we need to Enable it in Google Assistant as shown.

Finishing Touches of Part — 1

Initiate your Voice app by saying “Hey Google , Talk to Mr.Tello”
Now you can say “go forward for 100 cms” to update the value in database.

Congratulations on Successfully completing the part-1 and now let’s go with Part — 2

Here’s My LinkedIn Profile to Help , Share and Learn.