Adding Voice Recognition to Robots

How to use the Robot Control domain to add voice recognition to your programmable robot.

The Robot Control domain interprets voice intent and allows you to control your robot. If you have a robot or drone, you can program it to understand voice intent via the Robot Control domain. This tutorial will explain how to setup this domain for your robot or drone.

Overview

Before diving into the details, let’s talk about how the Robot Control domain works. The Robot Control domain interprets voice intent, but it is up to you to control the robot using client-side code.

If you have a robot that exposes a programmable API, you can use Houndify’s Robot Control domain to add voice functionality to it.

1. Create a Houndify Client and enable the Robot Control domain.
2. Download a Houndify SDK, and add the Client ID and Key from Step 1.
3. Capture a voice command using the SDK, such as “move forward 50 feet”. (we’ll go over all the supported commands below).
4. The voice command will be interpreted by the Robot Control domain, and a JSON response will be returned.

5. Interpret the JSON response and call a native method on your application layer to move the robot the desired amount.

Supported Commands

Here are the commands that are supported by this domain:

• Movement commands: It recognizes movement commands in the 2-D space, and a distance using any type of measurement. (for example: “Move forward 5 feet”).
• Rotation commands: It recognizes commands to “turn” or “rotate”, followed by the number of degrees. You can also ask it to “turn left/right/around”.
• Stop commands: Once a movement or rotation command has been specified, you can ask it to “stop” to immediately stop the last command. For example, “stop walking”.
• Understanding Custom Data: If you pass in custom data to the Robot Control domain (see below), it will be able to understand queries such as “Move to the living room”, where the living room is a custom object.

Sending Requests to the Robot Control Domain

To ensure that the Robot Control domain has full functionality, follow these steps:

1. Enable the Robot Control domain in Houndify, along with the Query Glue domain. You can also enable other domains that you require.
2. When sending a Houndify request, always pass in the RobotInfo object within the RequestInfo object. This should exist for every request.

Here’s an example of what this object looks like, although complete documentation about relevant fields can be found here.

"RobotInfo": {
 "RobotConfiguration": {
  "LegCount": 2,
  "ArmCount": 2,
  "IsMobile": true,
  "Sensors": {
   "HasCamera": true,
   "HasVideoCamera": true
  }
 }
}

How is configuration data used?

Passing in the configuration object above is important because the Robot Control domain will use it to determine whether a given action is feasible.

For example, if RobotConfiguration.HasCamera = false , the Robot Control domain will not respond to a query such as, “Take a picture”.

Setting Rooms

Clients integrating this domain can also upload an array of Room objects, to support queries such as “go to the kitchen”. This will only be useful however, if your robot is able to map out waypoints, store them, and navigate to them from an arbitrary location.

In order to add support for Rooms, invoke a text query with “set_known_rooms” and include a RobotInfo field similar to the above, with the addition of the KnownRooms field within the RobotState, as seen below.

{
    "RequestInfo" :
    {
        ...
        "RobotInfo" :
        {
            "RobotConfiguration" :
            {
                ...
            },
            "RobotState" :
            {
                "KnownRooms": [
                    {
                        "Name" : "Living Room"
                    },
                    {
                        "Name" : "Kitchen"
                    }
                ]
            }
        }
    }
}

Note: It’s important to make sure that the RobotInfo and RobotState objects that you pass into the RequestInfo for every request are kept up-to-date. The Robot Control domain will use the values in these objects to determine if a given action is possible or not.

For example, if you set RobotState.CurrentAction = “Turning” , the command “Stop Turning” will be matched. Otherwise, it won’t. Refer to the RobotInfo and RobotState documentation to view all the available fields that can be edited.

Handling Responses from the Robot Control Domain

The Robot Control domain interprets voice intent, but it is up to you to control the robot using client-side code.

When the domain recognizes a given command, it will return one of a few possible RobotCommandKind values. Each command type will also return a specific JSON object that will offer more information.

Here’s an example of what you may get back if you make the query: “move forward 10 meters”:

"AllResults": [
    {
      "CommandKind": "RobotCommand",
      ...
      "ClientActionSucceededResult": {
        "SpokenResponse": "Moving ten meters towards the front.",
        "SpokenResponseLong": "Moving ten meters towards the front.",
        "WrittenResponse": "Moving 10 meters towards the front.",
        "WrittenResponseLong": "Moving 10 meters towards the front."
      },
      "ClientActionFailedResult": {
        "SpokenResponse": "Could not move.",
        "SpokenResponseLong": "Could not move.",
        "WrittenResponse": "Could not move.",
        "WrittenResponseLong": "Could not move."
      },
      "RobotCommandKind": "MoveCommand",
      "Direction": 0,
      "Distance": 10,
      "DistanceUnit": "meter"
    }
  ],

In this case, we see that RobotCommandKind: "MoveCommand" along with values for Direction , Distance , and DistanceUnit . You can pass these values into your Robot’s API to move it.

In addition, we get two objects, ClientActionSucceeededResult and ClientActionFailedResult. These can be used based on whether your robot’s API is able to move it successfully or not.

Although they are listed here, the most up-to-date documentation can be found on the Robot Control Domain’s documentation.

• MoveCommand: Robot was asked to move.
• TurnCommand: Robot was asked to rotate.
• StopCommand: Robot was asked to stop its current action.
• TakePictureCommand: Robot was asked to take a picture.
• TakeVideoCommand: Robot was asked to record a video.
• StopRecordingVideoCommand: Robot was asked to stop recording a video.
• GoToRoomCommand: Robot was asked to go to a specific room. Only rooms recognized by the robot via the KnownRooms array in the RequestInfo will be recognized.

The following commands allow the user to get and set information related to the Robot’s state.

• RobotListCapabilitiesCommand: Robot was asked to list its capabilities. This will come from the RobotInfo object passed into the RequestInfo (see above).
• RobotListKnownRoomsCommand: Robot was asked for all the rooms that it knows. This will return the data in RobotInfo.RobotState.KnownRooms.
• RobotSetKnownRoomsCommand: If you send a special text query with “set_known_rooms”, the KnownRooms will be set with the object provided in the text request.

We’re always looking for new ideas on how to improve domains. If you have any commands which you think would be useful to add to Robot Control or have general feedback, comment below or contact us with your feature request.

Happy coding!