Open in app

Sign in

Write

Sign in

Voice Panel

Michael Ritchie
Snips Blog
Published in
11 min readFeb 22, 2019

A guest blog post by Michael Ritchie.

Voice Panel is an open source Android Voice Assistant for Home Assistant powered by the Snips Voice Platform. Snips provides a private, powerful, and customizable voice assistant technology that processes all language input on-device, with no data ever sent to the cloud.

Voice Panel User Interface which supports Darkmode

Voice Panel uses Snips to act as a voice interface for Home Assistant. You can control your alarm system, lights, windows, blinds, switches, check status, get the date/time, and retrieve the weather information. Here is a list of supported Snips apps:

Supported Snips apps for Voice Panel

Currently the application has a few limitations. The Snips Android SDK does not work as a satellite, so all messages must be forwarded to Home Assistant using MQTT, and speech is processed on the Android device using TTS. The Snips Android SDK does not support custom wake-words at this time. To initiate a conversation, you must say “Hey, Snips” or you can try the face detection wake-word feature which starts listening when the device camera recognizes a face.

Features

  • Face activated wake-word (no need to say “Hey, Snips”).
  • Control Home Assistant components using voice commands (“Turn on the kitchen lights”).
  • Stream video, detect motion, detect faces, and read QR Codes.
  • Support for MQTT Alarm Panel Control to control your alarm system.
  • MQTT commands to remotely control the application (speak text, play audio, send notifications, alerts, etc.).
  • Device sensor data reporting over MQTT (temperature, light, pressure, battery, etc.).
  • MQTT Day/Night mode based on the sun value reported from Home Assistant.
  • MQTT weather data to display weather reported from Home Assistant.

Installation & Requirements

  • Android Device running Android OS 5.0.1 (SDK 21) or greater. Though performance on older devices may vary.
  • Google Play services may also be required, this may affect some Amazon Fire Tablet devices.
  • Ideally you want to keep your device unlocked and use Daydream as a screensaver, waking the device when you either say the wake-word or with motion or face detection.
  • You can download and install the latest release from Github project release section or if you prefer, install the application from the Google Play.

Assistant Setup

You first need to setup a MQTT Broker in Home Assistant by adding the HASS MQTT Broker add-on to Home Assistant. This allows the two-way communication between Home Assistant and Voice Panel using the MQTT messaging protocol.

For the Voice Assistant to control Home Assistant components, you need to install either the Snips component or the Snips Add-On.

Home Assistant already has bundled scripts included when you add the Snips platform to Home Assistant. These scripts will allow you to turn on/off components (lights, switches, etc.), open/close components (garage, blinds, etc.), set the color of lights, add items to your shopping list, and retrieve items from your shopping list. So you can say “Turn on the kitchen lights” to control a the component lights.kitchen.

For Snips you basically need to add snips: to you configuration.yaml file and then add the custom intents you want to work with. The basic intents bundled with Home Assistant will already allow you to work with your HA components.

Custom Assistant Commands

Voice Panel also has custom scripts that extend the basic functionality. Currently Voice Panel can control your MQTT alarm panel in Home Assistant, get the status for any component by name (“What’s the status of the front door?”), get the time/date, get the weather, and control the thermostat.

To use custom scripts you need to create a file to store your intent scripts and customize the assistant’s behavior. Create an “intents.yaml” file in your configuration directory, then link this from the configuration.yaml file by adding this line at the bottom:

intent_script: !include intents.yaml

You will place intent scripts within the new “intents.yaml” file to work with the various components currently supported by application. Home Assistant already has bundled scripts included when you add the Snips platform to Home Assistant. Voice Panel also has custom scripts to extend upon the basic functionality. The additional scripts can be added to your intents yaml file to use these features.

HA Alarm Control

To control a Home Assistant alarm control panel using voice, you need to add the alarm intents scripts to your “intents.yaml” file. This will allow you to set the alarm status, disarm the alarm, and set the alarm to “home” or “away” modes by voice. Here are some sample voice commands:

  • “Set the alarm to home.”
  • “Disable the alarm.”
  • “Set the alarm to away mode.”
  • “What’s the status of the alarm?”

Alarm Panel Intents Scripts

HaAlarmStatus:
  speech:
    type: plain
    text: >
      {% if is_state("alarm_control_panel.ha_alarm", "disarmed") -%}
        The alarm is not active.
      {%- endif %}
      {% if is_state("alarm_control_panel.ha_alarm", "armed_home") -%}
        The alarm is in home mode.
      {%- endif %}
      {% if is_state("alarm_control_panel.ha_alarm", "armed_away") -%}
        The alarm is in away mode.
      {%- endif %}
HaAlarmHome:
  speech:
    type: plain
    text: > 
      OK, alarm set to home.
  action:
    - service: alarm_control_panel.alarm_arm_home
      data:
        entity_id: alarm_control_panel.ha_alarm
HaAlarmAway:
  speech:
    type: plain
    text: >
      OK, the alarm set to away, you have 60 seconds to leave.
  action:
    - service: alarm_control_panel.alarm_arm_away
      data:
        entity_id: alarm_control_panel.ha_alarm
HaAlarmDisarm:
  speech:
    type: plain
    text: >
      The alarm has been deactivated.
  action:
    - service: alarm_control_panel.alarm_disarm
      data:
        entity_id: alarm_control_panel.ha_alarm

Weather Updates

If you would like to get weather updates, you will need to add a weather component to Home Assistant. Voice Panel can report the weather by adding a new intent that will speak the current weather conditions to your “intents.yaml” file. Here is a sample voice command:

  • “What’s the weather today?”

Example Dark Sky Weather

searchWeatherForecast:
  speech:
    type: plain
    text: >
      The weather is currently 
      {{states('sensor.dark_sky_temperature') | round(0)}} 
      degrees outside and 
      {{states('sensor.dark_sky_daily_summary')}} 
      The high today will be 
      {{ states('sensor.dark_sky_daytime_high_temperature') | round(0)}}
      and 
      {{ states('sensor.dark_sky_hourly_summary')}}

Status Updates

You can get the status of Home Assistant components using voice commands. First you need to add the intent scripts to the “intents.yaml” file to handle script information. The status intent has two slots, one for the entity location (such as kitchen, living room, deck, upstairs, etc.) and one for entity id (door, camera, light, etc.). These slots can be used to provide a specific response for the item status.

  • “What’s the status of the main door?”
  • “Main door state?”
  • “Are the doors secure?”

Example Status Intent Script

getStatus:
  speech:
    type: plain
    text: >
      {% if entity_locale == 'main' and  entity_id == 'door'  %}
        The {{ entity_locale }} {{ entity_id }} is {{states(sensor.main_door)}}
      {% endif %}

In the above example, “entity_locale” and “entity_id” are used to provide a specific sensor status. In this case we asked “what is the status of the main door?” and Snips responds with “The main door is closed”, which is the status of the sensor.main_door entity.

Time/Date

You can get the current time and date from Home Assistant components using voice commands. Add the following intent scripts to the “intents.yaml” file to handle intents for Time/Date.

  • “What’s the date?”
  • “What day is it?”
  • “What time is it?”

Example Status Intent Script

getCurrentTime:
  speech:
    type: plain
    text: > 
      The time is {{ now().hour}}  {{ "%0.02d" | format(now().strftime("%-M") | int) }}
getCurrentDate:
  speech:
    type: plain
    text: > 
      It is {{ now().weekday }}, the {{ now().day }} of {{ now().month }}, {{ now().year }}
getCurrentDay:
  speech:
    type: plain
    text: > 
      Today is { {now().weekday }}

MQTT Alarm Panel Control

The alarm panel can be controlled using only voice, however, it also comes included with a manual way to set and disable the alarm using MQTT messaging. To use this feature, you need to install the Manual Alarm Control Panel with MQTT Support. This component allows for two-way control of the Home Assistant alarm panel component using MQTT messaging.

To enable the MQTT alarm feature, under settings (the gear icon) select Alarm Settings. Once active, you will see a lock icon at the bottom of the main screen which displays the current alarm mode and provide a manual means for arming and disarming the alarm in addition to the voice controls. The Alarm Settings has options to change the MQTT topic and commands if you do not wish to use the defaults.

Supported Command and Publish States

  • Command topic: home/alarm/set
  • Command payloads: ARM_HOME, ARM_AWAY, DISARM
  • Publish topic: home/alarm
  • Publish payloads: disarmed, armed_away, armed_home, pending, triggered (armed_night not currently supported).

Example Home Assistant Setup

alarm_control_panel:
  - platform: manual_mqtt
    state_topic: home/alarm
    command_topic: home/alarm/set
    pending_time: 60
    trigger_time: 1800
    disarm_after_trigger: false
    delay_time: 30
    armed_home:
      pending_time: 0
      delay_time: 0
    armed_away:
      pending_time: 60
      delay_time: 30

— If I set the the alarm mode home, the alarm will immediately be on without any pending time. If the alarm is triggered, there will be no pending time before the siren sounds. If the alarm mode is away, I have 60 seconds to leave before the alarm is active and 30 seconds to disarm the alarm when entering.

— Notice that my trigger_time is 1800 and disarm_after_trigger is false, this means the alarm runs for 1800 seconds until it stops and it doesn’t reset after it’s triggered.

MQTT Weather

Weather from Dark Sky

You can also use MQTT to publish the weather to the Voice Panel application, which it will then display on the main view. To do this you need to set up an automation that publishes a formatted MQTT message on an interval. Then in the application settings, enable the weather feature. Here is a sample automation that uses Dark Sky data to publish an MQTT message:

- id: '1538595661244'
  alias: MQTT Weather
  trigger:
  - minutes: '/15'
    platform: time
  condition: []
  action:
  - data:
      payload_template: '{''weather'':{''summary'':''{{states(''sensor.dark_sky_summary'')}}'',''precipitation'':''{{states(''sensor.dark_sky_precip_probability'')}}'',''icon'':''{{states(''sensor.dark_sky_icon'')}}'',''temperature'':''{{states(''sensor.dark_sky_apparent_temperature'')}}'',''units'':''{{states.sensor.dark_sky_apparent_temperature.attributes.unit_of_measurement}}''}}'
      topic: voicepanel/command
      retain: true
    service: mqtt.publish

The resulting payload will look like this:

{"topic": "voicepanel/command","payload":"{'weather':{'summary':'Partly Cloudy','precipitation':'0','icon':'partly-cloudy-day','temperature':'22.5','units':'°C'}}

MQTT Day/Night Mode

Similar to how weather works, you can control the Voice Panel to display the day or night mode by sending a formatted MQTT message with the sun’s position (above or below the horizon). To do this add the sun component to Home Assistant, then setup an automation to publish an MQTT message on an interval:

- id: '1539017708085'
  alias: MQTT Sun
  trigger:
  - minutes: '/5'
    platform: time
  condition: []
  action:
  - data:
      payload_template: '{''sun'':''{{states(''sun.sun'')}}''}'
      retain: true
      topic: voicepanel/command
    service: mqtt.publish

The resulting payload will look like this:

{
  "payload": "{'sun':'below_horizon'}",
  "topic": "alarmpanel/command"
}

You can also test this from the using the “mqtt.publish” service under the Home Assistant Developer Tools:

{
  "payload_template": "{'sun':'{{states('sun.sun') }}'}",
  "topic": "voicepanel/command"
}

If you wish, you can use an offset to change the day or night mode values or send a MQTT message at the desired time with “above_horizon” to show day mode or “below_horizon” to show night mode. If you wish to always be night, you need only send one MQTT message with “below_horizon” and the app will not switch back to day mode. Be sure to turn on the Day/Night mode under the Display settings in the application.

MQTT Sensor and State Data

If MQTT is properly configured, the application can publish data and states for various device sensors, camera detections, and application states. Each device requires a unique base topic which you set in the MQTT settings, with the default being“voicepanel”. This distinguishes your device if you are running multiple devices.

Device Sensors

The application will post device sensor data per the API description and Sensor Reading Frequency. Currently device sensors for Pressure, Temperature, Light, and Battery Level are published.

Sensor Data

Sensor Data Published

Sensor values are constructed as JSON per the above table and Voice Panel publishes all sensors to MQTT under voicepanel/sensor. Each sensor publishes to a subtopic based on the type of sensor. An example topic would be voicepanel/sensor/battery .

NOTE: Sensor values are device-specific; not all devices will publish all sensor values.

Home Assistant Examples

sensor:
  - platform: mqtt
    state_topic: "voicepanel/sensor/battery"
    name: "Alarm Panel Battery Level"
    unit_of_measurement: "%"
    value_template: '{{ value_json.value }}'
    
 - platform: mqtt
    state_topic: "voicepanel/sensor/temperature"
    name: "Voice Panel Temperature"
    unit_of_measurement: "°C"
    value_template: '{{ value_json.value }}'  - platform: mqtt
    state_topic: "voicepanel/sensor/light"
    name: "Alarm Panel Light Level"
    unit_of_measurement: "lx"
    value_template: '{{ value_json.value }}'
    
  - platform: mqtt
    state_topic: "voicepanel/sensor/magneticField"
    name: "Alarm Panel Magnetic Field"
    unit_of_measurement: "uT"
    value_template: '{{ value_json.value }}'  - platform: mqtt
    state_topic: "voicepanel/sensor/pressure"
    name: "Alarm Panel Pressure"
    unit_of_measurement: "hPa"
    value_template: '{{ value_json.value }}'

Motion, Face, and QR Codes Detection

In additional to device sensor data publishing, the application can also publish states for Motion Detection and Face Detection, as well as the data from QR Codes derived from the device camera.

Motion, Face, and QR Code Detection

Sensor values are constructed as JSON per the above table and Voice Panel publishes all sensors to MQTT under voicepanel/sensor. Each sensor publishes to a subtopic based on the type of sensor voicepanel/sensor/motion.

Home Assistant Examples

binary_sensor:
  - platform: mqtt
    state_topic: "voicepanel/sensor/motion"
    name: "Motion"
    payload_on: '{"value":true}'
    payload_off: '{"value":false}'
    device_class: motion 
    
binary_sensor:
  - platform: mqtt
    state_topic: "voicepanel/sensor/face"
    name: "Face Detected"
    payload_on: '{"value":true}'
    payload_off: '{"value":false}'
    device_class: motion 
  
sensor:
  - platform: mqtt
    state_topic: "voicepanel/sensor/qrcode"
    name: "QR Code"
    value_template: '{{ value_json.value }}'

Application State Data

The application can also publish state data about the application such as the current dashboard url loaded or the screen state.

Application States

State values are presented together as a JSON block {"currentUrl":"http://hasbian:8123/states","screenOn":true}.

Voice Panel publishes a state to topic with a default topic of voicepanel/state.

MQTT Commands

Interact and control the application and device remotely using either MQTT commands, including using your device as an announcer with Google Text-To-Speech. Each device required a unique base topic which you set in the MQTT settings. This distinguishes your device if you are running multiple devices.

Commands

The base topic can be changed in the application settings. Commands are constructed via valid JSON. It is possible to string multiple commands together {"clearCache":true, "relaunch":true}. Voice Panel subscribes to default topic voicepanel/comma with a JSON payload.

Google Text-To-Speach Command

You can send a command using either HTTP or MQTT to have the device speak a message using Google’s Text-To-Speech. Note that the device must be running Android Lollipop or above.

Example format for the message topic and payload:

{"topic":"voicepanel/command", "payload":"{'speak':'Hello!'}"}

MJPEG Video Streaming

Use the device camera as a live MJPEG stream. Just connect to the stream using the device IP address and endpoint. Be sure to turn on the camera streaming options in the settings and set the number of allowed streams and HTTP port number. Note that performance depends upon your device (older devices will be slow).

Example format for browser URL:

http://192.168.1.1:2971/camera/stream

Home Assistant Example:

camera:
  - platform: mjpeg
    mjpeg_url: http://192.168.1.1:2971/camera/stream
    name: Voice Panel Camera

Additional Resources

Snips Voice Platform
Home Assistant
Issues

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A version of this blog post originally appeared on Michael Ritchie’s GitHub Page, available here. To learn about Voice Panel and other android hacks from Michael, follow his GitHub.

IoT
Android
Smart Home Automation
Voice Assistant
How To

--

--

Michael Ritchie
Snips Blog

Written by Michael Ritchie

5 Followers
Writer for

Software developer focused on mobile and embedded microcontroller devices.

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams