Firefox Voice intents workshop

Volunteer help to add new voice-control commands for Firefox

Irvin Chen
Mar 15, 2020 · 3 min read
People are trying to use TTS from another laptop to trigger the intent they’re making.

Firefox Voice is an experimental add-on adding voice-control for Firefox. You can play with it now by installing the add-on on the desktop version of Firefox from the below page.

During the Mozilla Berlin All-Hands, we had some discussion with the engineers about if a community volunteer can help to build intents (voice commands) for Firefox Voice. We were wondering if this topic will be interesting for volunteers to work on, and decided to run a pilot workshop at Taipei Community Space.

As a branch event of our regular browser add-on workshop, on Feb 29, five people joined the very first Firefox Voice intents workshop with us from 2 to 6 pm.

The combination of attendees were one experienced add-on developer and evangelism to lead the workshop, one people who come to the add-on workshop often, and two new people who is able to write JS without any add-on dev experience.

In the end, we have came up with two PRs (1128 & 1129), and here are the problems we had, during the workshop:

Set-up the extension developing environment

It took us 2 hrs for everyone to fully set up their web extension developing environment (on Two Mac, one Windows and one Linux). The problems we bumped-in includes:

  1. Need to install node/npm.
  2. A broken git on Win laptop.
  3. Need to download and install Firefox nightly.
  4. Windows laptop unable to use npm run start to pack the source and run with nightly (resolved by download the pre-packed xpi file, extracted and replaced the files in the repo, and run with web-ext run.
  5. npm run start is unable to find Nightly on Linux, because it’s named firefox-trunk instead of firefox-nightly, resolved by run web-ext run --firefox=firefox-trunk.
  6. On Mac, the mic privilege requesting dialog (for iTerm) showed up pretty late, cause some privilege problem.

Problems people had when working on the intents

While working on intents, these are the problems we had:

  1. The voice command panel is not working properly on the current master branch.
  2. Intent matching is quite tricky. It’s hard to know if the command is handled by the new intent we’re working on, or get caught by the current intents. Eg., the intent (about tab) I tried to implement was always caught by find.find, whatever the different words I try to use.
  3. The good-new-bugs the mentor had prepared to use as the examples had been patched before the workshop took place.
  4. The biggest problem — the current STT doesn’t recognize our tones. We will need to retry like more than ten times before the STT can output the sentences we would like to say.

We eventually use another laptop’s TTS, to vocalize the sentences we like to test. After the workshop, one of us eventually found we can, in fact, typing the sentences instead of trying to say it again and again.

Siri and Google come to rescue.

People did have lots of fun and positive feedback for the workshop. If you want to find an interesting topic to hack with the community member together, check some good-first-bugs for Firefox Voice now.

Mozilla related

Irvin's stories about Mozilla

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store