Talk the Talk and Let Pi Do the Walk: A Guide to Voice-Controlling Your Raspberry Pi with your Lolo-App!
Welcome to this tutorial on building a Lolo app that enables your Raspberry Pi to take voice commands!
With this technology, you can create a variety of interesting and useful projects, such as:
- Home automation: Use voice commands to control your smart home devices, such as lights, thermostat, and security cameras.
- Media center: Set up your Raspberry Pi as a media center and use voice commands to play music, movies, and TV shows.
- Smart mirror: Use your Raspberry Pi to build a smart mirror that displays information like the time, weather, news, and calendar events, all controlled by voice commands. This would be great together with Gesture recognition: As we explained in a previous tutorial — [Thumbs up if you’re ready to take your Raspberry Pi projects to the next level]
- Basic navigation: You can use voice commands to control the basic movement of your robot. For example, say “forward” to make your it go forward, “backward” to make it go back, “right” to make it turn right, and “left” to make it turn left. In this tutorial, we’ll build a simple demonstration of this capability using this voice recognition system.
You’ll learn how to train voice recognition software, program the voice commands, test and refine the commands, and add more functionality as you go. By the end of this tutorial, you’ll have a solid foundation for building a variety of voice-controlled projects with your Raspberry Pi.
So let’s start with some basic directional commands for your Raspberry Pi.
Just as previous post we will need some stuff so be sure you have the following:
- Lolo account (API-key)
- Edge Impulse account (API-key)
- Google account for use with colab
- A Raspberry Pi
- A microphone connected to the Raspberry Pi (USB Mic)
- Four LED diodes (Different colors)
- Four 100–330 Ohm Resistors
First head over to Edge-Impulse and make a new project copy the new API-key.
Second, we need to create a Google Colab notebook to download the dataset and add it to Edge Impulse. Copy and paste the provided code below into the notebook, but before running it, ensure that you have created a new Edge Impulse project for the dataset. Also, replace the placeholders for the Edge Impulse API keys with your own unique keys. Once you have done this, you can run the code and monitor the progress as the samples are added to your Edge Impulse project.
import os
import io
import random
import shutil
import librosa
import soundfile as sf
import numpy as np
import glob
import requests
import zipfile
import subprocess
api_key = 'ei_e18xxxxxxxxx' # Your Edge-Impulse API-key goes here
# Remove directories recursively if needed
shutil.rmtree('./data', ignore_errors=True)
shutil.rmtree('./edgeimpulse_dataset', ignore_errors=True)
shutil.rmtree('./edgeimpulse_output_data', ignore_errors=True)
# Download and extract dataset
print('Downloading and extracting speech dataset...')
url = 'http://download.tensorflow.org/data/speech_commands_v0.02.tar.gz'
response = requests.get(url)
with open('speech_commands_v0.02.tar.gz', 'wb') as f:
f.write(response.content)
os.makedirs('./data')
subprocess.run(['tar', 'xvf', 'speech_commands_v0.02.tar.gz', '-C', './data'])
# Define variables
num_samples = 1500
output_dir = './edgeimpulse_output_data'
# Words in the dataset
words = ['left', 'right', 'forward', 'backward', 'noise', 'unknown']
# Download noise and unknown sound from Edge Impulse dataset.
print('Downloading Edge Impulse dataset...')
url = 'https://cdn.edgeimpulse.com/datasets/keywords2.zip'
response = requests.get(url)
z = zipfile.ZipFile(io.BytesIO(response.content))
z.extractall('./edgeimpulse_dataset')
shutil.copytree('./edgeimpulse_dataset/noise', './data/noise')
shutil.copytree('./edgeimpulse_dataset/unknown', './data/unknown')
# Create output directories
print('Creating output directories...')
for word in words:
os.makedirs(f'{output_dir}/{word}')
# Process audio files
print('Processing audio files...')
for word in words:
file_list = glob.glob(f'./data/{word}/*.wav')
random.shuffle(file_list)
for i in range(num_samples):
src = file_list[i]
dest = f'{output_dir}/{word}/{word}.{os.path.basename(src)}'
s, sr = librosa.load(src, sr=16000, mono=True)
if len(s) < 16000:
s = np.append(s, np.zeros(16000 - len(s)))
sf.write(dest, s, 16000)
else:
shutil.copyfile(src, dest)
# Print number of samples for each word
num_samples_word = len(os.listdir(f'{output_dir}/{word}'))
print(f"Processed {num_samples_word} samples for {word}.")
# Install Edge Impulse CLI
print('Installing Edge Impulse CLI...')
subprocess.run(['npm', 'install', '-g', '--unsafe-perm', 'edge-impulse-cli'])
print('Edge Impulse CLI installed.')
# Upload samples to Edge Impulse
print('Uploading samples to Edge Impulse...')
for word in words:
sample_dir = f'{output_dir}/{word}/*.wav'
cmd = ['edge-impulse-uploader', '--api-key', api_key, '--label', word, sample_dir]
subprocess.run(cmd)
print("Done!")To train our model, we’ll be using the default settings provided by Edge Impulse. Simply follow these steps:
To create an impulse and train the model, follow the default settings for creating the impulse, MFCC, and classifier. Starting the training process using the default settings provided by Edge Impulse is usually sufficient to achieve great results. However, if your model needs further refinement, adding as few as 30 recordings of your own voice can significantly improve its accuracy. This simple step can be completed with ease in Edge Impulse. Once you have done this, you can test the model and experiment with various settings to further enhance your scores.
Well done!
Now let’s create a new Lolo-app and add the Edge Impulse node, four GPIO output nodes (like the ones marked with green nodes in the example), and two new functions (like the ones marked with red in the example). Next, we’ll add some logic so we can test if the model will follow our directions.
Start by adding the Edge-Impulse API-key to your Egdge-Impulse node by by double-click on your newly created Edge Runner function. From there, select ‘Params’ and proceed to add a new authorization by including your Edge Impulse API key.
Next we’ll create a nested object and add it to the app’s state so we can pass it through the pipeline of nodes. We’ll need to set some thresholds for the classification (which you may have to adjust based on your voice), as well as color codes and other settings.
Here is the code for the new function called we named “Evaluation and Threshold”
Just dubble click the node and add the code:
// The export handler will run every time the event is triggered
exports.handler = async(ev, ctx) => {
const { route, log, state} = ctx;
const { classification } = ev;
// If the classification for this direction is above the threshold, set its 'is' property to true
// and set the 'is' property of the opposite direction to false.
for (const key in ledState) {
if (classification[key] > ledState[key].threshold) {
ledState[key].is = true;
ledState[ledState[key].opposite].is = false;
log.info(`${key.toUpperCase()} ${ledState[key].color} ${JSON.stringify(classification[key])}`);
}
}
// Save the updated LED state to the app state
state.set('ledState', ledState);
// Route event data to the default output port
route(ev);
};
// The setup runs when the application is initialized
exports.setup = async ctx => {
const { } = ctx;
};
// Define the initial state of the LEDs and set the threshold
// Adjust the threshold value to optimize the LED response to your voice
// The threshold determines the level at which the LEDs should activate in response to sound.
// Decrease the threshold if the LEDs are triggering too easily,
// or increase it if they are not responding to your voice.
const ledState = {
forward: {
is: false, // Indicates whether the LED for this direction is currently on or off
threshold: 0.7, // The minimum classification score required for the LED to turn on
opposite: 'backward',
color: 'green'
},
backward: {
is: false,
threshold: 0.4,
opposite: 'forward',
color: 'red'
},
left: {
is: false,
threshold: 0.9,
opposite: 'right',
color: 'yellow'
},
right: {
is: false,
threshold: 0.3,
opposite: 'left',
color: 'blue'
}
};Now lets get the new state object in next node we called “updateLEDs” so we can pass it to the GPIO Out:
// The export handler will run every time the event is triggered
exports.handler = async (ev, ctx) => {
const { route, state } = ctx;
// Get the current LED state from the app state
let ledState = state.get('ledState');
// Loop through each direction in the ledState object
for (const key in ledState) {
// If the LED for this direction is currently on, update the event object
if (ledState[key].is) {
// Set the color of the LED for this direction to 1 (on)
ev[ledState[key].color] = 1;
// Set the opposite direction of movement to 0 (off)
ev[ledState[key].opposite] = 0;
}
}
// Route event data to the default output port
route(ev);
};
// The setup runs when the application is initialized
exports.setup = async ctx => {
const { } = ctx;
};Connect the default output of the “updateLEDs” node to all four GPIO Output nodes. And make sure to properly configure the GPIO pins and specify the corresponding color that the GPIO is controlling, such as {event.red} or the appropriate color value.
Now let’s test the Lolo-app by clicking the green run button and heading over to the Debug tab. Check if the right color is connected to the right pin and voice command. You should see the command you said being printed and the appropriate LED lighting up based on the command. This will ensure that the system is set up correctly and responding appropriately.
If everything is set up correctly, the Debug tab should print something similar to this, and the LED lights should switch from forward to backward and right to left when you command it.
And finally, we are using the same wiring as demonstrated in the previous tutorial, which you can access by clicking here → [ Thumbs up if you’re ready to take your Raspberry Pi projects to the next level ]
Well done, fast and easy!
Originally published at https://medium.com on July 4, 2023.
