How To Build a Candy Machine With Feelings

Published in

IBM watsonx Assistant

17 min readMar 18, 2016

Welcome to another episode of “Teaching Robots How To Love.”

If you only want the tutorial on how to build the candy machine, scroll down to the Step-by-Step Tutorial section.

In my previous post, I wrote a Python script combining IBM Watson’s Speech to Text with Watson AlchemyLanguage Sentiment Analysis to determine the sentiment of speech. At the end of the post, I mentioned how I’d like to further explore what it’d be like to interact with a a candy machine with a speech-sentiment interface. Specifically, I wanted to build a candy machine that could understand speech and dispense different types of candy based on the sentiment of the words. Well …

I present you the first prototype of the Watson Naughty-or-Nice Candy Machine!

Though we had exposed wires and handwritten notes all over the place, I’m actually very happy with how it turned out. This picture was shot right before the opening of IBM Interconnect 2016 in February. The candy machine was on display for four days from February 22nd to February 25th. I expected a lot to go wrong during the conference because the machine went straight from my living room to a busy expo floor. But it performed way above my expectations.

This is a wider shot of the entire setup.

If you were at the conference, here’s how you would have operated the candy machine.

You’d walk up to the microphone and press the blue button (Record) to start recording.
You’d then speak into the microphone.
And press the red button (Stop) when you were done talking.
This is what you’d see on the screen if you were nice:

or this if you were mean:

Note: The transcription of what you said is the small words below the buttons.

The machine would dispense sweet candy (sweet M&Ms) from the dispenser on the right if you said something positive (sentiment score > 0). The machine would dispense sour candy (sour Skittles) if you said something negative (sentiment score < 0). Nothing happened if you had said something neutral.

Okay, now that you have an idea of what it’s suppose to do, shall we start building? Yes, totally.

Step-by-Step Tutorial

Here’s an overview of what we’re doing.

We’re connecting a laptop running a Flask web application (served on localhost) to an Arduino board.
The laptop uses the Watson Speech to Text service for transcription and the Watson AlchemyLanguage sentiment analysis to score the sentiment of the words.
The laptop then uses serial communication to pass the sentiment info to the Arduino board via the USB port.
The Arduino board has a motor shield plugged on top of it. The shield is directly connected to the DC motors inside the candy dispensers.
Based on the sentiment info from the laptop, the Arduino code determines which candy dispenser to turn on by controlling the voltage on its IO pins.

Got it? Below are the steps in more detail.

Step 0: What You Need:

Code: All the code you need can be found in this repository. Go ahead and clone the repo and I’ll walk you through how to set everything up in the following steps. Do ‘git clone https://github.com/boxcarton/say-something-nice.git’
Credentials for the Watson Speech to Text service and the Watson AlchemyLanguage service. If you’ve never used them before, see my previous post on how to acquire the credentials. You need to get the credentials through Bluemix, but you’re not tied to Bluemix in any way afterwards. Note: Bluemix is IBM’s PaaS offering that let’s you deploy and manage your cloud applications.
Two Sharper Image Candy Dispensers: I got the dispensers on sale from Bonton for $20 each. Note: This candy dispenser is unequivocally a one-star product — to directly quote a review from Amazon, “the mechanism dispenses between 0 and 35 pieces at random.” But don’t be discouraged, the inconsistency comes from the batteries’ inability to source enough current for the DC motors. We’ll be powering the motors (through the Arduino board) using an 12V-1.5A power adapter, which solves this problem. Another Note: It seems like this dispenser has been sold out from Bonton, you can still get one from Amazon for $36.99 each. The slightly cheaper version should also work for this project.
Power Adapter: I found a 12V-1.5A power adapter under my bed. If you don’t have such superpowers, you can buy one like this on Amazon.
Arduino Uno: I bought this Starter Kit from Vilros, but I’m pretty sure any Arduino board will do fine here. If you don’t get the kit, make sure to get the jumper wires (see #11), they’ll come in pretty handy.
Motor Shield: I bought the Adafruit Motor/Stepper/Servo Shield. You can also buy it directly from Adafruit, but shipping is faster with Amazon. If you’re wondering why you need a motor shield, see this Quora post. TL;DR: More current for the motors and protects the Arduino from transient circuit behaviors.
Extra Header Pins: You’ll be soldering header pins to the motor shield so you can plug it onto the Arduino. The Adafruit Motor Shield comes with just enough header pins, which is a problem if you screw up the soldering and need extra pins.
Soldering Station: This is the soldering station I bought. It gets the job done. But if you’ll be soldering often, you’ll probably want something of higher quality. I recommend the Weller WESD51 Digital Soldering Station if that’s the case.
Solder: This is the one I bought. But I found it to be too thick for the header pins we’re working with. Try getting something with a smaller diameter if you can, maybe something like this.
Solder Wick (Optional): Mandatory if you’re a soldering newb like me.
Jumper Wires (Optional): Having these does make it easier to connect the motor shield to the candy machine. But if you already have copper wires, they should work fine.
Noise-Cancelling Microphone (Optional): I got this one on Amazon after doing some research. It got the job done on the expo floor, but you can probably do better with more money. A noise-canceling microphone is only optional if you’re running this at home. It’s absolutely mandatory if you plan on having this in a noisy area.
Extra Long Audio Stereo Cable (Optional): The Rode Video microphone above comes with an incredibly short cable because it’s meant to be attached to a video camera. You’ll need a longer cable like this one if you plan on having the microphone more than a few inches away from the laptop.
Audio Port Splitter (Optional): Mandatory if you plan on using one of the newer Macbooks. In these newer laptops, Apple has combined the headphone and microphone ports into a single port. This is to go with the new Apple headphones that have combined headphone/microphone. You’ll notice that the the new Apple headphone plugs have three rings while most audio plugs still have two. This splitter will split the 3-ring port on your Macbook into normal headphone and microphone ports. Important Note: For the whole setup to work, you need to perform three steps in this order. 1. Plug a pair of normal headphones into the headphone port. 2. Plug the microphone into the microphone port. 3. Once both are plugged in, THEN plug the splitter into your laptop. This is the only way for your laptop to detect the external microphone.

Note: Phew, that was a long shopping list. I hope you weren’t discouraged by that because the hard part is now over. Putting everything together is actually not very technically challenging.

Okay, now that you’ve got everything you need, it’s time to hack some stuff together.

Step 1: Flask App and .env file

Let’s create your .env file first. I hope you’ve already gotten your credentials from both the Watson Speech to Text Service and the Watson AlchemyLanguage service. Refer to my previous post for details on how to do that.

Note: I use python-dotenv to manage my credentials. You need to add a .env file to your folder, similar to the one below. Here’s what my .env file looks like:

STT_USERNAME=<Your Watson Speech to Text Username>
STT_PASSWORD=<Your Watson Speech to Text Password>
ALCHEMY_API_KEY=<Your Alchemy API Key>

The user name and password you get from the Watson Speech to Text service goes under “STT_USERNAME” and “STT_PASSWORD” respectively. Your Watson AlchemyLanguage API key goes under ALCHEMY_API_KEY, naturally. Again, my previous post has more details. Note: Also related to my previous post, you’ll notice there’s a speech_sentiment_local.py in the repository folder. It is the Python speech to text to sentiment script I built in my previous post. This script will save your speech into speech.wav and transcribe the audio file over HTTP instead of doing speech transcription via WebSocket in the front end.

Now let’s move on to the actual Flask app. The entire Flask app lives in server.py. It is a simple server that:

Enables logging
Loads credentials and creates the necessary objects using Watson Developer Cloud Python SDK.
Sets up serial communication to Arduino through the USB port.
Creates the necessary endpoints.

Let’s go through these functions one by one:

Enable logging to both console and a local flat file (candy.log):

logger = logging.getLogger(‘candy_logger’)
logger.setLevel(logging.DEBUG)fh = logging.FileHandler(‘candy.log’)
fh.setLevel(logging.INFO)ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)formatter = logging.Formatter(‘%(asctime)s — %(name)s — %(levelname)s — %(message)s’)
ch.setFormatter(formatter)
fh.setFormatter(formatter)logger.addHandler(ch)
logger.addHandler(fh)

2. Load the .env file to get the credentials, then create the Watson auth object and the AlchemyLanguage object.

load_dotenv(os.path.join(os.path.dirname(__file__), “.env”))
auth = WatsonAuthorization(
   username=os.environ.get(“STT_USERNAME”),
   password=os.environ.get(“STT_PASSWORD”)
)
alchemy = AlchemyLanguage(api_key=os.environ.get(“ALCHEMY_API_KEY”))

3. Set up connection to local USB port. Your laptop will communicate with the Arduino board serially using PySerial.

has_arduino = False # Stays False if user runs “python server.py”
if len(sys.argv) > 1 and sys.argv[1] == ‘arduino’:
    has_arduino = True #True if user runs “python server.py arduino”if has_arduino:
   # configure the serial connections 
   # (Parameters differ depending on the device being connected)
   ser = serial.Serial(
       port=’/dev/tty.usbmodem1421',
       baudrate=9600,
       parity=serial.PARITY_ODD,
       stopbits=serial.STOPBITS_TWO,
       bytesize=serial.SEVENBITS
   )
   ser.isOpen()
   ser.flush()

4. Create the Flask app. Then create one endpoint to serve index.html and another endpoint to provide the auth token for the front end. The auth token is for communicating with the Watson Speech to Text API, which is performed on the front end (see next step).

app = Flask(__name__, static_url_path=”/static”, static_folder=”static”)@app.route(“/”)
def index():
   return app.send_static_file(“index.html”)@app.route(“/token”)
def getToken():
   return auth.get_token(url=SpeechToText.default_url)

The last endpoint determines the sentiment of the text returned by the speech to text API.

@app.route(“/sentiment”, methods=[“POST”])
def getSentiment():
    global has_arduino
    text = request.form[“transcript”]
    result = alchemy.sentiment(text=text)
    sentiment = result[“docSentiment”][“type”]

And then tells the Arduino whether the sentiment was positive or negative using the characters “p” and “n”.

if sentiment == “neutral”:
    score = 0
else:
    score = result[“docSentiment”][“score”]
    if has_arduino:
        if score != 0:
            if float(score) > 0:
                ser.write(‘p’)
            else:
                ser.write(‘n’)
            ser.flush()

Step 2: Front End Using Watson Speech to Text Browser SDK

The front end is in the /static folder and basically consists of index.html, basscss.css and client.js.

Index.html and basscss.css are self-explanatory — they create this page with the two control buttons.

client.js uses the Watson Speech Javascript SDK to communicate with the Watson Speech to Text service via WebSocket.

function listen() {
    clearTimeout(timeout)    if(!sttStream) {
        sttStream = WatsonSpeech.SpeechToText.recognizeMicrophone({
            token: sttToken,
            model: model,
            objectMode: true,
        })
        sttStream.on(‘data’, onData)
    }
    
    $recordButton.disabled = true
    $stopButton.disabled = false
}

The transcribed text is then displayed on the screen:

function onData(data) {
  if(!isStopped) {
    $results.textContent = arguments[0].alternatives[0].transcript
  }
}

and also sent to the back end for sentiment analysis:

function sentimentAnalysis(transcript) {
    if(!transcript) {
        return reset()
    }    var xhr = new XMLHttpRequest()
    xhr.addEventListener(‘load’, function(evt) {
        sentiment = JSON.parse(evt.target.responseText).sentiment        if(sentiment === ‘positive’) {
            reset(‘rgba(164,198,57,0.25)’, true)
        } else if(sentiment === ‘negative’) {
            reset(‘rgba(255,76,76,0.25)’, true)
        } else {
            reset(‘’, true)
        }
        timeout = setTimeout(reset, 3000)
    })    var formData = new FormData()
    formData.append(‘transcript’, transcript)    xhr.open(‘POST’, ‘/sentiment’)
    xhr.send(formData)
}

At this point, you should test this part of the project. Try running

python server.py

and you should have the server running on http://localhost:5000.

Later on, when you connect the Arduino board, you can run

python server.py arduino

to actually have the script try to talk to the Arduino board.

Note: when you run the server.py with an Arduino, a likely error that comes up is this.

OSError: [Errno 2] No such file or directory: ‘/dev/tty.usbmodem1421’

This occurs because PySerial can’t find the USB file that lets you establish serial communication. This file is created by OSX in /dev when you plug something into the USB port. You can fix this by going to line 123 of the code and change it to the usb file that your OSX has assigned you after you’ve plugged in your Arduino. You can find the right USB file by looking in the /dev folder and search for a file that starts with tty.usb*. If you’re too lazy to do even that, opening up the terminal and running ls -1 /dev/tty.* or ls -l /dev/tty.usb* should give you the file.

Edit: The code has been updated to automatically detect the USB port.

Step 3: Code for Arduino

The last piece of code you need to get is candy_arduino.ino, which lives in the /arduino folder. You then need to upload this code unto the Arduino board.

If this is the first time you’re using an Arduino, you should check out the official Getting Started with Arduino page. Make sure to familiarize yourself with the board and download the Arduino Software IDE.

Once you have the Arduino IDE opened, connect the Arduino board to your USB port. The first thing you need to do in the IDE is to make sure you have the correct board and port selected.

Then you need to install the library for the Adafruit Motor Shield. This guide should get you there. Take this opportunity to explore the IDE and maybe do some simple tutorials that’s on the Adafruit motor shield home page.

Finally, simply copy paste candy_arduino.ino into the IDE and hit the “upload” button. Your IDE should tell you that you’ve successfully compiled and uploaded your code.

But of course since you’re a good developer, you’re not satisfied with simply having a functional prototype. You want to understand how it works. So let’s dive into the Arduino code a little bit. Note: I’ve removed some of the comments in the code for brevity.

The first part of the code includes the necessary libraries

#include <Wire.h>
#include <Adafruit_MotorShield.h>
#include “utility/Adafruit_MS_PWMServoDriver.h”

and creates the objects to manipulate the motors. Notice how we’re grabbing two motors, one for each candy dispenser. Notes: each Adafruit motor shield can control up to four motors. These shields are also stackable.

Adafruit_MotorShield AFMS = Adafruit_MotorShield();// Select which ‘port’ M1, M2, M3 or M4. In this case, M1
Adafruit_DCMotor *motor1 = AFMS.getMotor(1);
Adafruit_DCMotor *motor2 = AFMS.getMotor(2);

Then comes the setup. By convention, this is done in the setup() function. We initialize the serial port at the baud rate of 9600 bps to start receiving data from the laptop. We also initialize the motors with max speed (255) and direction (FORWARD).

Note: “FORWARD” and “REVERSE” directions are arbitrary. If it’s not the direction you want, simply swap the two wires connected to the motor.

void setup() {
    Serial.begin(9600); // set up Serial library at 9600 bps
    Serial.println(“Welcome To The Polite Candy Machine!”);    AFMS.begin(); // create with the default frequency 1.6KHz
 
    // Set the speed to start, from 0 (off) to 255 (max speed)
    motor1->setSpeed(255);
    motor1->run(FORWARD);
    motor1->run(RELEASE);   // Set the speed to start, from 0 (off) to 255 (max speed)
   motor2->setSpeed(255);
   motor2->run(FORWARD);
   motor2->run(RELEASE);
}

Then comes the main function. Also by convention, loop() is where you put the code that runs forever. In this case, we’re encapsulating everything in a while loop that’s constantly looking for new strings (“p” or “n”) in the serial port.

while (Serial.available() > 0) {
    data = Serial.readString();
    data = data[0];
    if (data == “p”) {
        Serial.println(data);
        motor1->run(FORWARD);
        delay(500);
        motor1->run(RELEASE);
    } else {
        Serial.println(data);
        motor2->run(FORWARD);
        delay(500);
        motor2->run(RELEASE);
    }
}

If the sentiment was positive, we get the character “p”. We then run the motor (FORWARD), wait for 500 milliseconds, then stop the motor (RELEASE). We run the other motor (motor2) if it’s not “p”.

Three things to note here:

By adjusting the amount of delay, we can control the amount of candy dispensed since the longer the motor runs, the more candy the dispenser dispenses.
We have

data = Serial.readString();
data = data[0];

because Serial.readString() gets “p” or “n” along with any possible newline character. data = data[0] throws everything else away except the character we care about.

3. You can find more information about the Adafruit Motor Shield Library here.

That’s all for the software! I hope you’re ready to roll up your sleeves and say goodbye to the virtual world.

Step 4: Motor Shield

The first thing you should do is solder the header pins onto the Adafruit motor shield. Conveniently for me, Adafruit has made a very nice guide on how to do that.

If you’ve never soldered before, here’s a good guide on making good solder joints. The main thing to watch out for is to not have the solder touch each other.

Once the pins are soldered on, you should be able to plug the motor shield on top of the Arduino. To make sure you’ve soldered all the pins correctly, power up the Arduino and the motor shield’s power light should go on as well.

Important Note: The Arduino board can draw power either from the USB port OR the DC Barrel jack. If you plug something into both the power adapter and the USB port, your Arduino should automatically select the external power source from the DC Barrel jack. To have the motor shield also draw from that external power, place a VIN jumper on the double terminal block next to the green Power LED.

Step 5: Candy Dispensers

Next, let’s rewire the candy dispensers and expose the connection to the DC motors. This is actually quite simple with this specific candy dispenser.

The first thing we need to do is open up the candy dispensers. If you turn the machine upside-down, you’ll see 6 screws. Unscrew all of them.

2. Now find the yellow PCB board with three white connectors plugged in. Unplug the DC motor connection. So far, every single dispenser I’ve opened has had the DC motor connection on the very right with yellow-red wires. Since it’s very easy to trace the wires to from the motor, you should do that just to make sure you’re unplugging the right set of wires.

3. Next, unscrew the small green PCB board at the back of the dispenser. This is where you would normally plug in the power adapter. Once you have the board unscrewed, shove the unplugged DC motor connector through the hole into the outside world.

4. Now you need to screw the PCB board back in. Since you now have the motor wires there, it’s gonna be a little tight. But this is good because you now have the added benefit of keeping the wires in place.

5. You’re pretty much done! With the DC motor connection exposed, you can now use the jumper wires to connect the motor to the motor shield.

Note: The dispenser only dispenses candy if you run the motor in the right direction. The right direction is easy to figure out — if you hear the motor run a few times but no candy comes out, just switch the two jumper wires connected to the motor. Duh. :P

6. Now repeat these steps for the other candy dispenser.

Step 6: Putting It All Together

We’re now ready to put the whole thing together! But let’s do it step by step so we can debug any problems.

You should have tested the web application part of the machine after step 3. Your browser should show you a streaming transcription of your speech as you speak. If you give a brief pause, the browser should turn green or red depending on the sentiment of your words (unless it’s neutral).
To test the Arduino portion, you can connect the candy dispensers to the motor shields, connect the Arduino to your laptop, open the Arduino IDE, and navigate to Tools -> Serial Monitor.

You should now be able to input strings into the Arduino. If you already have candy_arduino.ino uploaded, entering “p” into the serial stream should turn on one of the candy dispensers, while any other character should turn on the other candy dispenser.

3. Now you can finally connect the Arduino board to your laptop via USB. Navigate to your project folder can run:

python server.py arduino

Ok everyone, it’s the moment of truth. Open your browser at http://localhost:5000, and start speaking to your laptop/microphone. If you’ve done everything correctly up until this point, you should have a candy machine that reacts the emotion of your speech!

What’s Next:

So we had a great showing at Interconnect 2016 as we were quite popular with the attendees.

But this is not the end. Due to the machine’s popularity, we plan to keep working on this. Here’s my plan that hopefully nobody will hold me accountable to.

Replace the laptop with a RaspberryPi.
A better user interface via iPad or iPhone. Mouse is so basic.
Give a boost to my candy machine’s EQ by using additional Watson service such as Watson Emotional Analysis and Watson Tone Analyzer.
Add text to speech capability so there will be a more complex and engaging interaction.
More polish in general so the whole thing isn’t duct taped together.
Add death ray on top of each dispenser.

So follow me here on Medium if you’re interested in seeing where this candy machine goes.

Final Thoughts

So I have a theory on why people liked the candy machine. I believe the interaction is surprisingly engaging because candy has always been associated with reward and overall positive feelings. Because of this, we all have fond memories of candy machines growing up. When you say something nice or mean to a candy machine, it feels awkward because you’re investing a bit of your own emotion into an interaction with a soulless object. But then the machine actually reacts and dispenses candy back to you, and it feels like your feeling is reciprocated. This exchange makes the entire interaction seem almost human, and ultimately more engaging.

This theory seemed like a stretch at first, but seeing the people’s reactions has affirmed my belief. Many people kept on talking to the candy machine to see how it’d react to different elements of human speech, such as slangs and sarcasm. As I further develop the candy machine, I believe I can create an even more engaging experience.

Finally…

I need your help. At this point, I think the candy machine deserves a name. The first name that came to mind, was of course, Kandy. But I’d like to think that my candy machine actually went to college, and would be classy enough to throw a dinner party. So if you have a better suggestion, please leave me a comment below. Thanks!

And speaking of gratitude…

Special Thanks

This wouldn’t have happened without the help of two of my awesome colleagues. Nathan Friedly (http://www.nfriedly.com), who helped guide me through many foreign territories in hardware and Yacine Rezgui (http://www.yrezgui.com), who built the front end of the software.

Also shoutout to my teammate Ashley Hathaway for proofreading this ridiculously long tutorial.

I hope you enjoyed this article. If you have any questions feel free to reach out at joshzheng@us.ibm.com, connect with me on LinkedIn, or follow me here on Medium.