Open in app

Sign in

Write

Sign in

Snag’s FoodCam using a Raspberry Pi, Slack, & Facial Recognition

John Heng
Snagajob Engineering
Published in
12 min readJul 19, 2019

--

What problem does this solve?

As a person that hates waste but loves free food, I often see something happen: leftover food is trashed when it could instead be shared. Whether it’s leftover wings from last weekend or some kale for the juice cleanse you swear you were going to start, food just goes bad and is tossed out. It’s the same for an office environment. Food is catered for a big team meeting but not everyone is feeling sandwiches (or whatever was catered that day). The food sits around and is eventually thrown away. So, that’s where the FoodCam comes in!

What is the FoodCam?

As an organization, we have monthly hackdays where engineers are given a day to dabble in new or exciting things outside of their usual work.

The Snag FoodCam sends a picture of food to Slack (#freefood) at the press of a button. The camera was spun off MIT’s media lab FoodCam. After seeing MIT’s FoodCam, I thought it’d be within the realm of possibility for us here at Snag.

Press a button, take a picture of the food, send it to Slack. Easy. And if you thought it needed to be over engineered, it now also takes a picture of the person leaving the food and tries to recognize their face!

Final product featuring a picture of the leftover food, a message alerting people, and a picture of the person(s) leaving the food.

So let’s get to building the camera

The question I asked myself was “what do we need to power all of this?” There’s probably more than one way to skin this cat but I went with a Raspberry Pi 3 B+. To get everything wired up and working together, I had to get a few other things as well.

Raspberry Pi 3 B+

Full item list:

I went the extra mile and built a box using a laser cutter and mounted it to a wall in our office kitchen. More on that later.

Programming the box

Set up the Raspberry Pi with NOOBS

Once the Raspberry Pi is set up, plug in the camera module and enable it through the console

  1. sudo raspi-config
  2. When running that command, a window will pop-up where you’ll navigate to Interfacing Options > Camera to enable it
  3. Save your changes by selecting Finish
  4. If you did that successfully, running raspistill -o freefood.jpeg from the command line should output an image that you can see.

Setting up the Slack app

Next, you’ll have to set up a Slack app that can post these pictures on your behalf. You can create a Slack app here: https://api.slack.com/apps. What we want is an OAuth token to talk to Slack with.

  • To find your OAuth token, go to your app’s Basic Information and OAuth & Permissions should be on the sidebar

Creating the Python script

In this step, we’ll create a short Python script that takes a picture and sends it to Slack when we call it. I named the script free-food.py. In a later step, we’ll have a button press call this script.

We’ll need to import two libraries, one to have Python makes calls to the command line and another for making HTTP requests

import subprocess
import requests

Next, we’ll use that camera command from earlier to take a picture. Since we’re doing it from Python, we’ll create a string for the command then call it from our subprocess library.

cmd = "raspistill -t 1 -vf -o freefood.jpeg"
subprocess.call(cmd, shell=True)

-t 1 sets the camera’s timer to 1 second. By default, it’s 5 seconds. -vf vertically flips the image.

There are other parameters you can use to finetune the image some more https://www.raspberrypi.org/documentation/raspbian/applications/camera.md

Alright, so now that we’ve got our picture, we need to post it to Slack to notify people.

token = '{Slack OAuth Token}'
my_file = {
  'file': ('./freefood.jpeg', open('./freefood.jpeg', 'rb'), 'jpeg')
}
payload = {
  "filename": "freefood.jpeg",
  "token": token,
  "title": 'Free food sent from Snag FoodCam',
  "initial_comment": 'There is free food in Town Center! Come & get it! :hamburger: :runner:',
  "channels": ['#freefood']
}

Once we’ve set up our payload, all that’s left is to send it to Slack

r = requests.post("https://slack.com/api/files.upload",
                params=payload, files=my_file)

The result should be something similar to this

We started this in a #freefood-dev channel so that no one was asking where the food actually was.

Cool, so far we have set up the Raspberry Pi to use the camera, created a Slack app to integrate with, and created a Python script to send a picture through the app.

Wiring the Pi

Next up, we need to wire our ridiculously big button to the Pi so that people can actually put it to good use.

If you’re like me, this might be outside the realm of usual small projects. Totally fine, it’s not too hard and I’ve already made a lot of mistakes so you don’t have to. That being said, this project took more than one Pi because I broke the first one. At one point, I was so lost on how to wire this up that I considered getting an Amazon IoT dash button but thought that was overkill.

Let’s take a quick look at the actual switch itself. It’s comprised of an LED that sits on top of a switch (they’re separate in this product picture). The LED has negative and positive terminals while the switch has a power and a ground. If you don’t care about having the LED light up, you can definitely skip wiring that up.

To wire into these terminals, you’ll need quick connects like these which can be easily found at any auto parts shop (AutoZone, Pepboys, etc).

Using a jumper wire with at least one female end, we’ll use wire cutters to strip the non-female end off and splice it into the quick connects, crimping the collar of it down with a pair of needle nose pliers like in the image.

Now connect your wires to your switch. The bottom terminal of the switch is the ground and the side terminal is the power. There’s a diagram later if you want something more visual. I wired in the LED using trial and error to figure out which terminal was negative or positive. Okay, actually, I just guessed it right on the first try.

It should look like this. Disregard the fact that my image has male ends. I was originally using a breadboard but found that the breadboard was not doing anything and I swapped to female ends later on. Also maybe consider using resistors. At one point, I also had a resistor for the LED but found that it was super dim. I took the resistor out and it was as bright as I was originally expecting. This leads me to believe the LED probably has a resistor built in.

Okay, so we are finally ready to put this thing into the Raspberry Pi. On a Raspberry Pi, there are two rows of GPIO (general purpose I/O) pins. If you look up a diagram, you’ll find that each one does something specific. Some act as grounds, some are power, and some are I/O.

Here’s how I wired mine up.

Awesome. We’ve got our button and LED wired up. Now we can start listening for a button click.

I created a new script named startup.py.

What it does is pretty simple

  • Turn on the LED
  • Wait for a button click
  • If the button was clicked, run our previous free-food.py script
import RPi.GPIO as GPIO
import timeGPIO.setmode(GPIO.BCM)
GPIO.setup(23, GPIO.IN, pull_up_down=GPIO.PUD_UP)
GPIO.setup(4, GPIO.OUT)
GPIO.setwarnings(False)while True:
  input_state = GPIO.input(23)
  if input_state == False:
      cmd = "python freefood.py"
      subprocess.call(cmd, shell=True)
      time.sleep(0.2)

Notice in the diagram that I have my LED wired into pin #4 and I have a line in the script to turn pin #4 on. The same can be said about pin #23. I’m waiting for an input from the switch on that pin.

If you run this program using python startup.py, your button’s LED should turn on and when you press the button, you should expect to see a picture in your Slack channel. It works!

Run this script on startup

Now, say the Raspberry Pi loses power. When it regains power, you don’t want to run startup.py manually every time. What you can do is update your /etc/rc.local file and include your startup.py file in the bash script when your Pi turns on.

To do this, just run sudo nano /etc/rc.local to edit the file. Notice that I’m adding a line just to call my startup script. I’m also defining the absolute path so that it knows where my script is.

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.sudo python /home/pi/Documents/free-food/startup.py# Print the IP address
_IP=$(hostname -I) || true
if [ "$_IP" ]; then
printf "My IP address is %s\n" "$_IP"
fiexit 0

Save the file and reboot your Pi. You should expect to see the LED turn on automatically once it’s rebooted and clicking the button should send the message, all without having to manually run the script. You’ve got a script that automatically runs and you don’t have to plug in a screen or keyboard anymore!

Making the box

So I think that the making of the box is kind of out of the scope of this post but I figured some people might still be interested so I’ll gloss over it quickly.

In Richmond, there is a makerspace named HackRVA where they’ve got a laser-cutter. I used a website (https://www.festi.info/boxes.py/) to generate the cuts I’d need for a box. It was super easy to do once I measured the thickness of some scrap wood. On top of that, I added a hole for the button to screw into as well as a slot on the side for the Pi’s I/O and a hole for the camera.

Laser-cutting process and test-fitting the button. The logo I etched in wasn’t deep enough and is kind of faint now that the box has been sanded down.
All the different panels for both the box and the lid (for easy access/maintenance).
Making a box using wood glue takes a lot of clamps!

Once it was fully assembled, it was ready to be mounted to the wall with a lot of Command Strips and used by the office.

FoodCam mounted in Snag’s kitchen using Command Strips. Eventually I went with a piece of scrap wood bolted into the TV’s mounting plate because Command Strips on the textured wallpaper weren’t permanent enough.

Alright, so we’ve reached our minimum viable product, right? It’s functional, it does what it was intended to do, and it is in production. And yes, it works! But there were also corner cases that we weren’t thinking about.

What if someone decides to push the button because it’s a giant red button?

So now we’ve got another problem. We don’t know who pushed the button so we don’t know who to blame where there’s a false alarm or we don’t know who to thank when something delicious has been left.

Let’s iterate.

The rest of this post will be more pseudo-code now.

Taking a picture of the person and overlaying it onto the food

So this next part is really easy to do. All we need to do is find a USB webcam lying around the office and plug it into one of the USB ports on the Pi.

We’ll need two more libraries to do this.

sudo apt install fswebcam
sudo apt install imagemagick

Fswebcam is cool because it finds the webcam you have automatically and uses the webcam’s built-in software to take a picture for you. There are also additional parameters you can add to finetune the image. I added -S 20 to have the camera be turned on for 20 frames (about a second) before actually taking the picture. I noticed that if I didn’t do this, the picture would be completely white, as if the camera was still in the process of waking up.

I also fine tuned the raspistill command some more. I added -hf to horizontally flip the images because nobody wants to eat srevotfel.

Now we can imagine our workflow is something like this.

#take pic of user
fswebcam --no-banner -S 20 -r 640x480 user.jpg# take pic of food
raspistill -t 1 -vf -hf -o food.jpg#overlay pictures
convert food.jpg user.jpg -gravity southeast -geometry +10+10 -composite output.png# send image to slack
# slack API

That’s really easy to do. We’ve solved one of the big problems we set out to fix. We know who done it. But surely there’s a way to over-engineer this while we’re already here…

Adding facial recognition

Enter machine learning. I recently found a Python library for facial recognition (https://github.com/ageitgey/face_recognition) and wanted to include it in the FoodCam.

Given an image, the machine learning model can identify each face in the image.

import face_recognition
image = face_recognition.load_image_file("your_file.jpg")
face_locations = face_recognition.face_locations(image)

There is a full folder of examples in the Github repo. There was one named identify_and_draw_boxes_on_faces.py that I decided to use for our FoodCam.

Following the example, we load in an image for each coworker we want to be recognized. I used an array of image paths then passed it through the library using a for-loop. What is output is an encoding of each coworker’s face. Maybe think about it as the model’s way of putting a Python value to a face. I also create a second array for each coworker’s name. I make sure to keep the indexes of these the same because that is how we will be getting our names in the end.

images = [
    "john.jpg",
    “obama.jpg”
]known_face_encodings = []
for x in images:
    image = face_recognition.load_image_file(x)
    face_encoding = face_recognition.face_encodings(image)[0]
    known_face_encodings.append(face_encoding)names = [
    “John Heng”,
    “Barack Obama”
]

Now that we’ve gotten our Raspberry Pi familiar with some faces, we can load in the picture we took earlier so it can do some identification.

fswebcam --no-banner -S 20 -r 640x480 user.jpg

We process user.jpg and find the face in the image.

unknown_image = face_recognition.load_image_file("user.jpg")face_locations = face_recognition.face_locations(unknown_image)
face_encodings = face_recognition.face_encodings(unknown_image, face_locations)

Finally we attempt to match the unknown user encoding to a known user encoding.

matches = face_recognition.compare_faces(known_face_encodings, face_encoding)
face_distances = face_recognition.face_distance(known_face_encodings, face_encodings[0])
best_match_index = np.argmin(face_distances)
if matches[best_match_index]:
    name = known_face_names[best_match_index]

Of course, if anything is confusing, the full code for this will definitely be more helpful than this abbreviated version.

https://github.com/ageitgey/face_recognition/blob/master/examples/identify_and_draw_boxes_on_faces.py

Alright, so what we got out of that was a name. The example also has code for putting a box around the person’s face as well as the name in the box as well.

Now let’s update our HTTP request to include the person’s name in our Slack message.

payload = {
            "filename": "freefood.png",
            "token": token,
            "title": "Free food sent from Snag FoodCam",
            "initial_comment": name + " left some free food in Town Center! Come & get it! :hamburger: :runner:",
            "channels": ['#freefood']
        }

Conclusion

What we’ve got now is a camera that takes a picture of both food left underneath it as well as a picture of the person that left it. It does some processing and recognizes the person in the picture. Lastly, it overlays the two pictures together and sends it off to Slack for people to be notified.

This project has been really cool for me to work on. As a software engineer, I’m working in the nooks and crannies of a much bigger codebase. Here, I’m completely owning the entire process, from ideation to maintenance to enhancements. On another level, software engineering is usually just in code. In this case, I learned to do a little bit of wiring, a little bit of woodworking, and a little bit of Linux.

Thanks for reading!

Raspberry Pi
Slack
Facial Recognition
Hackday

--

--

John Heng
Snagajob Engineering

5 Followers
Writer for

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams