FoodAlert: The IoT/AI/GCP Bandit Detector you didn’t know you needed!
A story of 22hrs to win Slalom’s Hack the Office v2.0
In hindsight, you might say we *slightly* over-engineered our project… but what’s the fun in that when you challenge yourself to work with the latest in IoT and Google Cloud?
Our idea? Simple and silly… capture the Poptart bandit of our office¹.
The Problem
Our office is humbly supplied by our two favorite SlalomOps people (Thank you Jess and Corbin!). Amongst the supplies, there are different types of Poptarts. Poptarts are the most famous snack in the office, so much so that many of us replace our lunches with them… this means that demand is off the charts. Some bandits just open a pair of Poptarts, eat only one, and then put the open packet back. Some of them just horde the Poptarts for themselves under their desks. That results in a lot of us remaining Poptartless. We had to do something to get this whole situation under control!
Our opportunity to build a solution became obvious when we heard about the latest hackathon being held… Hack the Office v2.0!²
The Challenge
Ok, so we had found our opportunity (and excuse) to build a solution to this problem, using skills we (mostly) possess. The problem was, we had to build this within 22 hours. The Hackathon started at 2pm Thursday and ended 12pm Friday. Our benevolent organizers decided to start playing S1 of Stranger Things… either in an effort to scare us to complete faster, distract us from finishing, or really just to watch? 👀
Our team was comprised of a Solution Architect, a Solution Owner, and a Senior Engineer. Our expertise? Frontend and Backend development, Cloud and DevOps. You’ll notice a few skills we certainly did not have… namely IoT, hardware engineering, and even Google Cloud Platform (GCP). Well… we did know cloud (AWS!🧐 ), but in the spirit of a true hackathon, it was all about getting out of our comfort zones, learning something new, and having fun while being challenged.
Enter FoodAlert
Built entirely on GCP, FoodAlert is a series of serverless applications/services that automagically determine food left in the 51 pantry with IoT, identify the bandits who steal (Poptarts) with AI, and let people interactively query/view status of inventory (Web App, Voice/Chat Assistant Bot).
Over the course of the hackathon, we successfully built FoodAlert to observe, catch, and record Poptart bandits in action, safely saving many more for the greater Slalom XM Chicago office. Our initial plan was to 1) Determine the quantity of the remaining Poptarts with some kind of IoT device, and 2) Create an inventory application to monitor the Poptarts.
How could we achieve this? Place a force-sensitive sensor under the box of Poptarts, and create a simple dashboard to view the inventory. This way we would not have to walk all the way to the kitchen just to realize that the box was empty (and could avoid becoming extra disappointed as our thirst for sugar remained unsatisfied).
“I thought you said this was complicated?” you might be asking. Well, I’m glad you asked! Here is how we built FoodAlert… and kept building it even after the hackathon was over!
Phase 1: Building an Inventory of Poptarts
Basically our hardware plans started with an old box of electronics Miheer had from the last Slalom IoT hackathon. A quick analysis of parts led us to a subsequent pitstop to Microcenter… and a Particle Photon with OLED and Battery shields, Raspberry Pi3+ and PiCam, FSR Sensors, and breadboards/jumpers for days. Our initial approach started as the following:
- A Particle Photon with FSR sensors will be placed under the Poptarts box.
- The Photon will translate FSR resistance into weight measurements, and post to a Google Cloud Pub/Sub Queue component.
- From there a Google function will trigger capture the data and store them in Google Datastore.
- The UI will be hosted on Google App Engine standard which is serverless, and makes an API to other GCP Functions which return readings from the GCP Datastore.
The Photon makes it super easy for IoT hardware novices to create cool stuff in a timely manner. It comes with a simple mobile app that you can use to experiment with the ports of the Photon with ease. It also comes with a nice web application in which you can view your Photons and deploy applications on all of them with a click of a button³. Surprisingly, the photon also had a blast from my past college days… with all code written in C++! Oh those fun days of sending http requests through tcp sockets… ohhhh fun days! Fortunately, the Particle website has amazing documentation, and a large community that stays active with helpful guides! With all of these tools and resources we knew that we could undoubtedly setup our sensors super fast. But, how would we get the data from the Photon to the the Google Cloud? Oh wait, the Particle website has a 3 step Google Cloud pub/sub integration!
Along with my partner in front-end crime Garrett, I built the UI framework with React and Typescript. We also sprinkled in a lot of cool technologies like Redux, RxJs, Observables and more.
Can’t get enough UI? Look for my other post here!
As experts in the front end technologies and the Cloud, the rest was history. We built our phase 1 within 2 hours.
The hackathon was known to last 22 hours, and by this point our brains were hot and we couldn’t just stop there. While our solution could detect steep changes to the quantity of the Poptarts and users could preview the inventory, it still couldn’t capture the specific bandit. So phase 2 came along…
Phase 2: At the Scene of the Crime
The main purpose of phase 2 was to capture the bandit red-handed. Our idea was to take a picture on sudden changes on the Poptart quantities, and use this to shame the thief. In order to do that, we needed an IoT camera. While there are cameras out there, we wanted an immediate image capture without delays (and with a hackathon-oriented price tag!). We decided to use a Raspberry Pi 3 with a Pi Camera, not because we had experience with it, but because it would be challenging, fun, and it’s a hackathon after all so why not!
Design-wise, we now had to:
- Send a signal from the Photon directly to the Raspberry Pi…
- which captures a picture with the PiCam…
- that then posts to a GCP Function to store the evidence…
- which the UI then pulls for further analysis.
Phew. Only a few integrations…
And so entered one of our greatest engineering challenges of the night, promptly presenting itself around 12:00am. As expected, the daisy chain of integrations we wanted would also need to be in real-time, across multiple clouds, and multiple devices. Fun times!
One of the most challenging parts was the connectivity between the Pi and the Photon. We had to port-forward access to the Pi from the internet on our router level. As in every big company, we “rule-abiding” engineers have no control over the routers; our Seattle-based IT team controls all infrastructure. We could put in a request with IT to open the port, but it was 2am, and the hackathon would be over in 6 hours… not to mention there was a chance the request wouldn’t be approved. In order to quickly fix the issue, we tinkered with some ssh tunnel proxying from a VM hosted on Google Cloud compute engine.
Voila! The Photon now could directly hit the Pi with success and capture the picture!! Just like that, we were able to trigger our RaspPiCam, through a triggered weight on the FSR… and FoodAlert was TRULY born.
As a recap… our phase 1 gave us hardware and a webapp to report Poptart inventory, with phase 2 recording real evidence of our bandits… so mission accomplished right? Not to mention, we still had 6 hours to sleep, and a real working product! Alas… that wasn’t enough, and since we really wanted to win…… enter phase 3.
Phase 3
For phase 3, we split our goals: Miheer’s goal was voice assisted control, and my goal was AI facial recognition.
Phase 3a: Enable Voice for the Lazy
As an expert on Alexa skills from AWS, it wasn’t very hard for Miheer to familiarize himself with GCP Dialogflow, which allows you to create custom voice and chat bots that respond to dynamic conversation flows. Conveniently, he also had a AIY Google Home Kit, so why not pair them?
In order to make the rest of this phase a success, Miheer had to go out of his comfort zone.
He eventually had to create a GCP function to serve that voice assistance. He put his developer hat on (brushing off a LOT of cobwebs) at 3am, started coding in JavaScript, and deployed his very own GCP function. Next step was to write a Dialogflow conversation, with defined inputs and variables. Miheer followed conventions from the web app, assuming users would ask about inventory primarily. Following the addition of some error logic (critical in chatbots!), he integrated the bot with Google Assistant, and started testing. Thanks to the simplicity of GCP, in ten minutes we had a chat and voice bot running!
For those who want to do this themselves, note that unfortunately, the AIY kit was much less DIY and more ‘how is this cardboard still not assembling’. Note to Google, please stop using Cardboard in your assembly!! Anyways, we digress… following a sufficiently long box-creation period, Miheer setup the new Google Home, already loaded with FoodAlert and the “Hey Google, tell me about Poptarts” command was born!
The Google voice assistance was a success.
Phase 3b: Tag the Bandit
But, we still weren’t happy - We wanted more! I had the idea of using the picture we captured, and running facial recognition against it to automatically identify the bandit.
Facial recognition sounds difficult… and it turns out that thanks to GCP Machine Learning jobs, a ton of training data, and their great examples, it actually was easier than expected! I started by reading this Google article. Then I cloned their repo and dove straight into the code.
First I had to collect training data. I needed images of Slalom employees’ faces (totally for research purposes and opt-in only!) in order to train my very own Slalom Facial recognition Artificial Intelligence model.
I quickly created a web application hosted on the serverless Google App engine standard to run on any phone and capture (with consent) pictures of the people who were in the room (given it was 3am at the time… about 4 very willing participants!). Then, a GCP Function was triggered to store the pictures in S3 for my evaluation and training data, with the user id as metadata of the picture, to tag it for training.
After I collected enough data, it was time for training my AI model. As expected, and even with high performance hardware, the image transformation process took around 1–2 hours to finish running on Dataflow runner, followed by 1–2 hrs for the training process on GCP Machine Learning jobs. That was the perfect time for a quick nap.
When we woke up, my model was complete and ready for testing under the machine learning model in GCP console. I took a selfie and ran it through the prediction model. Unfortunately, at first our prediction was not as accurate as we hoped, so we proceeded to tinker:
- Add “no face” use case: Added extra training data of empty pictures (without a face) and labeled them as empty.
- Adjust sizing of expected training set: edited some of the arguments to match the size of my training data.
After the changes and another quick round of retraining, the prediction accuracy was increased; I hooked it up to my GCP function (with much difficulty, by using the gcloud ml-engine predicttool, as there is no NodeJS library to directly call the machine learning models!)… and tada!
We could now identify bandits by their face!
The Big Test
With all phases (required and bonus) finished at this point, we had a working product to demo! Problem was… we still needed to tie everything together, test it, and make sure it worked as intended.
To make it worse, it was 10am, and we only had about 2 hours left… Enter “all hands on deck” mode. All of us took ownership over something, trying to get as many end-to-end tests in as possible.
Miheer quickly created a team logo and presentation deck, I worked to integrate the new AI model, and Garrett started upgrading the UI to display all recorded bandit pictures.
Finally… at 11:59am, the team performed our final test (and took our wonderful group photo!), and quickly snuck into the presentations to support our coworkers until our turn came around.
Eventually: “Up Next… Team FoodAlert!”. And off we went!
The victory
After the blur of our presentation, we waited for the final results. The levels of stress, adrenaline, excitement and caffeine in our bodies were all extremely high. The team had worked really hard for this to happen, and we were still in awe that we actually got our demo to work. Despite all the late (but wanted!) additions, we actually got our product to function exactly as intended!
No small feat indeed, with us wrangling challenges including basic IoT hardware setup, server-to-cloud-to-router-to-cloud communication, literal flexibility of cardboard, and overall GCP documentation issues… wow, it actually works!
Finally… the moment had come… “The winner of the hackathon is… FoodAlert!”. Despite all the clapping and cheering, we all (and in tandem), forgot to show any excitement or sentiment. To be honest, we were still on the high of getting everything to work, and after seeing the amazing work of our peers, had no expectation to win as EVERYONE had awesome demos and ideas!
It may also be because we were barely functioning at that point… though thoroughly content (and already winners!) with the amount we had learned in such a short timespan.
In the end… Congrats to Team FoodAlert, we did it guys!
My Takeaways
I have participated in 4 Hackathons, 3 Slalom-hosted and 1 hosted by a third party, winning 3 out of 4. As in every other Hackathon, the 2017 Hack the Office v2.0 Slalom Hackathon was a great opportunity for me to learn something new and enhance my engineering skills based on my long term life goals of owning the Clouds!
Specifically at this hackathon I enhanced my skills by:
- Using GCP extensively. It really aligns to my personal goal of owning all the Clouds! I got to host a fully serverless website with ease by using app engine and functions.
- Having the opportunity to play for the first time with some of the easy IoT technologies, Photon and Raspberry Pi, and sensors which made me very proud and excited.
- Getting my hands dirty with AI and Machine Learning. I know I just scratched the surface of it, but made me even more excited to experiment in depth on structuring and training my very own models!
Hackathons also prove that if you put a bunch of smart people in a room and let them create, they will work hard and with excitement, and come out of it with more knowledge, skills, and willingness to learn.
Tech companies out there, I encourage you to host more Hackathons and let your engineers fly in their own ideas and inspirations!
Acknowledgements
I would like to thank and congratulate my team members Garret Jones and Miheer Munjal for all their time and dedicated work.
A huge shoutout to our coworker Tim Knapp on his contribution with SSH tunneling!
Thank you as well to our coworkers for their interest and support on our product, especially late overnight!
Finally, a big thank you to the Slalom practice and leadership who continue to provide opportunities for us to innovate and play with all the new cool toys of technologies out there through Hackathons, Bootcamps, Conferences and a lot more…
[1] yes this is a real thing in our office. Definition: Slalom-er who takes 1+ too many poptarts and/or is a fiend who takes opens a bag, only toasts one, and leaves the other for a mysterious second helping. it brings some humaneness to our office :)
[2] v2 was because v1 was a couple years ago, resulting in many cool office hacks like GTFO and our ever popular Occupy Stall Street.
[3] We looked at Google IoT to manage the Photons, but it was pretty much overkill for our needs. Surprisingly, this was one of the over-engineered areas we didn’t wish to tackle. It’s now saved for v2!
