Bringing Machine Learning to the Connected Home
By Aren
A TensorFlow & Nest Cam Codelab
Machine Learning and Artificial Intelligence are exciting topics with developers, but few of us have actually gotten our hands dirty and built something with these technologies. This week at Google I/O, we wanted to offer developers an inside look at how to build something useful with Google’s Machine Learning library, TensorFlow.
So we’re deploying a Nest Cam & TensorFlow Codelab. Read on for additional resources.
Google released TensorFlow in 2015. Since then, it has seen industry-wide usage in a variety of applications, including Optical Character Recognition (OCR), recommendation engines (Google search results, Facebook news feed), and image recognition. The library has drastically reduced the complexity involved in setting up and training neural networks, which has turned incredibly complex tasks into everyday activities.
So, how does it work? When working with neural networks, you repeatedly train the network with known inputs and expected outputs. The output is effectively the network “learning.” The more the network learns, the stronger the output. To save developers from having to train all inputs themselves, TensorFlow provides a series of open-sourced, pre-trained resource models. This eliminates the tedious step of training the neural network; instead, we use one of the provided models called Inception-v3.
Inception-v3 was trained to classify images as part of the Large Visual Recognition Challenge from 2012. The challenge was to classify images into 1000 discrete classes at a large scale, and focus on image indexing and labeling, including things like “golf ball,” “tennis ball,” “broccoli,” etc.
In the Nest TensorFlow codelab, developers build a small Python web application. The app is designed to fetch imagery from a remote Nest Cam. The JPEG image is fetched via the Works with Nest REST API (learn about our API). In front of the Nest Cams field of view, we have placed objects that the Inception model is trained to recognize.
So, for instance, given the following image in front of the Nest Cam, the codelab application is able to say with near certainty that this image is a teddy bear.
It is important to note here that the Inception model has been trained with imagery drastically different than what Nest Cams see in the codelab–that’s what makes this truly incredible. The neural network is able to recognize the image contents in a fashion similar to a human brain.
There are all sorts of cool things that could be built with Machine Learning and Works with Nest. The Nest API offers access to a rich data set for our thermostats, cameras, and smoke + CO alarms.
Machine Learning is an area of rapid development with game-changing impact in software development — including in the connected home. Our hope is that this codelab serves as an interesting intro to TensorFlow and Works with Nest, and as a starting point for your curiosity.
If you’re in Mountain View, California this week for Google I/O, visit us at the Codelab space. Or find it here.
I can’t wait to see what you will build.
Join the conversation at Nest’s Developer community. Get started with Nest Developers. Or explore careers with Nest.
The information contained in this blog is provided only as general information for educational purposes, and may or may not be up to date. The information is provided as-is with no warranties. This blog is not intended to be a factual representation of how Nest’s products and services actually work. No license is granted under any intellectual property rights of Nest, Google, or others.
