My first experience creating an API

A weekend side project to build a Home Automation API

Earlier this year I joined Mashape, an API platform and marketplace allowing you to distribute, monitor, and manage your APIs. While I was familiar with what an API was, I had never made one myself. I learnt programming when I was in college, and had some experience with Python at work (mainly scripting). Over the past couple of weekends, I took it as a personal challenge to learn how to create my own API, as well as experience some “real-world” coding. I worked on what I call the “Smarthome API”, an API for home automation that can control the devices in my house, such as the lights, fan, WeMo, door lock, and climate (via Nest).

In this post, I have documented my journey, the steps I have taken, the learning, and experiences along the way (mostly of my housemates who had to live in darkness while I was testing the app).

1. Defining the problem

My housemate recently installed several flavors of home automation hardware:

1. Z-wave switches and locks that enable home automation over a wireless network. We use Vera, a smart home gateway, to interact with the switches and devices.
2. A Nest thermostat, of which we use the mobile app to adjust the temperatures and switch between “Home” and “Away” modes.
3. Belkin WeMo appliances.

All these products turned out great when we found ourselves too lazy to get out of bed to turn off the light switch, or check if the door is locked. However, it was rather cumbersome to manage different apps, or to manage every device individually.

It also helped that the Vera web interface was not the most aesthetically pleasing.

I wanted to have more flexibility and control over these devices. For a start, I scoped out these basic requirements for my API:

• Be able to retrieve the list of all the switches/ devices (lights, locks, Nest) in the house — GET method
• Be able to retrieve information on each individual switch/ device, e.g. Name, current state, etc — GET method
• Be able to modify the state of each switch/ device, e.g. turn a light on or off — PUT method

2. Choosing a language and framework

This was easy. Since I was most familiar with Python, I decided to go with Flask (a Python micro-framework), because it is lightweight. Of course, there were many other options, but I’ll not be going into details on the pros and cons of each.

PS. For the rest of this post, I’ll be referring to the switches/ devices as Lights. In reality, they control not just the lights, but also the fan, WeMo, etc. “Lock” refers to the door locks, while “Nest” refers to the climate control.

3. Building the scaffolding / creating the models

Once I had everything set up and managed to get my “Hello World” running, it was time to get my hands dirty.

I created the model class for a Light object (on hindsight, I probably should have named it “DeviceSwitch” instead) with 4 properties:

• id — device id as specified by Vera
• name — name of the device/ switch to be controlled
• room — name of the room which the device/ switch is located in
• status — 0 for when the device/switch is off, 1 for when it is on

Next, I created my first endpoint: GET /lights which was supposed to return a JSON with information of all the Lights. I fed it with some dummy data, and saw that my method was working correctly.

Pretty much my reaction when I realized that the endpoint was working, albeit with dummy data.

Sample response for GET /lights/1 endpoint

The second endpoint: GET /lights/<id> involved returning a JSON with properties of the requested light. Once again, the method was verified to work correctly with mock data.

Changing the state of the Light was slightly more complicated, as the PUT request would need to take in an <application/json> content type with the desired state. For this, I made use of the Postman Chrome App which was able to verify that my method was working correctly. Success!

At this point, I pretty much went around the house and showed off my wonderful “working” API to my housemates, although it didn’t exactly do anything. They were not impressed ☹

4. Figuring out the Vera interface

The difficult part came when I had to figure out how to make requests to Vera to actually control the hardware. I opened the Vera control panel and used Chrome’s inspector to filter the data packets that were sent and received while turning lights on and off.

For a start, I figured out the request URL which Vera uses to retrieve all the devices.

HTTP Request and response for retrieving information about Vera devices

It turned out that the request URL followed a simple pattern, and the I could request device information with just two lines of code:

p = {‘rand’: random.random()}
response = requests.get(“http://192.168.1.88/port_3480/data_request?id=user_data", params = p)

I used the same method to figure out the requests that I needed to make to Vera in order to change the state of a Light switch.

HTTP Request and response for modifying the state of a Vera Light switch

All that was required was the id of the device, the target state to change to, as well as a random number parameter.

p = {‘DeviceNum’: id, ‘newTargetValue’: <the new state>, ‘rand’: random.random() }
response = requests.get(“http://192.168.1.88/port_3480/data_request?id=lu_action&output_format=json&serviceId=urn:upnp-org:serviceId:SwitchPower1&action=SetTarget", params = p)

On hindsight, this was much easier to explain, but it took me some effort to learn and understand exactly what I was doing then.

5. Making it work

Since I had figured out how the HTTP requests worked, all that was left was making my API perform the desired action and return the correct response. (Yes, my API was still not really “working” after all the work ☹)

To list the lights in my home, I inspected the JSON response from the HTTP request, retrieving the values of id, name, room, and state of the device. I parsed the variables into a newly created Light object. Iterating through the entire response, I was able to build a list of Light objects which were then returned by the GET /lights method as a JSON.

Multiple loops and nested if-s which were not very pretty, but it got the job done.

To switch the state of a light, I decided to have my API accept a PUT request with the id in the URL parameters and the requested state in the JSON body. Implementing this was a 2 step process:

1. Verifying the legitimacy of the JSON request
2. Performing the actual request to Vera and returning a legit response

For a start, I only verified that the JSON request defined a proper state which could be used to control the Light device. If the state was specified, I’d send the request to switch the state. If there was no error, the method would return an OK response. Otherwise, it should return the error.

Once again, very messy and ugly code, but it works, so we’ll leave the refactoring to later.

Although my code was somewhat messy and ugly, it worked. And I made sure everyone in the house knew it by randomly switching on and off the lights in their room ;)

6. Doing the same thing for other devices

Once the three methods worked for Lights, it was easy to replicate them for the Locks. Most of the code was pretty straightforward and looked pretty much the same, except for some minor URL changes in the Vera HTTP request. However, I did enforce that a password needed to be parsed in the JSON request when changing the status of the lock. (I definitely didn’t want random hackers being able to remotely lock and unlock my front door!)

7. Refactoring the code

Getting to this point was a proud moment for me. However, my code was messy and ugly, with inconsistent variable names, multiple nested loops and conditions, and a ton of duplicated code. One key issue was that my JSON response was returning the room id instead of the actual room name. I needed to retrieve the room names from the Vera API once again, and return them in my response.

Sample response body of GET /lights endpoint returning room ID instead of room name

8. Adding verification functions and increasing sleep time

At this point my API worked for switching individual devices on and off. I tried to create an endpoint that would turn all the lights off. However, I found that not all the lights were being turned off! Upon investigation, I realized that all my requests to the Vera API was succeeding and returning an OK response even though not all the lights were turning off in reality ☹

While I grappled with the issue, my unfortunate housemates had to suffer in darkness, since I refused to switch on the lights for them until I fixed the bug.

To resolve the issue, I decided to add a verification function to ensure that the device state switch was successful. Otherwise, I would send the request again. I also added a sleep time just in case the updated device state was not reflected soon enough in the API response.

Verification function to ensure that the device was switched to the target state

9. Adding Nest and creating a Devices superclass

It was time for me to add the Nest device into the mix. For the sake of simplicity, I chose to control Nest through the Vera API, instead of the native API which was recently released. The Nest object that I created was pretty similar to the existing Light and Lock objects. Hence, I decided to create a Devices superclass, in which the Lights, Locks, and Nests would inherit from.

The Vera API considers the Nest controller as two different devices.

The Nest device was slightly more complicated, as the Vera API accounted for each Nest controller with a different id — one id for the Nest device controlling the temperature range, and another to switch between the states. I chose to expose only the id of the main Nest object controlling the temperature, and stored the other controller id as a variable within the main object class.

Upon creation of the Devices super class, I moved the update & verify state functions (functions controlling the PUT endpoints) into the class definition. This way, I had cleaner and prettier code with less repetition.

10. In comes Redis, Custom JSON Encoder, and Device Decoder

I had created endpoints to turn all the lights on and all the lights off (which I called the home and away modes). I added in the “features” to lock doors and turn the A/C down to the away mode (and a corresponding set of states for the home mode).

But manually hard coding the “Home” and “Away” modes for all the devices was not very efficient. I wanted to provide a more extensible solution for creating custom modes, where the API users would be able to store and reset the house to previous modes.

To do this, I needed to store the current states of all the devices (Lights, Locks, Nests), and be able to retrieve them when necessary. I needed some form of persistent storage… and this was when Redis came into the picture.

To store a mode, I created a new PUT method for /states/<name of slot> which stored the state of all the devices in Redis. To do that, I wrote a custom JSON Encoder which included a special “_type” variable in each representation of object to determine if it was a Light, Lock or Nest object. This was necessary as I needed to recreate the object based on its type during decoding.

Custom JSON Encoder which added a “_type” variable and custom Device Decoder which recreated the objects based on their “_type” variable

Once I was able to deserialize the JSON that was stored in Redis, creating the PUT method for loading a saved state was as simple as iterating through the states, and calling the respective class function to switch the state of each device. All these was done with lots of code refactoring, of course ;)

Future plans?

So there, over a couple of weekends, I managed to create my first API to make my house a smarter home. There are definitely areas of improvements, for example:

• Using ORM for creating models instead of writing custom models
• Some kind of parallelism to set the state of the devices, locks, and Nest all at the same time instead of sequentially. This will greatly speed up the response of the API.
• Verify the state of all devices at once instead of verifying them individually as their states switch. This will also reduce the number of requests being sent to Vera.
• Ability to roll back to the original state if there is an error in switching the state of the device — This will involve saving the original state before a change is made, and then rolling it back if an error occurs.

While some of these changes can be made pretty quickly, I’m calling this done for now as I work on my next project.

What’s happening to the API?

This project doesn’t end here! Now that I've got the API ready, my housemate, Ben, is in charge of the front-end work of creating an Android app using my API.

For now, this is what the app looks like — it is very basic and lacking in design, but does exactly what we need from one place, and it’s still a work in progress.

I also heard that he had hacked up a bitcoin integration where sending money to him would result in the door opening ;)

I hope to update this with more screenshots once the app is completed! ☺

Update (2/15/2015): Omg! Someone made a pull request to my repo and cleaned up the mess! ☺

Update (4/15/2015): About 6 months ago, I bought an iPhone after being an avid Android user for the past 6 years (yes, I owned the G1!). The first thing I missed in iOS was, of course, my wonderful smarthome app. I decided to take the chance to learn some iOS programming by porting the app over to iOS, and here’s what the app looks like. The code isn’t the prettiest, but hey, it works! ;)

Acknowledgments

All this would not have been possible without the patient guidance of Ben who taught me everything I needed to know to get this far. The github repo for the API is here: https://github.com/christinang89/smarthome, although it currently only works for my house. The repo for the Android app is available at https://github.com/bencxr/kotl, and for the iOS app: https://github.com/christinang89/Unicorn

This has been fun side project. While I set out expecting to learn about creating APIs, I ended up taking away a lot more. I learnt how to sniff data packets. I learnt how to write actual code outside of a programming assignment context. I learnt about Redis/ persistent storage, and custom serializers and deserializers. I learnt many design patterns in developing a web application, some coding best practices, PEP 8 (Sorry, I know my code isn’t exactly conforming to it), and how to solve a large problem in simple, manageable pieces.

So I would say, if you’re learning how to code, learn how to make an API! ☺