Building a Quantified Self API using AWS Lambda and AWS API Gateway

7 min readMay 30, 2018

I’ve been tracking my health, wealth and fitness in various forms for more than a decade now. I like being able to see my progress over time and have been recording everything from bills, workouts and spending in a notepad since I was a teen. Eventually I tracked everything in a spreadsheet but even that has slowly evolved into using various apps, services and devices. To name a few, the various app I’ve tried in the past are:

Finance: Mint, YNAB, CryptoPro, CoinTracking, Yahoo Finance, Google Finance, GNU Cash, Plaid
Habits: Coach.me, Way of Life
Health: MyFitnessPal, Strava, Apple Watch, Hello Sense, Oak, Headspace, Nokia Healthmate
Travel: TripIt, AwardWallet, Google Trips

While on their own these apps work great, they can be a walled garden with all my personal data trapped in separate silo’d services without an API to access them. Also, it can be tedious to check multiple different apps to see the information you are looking for in one glance. I wanted a way to see all my personal information on a single pane of glass on infrastructure that I control. I played with MagicMirror for a bit and it was great, but it just lacked some of the modules I was interested in.

Since I know Python and AWS quite well, I set out to build my own solution. While building an API in of itself isn’t that unique, I wanted to try out Amazon API Gateway and Lambda which I will demonstrate at the end of the article.

I should mention that I did look around for other solutions, but the closest I found was Google Now. Google Now does a great job of bringing information to you when you need is such as sports highlights, commute, flight information and restaurants nearby, it was lacking some of the other information I was actually interested in. Plus, I’m mostly in the Apple eco system, so without an Android as my primary device it wasn’t as useful.

Body Weight

I have a Nokia Body+ digital scale. The scale is Wifi connected and measures body fat (%), muscle mass and body water. As soon as your measurements are made they are pushed up to Nokia. Luckily, Nokia has an API that I was able to query.

Library: https://github.com/orcasgit/python-nokia

Exchange Rate

Having lived in UK and Australia, I still keep an eye on the exchange rates there. While browsing through Home Assistant components found this great API — Fixer, which allows you to get exchange rates for several currencies for free.

API: https://fixer.io

Stocks

This one should have been easier than it sounds. Since Yahoo Finance shut down its API and Google Finance closed it portfolio users really have no other option to track their portfolio via an API. I read this article of a guy that also built his own portfolio tracker in a single index.html. Through that article I discovered the IEX API

API: https://iextrading.com/developer/

Plants

Read my previous article on how I build my own API to monitor my plants

API: Homegrown

Crypto Currency

I normally track crypto currencies using Crypto Pro on iOS and Coin Tracking , but why have a separate app. I included crypto prices in my dashboard using CoinMarketCap’s API

API: https://coinmarketcap.com/api

Public Transit

I find it useful before I leave to work in the morning, when my next bus or streetcar will arrive. Fortunately, Nextbus provides an XML (unfortunately) API for real time bus tracking in various cities.

API: http://www.nextbus.com/xmlFeedDocs/NextBusXMLFeed.pdf

Commute

While I may know when the next bus will arrive, service may be delayed on the subway impacting my commute. Google Transit API actually estimates how long it will take to get from home to work.

API: https://developers.google.com/maps/documentation/directions/intro

Weather

One of my favorite weather apps on iOS is Weather Line. What makes it unique compared to the default weather app on iOS is that it shows hourly and daily weather in a line graph. While that doesn’t sound like much, I find it really useful for a quick glance of the weather ahead. I tried to replicate the same idea on my dashboard.

API: https://darksky.net/

Calendar

Google makes an API available for Google Calendar. I found it one of the hardest to work with but got it working with some trial and error.

API: https://developers.google.com/calendar/

Building the API

I wrote the backend using Flask because its simple and easy to use. I also used the Flask-Restful library which Twilio open-sourced back in 2012. The structure of my project looks as follows, notice the /lib directory containing libraries I wrote to connect to each API described above

.
├── __init__.py
└── api
    ├── __init__.py
    ├── lib
    │   ├── __init__.py
    │   ├── api_coinmarketcap.py
    │   ├── api_darksky.py
    │   ├── api_fixer.py
    │   ├── api_googlecalendar.py
    │   ├── api_googlemaps.py
    │   ├── api_iex.py
    │   ├── api_nextbus.py
    │   ├── api_nokia.py
    │   └── api_plant.py
    └── resources.py

Inside the /lib directory there is an api_(service).py file for each backend API I connect to. For example my Darksky library looks like this:

## api_darksky.pyfrom dotenv import load_dotenv, find_dotenv
from datetime import datetime
try:
    load_dotenv(find_dotenv())
except IOError:
    print("could not find env file")
    passdef get_weather():
    url = "https://api.darksky.net/forecast/{0}/49.641579,-89.4180915?units=si&exclude=minutely,alerts,hourly,flags".format(os.environ.get('DARKSKY_KEY'))
    data = requests.get(url).json()
    response = { 
        'last_updated': datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
        'data': data
    }
    return response

Once I have created a library for each api, I import it then then create a Resource

## resources.pyfrom flask import g, current_app
from flask_restful import Resource, fields, marshal_with, marshal
from flask_restful import reqparse, abort
from hashlib import md5
import re
from lib.api_darksky import get_weatherclass Weather(Resource):def get(self):
        r = get_weather()
        return r

I’ll create a blueprint for the /api directory

## lib/__init__.pyfrom flask import Blueprintapi = Blueprint('api', __name__)

Finally, I’ll add a route for /weather and the blueprint to my __init__.py

## __init__.pyfrom flask import Flask
from flask_restful import Api
from config import configfrom api.resources import Weatherdef create_app(config_name):
    app = Flask(__name__)
    app.config.from_object(config[config_name])
    config[config_name].init_app(app)from .api import api as api_blueprint
    api = Api(api_blueprint)#routes
    ...
    api.add_resource(Weather, '/weather')
    #
    app.register_blueprint(api_blueprint, url_prefix='/api/v1')return app

Deploying to AWS API Gateway and AWS Lambda

With the API written I still needed somewhere to run this thing. While I could setup a Docker container or an EC2 instance, it would be much cheaper and less overhead for it to be serverless using Lambda. Luckily, I found the awesome project Zappa

Enter Zappa

Zappa lets you build and deploy serverless Python applications. To get started you just need to run two commands

pip install zappa
zappa init

Zappa init will detect your application type and create a default config file for you. For my Flask project my zappa_settings.json looked something like this

{
 "dev": {
 "app_function": "manage.app",
 "aws_region": "us-east-1",
 "project_name": "api",
 "profile_name": "zappa",
 "runtime": "python2.7",
 "s3_bucket": "zappa-mybucket",
 "domain": "api.domain.io",
 "certificate_arn": "arn:aws:acm:us-east- 1:0000000:certificate/12345abcdef67890",
}
}

Another cool thing, is using AWS ACM, you are not charged for any certificates created using ACM that are attached to AWS resource. So I was able have my API run over HTTPS without much effort.

With the config file all setup, to deploy you code all you need to do is run

zappa deploy dev
Deploying..
Your application is now live at: https://7k6anj0k99.execute-api.us-east-1.amazonaws.com/dev

Running the deploy command Zappa will zip up your code, deploy it to an S3 bucket, create a Lambda, create an API Gateway, and attach the Lambda to the Gateway as seen below.