CodeX
Published in

CodeX

Choosing the AWS serverless stack for a prototype (Detecting Paris’ locked bicycle stations 1/5)

Photo by Guillaume Bontemps / Ville de Paris

Lockdown and dead stations

The Velib API

  • /station_status.json which returns, in “real-time” (refreshed every minute) the content and status of each station.
  • /station_information.json which returns the geo-coordinates and name of each station.
//station_status.json simplified structure
{
"data": {
"stations": [
{
"is_installed": 0,
"is_renting": 0,
"is_returning": 0,
"num_bikes_available_types": [
{
"ebike": 0,
"mechanical": 0
}
],
"num_docks_available": 0,
"station_id": 0
}
]
}
}
//station_informations.json simplified structure
{
"data": {
"stations": [
{
"lat": 0,
"lon": 0,
"name": "string",
"station_id": 0
}
]
}
}

One event at a time

A Kafka topic with multiple consumers
Fictional Kafka topology: Youtube’s video import process

The cost of Kafka

The Kafka-based prototype architecture
Simplified Kafka topology
  • Fully managed: I could use AWS MSK. The smallest instance (Kafka.t3.small) cost 0,0456 USD per hour. I need to run it full time, so that means… $400 per year
  • Self-managed: I could just use an AWS EC2 instance with ~2GB of memory and install Zookeeper and Kafka on it. A t2.small cost 0,023 USD per hour, $200 per year.
  • Container: I could also leverage Docker and use a similar EC2 instance as part of AWS ECS. That would simplify the setup but still be $200 per year.
  • AWS Kinesis: it’s very similar to Kafka, it’s fully managed, it seems way cheaper. But it’s not compatible with Kafka Stream, which would mean rewriting a big part of the application and choosing a new streaming framework.
  • Raspberry Pi? I already have a Raspberry Pi lying around somewhere. I could surely install my Kafka cluster on this, open the right ports on my home gateway and host the rest of the application on AWS. I would also need to pray to the computing gods that nothing bad happens while I’m not home to fix it.

Back to the drawing board

Photo by Diana Polekhina on Unsplash
  • It should run “in production”, on AWS.
  • Velib works 24/7, so should my application.
  • I want to spend 200€ max per year if I have less than 100 monthly users.
  • The day-to-day operations should be minimal.
  • The UI should look OK.
  • Kafka is not mandatory. At this point, I feel like I have already learned a lot about it, and would rather focus on delivering something.
  • The locked station detection does not need to be accurate. I don’t even know if the Velib API’s data are accurate.
  • The quality of code is not my concern. I’m more interested in the architecture and learning more about AWS.
  • The UI will be french only, as Velib only operate in Paris.
  • Nothing fancy for the front end. I’m already familiar with Angular, so that will be enough.

λ function

//a basic javascript lambda performing a REST call
const https = require('https')
let url = "https://example.com"
exports.handler = async function(event) {
const promise = new Promise(function(resolve, reject) {
https.get(url, (res) => {
resolve(res.statusCode)
}).on('error', (e) => {
reject(Error(e))
})
})
return promise
}
Cost comparison between the smallest EC2 instance and the smallest Lambda

The serverless offering

  • A database? DynamoDb (you pay for the number of items created/read)
  • A file server? S3 (you pay for the number of files created/read)
  • A load balancer/router? API Gateway (you pay for the number of requests being routed)
The current AWS serverless offering
AWS Free Tier for Dynamo and Lambda
  • Part 1: Choosing the AWS serverless stack for a prototype
  • Part 2: The backbone of a serverless app: Lambda functions and DynamoDb tables
  • Part 3: Implementing a real-time detection algorithm with Lambda functions and DynamoDb streams
  • Part 4: Creating a serverless API and hosting a frontend with S3
  • Part 5: Performance tuning for a Lambda-based API

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store