Monitoring Electric Scooters in Paris with Datadog and NodeJS
Electric Scooters are ubiquitous in Paris, a recent development for a city that’s over 1,000 years old. They’re a staple for Parisian commuters, and used as last mile transport journeys for getting to just about everything, whether it’s work, cafes, or the market. This naturally made me curious to see if I could use *computers* to track the distribution of electric scooters in Paris among the different Arrondissements. When and where are people using electric scooters? How do the battery levels hold up over the course of the day? I spent this weekend poking around the various APIs of the different scooter providers to see if I could learn anything.
It turns out Bird Scooters has an undocumented API with public endpoints with information about nearby scooters, including unique ids and current battery level. I decided to use this API to gather electric scooter distribution and battery levels for Paris over the course of the day. I formatted these details into consistent metrics that I collected with Datadog so I could visualize and monitor them in a dashboard as time series data. Lastly, I tried to abstract these functions into reusable components that people can use for their own zip codes as they see fit, which I’ll share below along with some anecdotal observations.
You can find my Github repo here
Background
Paris is split into 20 different arrondissements, or municipal districts, forming a sort of spiral that starts in the city center and moves outward. The 1st is where you have the big tourist stuff like the Louvre and the Tuileries, and the teens are the more recent additions to Paris. Like the 17th , a former village on the western edge of the city where Manet used to live, or the 20th in the east, home to Père Lachaise Cemetery where Honore de Balzac, Chopin, and Oscar Wilde now reside. So that’s the city in a nutshell.
Code
Each arrondissements has its own zip code, but the Bird Scooter API endpoint only accepts latitude and longitude, so I had to translate these. I found a service at geonames.org with a public API that returns a latitude and longitude for a zip code. It’s free to use (although you have to signup and create a username for API access). To interact with it I used axios, a popular http request library for Node.
async (req) => {
try {
// get zip code lat/lng and format, throw error if missing
let [zip, country] = [req.query.zip, req.query.country] if (zip === undefined || country === undefined) { throw "please submit a zip and country code" } let geoApiDetails = `http://api.geonames.org/postalCodeSearchJSON?maxRows=1&username=${config.geonames_username}&country=${country}&postalcode=${zip}`
let geoInfo = await axios.get(geoApiDetails)
let mappedGeoInfo = {[zip]: { "lat": geoInfo.data.postalCodes[0]["lat"], "lng": geoInfo.data.postalCodes[0]["lng"]} }
} catch (err) {
console.log('error', err);
}
}
The Bird API is a bit wonky in that first you have to retrieve a valid access token (each token is good for about 15 minutes), and then make GET requests (as opposed to POST requests) to the /bird/nearby endpoint with the specific headers, including the appropriate latitude and longitude details in JSON as a request header. Given the unusual request formatting, I decided it would be better to use an existing library instead of trying to roll my own API wrapper, so I used node-bird, a NodeJS wrapper for the various Bird API Endpoints. I settled on collecting two distinct metrics throughout the day, the number of scooters in a given arrondissement, and the average battery level of those scooters.
//register token w/any email then get+format scooters in 500m radius of zip code lat/lng
await bird.login(faker.internet.email());
let results = await Promise.all([bird.getScootersNearby(mappedGeoInfo[zip]["lat"],mappedGeoInfo[zip]["lng"])])
let mapped = results.reduce( (mapping, zip_scooter_list) => {
if (mapping[zip] === undefined) { mapping[zip] = {} } mapping[zip]['avg_scooter_battery_level'] = average(zip_scooter_list.map( (scoot) => scoot.battery_level))
mapping[zip]['scooter_count'] = zip_scooter_list.length
return mapping
},{})
Finally, I collected and sent those metrics to Datadog with dogapi, a Datadog API NodeJS Client. What's useful about collecting metrics in Datadog is that I can also tag these metrics, so I can just define a few base custom metrics (bird.scooterCount, bird.avgScooterBatteryLevel), and then tag them with the different zip codes so they’re easy to group and sort.
//send bird custom metrics to DD with zip code tags
await Promise.all( Object.keys(mapped).map( (x) => dogapi.metric.send("bird.scooterCount", mapped[x]['scooter_count'], {tags: "zip:"+x})))
await Promise.all( Object.keys(mapped).map( (x) => dogapi.metric.send("bird.avgScooterBatteryLevel", mapped[x]['avg_scooter_battery_level'], {tags: "zip:"+x})))At first I messed around with deploying this as a serverless functions with one of the cloud providers like Microsoft Azure, then coupling them with a free service like cron-job.org to invoke them at regular intervals (every 10 minutes or so would probably be fine). Azure was a breeze to setup and deploy, however it turns out Bird does some rate limiting on IP Ranges for the major cloud providers. So, rather than complicate things I've just deployed locally using expressJS on my laptop and then set a cronjob to ping it every 10 minutes for each arrondissement I wanted to monitor
*/10 * * * * curl -k http://localhost:3000?country=FR&zip=75001Observations
The data revealed some interesting trends. Generally it seems like Birds Scooters had high availability throughout the day, and average Battery life stayed pretty consistent during the day too, usually not dropping below 70% for a particular zip code. They stayed available until pretty late into the evening too, about 10pm, before the API stopped returning scooter information. I assume this was when the scooters had been removed from the streets by chargers in order to charge them for the next day. Visualizing this data in Datadog also allowed me to surface a few interesting observations.
For Battery Levels, it was interesting to see the lowest average battery levels throughout the day in tourist heavy areas like the 1st, 2nd, and 7th arrondissements. This makes sense to me, as places like Musee Orsay, the Louvre, or even Balencegia (Women be shopping!), would attract commuters from far away or who have already spent significant time on the Scooter sightseeing.
For Scooter Counts, the lowest counts tended to be the low traffic arrondissements like the 20th, and surprisingly also the 10th and 4th, home to major railway and tourist destinations. Some scooters tended to show up in overlapping zip codes as the search radius parameter of the Bird API sometimes exceeded the size of the zip code, so this may have been a case of needing to clean the data a bit better.
I made a small Github repository which includes an ExpressJS Server for you to run and monitor your own endpoints, which can be found here. Electric Scooters are here to stay in most major cities, and it’s fun to see what the data can tell us about how people increasingly use them in the day to day lives.
Thanks for reading and good luck with your monitoring and your commuting!
-eric mustin
