Some data (and geek stuff) about Roomdi.com
Here is a day of statistics from Sunday, 18 June 2017:
- 12,980,710 HTTP requests to our Hotel’s availability API ( ~ 720 hours spent processing)
- 305.623 web page loads ( ~ 61,61 hours spent processing pages)
- 5,601,2628 Redis hits (~ 1.6 hours spent on Redis hits)
- 4,608,427 documents written in MongoDB (~ 22.6 hours spent written)
- 41,475,727,602 bytes (41,47 GB) sent from backend to web.
- 134,233,861,761 bytes (134,23 GB) total sent to our partners (hotel availability)
Web
The current infrastructure is based in 3 web servers, running with nginx and gunicorn. Each of this is configured with 1GB RAM, 1 vCpu
and 20GB of HDD.
Backend
Our backend API is running under Falcon with 16GB RAM, 4 vCPU and 20GB of HDD. This application’s function is manage all connections between WEB and databases, redis and different APIs, for example Booking Engine.
Falcon
Falcon is a very fast, very minimal Python web framework for building microservices, app backends, and higher-level frameworks. The Falcon web framework encourages the REST architectural style, meaning (among other things) that you think in terms of resources and state transitions, which map to HTTP methods. Falcon can be deployed in a variety of ways, depending on your needs.
Tracking
Data is a big deal at Roomdi.com. Our business development and marketing teams rely on data to make critical business decisions. Currently we handle over 10,000,000 of events per day.
Kafka
We use Kafka as our data transport layer to achieve loose coupling between data producers and consumers. With this type of architecture, Kafka acts as an abstraction between the producers and consumers in the system. Instead of pushing data directly to consumers, producers push data to Kafka. The consumers then read data from Kafka. This makes adding new data consumers trivial.
Storm
Storm is a battlefield-tested solution that integrates very well with Kafka. It allows creating a custom processing flow, i.e. a topology. In our case, as we are a “python” company, we’ve coded our bolts in python — then encapsulated them with the Bolt in Java. Below is a run-time visualization of the topology that runs on our Storm cluster.
Partner Integration
To connect via HTTP with our partners — Trivago, Tripadvisor, Google — our application is built under Falcon with the following configuration: 30 GB disk, 6GB ram and 4 vCPU. To handle ~ 13,000,000 requests per day, we’ve a cluster of 13 servers — just at 30% of total capacity. This cluster hold more servers than current demands as we will double traffic on the next few weeks.
Real-time Monitoring and Alerting
Business and API events are stored in Elasticsearch for real-time monitoring and alerting. We use Kibana to visualize the performance of processes, and the performance of partner APIs in real-time.
Finally
We’ve another services that are used to run marketing processes, analyze data, train algorithms or fill our BI tools but about this topic I’ll write a post later. So in aggregate our applications are running over 6,1 TB of HDD, 84GB RAM and 263 vCPU.
