FlightControl — Logging and instrumentation for multi-tiered apps
Before we get started, let me just say — multi-tiered / microservice architecture is great. It gives you great freedom to work on small features in isolation, building up to large feature sets without having the headache of learning how the whole platform works every time you want to work on a small feature.
However — running a large number of smaller systems make tracing a lot harder. It can become tricky to nail down a bug when 5–6 systems are involved in a single client-side transaction. Log aggregation with tools like Kibana/ES only solve half the problem — you still have the potential for a ton of requests to be within your search window and this makes it quite difficult to follow a whole transaction through the many apps.
We came up with a better idea at Urban Massage which has been a spare-time project — it’s still very much a work in progress but when it’s ready, we’re planning to open source the project.
We set off with 2 main features in mind — the ability to see what request / transaction your logs are associated with, and when an app dispatches a request to another app to perform some task, linking the parent transaction to the new request (and obviously doing this to an infinite depth).
Here’s a quick (pseudo) example:
POST bookings.urbanmassage.com/create-booking (200ms)
>>> Log ‘Input request body is {“test”: “body”}’
>>> POST logistics.urbanmassage.com/obtain-lock (10ms)
>>>>>> REDIS SET test.lock.key (9ms)
>>> POST payments.urbanmassage.com/charge-card (90ms)
>>>>>> Log ‘Submitting card charge for £65.00’
>>>>>> Braintree Charge (86ms)
>>> MySQL INSERT INTO bookings... (30ms)
etc etc etc….
Hopefully the above gives you an example of the power of linking these different transactions in different apps together — stored in a structured format, we can do even more than just nail down bugs — we can quickly find bottlenecks in production, get a better idea of how to plan our scale-out and much more.
At the moment, we have a basic working NodeJS client — this is exposed as a library you init with the location of where the FlightControl server runs and the app name, and gives you a piece of express middleware for convenience so you can do things like req.logger.write(…) to automatically track logs against the appropriate transaction.
The cross-app transaction nesting works by just setting an extra header in HTTP requests, or attaching an extra property when we do RPC over RabbitMQ.
At the moment we’re looking for contributors to help create some clients for other languages and frameworks, as well as doing some work on the UI, so get in touch if you’re interested!
