Headers propagation with hpropagate

Since starting using Istio (see Part 1, Part 2 and Part 3), we’ve been very interested in using it to deploy multiple branches of our micro-services at the same time (as mentioned in Part 3).

To be able to do that, we need to propagate a custom HTTP header across our services. We’ve decided to use a header named x-variant-id for that purpose. Propagating custom headers is something we were already doing to propagate our custom x-correlation-id header in our codebase (which we use to correlate requests together, useful in logging and for debugging). It is currently a mix of using an Express middleware, keeping a handle of the header value and remembering to explicitly set the header value on all the outbound calls: in other words, a bit of a pain and prone to error.

So we started thinking about a way to automatically propagate headers. We had in mind the way the NewRelic or Elastic APM npm packages work: you simply require a library and some “magic” happens out of the box.

As an added bonus, this would also allow us to use Istio tracing which requires 6 specific headers propagated across services.

This is how hpropagate was born, as an attempt to automatically propagate a specific set of HTTP headers as follows:

• x-correlation-id used for logging/debugging. If not set, hpropagate will generate a new UUID value.
• x-variant-id used to identify which version of the service Istio should route traffic to.

And to enable Istio distributed tracing:

• x-request-id
• x-b3-traceid
• x-b3-spanid
• x-b3-parentspanid
• x-b3-sampled
• x-b3-flags
• x-ot-span-context

How does it work?

hpropagate was inspired by the following talk and module. We found these when researching how NewRelic and Elastic APM wrote their Node.js modules. I won’t go into too much detail here as it’s well explained in the short video below.

What we’re trying to do is:

1- When an incoming HTTP request is received by the server, we want to create a trace object used to record the headers (the trace is a 1-to-1 mapping with the incoming request).

2- When an outbound HTTP call is made, we want to transfer the HTTP headers collected on the incoming request to the outbound call.

Both are done easily by instrumenting the core http lib (in other words, by wrapping the original functions to add extra logic). The first challenge is that we need the trace object to be globally accessible so that we can share the data between incoming and outgoing requests.

As we might be dealing with multiple incoming HTTP requests at one point in time, another challenge is to make sure that we are using the correct trace when making an outbound call to prevent mixing requests.

For this, we rely on the Node.js module async_hooks (it provides the following hooks init, before, after and destroy that will be invoked on all asynchronous operations. It also provides a way to identify an asynchronous operation with an id and a parent id that initiated its creation).

Its job is to set the correct trace before an asynchronous callback is invoked. We use the global trace object when we instrument the core http methods to share data between incoming and outgoing requests.

More importantly, we also use it to associate all children async operations to our incoming HTTP request (this will become clearer in the example below).

So async_hooks provides the glue. Now we just have to instrument http.createServer to wrap the HTTP listener in order to create a new trace and collect HTTP headers for each incoming request:

We also instrument http.request to inject the headers on outbound calls.

Lifecycle of a request: An example

To understand how it’s all working together let’s look at the full lifecyle of a request.

As a new request comes in, a new trace object is created by the wrapped HTTP listener. That trace is available on a global tracer object and is associated with the current async id:

Now we can use the hooks providing by async_hooks to provide the correct trace object to the current execution context.

The critical part is in the init function. When invoked it has the asyncId of the current operation but also the triggerAsyncId which is the id of the operation that triggered its creation. Because we link the trace object to the asyncId of the incoming request, we can now link the same trace object to all children async operations. Therefore, they will all share the same trace object (and be able to share headers’ values).

The before function then simply sets the current trace to the current execution context (re-using the asyncId). This is how our wrapped http.request can access the headers’ values (through the tracer.currentTrace ref)

Instrumenting services

So now we have to instrument our services. For that we simply add the hpropagate module and require it (as early as possible in our apps, at the very least before requiring express)

Now, x-correlation-id headers will be propagated (or set with a new UUID) and also added to the HTTP response.

The x-variant-id if present, will be propagated to outbound calls enabling us to deploy and route to multiple versions of our services.

Added bonus: Istio tracing

As the tracing headers are now also propagated, we can get, out of the box with Istio, access to traces:

It’s open source

We have only tested hpropagate with our Express apps and our clients use request-promise-native or axios to make HTTP calls.

You can also customise the list of headers to propagate but there is plenty of scope for improvements and additions.

Please give it a try and as always feedback/PR are more than welcome!