Enabling Event Traceability Between Java and Other Language Modules (Elastic APM)

In the dynamic realm of software development, establishing traceability between modules written in various languages often resembles navigating a labyrinth blindfolded. This challenge is particularly pronounced when dealing with Kafka events, where maintaining traceability becomes essential for facilitating seamless communication across modules. While Java modules enjoy inherent cross-module traceability through their reliable APM agent in the case of Elastic APM, extending this seamless integration to other languages proves to be a formidable task — akin to herding cats.

The Challenge

For Elastic APM users, the absence of traceability on the APM dashboard when Kafka events traverse between languages — be it from Java to Node.js or Python to Java — is conspicuous. While Java modules benefit from their APM agent appending special headers to events, other language modules are left in the dark. Attempting to utilize Java headers elsewhere without proper context is akin to deciphering a secret code without the requisite decoder ring.

Crafting the Solution

A direct approach to addressing this challenge involves implementing the custom encoding and decoding logic utilized by Java’s APM agent in other languages.

Upon scrutinising Elastic’s GitHub repository, it becomes apparent that they employ the proprietary ‘elasticapmtraceparent’ header for trace-context propagation, based on the W3C Binary Trace Context. The Java Elastic APM agent automatically includes this encoded header in Kafka events and decodes it upon message consumption. Detailed information can be found here.

The Code

Let’s tackle the challenge using TypeScript.

Parsing the Java headers

Inspecting a message originating from a Java module reveals the presence of the ‘elasticapmtraceparent’ header:

{
	"elasticapmtraceparent": "\u0000\u0000����M��2�\u0017H[�\u0002:�\u0001ú���]ٵ\u0002\u0001"
}

Attempting to parse this with conventional methods yields unintelligible output. Let’s create a custom parsing method armed with insights gleaned from documentation and code analysis:

function encodeDigits(
  buffer: Buffer,
  start: number,
  length: number,
  traceparentHeader: string[],
  offset: number,
) {
  for (let i = 0; i < length; i++) {
    const hex = buffer[start + i].toString(16).padStart(2, '0');
    traceparentHeader[offset + i * 2] = hex[0];
    traceparentHeader[offset + i * 2 + 1] = hex[1];
  }
}

// Custom parsing method for Kafka headers
const parseKafkaHeader = (buffer: Buffer): string | null => {
  if (buffer.length < 29) {
    console.warn('The buffer has to be at least 29 bytes long, but is not');
    return null;
  }
  try {
    const traceparentHeader: string[] = new Array(55).fill('0');
    traceparentHeader[2] = traceparentHeader[35] = traceparentHeader[52] = '-';
    encodeDigits(buffer, 2, 16, traceparentHeader, 3); // encode 16 byte traceid
    encodeDigits(buffer, 19, 8, traceparentHeader, 36); // encode 8 byte parentid
    const flags = buffer[28].toString(16).padStart(2, '0'); // flags
    traceparentHeader[53] = flags[0];
    traceparentHeader[54] = flags[1];
    return traceparentHeader.join('');
  } catch (e) {
    console.warn('Failed to parse legacy buffer', e);
  }
  return null;
};

Using our function with our ‘elasticapmtraceparent’, we get the following output.

00-efbfbdefbfbdefbfbdefbfbd4defbfbd-bfbd32efbfbd1748-ef

Now that’s something we can work with!

Creating our own headers

To facilitate seamless communication between Java and other language modules, we need to prepare our own ‘elasticapmtraceparent’ header. Let’s convert the parsed traceparent back into a buffer format suitable for sharing across modules:

function decodeDigits(
  input: string,
  start: number,
  length: number,
  buffer: number[],
  offset: number
) {
  const hex = input.substring(start, start + length);
  for (let i = 0; i < hex.length; i += 2) {
    buffer[offset + i / 2] = parseInt(hex.substring(i, i + 2), 16);
  }
}

// Custom encoding method for traceparent header
export const prepareHeader = (
  traceparentHeader: string
): Buffer | null => {
  try {
    const buffer: number[] = new Array(29).fill(0);
    buffer[0] = 0; //version
    buffer[1] = 0; //trace-id field-id
    decodeAscii(traceparentHeader, 3, 32, buffer, 2); //read 16 byte traceid
    buffer[18] = 1; //parent-id field-id
    decodeAscii(traceparentHeader, 36, 16, buffer, 19); //read 8 byte parentid
    buffer[27] = 2; //flags field-id
    decodeAscii(traceparentHeader, 53, 2, buffer, 28); //flags
    return Buffer.from(buffer);
  } catch (e) {
    console.warn("Failed to encode for legacy buffer", e);
  }
  return null;
};

Using the functions in code

With the parsing and encoding functions in place, we can seamlessly integrate them into our codebase to enable traceability:

import apm from "elastic-apm-node";

// Parse Kafka header and start transaction
const traceparent = parseKafkaHeader(
  message.headers.elasticapmtraceparent as Buffer
);

// If traceparent exists (is not null) we use the traceparent
traceparent
  ? apm.startTransaction(transactionName, { childOf: traceparent })
  : apm.startTransaction(transactionName);

And that’s it!

Conclusion

By implementing custom encoding and decoding logic, we’ve successfully harnessed the ‘elasticapmtraceparent’ header, transforming it into a universal translator for traceability across modules. This approach can be replicated in any language, offering a standardized solution for seamless cross-module tracing.

Thank you for embarking on this journey with us. Until next time, happy coding!