Merging OpenTracing and OpenCensus: A Roadmap to Convergence

Authored by Ted Young, Pritam Shah and the “technical specification committee” (Carlos Alberto, Bogdan Drutu, Sergey Kanzhelev, and Yuri Shkuro).

Update: The merged project now has a name! http://opentelemetry.io

The short, short version

• We are creating a new, unified set of libraries and specifications for observability telemetry. It will merge the OpenTracing and OpenCensus projects, and provide a supported migration path.
• The Java reference implementation will be available on April 24th, and cross-language work will begin in earnest on May 8th, 2019. Check out a timeline for the merger here.
• By the September 2019, we plan to reach parity with existing projects for C#, Golang, Java, NodeJS, and Python. There is a lot to do, but we can make it if we parallelize. If you are interested in being part of this project, please sign up and tell us how you would like to contribute.
• When each language reaches parity, the corresponding OpenTracing and OpenCensus project will be sunset. This means that the old projects will be frozen, but the new project will continue to support existing instrumentation for two years, via a backwards compatibility bridge.

Project Overview

We’re merging! The high level goal is to synthesize the OpenTracing and OpenCensus projects into a single, unified project. The core of the new project will be a set of clean and well thought interfaces, and a canonical distribution of libraries implementing those interfaces, referred to as the SDK. Recommended standards for data and wire protocols, along with common pieces of infrastructure, complete the project.

The result will be a complete telemetry system — suitable for monitoring microservices and other types of modern, distributed systems — and will be compatible with most major OSS and commercial backends.

Major Milestones

4/24 The reference candidate is presented for review.

5/8 Teams form, and work begins in all languages.

5/20 Project officially launched at Kubecon Barcelona.

9/6 C#, Golang, Java, NodeJS, and Python reach parity with existing projects.

11/6 OpenTracing and OpenCensus projects are officially sunset.

11/20 Sunset party at Observability Summit, Kubecon San Diego.

Timeline to Convergence

The migration for each language involves building a production-ready SDK, instrumentation for popular libraries, documentation, CI, a backwards compatibility bridge, and closing down the associated OpenCensus and OpenTracing projects (“sunsetting”). We have set an ambitious goal of September 2019 as the target date for parity in C#, Golang, Java, NodeJS, and Python. If a language is not ready by that time, we will push back the sunset date until it is. But we prefer this not to happen.

When checking out the goals below, think about how you would like to get involved. Let us know by filling in the signup sheet, or saying hello on the OpenTracing and OpenCensus Gitter chat rooms.

Click here to view the timeline as an infographic.

Goal: First draft of Cross-Language Specification (due May 8th)

It is important that the project remain cohesive, even as we work in parallel on separate languages. The Cross-Language Specification provides this guidance for the project. It is written in prose, and ensures that we maintain a coherent system which feels familiar regardless of programming language.

The first draft of the x-lang spec requires the following:

• Definitions of common terminology.
• A model for describing distributed transactions, stats and metrics.
• Clarifications on important issues encountered during implementation.

This goal blocks all other work, and a first draft must be completed by May 8th.

Goal: First draft of Data Specification (due July 6th)

The Data Specification defines a common data format for tracing and metrics, so that data exported by all processes can be processed by the same telemetry infrastructure, regardless of how the process generated the data. This includes a data schema for the tracing model described in the Cross-Language spec. But it is also includes metadata definitions for common operations that tracing is used to capture, such as HTTP requests, errors, and database queries. These semantic conventions are an example.

The first draft is based on the current OpenCensus data format, and will contain the following:

• Data Schema which implements the cross-language specification
• Metadata definitions for common operations.
• JSON and Protobuf definitions.
• Reference client implementations.

Note that there is also a wire protocol for propagating traces in-band, which we would like to standardize on. The Trace-Context propagation format is being developed via the W3C, and can be found here.

Goal: Parity in all currently supported languages (due Sep 6th)

In order for the new project to replace the current ones, we must achieve parity with existing language ecosystems.

• Interface definitions for tracing, metrics, and context propagation, based on the Cross-Language specification.
• A production-ready SDK, which implement these interfaces and export Trace-Data. When available, the SDK will be created by porting the existing OpenCensus implementation.
• Instrumentation for popular libraries currently covered by OpenTracing and OpenCensus.

We also value backwards compatibility, and ensuring a smooth migration from the existing projects.

• The new SDK will contain an OpenTracing bridge, which implements the current OpenTracing interface. This allows legacy OpenTracing instrumentation to work together with new instrumentation in the same process, so users may migrate their instrumentation code incrementally.
• When the new SDK is ready, an upgrade plan for existing OpenCensus users will be available. Like with OpenTracing, legacy instrumentation will continue to work alongside new instrumentation.
• By November, both OpenTracing and OpenCensus will be sunset. No more commits will occur from this point on. Backwards compatibility with legacy instrumentation will be supported for two years.

It is a lot of work to create a best-in-class SDK for each language, and this is where we need the most help.

Goal: Core Documentation (due Sep 6th)

Documentation is critical to the success of any open source project. We really want documentation and learning to be a first-class component, and people who write documentation are considered primary committers to this project. Teaching developers how to observe their software correctly is a critical part of the impact we want to make on the world.

At minimum, we need the following to get started:

• Project Orientation
• Observability 101
• Getting Started
• Language Guides (one per language)

Writers of all backgrounds and skill levels are welcome! The new website is based on Hugo, written in markdown, and is very easy to contribute to.

Goal: Registry v1.0 (due July 6th)

Another critical component is the Registry, an improved version of the OpenTracing Registry.

• Easily find libraries, plugins, installers, and other components.
• Easily add and manage registry components.
• Understand what SDK features are available in each language.

If you’re interested in design, front end, and UX, this is a great project to be a part of.

Goal: Testing and release infrastructure (due Sep 6th)

We have a project wide commitment to building a quality testing and release pipeline, to ensure we continue to ship safe code we can rely on. If you care about testing, benchmarking, and release pipelines, please let us know. We will clearly mark the level of production readiness and the maturity of the testing infrastructure will be a major deciding factor.

Goal: Sunset OpenTracing and OpenCensus (due Nov 6th)

By September 6th, if the new project has reached parity, the existing projects will begin to be shut down. Two months after each language reaches parity, the corresponding OpenTracing and OpenCensus projects will be sunset. This means the following:

• The repositories will be frozen, and no more commits will be made.
• There will be a two year support plan for existing instrumentation.
• Users can upgrade to the new SDK while continuing to use existing instrumentation.
• It will be possible to migrate instrumentation over incrementally.

Get Involved

This is a huge project, and we need all hands on deck. If you’ve been curious to learn about observability, this is the perfect time to get involved!

• Fill in the sign up form to join a team and get directly involved.
• Contact us on the OpenTracing and OpenCensus Gitter channels.
• Check out the Java reference implementation currently under development.