What you need to know before creating your first OpenTelemetry pipeline for tracing
Some time ago in adidas we had to start thinking about creating our own tracing platform for our observability scope. My colleague Juan Luis Gonzalez made a deeper dissertation about this process and which is the current context of this platform. You can read more about in this Medium blog entry.
When we started to think about the different possibilities that we could use to create our own tracing platform, we decided with no doubt to start to work with the OpenTelemetry standard for our end to end solution (instrumentation, collection, processing…).
As we talk about the processing of the traces in the pipelines we cannot help but think of sampling.
Inside the sampling we have two different parts in the process, the head sampling and the tail sampling. And, after explaining some concepts regarding the sampling and the processing in general, we will talk about our concrete case.
Tail sampling vs Head sampling
Regarding tracing, sampling is the process by which some traces are selected and others are discarded.
This process can be arbitrary when we apply a probabilistic sampling for example, or we can decide which kind of traces we want to see always in our backend and which ones not.
Head sampling
The head sampling is the process of sampling that we are going to do in the application level, regardless if we are using auto or manual instrumentation.
We can decide in this level if we always sample, if never sample, if we set a probability of sampling, or if we are going to sample all the traces that are sampled by the parent process who’s calling this application.
Tail sampling
On the other hand, we have the tail sampling, which is applied in the OpenTelemetry collectors. There we have three main different parts: receivers, processors and exporters.
The tail sampling rules are applied in the processor part. The behaviour is quite similar to the head sampling but with a couple of big differences:
- We can apply it multiple times, i.e. we can have multiple collectors with tail sampling and even multiple tail sampling policies in the same collector.
- There are many more policy possibilities to be applied in tail sampling than in head sampling. And we can even combine them to make more complex policies for our pipelines.
Why not both?
Once we have seen the differences between head and tail sampling we can discuss the purpose of both of them.
The head sampling is meant to be an initial discarding process of some traces that we are sure we are not going to be interested at the application level.
In the opposite side we have the tail sampling, which is happening in the collectors so it can be reached by different applications, we may want to discard some traces depending on the source application or depending on the attributes of the span related to the application where it is coming from.
The best solution might be a mix of both, head and tail sampling in order to get a great sampling process.
To know more in detail the behaviour of the head and tail sampling and how to combine them, you can follow the documentation in the Opentelemetry site.
What we needed for our cases
Regarding our concrete cases we found some issues that were important for us to take into account and to have them under control.
Limit number of traces
Since in a very initial scenario the teams were not very familiar with the pipelines configurations, we were searching a good way to manage the amount of traces that we were going to send to our backend. A very usual way to manage it is with the probabilistic sampling so we tried that solution.
When we use the probabilistic sampling we just set a percentage, and this will be the the percentage of the traces that are going to reach the backend, so if we set a 0.5 we are going to send the half of the traces that we are generating and have been processed by the previous pipelines.
But, when it comes to traces we can find a problem with this strategy, since we may not always get the whole trace in the same collector and if we are using a probabilistic sampling, eventually we might have a problem of consistency with incomplete traces, because we are discarding some spans in a collector that are not being discarded in the other pipelines.
On the other hand, we find out that not all the use cases for the tracing have the same criticality, and they probably are not going to need the same amount of traces in the backend. Some of them even may need the full set of traces available in order to track the business properly.
So after seeing all this stuff we have to consider to treat each pipeline in a different and specific way and pay attention on the distribution of the spans that are being part of the trace, because they may come from different sources and they may be processed by different collectors.
Furthermore, we can not provide “free for all” traces, so we have to limit the number of traces that are going to be sent by every collector based on a sizes system. In the same way, we set the probabilistic policy to sample the traces we can set a policy that sets the maximum amount of traces that are going to be processed every certain time and that is how we do it.
Data Quality
When we are talking about tracing we can set attributes for the spans making up the trace so that we can identify them, and then, we can query and filter them from the backend visualisation point of view.
Having stated that we can not set all the attributes in the same part of the process, some of them should be set in the agent part (application side) and others should be set in the collectors side (next to the backend). This is because we may use the the same collector for multiple applications or services and we want to differentiate the spans depending on where they are coming from (we set attributes on agent side) but we can have also the same properties for different applications (we set attributes on collector side).
Let’s say we have a collector for all the non-production deployments of one application, so we want to query the traces based on the environment they are coming from, in that case, we should set the environment attribute on the agent side (for instance env: [dev|stg|playground]). On the other hand we may have an application name involving multiple services that are sending the spans of the traces to the same collector, then we should add the application name attribute in the collector side.
How to set the pipelines
Now, let’s see some possible configurations for the collector to set some of the options we’ve been talking about previously.
Tail sampling
In the following example we can see how to set a tail_sampling policy to have in our backend only the traces that have at least one span that have an STATUS CODE of OK and also the latency is greater than 5000 ms:
processors:
tail_sampling:
decision_wait: 10s
num_traces: 100
expected_new_traces_per_sec: 10
policies:
[{
name: and-policy-1,
type: and,
and: {
and_sub_policy:
[
{
name: test-and-policy-1,
type: latency,
latency: {threshold_ms: 5000}
},
{
name: test-and-policy-2,
type: status_code,
status_code: {status_codes: [OK, UNSET]}
},
]
}
}
]As you can see there is a policy whose type is “and” whose usage is to create a policy formed by two (or more) other policies, so that both conditions have to be satisfied for the trace to be sampled.
Another example is the following:
tail_sampling:
decision_wait: 10s
policies:
[
{
name: default-tail-sampling-probabilistic,
type: probabilistic,
probabilistic: {sampling_percentage: 10}
},
{
name: actuator-tail-sampling-policy,
type: string_attribute,
string_attribute: {key: http.target, values: [\/actuator\/prometheus], enabled_regex_matching: true, invert_match: true}
},
]In this example we set a policy to drop all the traces related to the call of our application to the Prometheus actuator and also a policy to get only the 10% of the traces that we are receiving in our collector.
Tail sampling is a trace based process, which means that if one span of a trace meets the condition or conditions the whole trace will be sampled, otherwise not.
Filter processor
In contrast to tail sampling, filter processor is span based. This means that only the spans meeting the criteria will be in the next step of the pipeline. That can generate incomplete spans, so we have to be careful with the conditions that we define.
One example of filter processor is the following:
processors:
filter/spans:
spans:
include:
match_type: strict
services:
- inventory-service
exclude:
match_type: regexp
services:
- chart-service
- orders-service
span_names:
- auth
- auth/login
attributes:
- key: container.name
value: (chart-service-cont | orders-service-cont)Here we can see two parts, the exclude and the include. In the include we define that every span with the inventory-service as service name will be kept and the exclude we define that every span that meets the criteria define in the different sections.
Example of a whole pipeline
Now let’s define a whole pipeline with the receivers, processors and exporters. (For more info about all this available configurations for our pipelines we can go here)
receivers:
otlp:
protocols:
grpc:
http:
processors:
tail_sampling:
decision_wait: 10s
policies:
[
{
name: default-tail-sampling-probabilistic,
type: probabilistic,
probabilistic: {sampling_percentage: 10}
},
{
name: actuator-tail-sampling-policy,
type: string_attribute,
string_attribute: {key: http.target, values: [\/actuator\/prometheus], enabled_regex_matching: true, invert_match: true}
},
]
filter/ottl:
spans:
include:
match_type: strict
services:
- inventory-service
exclude:
match_type: regexp
services:
- chart-service
- orders-service
span_names:
- auth
- auth/login
attributes:
- key: container.name
value: (chart-service-cont | orders-service-cont)
exporters:
otlp:
endpoint: your-next-collector:4317
tls:
insecure: true #Here we should include our certificates if necessary
service:
pipelines:
traces:
receivers: [otlp]
processors: [filter/ottl, tail_sampling]
exporters: [otlp]What else do we have to take into account?
We’ve been talking about the theory and creating our own pipeline using the filter processor and the tail sampling, but I would like to comment some points that we have to consider before to start to deploy our pipelines.
Versioning
One problem that we found when we started with the filter processing is that this feature has not been enabled since the beginning, and the definition of the configuration has changed throughout the different versions of the opentelemetry-collector-contrib. Due to that we have to be careful to use the right version and also check the changelog in order to use the right definition of the different features.
Load Balancing
If we are using the load balancing in the exporter definition we have to take something into account. Since the version 0.77 we can balance the load of our pipelines by the Kubernetes service name. Previous to this version we had a feature whose name was dns and it could lead us to make a mistake. Let’s see both options:
resolver:
dns:
hostname: backend-1
port: 4317resolver:
k8s:
service: my-collector-service-name.my-namespace
ports:
- 4317The second option is the right one when we are running our collectors in Kubernetes. In order to deploy the collector we have to deploy additional Kubernetes components as it is explained here.
Conclusions
The OpenTelemetry environment is something that is constantly changing in order to offer more and better features.
The community is contributing with amazing features in short term updates since it is providing the standards for the present and future of the tracing.
Here we have given a view on some details we have found out in our case that can help for yours, but the current available number of features is quite high, so I encourage you to research and see which of them can help in your concrete cases.
The views, thoughts, and opinions expressed in the text belong solely to the author, and do not represent the opinion, strategy or goals of the author’s employer, organization, committee or any other group or individual.
