Quick-Start to Integrating OpenTelemetry with FastAPI (Part 3)

Traces, and logs

Traces and logs are what I would consider absolutely mandatory if you integrate Otel with FastAPI. If you are not sending both of these to your otel backend, you should be buried in hot sand, have honey poured on you, and a box of red fiery ants released near you!!!

Traces allow you to establish your spans. I found it easy to think of spans as a transaction (yeah, in the database sense). This means:

1. A user requests for an ice-cream, they click on a button in your app.
2. Your app grabs the request, determines which backend services to invoke (e.g. shipping, pricing, shopping basket, invoicing, etc) and makes calls to them.
3. Each of the backend services complete their work and return with a response.
4. You collate the responses and provide some UI feedback to the user — “Thank you, your Surströmming flavoured ice-cream is ready and will be shipped to you by 2pm!”

A span would be a single transaction that encompasses all the steps above into a logical group. This makes it so much easier to have visibility into your FastAPI APIs as they react to calls.

Logs on the other hand are exactly what you think they are. Every time your code needs to spit out some status / message, such as:

• Compositing SQL statement with parameters (a = 55, b = “Yes”) for submission to back-end RDBMS.
• Submissing composited SQL statement.
• Query processed. 573 records returned.
• etc.

These are treated as logs. You should absolutely use the Python logging module (https://pypi.org/project/logging/) for this because it is nicely integrated into OpenTelemetry.

Recommendations

Here’s what I would recommend as a quick-start to otel + FastAPI:

• In your FastAPI startup .py file, usually called main.py, create a trace provider:

trace.set_tracer_provider(
       TracerProvider(
           resource=Resource.create({}),
       ),
    )

• Set a verbose format for your logs. You can tune it down later if you like. I like to include Python file name, line number, and function name in every log I generate. This makes troubleshooting so much easier. You can do this using logging’s Formatter objects. With OpenTelemetry, you take it one step higher and use the OTEL_PYTHON_LOG_FORMAT variable.
• At the start of EVERY function in your Python code (whether it’s in your core FastAPI code, or helper libraries), do this:

with trace.get_tracer(api_name).start_as_current_span(sys._getframe().f_code.co_name) as process_api_span:
    <then insert your function code here>

It will come out looking very nice in your Otel backend when you are exploring your traces and spans. For example, this is what I see in NewRelic:

You get to see the amount of time each Python function takes to complete. execution. Very handy for performance tuning.

If you need more detailed breakdown, for example you want to time a for-loop, just do this:

with trace.get_tracer(api_name).start_as_current_span(sys._getframe().f_code.co_name) as process_api_span:
    <then insert your function code here>
    ...blablabl

    with trace.get_tracer(api_name).start_as_current_span("my-for-loop") as process_api_span:
         for num in range(1, 10000):
             print(num)

• As you saw in the previous parts of this article, make your Otel variables configurable (set them as environment variables). Do not hard code them in your Python source. Why? This makes it easier for change. For example, if your SaaS otel backend suffers an outage, you can quickly redirect all otel output to another region of that provider.
• As an extension of the previous bullet point, we use Hashicorp Vault and ArgoCD in your CI/CD pipeline. We don’t use Vault as a secret-store only. We use it as a config-store as well. We insert all the otel environment variables (even the not-so-secret ones) into Vault. ArgoCD then reads them from Vault and deploys the package to our Kubernetes clusters. This means it’s so super easy to switch to another otel SaaS region of our provider — we only need to modify one secret in Vault to affect a change across all our Kubernetes deployments where FastAPI is running. And what if an API key for our NewRelic instance is compromised? Easy! Just revoke it, issue a new one, and put the new API key in Vault. Here’s what it looks like in our kustomization.yaml file:

secretGenerator:
- name: me-otel-secrets-arrrrrr
  literals:
    - debug_log_otel_to_console=<path:kv-v2/data/prod/prod#debug_log_otel_to_console>
    - debug_log_otel_to_provider=<path:kv-v2/data/prod/prod#debug_log_otel_to_provider>
    - otel_endpoint_http_headers=<path:kv-v2/data/prod/prod#otel_endpoint_http_headers>
    - otel_endpoint_url=<path:kv-v2/data/prod/prod#otel_endpoint_url>
    - otel_python_log_correlation=<path:kv-v2/data/prod/prod#otel_python_log_correlation>
    - otel_python_log_format=<path:kv-v2/data/prod/prod#otel_python_log_format>
    - otel_python_log_level=<path:kv-v2/data/prod/prod#otel_python_log_level>
    - otel_resource_attributes=<path:kv-v2/data/prod/prod#otel_resource_attributes>

• Do not be afraid to log and create spans when you are first experimenting with OpenTelemetry + FastAPI. However, do becareful with the different log levels you use (info, warning, debug, critical, etc). They can either make your life a living nightmare or a walk-in-the-park when you are troubleshooting using your otel telemetry. Remember: the logging module can be configured to spit out logs are different levels depending on the verbosity you want. This may also impact your otel backend due to the volume of data being ingested.