Crafting Your Custom Logger in Python: A Step-by-Step Guide

Python’s built-in logging module is a powerful tool for tracking and understanding the flow of your application. However, there are situations where a custom logger tailored to your specific needs can add significant value. In this article, we’ll explore the process of creating a custom logger in Python, providing you with more control and flexibility over your application’s logging mechanism.

Why a Custom Logger?

While Python’s default logging capabilities cover a wide range of scenarios, a custom logger allows you to fine-tune logging behavior to meet your project’s unique requirements. Whether you need to format log messages in a specific way, filter messages based on criteria, or integrate with an external logging service, a custom logger can be your go-to solution.

This one is a basic version where I am overriding a method. Basic concepts of object-oriented programming. I can use the same behavior of info from the module library and add some functionalities. I am setting the value extra in the new Logger class, so I don’t need to pass every time.
We can override in the same way the methods debug, error, exception, etc.

import logging

class Logger(logging.Logger):
  def __init__(self, name, level=logging.NOTSET):
    super().__init__(name, level)
    self.extra_info = None

  def info(self, msg, *args, xtra=None, **kwargs):
    extra_info = xtra if xtra is not None else self.extra_info
    super().info(msg, *args, extra=extra_info, **kwargs)

Storing runtime information within the Logger class proves beneficial, eliminating the need to pass it explicitly each time the logger is utilized.

Add Databricks information to the logger

If we use this logger in Databricks we can add in this way information related to the single runtime:

import logging
from datetime import datetime

class Logger(logging.Logger):
  def __init__(self, name, level=logging.NOTSET):
    super().__init__(name, level)
    self.extra_info = None
    self.dbutils_info()

  def info(self, msg, *args, xtra=None, **kwargs):
    extra_info = xtra if xtra is not None else self.extra_info
    super().info(msg, *args, extra=extra_info, **kwargs)

  def dbutils_info(self):
      # Get the notebook path
      self.notebook_path = dbutils.notebook.entry_point.getDbutils().notebook().getContext().notebookPath().getOrElse(None)
      # Extract the notebook name from the path
      self.notebook_name = self.notebook_path.split("/")[-1]
      self.notebook_id = json.loads(dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson())["tags"]["notebookId"]
      
      self.extra_exc = {
          'notebook_name': self.notebook_name,
          'notebook_id': self.notebook_id
      }

Add telemetry to our logger: opencensus and Azure

Another interesting use is additional functionality related to distributed tracing and integrates with Azure Application Insights for logging and tracing purposes. This code utilizes the OpenCensus library for tracing and Azure-specific exporters for logging. This method will be deprecated in September 2024, more info here.

from opencensus.trace.tracer import Tracer
from opencensus.ext.azure.log_exporter import AzureLogHandler
from opencensus.ext.azure.trace_exporter import AzureExporter
from opencensus.trace.samplers import ProbabilitySampler
import logging

class Logger(logging.Logger):
  def __init__(self, name, level=logging.NOTSET):
    super().__init__(name, level)
    self.extra_info = None

    # Add handlers (e.g., ConsoleHandler, FileHandler, etc.)
    handler = logging.StreamHandler()
    formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
    handler.setFormatter(formatter)
    self.addHandler(handler)
    
    self.addHandler(
      AzureLogHandler(
        connection_string=f'InstrumentationKey={appinsight_key}'
        )
    )

    self.tracer = Tracer(
      exporter=AzureExporter(
          connection_string=f'InstrumentationKey={appinsight_key};IngestionEndpoint=https://{region}.in.applicationinsights.azure.com/'),
          sampler=ProbabilitySampler(1.0),
    )

  def info(self, msg, *args, xtra=None, **kwargs):
    extra_info = xtra if xtra is not None else self.extra_info
    super().info(msg, *args, extra=extra_info, **kwargs)

Usage example

#Instantiate the class
logger = Logger('logger_name',appinsight_key, level=logging.DEBUG) # level=logging.INFO

logger.info("This is an info message.") # 2024–01–09 02:02:03,042 - INFO - This is an info message.
logger.error("This is an error message.") # 2024–01–09 02:02:03,042 - ERROR - This is an error message.
logger.exception("This is an exception message.")
logger.debug("This is an exception message.") # you need to set up level=logging.DEBUG

Further Customizations

Feel free to extend and adapt your custom logger based on your project’s needs. You can explore features like custom filters, and log levels, or even integrate with third-party logging services.
You could add a method in the error method to raise a message to your slack or Teams channel

def error(self, msg):
  super().error(msg)
  self.slack.send(msg)

Conclusion

Creating a custom logger in Python provides you with the flexibility to tailor your logging experience to match your project’s requirements. By following the steps outlined in this guide, you can easily craft a logger that suits your needs, enhancing the overall manageability and understanding of your application’s behavior.

Happy logging! 📜🐍