Python — Understanding Advanced Concepts with Ease: Day 10 (Logging)

Photo by David Clode on Unsplash

Welcome back to Day 10 of our Python — Understanding Advanced Concepts with Ease series! Let’s learn something new today.

Logging in Python

Python’s logging module provides a flexible framework for recording log messages from Python programs.

Let’s look into a basic example:

import logging

# Create a logger with a unique name
logger = logging.getLogger('example_logger')

# Configure logger to log messages of INFO level and above
logger.setLevel(logging.INFO)

# Define a handler to specify where the log messages should be sent
handler = logging.StreamHandler()  # Log messages to console
logger.addHandler(handler)

# Log messages
logger.info('This is a info message') 

Now let’s dive into the key components of Python Logging:

• Log Levels
• Loggers
• Handlers
• Formatters
• Filters

Log Levels

In Python logging, log levels are used to categorize log messages based on their severity or importance. Each log level corresponds to a specific numeric value, and log messages with a level equal to or higher than the configured level will be processed and logged. Python’s logging module defines the following standard log levels:

1. DEBUG: Detailed information, typically used for debugging purposes.
2. INFO: General information about the application’s operation.
3. WARNING: Indicates a potential issue or situation that may require attention.
4. ERROR: Indicates an error or unexpected condition that the application may recover from.
5. CRITICAL: Indicates a critical error or condition that requires immediate attention.

The below example shows how you can log messages at different levels:

import logging

# Create a logger
logger = logging.getLogger('log_level_change')

# Configure logger to log messages of DEBUG level and above
logger.setLevel(logging.DEBUG)

# Define a StreamHandler to log messages to the console
handler = logging.StreamHandler() 
logger.addHandler(handler)

# Log messages at different levels
logger.debug('This is a debug message')    # Shown
logger.info('This is an info message')      # Shown
logger.warning('This is a warning message') # Shown
logger.error('This is an error message')    # Shown
logger.critical('This is a critical message') # Shown

# Change the log level to WARNING
logger.setLevel(logging.WARNING)

# Log messages again
logger.debug('This is a debug message')    # Not shown (below WARNING level)
logger.info('This is an info message')      # Not shown (below WARNING level)
logger.warning('This is a warning message') # Shown
logger.error('This is an error message')    # Shown
logger.critical('This is a critical message') # Shown

As you can see from the above example, you can also change the log level using logger.setLevel in between your code.

Loggers

In Python logging, loggers are the primary components responsible for generating log messages. They serve as the entry points through which log messages flow into the logging system. Loggers are organized in a hierarchical namespace, allowing developers to categorize and manage log messages effectively.

Let’s look at an example:

import logging

# Create a logger with a unique name
logger = logging.getLogger('example_logger')

# Configure logger to log messages of INFO level and above
logger.setLevel(logging.INFO)

# Define a handler to specify where the log messages should be sent
handler = logging.StreamHandler()  # Log messages to console
logger.addHandler(handler)

# Log messages
logger.debug('This is a debug message')    # Not shown (below INFO level)
logger.info('This is an info message')      # Shown
logger.warning('This is a warning message') # Shown
logger.error('This is an error message')    # Shown
logger.critical('This is a critical message') # Shown

In the above code:

• We create a logger named example_logger using logging.getLogger().
• The logger’s log level is set to INFO using logger.setLevel(logging.INFO), meaning it will handle log messages of INFO level and above.

Handlers

Handlers are responsible for defining where the log messages should be sent. They determine the output destinations for log messages, such as the console, files, sockets, or external services. Python provides a variety of built-in handlers to cater to different logging requirements.

Let’s look at an example:

import logging

# Create a logger
logger = logging.getLogger('stream_logger')

# Configure logger to log messages of INFO level and above
logger.setLevel(logging.INFO)

# Define a StreamHandler to log messages to the console
handler = logging.StreamHandler() 
logger.addHandler(handler)

# Log messages
logger.info('This is an info message logged to the console')
logger.error('This is an error message logged to the console')

In this example:

• We create a logger named stream_logger and set its log level to INFO.
• A StreamHandler is defined, which sends log messages to the console.
• We attach the StreamHandler to the logger using logger.addHandler(handler).
• Log messages of INFO level and above are then logged to the console.

RotatingFileHandler:

The RotatingFileHandler is a type of logging handler provided by Python’s logging module. It is designed to manage log files by rotating them based on predefined criteria, such as file size or a specified interval. This handler is particularly useful for applications that generate a large volume of log messages, as it helps prevent log files from growing indefinitely and consuming excessive disk space.

• File Rotation: When a log file reaches a certain size or age, as specified by the developer, the RotatingFileHandler automatically rotates the log files. Rotation involves closing the current log file and renaming it with an extension indicating its sequence number or timestamp.
• Backup Files: The handler can be configured to keep a certain number of backup files. Once the maximum number of backup files is reached, the oldest backup file is deleted to make room for the new log file. This ensures that the disk space used by log files remains within a manageable limit.
• Usage: The RotatingFileHandler is typically used in scenarios where continuous logging is necessary, such as in web servers, long-running services, or batch processing applications. It helps maintain a structured and manageable log file system, making it easier to debug issues and analyze application behavior.

import logging
from logging.handlers import RotatingFileHandler

# Create a logger
logger = logging.getLogger('rotating_file_logger')

# Configure logger to log messages of INFO level and above
logger.setLevel(logging.INFO)

# Define a RotatingFileHandler to log messages to a rotating set of files
handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount=3) 
logger.addHandler(handler)

# Log messages
for i in range(10):
    logger.info(f'This is info message {i}')

In this example:

• We create a logger named rotating_file_logger and set its log level to INFO.
• A RotatingFileHandler is defined with the filename app.log, specifying a maximum file size of 10,000 bytes (maxBytes=10000) and keeping up to 3 backup files (backupCount=3).
• The RotatingFileHandler is then added to the logger using logger.addHandler(handler).
• Ten log messages are then logged, causing the log file to rotate when the maximum file size is reached.

Formatter

In Python logging, formatters are used to specify the layout and structure of log messages. They allow developers to customize the appearance of log entries by defining the format of timestamps, log levels, log messages, and additional metadata. Python’s logging module provides a variety of built-in formatters, and developers can also create custom formatters to suit their specific logging requirements.

Let’s look at an example:

import logging

# Create a logger
logger = logging.getLogger('basic_formatter')

# Configure logger to log messages of INFO level and above
logger.setLevel(logging.INFO)

# Define a basic formatter
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')

# Define a StreamHandler with the formatter
handler = logging.StreamHandler()
handler.setFormatter(formatter)
logger.addHandler(handler)

# Log messages
logger.info('This is an info message')
logger.error('This is an error message')

Here, a basic formatter is defined using the logging.Formatter class. The formatter specifies the format of log messages, including the timestamp (%(asctime)s), log level (%(levelname)s), and the actual message (%(message)s). Then, a StreamHandler is created to log messages to the console, and the formatter is set on the handler using handler.setFormatter(formatter).

Log messages of INFO level and above are then logged with the specified format.

Defining custom formatter:

import logging

# Create a logger
logger = logging.getLogger('custom_formatter')

# Configure logger to log messages of DEBUG level and above
logger.setLevel(logging.DEBUG)

# Define a custom formatter
class CustomFormatter(logging.Formatter):
    def format(self, record):
        formatted_message = f'[{record.levelname}] {record.msg}'
        return formatted_message

# Define a StreamHandler with the custom formatter
handler = logging.StreamHandler()
formatter = CustomFormatter()
handler.setFormatter(formatter)
logger.addHandler(handler)

# Log messages
logger.debug('This is a debug message')
logger.info('This is an info message')

Filters

In Python logging, filters provide a mechanism for selectively processing log messages based on predefined criteria. Filters allow you to include or exclude log messages dynamically based on attributes such as log level, logger name, or custom conditions. By using filters, you can fine-tune the behavior of logging handlers to meet specific requirements.

Here’s how filters work in Python logging:

1. Filter Class: Filters in Python logging are implemented using filter classes. These classes must inherit from the logging.Filter base class and override the filter method. The filter method takes a log record as input and returns a boolean value indicating whether the record should be processed by the logging handler.
2. Configuring Filters: Filters can be attached to logging handlers or loggers using the addFilter method. Multiple filters can be added to a single handler or logger, and they are applied in the order they were added. When a log message is processed, each filter's filter method is called sequentially, and the message is passed to the next filter only if the previous filter returns True.
3. Custom Filters: In addition to built-in filters provided by the logging module, you can create custom filters tailored to your specific requirements. Custom filters allow you to implement complex logic for filtering log messages based on application-specific criteria.

Let’s look at an example:

import logging
import sys

logger = logging.getLogger("module_name")

class NoParsingFilter(logging.Filter):
    def filter(self, record):
        return not record.getMessage().startswith("parsing")

def function_a():
    logger.debug("a message")

def function_b():
    logger.debug("parsing another message")

if __name__ == "__main__":
    logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)
    logger.addFilter(NoParsingFilter())

    function_a()
    function_b()

Output:

DEBUG:module_name:a message

Conclusion

Congratulations! Hope you learnt something new today. See you on Day 11.

Happy coding!