Setting up Logging in Python and Flask Application the right way

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. — Brian Kernighan, co-creator of the C language

But a good log file can make debugging easier and enjoyable. 😄

But setting up a good logging and alerting pipeline is an Art, because it needs to evolve as your application grows and it should scale.

The default Python logging is not helpful either. It either gives too much information or too little. Documentation is all over the place. Along with there are multiple ways that you can use to set it up.

If some other module is used in application which you will, then it’s a nightmare if you want some of those module logs to be printed too.

Different types of handlers provided by Python

• StreamHandler
• FileHandler
• NullHandler
• WatchedFileHandler
• BaseRotatingHandler
• RotatingFileHandler
• TimedRotatingFileHandler
• SocketHandler
• DatagramHandler
• SysLogHandler
• NTEventLogHandler
• SMTPHandler
• MemoryHandler
• HTTPHandler
• QueueHandler
• QueueListener

Python logging levels

DEBUG

Detailed information, typically of interest only when diagnosing problems.

INFO

Confirmation that things are working as expected.

WARNING

An indication that something unexpected happened, or indicative of some problem in the near future (e.g. ‘disk space low’). The software is still working as expected.

ERROR

Due to a more serious problem, the software has not been able to perform some function.

CRITICAL

A serious error, indicating that the program itself may be unable to continue running.

Current Implementation

1. Real time error notifications should be sent to Slack channel or Email (Getting to know that there is an error is the first thing that needs to be done before you can debug an error).
2. Full stacktrace should be saved in error files on the host that can be sent asynchronously to some centralized logging (Elasticsearch).
3. access log should be saved in log files for audit purposes.
4. Files must be rotated and deleted accordingly when logs get huge.
5. In debug mode everything must be printed in console.
6. In production only logs that are absolutely necessary should be printed to console.
7. Different formatters for different uses cases should be used.
8. And of course it should be easy to implement.

Setting up Logging in Python and Flask Application the right way

In the above implementation you can change handlers to switch between different channels.

References

https://gist.github.com/deepaksood619/99e790959f5eba6ba0815e056a8067d7

Setting up logging for Gunicorn in a Flask application using Python's dictConfig

Logging HOWTO - Python 3.8.2 documentation

Logging Cookbook - Python 3.8.2 documentation