Setting up Logging in Python and Flask Application the right way
Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. — Brian Kernighan, co-creator of the C language
But a good log file can make debugging easier and enjoyable. 😄
But setting up a good logging and alerting pipeline is an Art, because it needs to evolve as your application grows and it should scale.
The default Python
logging is not helpful either. It either gives too much information or too little. Documentation is all over the place. Along with there are multiple ways that you can use to set it up.
If some other module is used in application which you will, then it’s a nightmare if you want some of those module logs to be printed too.
Different types of handlers
provided by Python
- StreamHandler
- FileHandler
- NullHandler
- WatchedFileHandler
- BaseRotatingHandler
- RotatingFileHandler
- TimedRotatingFileHandler
- SocketHandler
- DatagramHandler
- SysLogHandler
- NTEventLogHandler
- SMTPHandler
- MemoryHandler
- HTTPHandler
- QueueHandler
- QueueListener
Python logging levels
DEBUG
Detailed information, typically of interest only when diagnosing problems.
INFO
Confirmation that things are working as expected.
WARNING
An indication that something unexpected happened, or indicative of some problem in the near future (e.g. ‘disk space low’). The software is still working as expected.
ERROR
Due to a more serious problem, the software has not been able to perform some function.
CRITICAL
A serious error, indicating that the program itself may be unable to continue running.
Current Implementation
- Real time error notifications should be sent to Slack channel or Email (Getting to know that there is an error is the first thing that needs to be done before you can debug an error).
- Full
stacktrace
should be saved in error files on the host that can be sent asynchronously to some centralized logging (Elasticsearch). access log
should be saved in log files for audit purposes.- Files must be rotated and deleted accordingly when logs get huge.
- In
debug
mode everything must be printed in console. - In
production
only logs that are absolutely necessary should be printed to console. - Different
formatters
for different uses cases should be used. - And of course it should be easy to implement.
In the above implementation you can change handlers
to switch between different channels.