How to save both ‘logs’ and ‘print’ statement outputs in the same file using Python?

Logs and Python

While doing one experiment, I needed to save the logs and the print statement outputs, in the same log file. Both the logs and print statement outputs were being printed in the terminal, and no file was being saved. While it is fairly easy to use the Python logging library to instantiate a StreamHandler or FileHandler with the logger, redirecting the print statement outputs to the same file was not easy, and requires expertise I didn’t have. After searching the web for a while, I came across two resources, one on the blog Electric Monk, and another on StackOverflow. While I later found out, that there are other resources of a similar nature, but combining these two seemed to work for me. Hence, in this article, I am going to present the piece that worked for me.

First, we will define a wrapper for the Python logger, named StreamToLogger.

import logging
import sys


class StreamToLogger(object):
    """
    Fake file-like stream object that redirects writes to a logger instance.
    """
    def __init__(self, logger, log_level=logging.INFO):
        self.logger = logger
        self.log_level = log_level
        self.linebuf = ''

    def write(self, buf):
        temp_linebuf = self.linebuf + buf
        self.linebuf = ''
        for line in temp_linebuf.splitlines(True):
            if line[-1] == '\n':
                self.logger.log(self.log_level, line.rstrip())
            else:
                self.linebuf += line

    def flush(self):
        if self.linebuf != '':
            self.logger.log(self.log_level, self.linebuf.rstrip())
        self.linebuf = ''

Then, we are going to instantiate a logger and set up the basic configurations we would like the logs to be in.

Logger Instantiation

logger = logging.getLogger('logs')

Instantiate FileHandler

logger.handlers = []
fh = logging.FileHandler(filename=f’{BASE_PATH}/out.log’, mode = ‘a’)
formatter = logging.Formatter(‘[%(levelname).1s] %(name)s >> “%(message)s”’)
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.setLevel("WARNING")

For redirecting the print statement outputs to the log file, we are going to wrap the logger in the StreamToLogger object

wrapped_logger = StreamToLogger(logger, logging.INFO)

Finally, we need to set STDOUT to the wrapped logger

sys.stdout = wrapped_logger

Now, we can save both logs and print statement outputs to a single file.

logger.info("Logger instantiated")
print("Print statement logging to file")

We can test if it works by printing the lines that have been appended to the text file. However, we first need to restore the functionality of sys.stdout.

sys.stdout = sys.__stdout__
with open(‘./log2.txt’) as f:
  l = f.readlines()
  for line in l:
    print(line)

sys.__stdout__ stores the original values of sys.stdout.

For IPython kernels, however, we need to run

oldstdout = sys.stdout

before assigning the wrapped_logger to sys.stdout. And then restore it before executing any print statements.

sys.stdout = oldstdout
with open('./log2.txt') as f:
  l = f.readlines()
  for line in l:
  print(line)

The output should look like this

[I] logs >> “Logger instantiated” 
[I] logs >> “Print statement logging to file”

A note from the StackOverflow answer:

From the io.TextIOWrapper docs: On output, if a newline is None, any '\n' characters written are translated to the system default line separator. By default sys.stdout.write() expects '\n' newlines and then translates them, so this is still cross-platform.

Readers can have a look at the complete example here

The code is also available in Colab.

There are other such codes available on the web to do the same job, like streamtologger. However, the flush() is not implemented which may cause issues. But the idea is the same.

Clap and share if you like this post.