Three ways to close buffer for stdout, stdin, stderr in Python

A quick guide to set the unbuffered mode

@joshrh19

What is Buffer

I already explain this concept in the Python file scenario in the below article.

Understand the Buffer Policy in Python

Here is a little different. The scenario is for standard input/output in Python. But the basic idea of the buffer is the same. You can think the buffer is a middle layer between what the program has done and what the program has shown to us, and buffer policy decides what we can see.

When and Why we want to close buffer

In the above article, we talked about the advantages of buffer. It can reduce the number of hard disk writes. But when the scenario comes to the interaction program, the disadvantage also comes up. We cannot see the result in real time.

If you run the below Python script, you will find all number is shown together. This is because of the buffer policy.

In [1]: import time
In [2]: for i in range(10):
   ...:     print(i, end=" ")
   ...:     time.sleep(.2)
   ...: print()
   ...:
0 1 2 3 4 5 6 7 8 9

But if we close buffer with flash, we can see the number shown up every 2 seconds. So we can see the result in real time.

In [3]: for i in range(10):
   ...:     print(i, end=" ", flush=True)
   ...:     time.sleep(.2)
   ...: print()
   ...:
0 1 2 3 4 5 6 7 8 9

How to set unbuffer?

Method 1: Command line

You can run the python script by python -u option

-u     : unbuffered binary stdout and stderr, stdin always buffered;
         also PYTHONUNBUFFERED=x

Method 2: flush

If you just use the print and not modify too many parts, you can add flush in each print statement.

In [3]: for i in range(10):
   ...:     print(i, end=" ", flush=True)
   ...:     time.sleep(.2)
   ...: print()
   ...:
0 1 2 3 4 5 6 7 8 9

Method 3: change the default setting in code

We set the buffering as 1, this is the line-buffering. It buffers every line. Available in text file mode.

import os,sys
import time
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', buffering=1)
sys.stderr = os.fdopen(sys.stderr.fileno(), 'w', buffering=1)
sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', buffering=1)
print('out')
line = sys.stdin.readline()
for i in line:
    print(i)
    time.sleep(.2)

If we set buffering=0, it means the unbuffered. But it is only available in binary file mode (w->wb). We can use the line buffer to achieve a likely effect as the unbuffering policy.

Check out my other posts on Medium with a categorized view!
GitHub: BrambleXu
LinkedIn: Xu Liang
Blog: BrambleXu