Displaying real-time webcam stream in IPython at (relatively) high framerate
Working on my side project involving reading video stream using OpenCV in Python, I realized I could easily grab an image from webcam as Numpy array, modify it and then display the resulting stream using cv2.imshow(). OpenCV will create a window and push my frames there. However, this is not working in a IPython notebook.
I found few solutions to implement display functionality, but they all were very slow for a real-time processing (about 250 ms per frame). Here, I combine and modify these two examples to achieve ~10 times higher framerate:
1. Showing webcame image using OpenCV and matplotlib. Here previous frame is cleared from the screen using IPython.display.clear_output().
2. Minimal code for rendering a numpy array as an image in a Jupyter notebook in memory. It uses PIL to convert NumPy array to .PNG format in order to display it with IPython.display.display()
Both methods are relatively slow. The slowest part in the first one is the matplotlib rendering using matplotlib.pyplot.imshow() and the second one spend most of the time converting array data to PNG in PIL.Image.save().
But converting to PNG is not the fastest and only gives 2–3 FPS. If I use JPEG instead, frame rate goes up to 36 FPS, which good enough to work in the real-time.
#Use 'jpeg' instead of 'png' (~5 times faster)
def showarray(a, fmt='jpeg'):
f = StringIO()
PIL.Image.fromarray(a).save(f, fmt)
IPython.display.display(IPython.display.Image(data=f.getvalue()))
The main loop is given below, and is mostly a copy from [1] with modified call to display the NumPy array. The entire notebook is on GitHub, be sure to try it and follow for more updates!
cam = cv2.VideoCapture(0)
try:
while(True):
# Capture frame-by-frame
frame = get_frame(cam)# Convert the image from OpenCV BGR format to matplotlib RGB format
# to display the image
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) showarray(frame)
print("%f FPS" % (1/(t2-t1)))
# Display the frame until new frame is available
clear_output(wait=True)
except KeyboardInterrupt:
cam.release()
print "Stream stopped"
