Interactive GrabCut Implementation in Flask (Python)

GrabCut is a computer vision algorithm for extracting an object from an image. It doesn’t always work on the first try, so Interactive GrabCut allows users to indicate how to refine the output. There are several implementations of GrabCut on the web — some run only in python (no web interface), some are not interactive.

Here is an implementation of Interactive GrabCut that runs in Flask. It has two additional dependencies: opencv to run Grabcut and fabric.js to draw on the image to indicate where the foreground and background are. It has two steps:

1. The user draws a rectangle that includes the entire object in the foreground. GrabCut makes an attempt to extract the object.
2. The user draws lines on the image to indicate additional background pixels to remove. GrabCut is run again to refine the object extraction.

Step 1: Draw Rectangle, Run GrabCut

Our goal is to extract the red king from the following image:

First, the user draws a green rectangle on a fabric.js canvas. They can move and resize the rectangle after drawing it on the canvas, to get a better fit. Make sure to get all the foreground in the rectangle.

The implementation works as follows:

• To run GrabCut with the rectangle drawn on the canvas, send rectangle coordinates and the scale factor used to make the image fit on the canvas to the flask server.
• Scale the rectangle coordinates to match the original image on the server side. Run GrabCut using the rectangle coordinates and the option GC_INIT_WITH_RECT.

cv2.grabCut(img,mask,rect,bgdModel,fgdModel,5,cv2.GC_INIT_WITH_RECT)

• The inputs mask, bgdModel and fgdModel are all empty 2D arrays that GrabCut later modifies.

GrabCut can also be used to extract parts of objects, like getting the meat, lettuce, tomato and cheese between the hamburger buns. Here is the first pass of GrabCut:

Step 2: Draw lines, Refine Grabcut Result:

Often the output of GrabCut still has some background pixels in it. To get a better output, we need to refine GrabCut. For the refinement step, allow users to draw on a second canvas, and draw lines where the background should be removed. For the drawing instead of running cv2.GC_INIT_WITH_RECT we use cv2.GC_INIT_WITH_MASK.

Here is an overview of the implementation:

• To run GrabCut refinement, we use the first rectangle and the drawn lines to run grab cut again.
• Send both the rectangle coordinates on the first canvas and the drawing (an array of lines which each include a path that is an array of point coordinates) to the server. Scale the rectangle and line coordinates, along with the line thickness of the line to match the original image.
• Run GrabCut using GC_INIT_W_RECT first, using the rectangle coordinates.
• Clear the foreground and background models.
• Convert the drawing to a mask, using:

cv2.circle(mask, (x1,y1), int(thickness), 0, -1)

for every point, for every line in the drawing. The 0 value indicates that all the points are sure background. (The argument -1 fills the circle shape.)

• Run GrabCut with both rect and mask using GC_INIT_WITH_MASK

cv2.grabCut(img,mask,rect,bgdModel,fgdModel,5,cv2.GC_INIT_WITH_MASK)

Here’s how we refine GrabCut to extract the king, and the hamburger filling:

This implementation of iterative GrabCut only uses removing background to refine the initial GrabCut output. Technically, we could also add foreground (by changing the 0 to be either 1 for sure foreground and 3 for probable foreground when drawing circles on the mask), but we found that adding background works better than adding foreground.

See code here: https://github.com/eozmen410/interactive_grabcut