Preprocessing with Computer Vision Part VIII: Image Annotation
The key to artificial intelligence has always been the representation. — Jeff Hawkins
The primary objective of this article is to enhance the accuracy of ML models trained for object detection by annotating unique objects within an image . We discuss how different shapes used for image annotation within a singular image can lead to smart , clutter free and better user experience.
Image processing : Basics
Following code demonstrates the scaling and changes in resolution of images.
#read the image
image = cv.imread('traffic.jpeg')#define percentage for scaling the image (resolution changes)
percentage = 50#calculate width and height of image according to percentage
width = int(image.shape[1] * percentage/ 100)
height = int(image.shape[0] * percentage/ 100)#resize the image with defined width and height
image_resize = cv.resize(image, (width, height), interpolation = cv.INTER_AREA)#used when capturing image from camera with a resolution
# Note : Only useful when downscaling camera. Upscaling is used only when camera supports.imag_capture = cv.VideoCapture(0)def make_1080p():
cap.set(3, 1920)
cap.set(4, 1080)def make_720p():
cap.set(3, 1280)
cap.set(4, 720)def make_480p():
cap.set(3, 640)
cap.set(4, 480)
Masking
Masking is the process of creating a custom filter to extract an object from the image with a specific color or shape. For masking operation, the image is initially converted to Hue, Saturation and Value (HSV) range. HSV is an alternative human vision representation of Red, Blue, Green (RGB) color range. As image annotation techniques are widely used in real-world applications, it is essential to use the human vision perspective of colors rather than the original RGB components. Therefore, the image is converted to HSV range before the annotation process. We then extract the region with the color mask by performing the bitwise operation between image and color mask. This gives us the region of interest.
Let us see how this is done in python.
# convert the image to hsv
#cvtColor(image, operator): returns the resulting image after applying the specifying operator
rgb_to_hsv = cv.cvtColor(image, cv.COLOR_BGR2HSV)#define the lower and upper range for red color
lower_yellow = np.array([30,150,50])
upper_yellow = np.array([255,255,180])#create mask for yellow color using the range for hsv values
mask_for_color = cv.inRange(rgb_to_hsv, lower_yellow, upper_yellow)# bitwise and of the image and mask
bitwise_result = cv.bitwise_and(image, image, mask= mask_for_color)
Annotation with OpenCV using different shapes
Different shapes with different sizes may require each object in an image to be annotated differently. Following snippet demonstrates using different shapes — rectangle, circle and polygon using OpenCV library in python.
# array for storing the co-ordinates of the region of interest# for rectangle
array_points = []
#for circle
array_circle = []
#three arrays for polygon
current = []
points = []
#boolean to check whether drawing polygon is done
done = False#boolean value for cropping a region
cropping = Falsedef draw_rect(event, x, y, flags, param):
#reference to the global variables
global array_points, cropping
#check whether the mouse button is clicked
if event == cv.EVENT_LBUTTONDOWN:
array_points = [(x, y)]
cropping = True
#check whether the button is released
elif event == cv.EVENT_LBUTTONUP:
# if the bbutton is released, note the points of the co-ordinates
array_points.append((x, y))
cropping = False
# draw rectangle with mouse positions
cv.rectangle(image, array_points[0], array_points[1], (0, 255, 0), 2)
cv.imshow("Draw_Rect", image)
def draw_circle(event, x, y, flags, param):
#reference to the global variables
global array_circle, cropping
#check whether the mouse button is clicked
if event == cv.EVENT_LBUTTONDOWN:
array_circle = [(x, y)]
cropping = False
# draw circle with mouse positions(here, we require x, y and radius of the circle)
cv.circle(image, (array_circle[0][0], array_circle[0][1]), 40, (0, 255, 0), 2)
cv.imshow("Draw_Rect", image)
def draw_polygon(event, x, y, flags, param):
global current, points, done
#reference to the global variables
if event == cv.EVENT_MOUSEMOVE:
current = (x, y)
elif event == cv.EVENT_LBUTTONDOWN:
# if the bbutton is released, note the points of the co-ordinates
points.append((x, y))
elif event == cv.EVENT_RBUTTONDOWN:
# Right click means we're done
done = True
while(not done):
if (len(points) > 0):
# draw the polygon with all the points clicked
cv.polylines(image, np.array([points]), False, (0, 255, 0), 1)
# show each line being drawn with the co-ordinates
cv.line(image, points[-1], current, (0, 255, 0))
cv.imshow("Draw_image = cv.imread('traffic.jpeg')
clone = image.copy()
cv.namedWindow("Draw_Rect")
cv.setMouseCallback("Draw_Rect", draw_circle)#loop for function of draw and crop
while True:
cv.imshow("Draw_Rect", image)
key = cv.waitKey(1) & 0xFF
if key == ord("r"):
image = clone.copy()
elif key == ord("c"):
break
elif key == ord("s"):
rectX = (array_circle[0][0] - 40)
rectY = (array_circle[0][1] - 40)
roi = clone[rectY:(rectY+2*40), rectX:(rectX+2*40)]
cv.imshow("ROI", roi)
cv.waitKey(0)
else:
continuecv.destroyAllWindows()cv.imshow("Draw_Rect", image)
if cv.waitKey(50) == 27: # ESC hit
done = True
Extracting all the objects from the image
Extracting objects with different shapes requires different procedures. Rectangle requires only 4 co-ordinates, while polygon requires multiple co-ordinates while a circle requires the diameter. Following code snippet demonstrates the cropping of such different objects with different shapes from an input image.
# crop rectangle
image = cv.imread('traffic.jpeg')
clone = image.copy()
cv.namedWindow("Draw_Rect")
cv.setMouseCallback("Draw_Rect", draw_circle)#loop for function of draw and crop
while True:
cv.imshow("Draw_Rect", image)
key = cv.waitKey(1) & 0xFF
if key == ord("r"):
image = clone.copy()
elif key == ord("c"):
break
elif key == ord("s"):
rectX = (array_circle[0][0] - 40)
rectY = (array_circle[0][1] - 40)
roi = clone[rectY:(rectY+2*40), rectX:(rectX+2*40)]
cv.imshow("ROI", roi)
cv.waitKey(0)
else:
continuecv.destroyAllWindows()# crop circle
image = cv.imread('traffic.jpeg')
clone = image.copy()
cv.namedWindow("Draw_Rect")
cv.setMouseCallback("Draw_Rect", draw_circle)#loop for function of draw and crop
while True:
cv.imshow("Draw_Rect", image)
key = cv.waitKey(1) & 0xFF
if key == ord("r"):
image = clone.copy()
elif key == ord("c"):
break
elif key == ord("s"):
rectX = (array_circle[0][0] - 40)
rectY = (array_circle[0][1] - 40)
roi = clone[rectY:(rectY+2*40), rectX:(rectX+2*40)]
cv.imshow("ROI", roi)
cv.waitKey(0)
else:
continuecv.destroyAllWindows()#crop polygon
image = cv.imread('traffic.jpeg', -1)
clone = image.copy()
cv.namedWindow("Draw_Rect")
cv.setMouseCallback("Draw_Rect", draw_polygon)#loop for function of draw and crop
while True:
cv.imshow("Draw_Rect", image)
key = cv.waitKey(1) & 0xFF
if key == ord("r"):
image = clone.copy()
elif key == ord("c"):
break
elif key == ord("s"):
mask = np.zeros(image.shape, dtype=np.uint8)
channel_count = image.shape[2]
ignore_mask_color = (255,)*channel_count
cv.fillPoly(mask, np.array([points]), ignore_mask_color)
masked_image = cv.bitwise_and(image, mask)
cv.imshow("ROI", masked_image)
cv.waitKey(0)
else:
continuecv.destroyAllWindows()
Takeaways
The users are cautioned against manual annotations as they are time consuming in the space of automated annotation tools. The readers are encouraged to use customizing shapes for annotating images. Also this week’s python code is equipped with additional functionalities like playing around with colors, extracting objects of a particular color, extracting similar objects etc. We invite users to explore and get more equipped with similar functionalities.
The entire code for this article can be found here.
Do you have any questions?
Kindly ask your questions via email or comments and we will be happy to answer.
