Using AI to calculate our happiness

Since June 2017, we use a physical panel called “weather board” to report the working mood of all employees. The board has scales ranging from 1 to 10, where 1 stands for extremely negative mood and 10 means highly positive. Every two weeks all colleagues are asked to anonymously put one green LEGO brick on the weather board between 1 and 10 according to his/her happiness of working in Digital Unit. We use it as a basis to continuously improve how we work together (Figure 1 with sample data).

Figure 1. Weather Board

In the beginning, we collected all scores and calculated our “happiness” (i.e. the mean of the scores) manually. But this is so BOOORING! and we don’t want to be bored. This needed to be automated!

Artificial Intelligence to the rescue.

How cool would it be to take a picture of the board and let AI do the rest?

Here you go: Read on to see how this works with some cool Python3 and OpenCV to replace the annoying manual operation.

The steps of the automated method

  1. Define the color spaces of the LEGO bricks and scale lines.(One-time)
  2. Locate the scale lines/LEGO bricks
  • Apply mask
  • Morphological transformations
  • Contour(boundary) detection

3. Calculate the score of each brick and mean/median

Define the color spaces of the LEGO bricks and scale lines

To define the color spaces of LEGO bricks and scales, we upload several weather board images to imagecolorpicker.com and manually pick colors of about 150 pixels. The upper/lower HSV bounds of LEGO bricks and scale lines are used as their color spaces, separately. HSV(Hue, Saturation, Value) is an alternative representation of the standardRGB color model(Figure 2).

lego_green_normal_hsv_low=np.array([62,32,47])
lego_green_normal_hsv_high=np.array([105,78,71])
lego_black_normal_hsv_low=np.array([0,10.23,3.53])
lego_black_normal_hsv_high=np.array([65,100,20])
Figure 2. HSV color model

Apply mask

To cut LEGO bricks and scale lines out of the whole image, we should apply mask with the pre-defined color spaces. (Figure 3, 4)

raw_mask = cv2.inRange(hsv_img, lego_black_opencv_hsv_low, lego_black_opencv_hsv_high)
.
.
.
raw_mask = cv2.inRange(hsv_img, lego_green_opencv_hsv_low, lego_green_opencv_hsv_high)
Figure 3. LEGO brick mask
Figure 4. Scale line mask

Morphological transformations

Morphological transformations is a mathematical technology processing geometrical structures. Here two morphological operations: opening and closing are implemented. morphological transformations is used to remove noises: opening removes small objects, while closing removes small holes. (Figure 5,6)

kernelOp = np.ones((1,1),np.uint8)
kernelCl = np.ones((11,10),np.uint8)
.
.
.
ret,imBin= cv2.threshold(raw_mask,200,255,cv2.THRESH_BINARY)
mask = cv2.morphologyEx(imBin, cv2.MORPH_OPEN, kernelOp)
mask = cv2.morphologyEx(mask , cv2.MORPH_CLOSE, kernelCl)
.
.
.
ret,imBin= cv2.threshold(raw_mask,200,255,cv2.THRESH_BINARY)
mask = cv2.morphologyEx(imBin, cv2.MORPH_OPEN, kernelOp)
mask = cv2.morphologyEx(mask , cv2.MORPH_CLOSE, kernelCl)
Figure 5. LEGO brick mask after morphological transformations
Figure 6. Scale line mask after morphological transformations

Contour(boundary) detection

To extract each LEGO brick and scale line separately, their contours(or outer boundary) are detected. (Figure 7,8)

_, contours, _  = cv2.findContours(mask,cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
.
.
.
_, contours, _ = cv2.findContours(mask,cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
Figure 7. Contours of LEGO bricks
Figure 8. Contours of scale lines

Calculate the score of each brick and mean/median

After finding the contours of each brick and scale, we can get the coordinates as well as scores easily.

The code and example of this project can be found here: https://github.com/costacruise/auto-weather-report