Child’s Play: Turning Toy Boxes Top-Down with Homography

My wife went to a toy store to buy some Christmas gifts and sent me this picture. I could get the general idea of the what the boxes were showing, but thought, wouldn’t it be great if I could get a top-view of all these boxes? But my wife isn’t that tall, so we could probably be use some homography techniques instead.

The first step is to try and find the four corners of a box that is more or less on the same plane as the other boxes. In this case, I choose the red box on the bottom left. An important note is that the ordering of the points is important. If the before and after points are not in the same order, then the image will get twisted out of shape.

from skimage.io import imread, imshow
from skimage.color import rgb2gray
import matplotlib.pyplot as plt
import numpy as np

toys = imread('toys.png')[:,:,:3]

# convert the toys image to grayscale
toys_gray = rgb2gray(toys)

src = np.array([35, 2775,
                205, 2030,
                1110, 1930,
                1180, 2570,
])

dst = np.array([500, 2700,
                500, 2300,
                1000, 2300,
                1000, 2700,
])

fig, ax = plt.subplots(figsize=(12, 10))
imshow(toys_gray, cmap='gray')
ax.scatter(src[::2], src[1::2], color='red')
ax.scatter(dst[::2], dst[1::2], color='blue')
plt.show()

src = src.reshape((4, 2))
dst = dst.reshape((4, 2))

I’ve marked the corners of my reference box with red dots, and marked with blue dots where I estimate the box would be if I was looking from a top-down perspective. Then we use the transform to see how it projects to the new plane.

from skimage import transform

tform = transform.estimate_transform('projective', src, dst)
tf_img = transform.warp(toys, tform.inverse, output_shape=toys.shape)

fig, ax = plt.subplots(figsize=(15,15))
ax.imshow(tf_img)
_ = ax.set_title('projective transformation')

Amazing! The red box is as we predicted, as are most of the other boxes on the same plane. Of course, the toys which were standing up were heavily distorted, and we also have these black areas that weren’t in the frame of the original picture.

We’ve shown you how to change your view with homography, a technique that transforms an image from one plane to another, using only four points. This would be most useful for flat images such as paintings or signs, but 3D objects could be included as well, if you’re willing to accept some distortion. Hopefully, you enjoyed this blog and learned to look at things from a different perspective.