Analytics Vidhya
Published in

Analytics Vidhya

Building Hari Puttar’s missing invisibility cloak with OpenCV

Hari, the Indian cousin to J. K. Rowling’s Harry Potter has landed himself in trouble again. “Learn at least something from Hermoine!” is what his mum said when he broke the news.

Recommended music with this content:

Can you believe it?! I mean: how irresponsible do you have to be in order to lose your only Invisibility Cloak days before your cousin’s big day?! Hari said he must have misplaced the cloak while rehearsing his performance at Gigi’s wedding the following week.

Due to the pandemic, the shops at Diagon Alley had taken an unscheduled vacation. We couldn’t order a new one from Alibaba either per the ban on import of non-essential items from China. I checked WizKart but they couldn’t have delivered the cloak in time for the event.

However, just as every cloud has a silver lining, there is an opportunity hidden inside every obstacle. In order to maintain social isolation, Gigi’s father decided to conduct the ceremony remotely -Exactly the break I needed!

I was fairly confident that such a cloak could be created on-screen using OpenCV and any solid-colored blanket. So, I set out to build a fake invisibility cloak for our graceless protagonist! The plan was simple:

  • Zeroth, create an interface to read, write and display video-streams and video-files. Provide basic user-interaction such as selecting the objects to be removed or tuning the thresholds for best results.
  • First, have the user select a background image which they would like to display in place of the “vanished” objects.
  • Second, Color-Segmentation. Since the blankets used are of solid color, we need to perform accurate color-segmentation followed by replacing pixels representing the cloaks by corresponding ones from the back-ground image.
  • Additionally, since the Color-Segmentation may produce irregular masks, performance tweaks (mainly Morphological Opening and Closing) were employed which are too main-stream to mention in the blog. However, interested people should refer the documented code available on GitHub! (Scroll down for the link)
Well then, Welcome aboard ‘The Hogwarts Express’; Source: https://unsplash.com/photos/DmDYX_ltI48

The empty-pipeline

I began with acquiring the ability to read frames from a source, display them on the screen and write them into a file. Since Hari uses a Windows laptop, I had to make sure the code runs on Windows 10 as well. Turns out this isn’t that simple because OpenCV uses FourCC’s codecs which must be installed separately. After some trial-and-error, I zeroed in on the “.mp4” file format and a couple of pre-installed codecs for each OS:

Snippet 1: Read, display and write to “.mp4” files on Windows 10 and Ubuntu 16.04 (LTS)

Hue-based color-spaces for segmentation

It was time to start thinking about ways to make a normal cloak invisible. Since, we had only two solid blankets available (dark-blue and dark-pink), I figured it would be better to write code that could handle cloaks of any color. (Can never be too cautious when dealing among wizards with clumsy hands!)

Since HSV(Hue-Saturation-Value) and HSL(Hue-Saturation-Luminosity) color-spaces are better at clustering different shades with similar “Hue” together [1], I decided to use them. I am planning to write a small snippet to have an interactive version of Figure1 that demonstrates how this color-separation is better in Hue-based color-spaces, follow me or keep checking the edits on this blog if you’re interested! Vectorized conversions to-and-from HSV are implemented in openCV, so it’s better to go with HSV!

Figure 1: Hue separation in HSV color-space (Source: https://stackoverflow.com/a/48367205)

I needed to find some way to compute lower and upper bounds for a color that could encircle its different shades arising from shadows, etc. At the same time, I couldn’t complicate the parameters too much else Hari would never quite learn how to use the system! With some iterations, I found that the scheme described in Snippet 2 to was an good solution.

##Snippet 2: How to get upper and lower bounds for any color in HSV
#Let us take one HSV color with hue=HUE, saturation=SAT &
#value=VAL. Also, lets have a thresh=THRESHOLD
HSV_COLOR = [HUE, SAT, VAL]
thresh = THRESHOLD
#We make two copies, one for each upper and lower bounds of this #colorlower = HSV_COLOR[:]
upper = HSV_COLOR[:]
#As we see in Figure1, vertical rectangles with height twice as much #as width fit well for most colors, let's model that in our bounds!lower[0] = max(lower[0] — thresh//2, 0)
lower[1] = max(lower[1] — thresh, 0)
lower[2] = max(lower[2] — thresh, 0)
upper[0] = min(upper[0] + thresh//2, 179)
upper[1] = min(upper[1] + thresh,255)
upper[2] = min(upper[2] + thresh,255)
#Ideally, we should have separate threshold for each dimension. We #have chosen to go with only one param since it greatly enhances #user-experiences while leaving relatively less performance on the #table.##Snippet 2: How to get upper and lower bounds for any color in HSV

Performance Enhancement -Morphology

Whether you’re trying to remove background noise from an image, smooth out the edges of an irregular contour, remove “holes” from a polygonal mask or maybe you’re just trying to make your masks look “prettier”, morphology is the one cure that solves it all! Let’s briefly describe the morphological techniques used to improve our results.

As shown in Figure 2, there are two main types of defects when we’re trying to process segmentation masks of any type(color, instance, semantic). They are referred to as “holes” and “noise”.

Figure 2: “Holes”(Left) and “Noise”(Right) in a segmentation mask

In Morphology, the Open-Close transform is widely used to remove these kinds of defects. The Open-Close transform has two phases, the Opening phase and the Closing phase. Figure 3 breaks down the transform and shows the intermediate output after each phase.

Figure 3: Morphology Pipeline — Input mask (Left); Mask after Morphological Opening (Center); Mask after Morphological Closing

As we can see, both the types of defects are present in the input image. After morphological opening (which is erosion followed by dilation), we see that the “noise” disappears but at the same times the holes are more pronounced. However, after morphological closing, we see that none of the “holes” survive. We don’t go into the details here, but there is an amazing article that explains the technicalities of Morphology available here.

Morphological Opening can be views as Black pushing at all it’s boundaries as White retreats. In this process tiny pockets of White do not survive while the large ones are eroded. Morphological Closing is the same except this time White is pushing at the boundaries. When Morphological Opening and Closing are applied in succession, tiny pockets of neither White nor Black survive but the larger contours remain unaffected.

And Presto!

Link to GitHub repo: https://github.com/IshanBhattOfficial/HarryPotter-InvisiblityCloak.git

Hope this helps you :)

--

--

--

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Recommended from Medium

CS371p Spring 2022: Max Thomas

ASP.Net Core 2 with Postgresql on Linux Part 10: Program, Startup, appsetting.json, etc.

October noon

October noon

PIC 18 TMR1 Max Counting

Scalability and Cost Efficiency with AWS

How to add a background image to your footer on Dawn — Shopify 2.0 Tutorials

How to Build Babel Tower Out of Airtable Blocks

a man drawing flags of different countries with chalk on pavement

Introducing Jira CLI: The Missing Command-line Tool for Atlassian Jira

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ishan Bhatt

Ishan Bhatt

graduate student (MS CS) & research assistant (PICTure Lab) @ NCSU, Raleigh; Ex systems engineer @ TCS — R&I, India; B. Tech. (Computer Engg.) from BVM, India.

More from Medium

Use CUDA 11.0 for RAPIDS 21.12 with TensorFlow 2.4 in Ubuntu 18.04

Estimating depth for YOLOv5 object detection bounding boxes using Intel® RealSense™ Depth Camera…

Visualizing Pathologies in Ultrasound Image using OpenCV and Streamlit

Source: Omdena Visualizing Pathologies in Ultrasound Image using OpenCV and Streamlit

XMAS-Project — Part 5: Teachable Machine by Google to solve Image Classification