Visualizing with OpenCV

What is OpenCV?

Have you ever written an online exam with webcam face detection, and wondered how the website automatically detects your face? Probably not. Because you’re under the pressure from the exam and you wouldn’t be thinking about face detection, right? But how does it recognize your face? And how is this facial recognition achieved? And also have you ever wondered how self-driving cars know how to navigate roads and steer through traffic with no issues? How do computers SEE and UNDERSTAND the world around them and process information from digital images, videos, and other visual inputs? It is possible with the help of an amazing library called OpenCV. That’s what I’ll be discussing in this article.

OpenCV (Open-Source Computer Vision Library) is a free and open-source software library for computer vision and machine learning. It provides a comprehensive range of both traditional and state-of-the-art computer vision and machine learning techniques, with over 2500 optimized algorithms. These algorithms can be used to detect and recognize faces, identify objects, classify human actions in videos, track camera movements, track moving objects, extract 3D models of objects, produce 3D point clouds from stereo cameras, stitch images together to produce a high-resolution image of an entire scene, remove red eyes from images taken with flash, follow eye movements, recognize scenery, and establish markers to overlay. Amazing, isn’t it?

Controlling video games using hand gestures

Have you ever used the Xbox Kinect and wondered how cool it would be if you could use hand gestures to control PC games as well? That’s exactly the project I worked on in my first sem. It was a project based on OpenCV. I made a program called CENGO to use hand gestures instead of mouse and keyboard to control PC video games (like Xbox Kinect but for PC). I take input from the player using their webcam, and use hand gesture recognition (for the character movement) and color recognition (mouse movement) to emulate keyboard presses and mouse movement and clicks to control video game characters. It’s a simple project, but it was fun to make something new and exciting to explore different possibilities and approaches to the project. All of this was possible using OpenCV. The project is demonstrated below

In the project, I have used Mediapipe library by Google to recognize hand gestures and to get the landmarks of various parts of the hand. Using these landmarks, I have coded different hand positions to emulate different key presses to control the character. The key presses are emulated using PyDirectInput library. For the mouse movements, I have used red color recognition and the red surface with the largest area in the webcam acts as the mouse. Using win32api.mouse_event() function, the mouse is moved when the red surface is moved around. The combination of these two allows for video games to be played by using your hand gestures instead of mouse and keyboard.

Conclusion and future scope

This is just a small example of what somebody in their first semester with very little AI ML knowledge can do using OpenCV. I have used very basic tools to implement something that interested me, which in turn helped me gain a lot of knowledge as well. The FPS while playing a few games is poor and unsatisfactory but it can be improved upon in future iterations. I am researching alternate python libraries that I can use to fix the framerate issues. OpenCV is a very powerful but also a fun tool to develop amazing and creative projects that use Computer Vision. The applications of OpenCV are limitless bound only by the programmer’s creativity.

This blog is written by Sai Ruthvik Athota, AI ML Project Lead at RoboVITics, The Official Robotics Club of VIT. If you liked this project, please check it out and go through the project description on GitHub for more information.

GitHub Link — https://github.com/RoboVITics/Cengo

Feel free to contact me through my mail ID — sairuthvikathota2002@gmail.com. Also, feel free to drop a like if this project has interested you. Follow us on Medium for more amazing content.