Building an Eye-Controlled Mouse using Python and OpenCV

You are going to create Eye Controlled mouse in 5 steps

Introduction:

In this article, we’ll explore how to create an eye-controlled mouse using Python and OpenCV. This project is not only fun but also showcases the potential of computer vision in creating innovative solutions. By tracking the movement of your eyes, we’ll be able to control the mouse cursor on your screen. Let’s dive in!

Step 1:

Setting up the Environment To begin, ensure you have Python installed on your system. We’ll also need to install the necessary libraries: OpenCV, Mediapipe, and PyAutoGUI. You can install them using pip:

pip install opencv-python mediapipe pyautogui

Step 2:

Capturing Video from Webcam We’ll use OpenCV to capture video from the webcam. This will serve as the input for our eye-tracking algorithm.

import cv2

cam = cv2.VideoCapture(0)

Step 3:

Detecting Facial Landmarks Next, we’ll use Mediapipe to detect facial landmarks, particularly focusing on the eyes. This will help us determine the position of the eyes in the video frames.

import mediapipe as mp

face_mesh = mp.solutions.face_mesh.FaceMesh(refine_landmarks=True)

Step 4:

Tracking Eye Movement Using the detected facial landmarks, we’ll track the movement of the eyes. We’ll calculate the position of the eyes relative to the screen size and move the mouse cursor accordingly.

import pyautogui

screen_w, screen_h = pyautogui.size()

while True:
    _, frame = cam.read()
    frame = cv2.flip(frame, 1)
    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    output = face_mesh.process(rgb_frame)
    landmark_points = output.multi_face_landmarks

    # Eye-tracking logic here

    cv2.imshow('Eye Controlled Mouse', frame)
    cv2.waitKey(1)

Step 5:

Interacting with the Screen Finally, based on the eye movements detected, we’ll interact with the screen. For example, if the user blinks, we can simulate a mouse click.

if (left[0].y - left[1].y) < 0.004:
    pyautogui.click()
    pyautogui.sleep(1)

we’ve learned how to create an eye-controlled mouse using Python, OpenCV, Mediapipe, and PyAutoGUI. This project demonstrates the power of computer vision in developing innovative human-computer interaction systems.

Full Code for Who Needed:

import cv2
import mediapipe as mp
import pyautogui
cam = cv2.VideoCapture(0)
face_mesh = mp.solutions.face_mesh.FaceMesh(refine_landmarks=True)
screen_w, screen_h = pyautogui.size()
while True:
    _, frame = cam.read()
    frame = cv2.flip(frame, 1)
    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    output = face_mesh.process(rgb_frame)
    landmark_points = output.multi_face_landmarks
    frame_h, frame_w, _ = frame.shape
    if landmark_points:
        landmarks = landmark_points[0].landmark
        for id, landmark in enumerate(landmarks[474:478]):
            x = int(landmark.x * frame_w)
            y = int(landmark.y * frame_h)
            cv2.circle(frame, (x, y), 3, (0, 255, 0))
            if id == 1:
                screen_x = screen_w * landmark.x
                screen_y = screen_h * landmark.y
                pyautogui.moveTo(screen_x, screen_y)
        left = [landmarks[145], landmarks[159]]
        for landmark in left:
            x = int(landmark.x * frame_w)
            y = int(landmark.y * frame_h)
            cv2.circle(frame, (x, y), 3, (0, 255, 255))
        if (left[0].y - left[1].y) < 0.004:
            pyautogui.click()
            pyautogui.sleep(1)
    cv2.imshow('Eye Controlled Mouse', frame)
    cv2.waitKey(1)

Good luck!

PS: If you’ve gotten this far, you might as well just click the heart button below :)

Thanks for reading! My name is Manchala Sreekanth and I mostly write about learning new stuff.

Feel free to reach out to me if you have and questions or would like to connect, either through Linkedin, GitHub or just plain old email: manchalasreekanth999@gmail.com.