Detecting Pupil Dilation Using Smart phones
Using smart phones to accurately analyse pupil dilation
We recently encountered a need to identify a reliable technique for analysing pupil dilation using a smart phone.
We conducted an in-depth investigation into current pupil detection techniques. After discovering many different processes, we chose an approach outlined in a research paper titled “Robust Pupil Detection for Real-Time Pervasive Eye Tracking.”
The study’s results show that we can achieve an impressive 95% accuracy in real-time situations, giving us dependable outcomes. This breakthrough has profound implications, as it paves the way for exciting applications in healthcare, user experience improvement, and biometric authentication. This technology can have a huge impact and change many industries for the better.
While various approaches employ OpenCV and computer vision models for pupil detection, none have been specifically optimised for mobile devices. This is primarily due to challenges presented in the mobile environment, including fluctuating lighting conditions, processing speed limitations, and privacy considerations.
Addressing these challenges can make it possible to measure pupil diameter using mobile devices. The process can be broken down into several key steps:
· Capture Eye Area: The main smartphone camera is used to capture the eye area (pupil and iris) only, so we must ensure proper lighting conditions for accurate detection. By focusing on the eye area instead of the eye region (eyelids and eyebrows), the algorithm can reduce the impact of variable lighting on the overall accuracy.
· Image Pre- Processing: The captured eye area image is then processed using appropriate image processing tools, such as OpenCV. Pre-processing techniques like image thresholding, edge detection, and noise reduction are applied to transform the raw image to different formats and prepare it for pupil detection.
· Use Computer Vision Model: To detect the pupil within the processed image, we employ an appropriate computer vision model. This would be a convolutional neural network (CNN) for eye feature recognition. Using the right model ensures accurate and reliable pupil detection.
· Visualizing Pupil Detection Fluctuations: The objective here is to visualize how the pupil size changes over a period. This can involve tracking variations in pupil dimensions against the passage of time.
The main goal is to achieve accurate pupil detection. To achieve this utilising the right computer vision model significantly improves how accurately we can detect pupils when we apply it to processed images.
Before transitioning to the Computer Vision model, it’s important to ensure its compatibility with mobile devices. A solution that emerged involves converting the model into a lighter version using tools like TensorFlow or PyTorch. Firebase ML can then be utilised to host the model, making it accessible for mobile app users. This setup allows the model to be downloaded to the mobile app whenever needed, enabling the integration of the Computer Vision model on the mobile side.
Use Computer Vision Model
Based on the findings, there are two straightforward assumptions that help define the valid range of pupil dimensions without needing specific predefined values. These assumptions are:
1) the eye’s corners are within the image,
2) the eye’s corners cover at least two-thirds of the diagonal of the image.
It’s important to note that these assumptions are flexible — the method still functions effectively even if they are slightly violated.
The approach primarily focuses on detecting edges, specifically selecting curved edge segments that likely make up crucial parts of the pupil outline. These chosen segments are then combined under specific conditions to create additional options that could represent a reconstructed pupil outline. For each option, an ellipse is fitted, and its properties are assessed based on the roundness of the ellipse, the distribution of its edges in relation to the ellipse, and the section of the ellipse’s outline that supports the notion of it being the pupil. This assessment generates a confidence score for each option’s likelihood of being the pupil. The option with the highest confidence score is ultimately chosen as the pupil.
Edge Detection
The Canny edge operator (Canny, 1986) is used to find edges in the image. The edges are then processed using a morphological approach to make them thinner, straighter, and break connections. This method, following the steps of Fuhl et al. (2016c), results in an image with separate and streamlined edge sections.
Next, each of these edge sections is simplified using the k-cosine chain approximation method (Teh and Chin, 1989), which helps fit ellipses and improves accuracy, especially when separating the pupil from nearby edges is tricky. After this, different rules are applied to remove edge sections that probably aren’t part of the pupil outline. This step sharpens our focus on the important parts for further study.
Measuring Confidence
These three important factors can be considered to confidently determine the pupil.
Roundness of the Ellipse: This checks if the ellipse is round. It prefers round shapes, which happen when the camera is placed correctly. It’s like comparing the width to the length of the ellipse.
Spread of Edge Points: This checks if the edge points are spread out nicely around the ellipse. If they are, it means the shape is more like an ellipse, like a pupil. We just count how many edge points are in different parts of the ellipse.
Contrast of the Ellipse Outline: These measures how different the inside and outside parts of the ellipse are. We pick some points around the edge and see if the inside is darker than the outside, like a pupil’s appearance.
If the ellipse outline isn’t the right size or less than half of the outline contrast supports the potential pupil, the confidence score is zero. This way we do the validation part.
Conditional Segment Combination
Next, we take the remaining potential pupil segments and pair them up to create more potential pupils. This step helps reconstruct the pupil outline, which can be fragmented due to things like reflections or eyelashes getting in the way. Let’s call the sets of points for two segments D1 and D2, and the squares around them S1 and S2. We combine the segments if these squares overlap but aren’t completely inside each other — basically, some parts of S1 are in S2 and vice versa.
The resulting merged segment is subsequently evaluated based on the validation criteria discussed in the Edge Detection validation section above. Its confidence is determined using the method described in the Measuring Confidence section above. This approach allows us to accurately determine the pupil.
These methods underwent thorough testing using a large dataset of over 316,000 images captured with four distinct head-mounted eye-tracking devices. The results were quite impressive, showing a significant improvement in pupil detection compared to other approaches.
Right now, we’re actively working with this model to create a reliable pupil dilation system.
Capture Eye Area
To capture the eye area, we can record a short video focused on the eye. This video can then be converted into individual image frames. Each frame can be processed independently to analyze and detect the pupil accurately. This approach allows us to utilize multiple frames and ensures comprehensive coverage of the eye region for more reliable pupil dilation.
Image Pre- Processing with OpenCV
Image Processing with OpenCV can be achieved on the mobile end using platform-specific languages. For Android applications, developers can use Kotlin, while for iOS, Swift is the preferred choice. In cases where developers choose cross-platform frameworks like React Native or Flutter, they have the option to check whether OpenCV library support is available for their respective frameworks.
If a developer chooses Flutter or React Native and finds that OpenCV is not directly supported, they can still access the OpenCV library by communicating with the native side of the framework. This involves writing platform-specific code in Java/Kotlin for Android or Objective-C/Swift for iOS to access the OpenCV library. The communication between the cross-platform framework and the native side allows developers to leverage the powerful image processing capabilities of OpenCV.
When using OpenCV in iOS development, developers typically need to create a bridging header to facilitate communication between Objective-C and Swift code.
OpenCV is primarily written in C++ and provides Objective-C wrappers for iOS. Therefore, if you are using Swift in your iOS project, you’ll need to create a bridging header to access the Objective-C code and use the OpenCV functionalities.
The bridging header acts as an interface between Swift and Objective-C, allowing seamless communication and interoperability between the two languages. This enables you to access the OpenCV libraries and utilize their image processing capabilities in your Swift codebase. You can follow this tutorial to add OpenCV lib to iOS.
We can utilize various image processing methods to enhance the eye image and isolate the pupils. Here are some common methods we can use in OpenCV for the image processing part:
1. Grayscale Conversion: Convert the eye image to grayscale using the cv2.cvtColor() function. Grayscale images simplify processing and reduce computation time.
2. Histogram Equalization: Improve the contrast and visibility of the image by applying histogram equalization using cv2.equalizeHist().
3. Gaussian Blur: Reduce noise and smooth the image using Gaussian blur with the cv2.GaussianBlur() function.
4. Thresholding: Use image thresholding to create binary images that segment the pupil from the background. The cv2.threshold() function can be used for this purpose.
5. Edge Detection: Detect edges in the image using edge detection algorithms like Canny edge detection (cv2.Canny()). Edge detection can help identify the boundaries of the pupil.
Morphological Operations: Utilize morphological operations, such as dilation and erosion, to clean up the image and refine the detected pupils’ contours.
After applying various filters and image processing techniques, the eye image appears enhanced and refined, similar to the image depicted above. These filters and methods have effectively isolated the pupils, making them stand out from the background and improving the overall clarity of the eye region.
There are indeed several approaches to detect pupils using OpenCV, but achieving high accuracy can be challenging. Pupil detection is a complex task influenced by various factors, including lighting conditions, noise, and occlusions. As a result, some of the existing approaches might not provide the desired level of accuracy.
Conclusion
In summary, we have evaluated several approaches currently available and have selected the solution above due to its superior accuracy compared to others. As mentioned, this method achieves an impressive accuracy in real-time situations, marking a significant advancement. This discovery holds promising potential for transformative applications in healthcare, enhancing technological utilization, and bolstering security reliability.
Throughout our exploration, we encountered two main challenges. The first involved fine-tuning the CNN model to effectively detect pupils, while the second centered around deploying the model within a mobile environment.
In simpler terms, we want to highlight the fact that there aren’t a lot of ways or solutions available for measuring the size of pupils using a CNN model on mobile devices. Importantly, we have successfully devised an innovative approach to locally run the model within the mobile environment, eliminating dependency on external servers or tools.
Our methodology involves utilising short videos to concentrate on the eye region and meticulously processing individual frames for heightened precision. Leveraging OpenCV tools has significantly enhanced the quality of eye images, and our implementation incorporates specialised communication techniques optimised for diverse platforms.
Although we have made strides in enhancing eye image quality, achieving absolute accuracy remains challenging due to factors such as lighting conditions and other variables.
This research helps us get better at detecting pupils, which could have many useful applications.
Importantly, we are currently in the process of implementing this model. This means that we are actively working on turning this solution into a reality.
Thank you for making it to the end of this article!
Please let me know in the comments if you found this helpful or would like to share your experiences.
Reference — Robust pupil detection for real-time pervasive eye tracking
