Smart capturing of mobile front via mirror

One of the most prominent products offered by OneAssist is Mobile Phone Protection Plan against any kind of accidental damage. The customers pay an upfront premium and any physical damage to the phone is covered by us. We are responsible to collect the phone, get it repaired by authorised service centre and deliver it back at the doorstep. All this arrangement is similar to an insurance policy except we just not only bear the expenses but also provide the service.

While selling such a protection plan, there is always risk associated that the device being protected is already damaged. We need to have an inspection before onboarding any device under this protection plan. We need to make sure that screen of the phone doesn’t have any cracks. One way to do this is to let the customers upload front screen images and we verify that there are no cracks on it. While getting the images for verification, there’s a risk of forgery as well. So we have established a process to make sure that front screen image, that is being uploaded, belongs to the same device for which the plan is being purchased and it is captured live under our supervision. We call it the Fraud Detection process.

Customers need to upload the image of smartphone screens. Screenshots can’t be used here as screenshot doesn’t capture the damaged screen or edges of smartphone. So, customers can use another smart phone to capture the image of the device or place the mobile phone in-front of the mirror. We have developed both the capabilities. The scope of this article is limited to the mirror flow.

Now, let’s talk about some nuances of capturing the mobile phone image by placing it in-front of mirror. If we use the front camera of the device and put it in front of the mirror, it is not possible to click on a button on the screen to click the image because we need to capture the entire screen and nothing else should come in between. So we decided to build models which should run on the device itself and capture image at the right moment.

There are several checks which the models need to perform to make sure that the image is in right condition to be captured for a proper verification of cracks later on. Sometime captured images are shaky or blur. Sometime, user unintentionally covered the phone with fingers/hand. Sometime, only partial mobile phone are captured.

The models start running as soon as the customer starts the Smart image capture during the Fraud Detection process.

Real-Time model to Detect Mobile Phone

First step is object detection. We identify that whether there’s a mobile screen in front of the camera and the mark the bounding box around it.

While testing the model, we analysed data and identified major issues that a person faces while conducting the process. The idea was to guide the customer to take a corrective action and come to a state where there are no more issues with the image. The following cases shows the challenges faced by customer.

1. Some user places mobile phone too near or too far from mirror

Phone is too Near to mirror, causing partial capture

Phone is too Far from Mirror, resulted in blur image and cracks can’t be identified, if any

2. Background is too dark

background is dark, causes smartphone edges disappear and hence any damage at the edges also disappear

3. Some user covered the smartphone with fingers/hand which causes obstruction

Covering phone screen with fingers/hand

So to tackle the above challenges, we added new models to the pipeline, along with existing object detection model. A Near_Far model to guide the customer to place the phone at proper distance from mirror in real time. A Dark model provide the dark background information and prompt customer to move to brighter area or change background. An Obstruction model prompt the customer to move finger away from mobile phone. All the above models run to make sure we capture the clear and complete image.

Complete flow for capturing clear image

Also, all the prompts are available in real time in text form on the screen and also available in voice form. The voice support & text messages guide the customer in real-time and help in quickly capturing the clear and complete image. Some of the messages which displays on screen are “bring your phone close to mirror”, “tilt your camera slightly downwards”, “move the phone slightly back” etc.

image showing message to move the phone slightly back

Now its Puzzle time

I don’t speak, can’t hear or listen anything. But I will always show the truth. Who am I ?

Now, waiting for your answer in comment section.

Any kind of criticism or suggestion is always welcome. I will work on those and improve further.