Real time face recognition with TensorFlow Lite + MLKit in Android
Overview
Face recognition in Android refers to the capability of identifying and verifying individuals based on their facial features using a mobile device’s camera or other imaging sensors. It involves capturing an image of a person’s face, processing the image to extract facial features, and then comparing these features with a database or a pre-trained model to identify or verify the person.
Face Detection: The process begins with face detection, where the system identifies and locates faces within an image or video stream. This involves detecting the position and boundaries of faces in the captured frames.
How it works:
Face detection algorithms typically rely on machine learning models trained on vast datasets of human faces. These models analyze an image or video frame, searching for patterns and features characteristic of human faces. The process can be broken down into two main stages:
- Feature extraction: The algorithm identifies specific facial features like eyes, nose, mouth, and the overall shape of the head.
- Classification: Based on the extracted features, the algorithm determines whether the image region contains a human face and its location within the frame.
Face recognition: in Android refers to the capability of identifying and verifying individuals based on their facial features using a mobile device’s camera or other imaging sensors. It involves capturing an image of a person’s face, processing the image to extract facial features, and then comparing these features with a database or a pre-trained model to identify or verify the person.
Examples of applications using face recognition on Android include:
- Device Unlocking: Face recognition can be used as a biometric method for unlocking devices.
- Authentication: Apps may use face recognition for user authentication and authorization.
- Photo Organization: Some gallery apps use face recognition to automatically tag and organize photos based on individuals.
- Augmented Reality: Face recognition can be used in augmented reality applications for various purposes.
TensorFlow Lite (FaceNet): TensorFlow Lite is a framework developed by Google that allows machine learning models to run on mobile and edge devices with limited computational resources. It’s designed to be fast, efficient, and well-suited for tasks like image classification, object detection, and face recognition on smartphones. The converted model is often saved with a “.tflite” extension.
Benefits of using TFLite for Face Recognition in Android:
- Efficiency: TFLite models are smaller and more efficient than full TensorFlow models, allowing for faster execution and lower battery consumption on mobile devices.
- Offline Capabilities: Face recognition can be achieved even without an internet connection, as the model and database can be stored locally on the device.
- Customization: You can choose from various pre-trained face recognition models or even train your own model for specific needs.
How to add to the android app?
Add the (dependencies) to your project in the build.gradle.kts
val cameraX = "1.4.0-alpha04"
implementation("com.google.android.gms:play-services-mlkit-face-detection:17.1.0")
implementation("com.google.android.gms:play-services-vision:20.1.3")
implementation("androidx.camera:camera-camera2:$cameraX")
implementation("androidx.camera:camera-core:$cameraX")
implementation("androidx.camera:camera-lifecycle:$cameraX")
implementation("androidx.camera:camera-view:$cameraX")
implementation("org.tensorflow:tensorflow-lite:2.14.0")
implementation("org.tensorflow:tensorflow-lite-gpu:2.14.0")
implementation("org.tensorflow:tensorflow-lite-support:0.4.4")mapped face net model
private lateinit var faceNetInterpreter: Interpreter
try {
faceNetInterpreter = Interpreter(
FileUtil.loadMappedFile(context, "mobile_face_net.tflite"),
Interpreter.Options()
)
} catch (e: Exception) {
e.printStackTrace()
}setup the Face Detector and ImageProcessor into our app.
/*configure options for the Face Detection API. It allows you to customize
various aspects of face detection, such as performance mode,
landmark detection, contour detection, and classification */
private val realTimeOpts = FaceDetectorOptions.Builder()
.setPerformanceMode(FaceDetectorOptions.PERFORMANCE_MODE_ACCURATE)
.setContourMode(FaceDetectorOptions.CONTOUR_MODE_NONE)
.setLandmarkMode(FaceDetectorOptions.LANDMARK_MODE_ALL)
.build()
private val faceNetImageProcessor = ImageProcessor.Builder()
.add(ResizeOp(FACENET_INPUT_IMAGE_SIZE, FACENET_INPUT_IMAGE_SIZE,
ResizeOp.ResizeMethod.BILINEAR)) // Resize to 224x224 using Bilinear interpolation
.add(NormalizeOp(0f, 255f))//Normalize pixel values to range [0, 255]
.build()
private val detector = FaceDetection.getClient(realTimeOpts)Analyze image from CameraX OnImageCapturedListener
private class YourImageAnalyzer : ImageAnalysis.Analyzer {
override fun analyze(imageProxy: ImageProxy) {
val mediaImage = imageProxy.image
if (mediaImage != null) {
val image = InputImage.fromMediaImage(mediaImage, imageProxy.imageInfo.rotationDegrees)
detector.process(image)
addOnSuccessListener { faces ->
// Task completed successfully
}
.addOnFailureListener { e ->
// Task failed with an exception
}
//Convert the Bitmap to a TensorImage
val tensorImg = TensorImage.fromBitmap(faceBitmap)
val faceArray = Array(1) {
FloatArray(
192
)
}
Log.e(TAG, "Tensor Image : $tensorImg")
// Preprocess the input image
val faceNetBuffer = faceNetImageProcessor.process(tensorImg).buffer
// Run inference
interpreter.run(faceNetBuffer,facetArray)
// Postprocess the output as needed
Log.d(
"TAG",
"output array: " + faceArray.contentDeepToString()
)
}
}
}The purpose of this distance calculation is as follows: You have the result of face recognition model This result is in the form of arrays. The face recognition model does not know which face is which. The result of the array is essentially the coordinate of the dot in an N-dimensional graph. The value of n represents the dimension of the array. This value can be 128 or 192.
Get information about detected faces
[Face{
boundingBox=Rect(152,
136-345,
324),
trackingId=-1,
rightEyeOpenProbability=-1.0,
leftEyeOpenProbability=-1.0,
smileProbability=-1.0,
eulerX=12.741805,
eulerY=-4.3844852,
eulerZ=-8.575908,
landmarks=Landmarks{
landmark_0=FaceLandmark{type=0,position=PointF(238.09457,295.0142)},
landmark_1=FaceLandmark{type=1,position=PointF(196.6417,252.61476)},
landmark_3=FaceLandmark{type=3,position=PointF(179.13776,244.46556)},
landmark_4=FaceLandmark{type=4,position=PointF(216.6419,208.37985)},
landmark_5=FaceLandmark{type=5,position=PointF(212.7397,278.4354)},
landmark_6=FaceLandmark{type=6,position=PointF(245.9693,242.97098)},
landmark_7=FaceLandmark{type=7,position=PointF(293.00153,264.81573)},
landmark_9=FaceLandmark{type=9,position=PointF(327.2191,256.72867)},
landmark_10=FaceLandmark{type=10,position=PointF(286.96707,217.09705)},
landmark_11=FaceLandmark{type=11,position=PointF(269.56,286.7797)}},
contours=Contours{Contour_1=null,Contour_2=null,Contour_3=null,Contour_4=null,Contour_5=null,Contour_6=null,Contour_7=null,Contour_8=null,Contour_9=null,Contour_10=null,Contour_11=null,Contour_12=null,Contour_13=null,Contour_14=null,Contour_15=null} }]The remain is quite easy, all code is included and additional information can be found in the repository.
Repository:- github.com/variable12354/FaceRecognization-
Here is an example of face attendance ex. This is a company that has multiple employees and stores all employee picture data in the database. At the time of the employee attendance check, detect person is authenticated user or not.
__________________ Thank you so much for your time _________________
