Improving Computer Vision TensorFlow Lite Samples

GSoC 2021 TensorFlow Project

Published in

Geek Culture

5 min readAug 31, 2021

I got to participate in GSoC 2021 this year to help for improving the Computer Vision Sample Apps of TensorFlow Lite. The idea behind the project is to improve the sample apps of Computer Vision which uses TensorFlow Lite Task Library as well as TensorFlow Support Library. The main objective is to implement CameraX and update the app so that the developers from the community find it easy to integrate Machine Learning with Android Apps.

Mentors: George Soloupis and Margaret Maynard-Reid

I find myself lucky to have them as my mentors.

Link of the Project: https://summerofcode.withgoogle.com/projects/#4931401570320384

GitHub Link: https://github.com/sayannath/GSoC-Project-2021

About Me

Hello! I am Sayan Nath. I am a third-year undergraduate student at the Kalinga Institute of Industrial Technology. I am amongst the top contributors in GitHub from India. I am an aspiring Machine Learning Engineer at Codebugged AI, Lead of Dsckiit. I have also been selected as a Mentor from Google Developers. My other interests aside from Machine Learning are Backend Development, DevOps and Flutter. I have contributed to Keras Coding Examples, TensorFlow Repositories, ML-GDE Repositories, and Awesome TensorFlow-Lite Repository. I also love to give technical talks at events and mentor my fellow mates and my juniors.

GSoC Selection Story

This is the selection mail I got on May 17th, 2021. But this journey started a long time ago. I have wanted to do something big in open-source since the 12 standard. Doing GSoC was on my bucket list. Back then, I was an open-source enthusiast, but I got in touch with Sayak Paul in the last few months. He is an awesome mentor as well as a good human being. I consider myself lucky to have got a mentor like him. Soumik Rakshit reproduced a research paper known as MIR-Net. Sayak Paul made the TensorFlow-Lite file for it made available in TensorFlow-Hub. I used the TF-Lite to implement it in my Flutter App. Fortunately, this Flutter project was got mentioned by Google I/O ’21 in one of the meetups. The talk was on the Community Ecosystem of TFLite by Sayak Paul. Click here for the Deck. Sayak Paul reviewed my proposal for Google Summer of Code, a big shoutout to him once again.

Tasks

Implement CameraX and remove the usage of fragments with the existing Camera2 and Camera API.
Implement Support Library with TensorFlow Lite Interpreter.
Implement Data Binding.
Implement the Image to BitMap conversion and modify the Support Library and Task Library.
Modify the Test Code in Android
Add a Bounding Box Function in the tflite-support library (Stretch Goal)

First Evaluation

In my first evaluation, I completed the first three tasks under the mentorship of George Soloupis very smoothly. He is the best mentor one can ever get. His support was constant. I used to meet with my mentors i.e George Soloupis and Margaret Maynard-Reid once a week to discuss the updates of my project. Apart from that, George Soloupis and I used to chat on LinkedIn, with Margaret Maynard-Reid and Meghna Natraj on Google Chat and Google’s TensorFlow Lite Team in emails.

Challenges Faced

The main challenge of this project is to draw the bounding boxes when CameraX is implemented in place of Camera2. This is because the output format of Camera2 is Bitmap but for CameraX is Image. George Sir helped me with the preprocessing of the image before making any inference. This fixed the above problem.

Now it’s time for the result.

Results of First Evaluation

With this, I successfully passed my first evaluation under the guidance of George Soloupis and Margaret Maynard-Reid.

Final Evaluation

Tasks 4 and 5 are under second evaluation. I completed the task in a span of two weeks. George Sir helped me a lot with the conversion and with the android test. I finished my project a little earlier than expected and was enthusiastic enough to take up another thing under my belt. Margaret Maynard-Reid Mam suggested making the Drawing Class inside the support library so that the developers can easily use the class to draw the bounding boxes in case of Object Detection.

Challenges Faced

TensorFlow Support Library supports the input as Bitmap, but the CameraX output is Image, so we also develop a preprocessing function for TensorFlow Support Library. The second challenge needs to modify the test according to the output of the image for streamlining the process. George Soloupis Sir helped me as well.