Research @ OffNote Labs : My experience [Nihal]

OffNote Labs is a home for independent builders and researchers. Seasoned researchers helping budding researchers learn and grow.

Firstly, how is OffNote Labs unique?

• OffNote Labs is completely remote — there are no boundaries on the location. This also allows students to do research during their academic studies itself. So if you can balance studies with research, this is for you.
• The quality and frequency of interaction is also exactly the same as an in person research internship! There are daily conversations, be it on discussing experimental results or survey papers to read.
• It is open to students from all disciplines. This is in line with the free dissemination of knowledge, especially on AI/ML concepts online which allows anyone to learn, regardless of background.

Overall, it offers a chance to perform meaningful, exciting research free from any barriers.

More about my experience..

For better logical understanding, this is spread across the following sections:

• The project
• Writing a paper
• Main learnings
• Good practices I followed
• Some things which needed improvement
• Conclusion

The project

I worked on the problem of real-time 2D hand pose estimation. Given an input image containing a hand, the objective is to identify the positions of the keypoints as shown below in the image pixel coordinates.

An example output

I worked on this project from Feb — Aug 2020, under the guidance and mentorship of Dr. Nishant Sinha.

How it started:

I summarized a few papers on hand pose estimation (2D as well as 3D). These papers had git repos available — helpful for quick experimentation. Two important observations were made :

• Approaches to 3D hand pose estimation use 2D pose as intermediate representation.
• Even 2D hand pose estimation — very slow, 5-6 fps on NVIDIA GTX 1050 4GB.

I also noticed that all papers for hand pose estimation use cropped pictures of the hand. This was done by a pass through a separate hand detector before they could be used.

Running two separate, lightweight models in this series fashion, gave 6–7 fps speed.

existing hand detector architectures

The above figure shows the pipeline used in previous work : two models in order to detect keypoints in images. One model for hand localisation, another for keypoint detection.

However running two models in series results in increased latency. Would it be possible to use a single model for both tasks? Inspired by object detection architectures like Faster RCNN, RetinaNet, the idea was to add a ‘head’ on top of such an architecture.

With this idea, the project then roughly followed this order:

1.Training and optimizing an object detector for accuracy and speed.

2.Incorporating an head on to the object detector.

3.Training the keypoint head with the appropriate loss function.

The results obtained showed comparable accuracy with current methods, with a 2x increase in speed over using distinct lightweight models in series! The approach of using a single model for both tasks was unique— but now we had the results to back it up. Which led to…

Writing a paper

• Results were compiled around the end of July
• Paper deadline — Aug 10.
• Initially — I had the wrong idea of writing a paper. I was more focused on editing sections within the paper — not the flow between them.
• My mentor however changed my perspective — writing a paper is more like how one would write a story.
• Without any links between sections, the paper will end up looking a like a disjointed set of paragraphs. Though the paragraphs may be self contained and coherent within themselves , to the external reviewer it will be barely understandable as he/she may not get the ‘big picture’.
• Thus the importance of connections between sections and subtly repeating novelties cannot be overstated.

Main learnings from the project:

• Finding the right tools was essential. Finding a library (Detectron2) which could do object detection efficiently as well as providing scope for customization efficiently was crucial in saving time and effort.
• Datasets are not straightforward. They may be poorly documented, many do not have data-loaders provided and may even have be unsuitable for the exact problem you are trying to solve.
• Don’t be scared of writing papers in Latex ! Resume-making aside, writing a paper in Latex is completely different — the entire format is provided in a template, only content is to be added.
• Visualize the outputs after each sanity check. Though the loss may be decreasing the model might not be learning anything . May be due to an error in the loss function itself !
• Keep reading papers, especially surveys. Surveys are a bonus — you get a lot of relevant papers at one place.

Some good practices which I felt were beneficial

• Summarizing papers while reading them. Has two uses — forces an understanding of key points of the paper. Also helps in writing the literature review section!
• Just ask for help if you’re stuck even after a few days. Going through all the steps as a pair, in logical order , makes things more clear.
• Add all updates to a common hub, like Notion. Easier to trace the project, especially when coming back to it after some time.

And things I could have done better….

• Tracking experiments effectively. This was a bit difficult to do as I depended on Google Colab for compute. One cannot see the entire history of the notebooks, thus tracing changes becomes difficult. Each experiment in a new notebook is difficult to manage too.
• Push even the smallest of changes to Git. I did not update the Git repo for the project when there were minor changes in the code. This caused a disconnect initially.

Conclusion

I remain indebted to Nishant sir for his mentorship — of course you can go far by yourself, but having someone to help you when ‘running the marathon’ enables one to go even further.

The entire experience was truly rewarding. I constantly came across things to learn and adopt. This was throughout the research cycle , from optimizing the initial literature survey to writing papers in a fluid manner. It was also my first remote research project — ensuring no communication gaps with your mentor in such cases may seem daunting but is in fact simple to solve with the right tools!