Google Summer of Code 2020

Project title: Deep learning using geometric features

Developers:

Student: Joanna Stachera

Mentors: Dimiter Prodanov, Sumit Kumar Vohra

Organization:
International Neuroinformatics Coordinating Facility

About Active Segmentation : https://sumit3203.github.io./

The aim of the Active Segmentation is to provide researchers an extensible toolbox enabling them to select custom filters and machine learning algorithms for their research. Semantic segmentation is one of the task that scientists are currently trying to efficiently deal with. Several approaches has been developed and implemented and recently, deep learning models has shown to achieve outstanding results. My objective was to enable non technical users to be able to efficiently use them and ultimately help in their research.

My contribution: https://github.com/jnnstchr/ACTIVESEGMENTATION/commits/testBranch

Project proposal for my Google Summer of Code project

This summer I had a pleasure to be working on my project Deep learning using geometric features as a part of Google Summer of Code 2020. I was incorporating deep learning models in the Active segmentation ImageJ plugin. I managed to incorporate implementation of U-Net — one of the most popular computer vision architecture and enable importing pretrained weights, as well as SegNet — another top-notch deep learning semantic segmentation model (which architecture already assumes use of transfer learning).

Image for post
Image for post
U-Net architecture. Source: https://arxiv.org/abs/1505.04597
Image for post
SegNet architecture. Source: https://arxiv.org/pdf/1511.00561.pdf

In order to implement it, I was using deeplearning4j library. One of the things that I haven’t expected is that it was much harder to implement deep learning in java than in python (mostly due to the fact that there are multiple tutorials available in python, which in java is not that straightforward). I really enjoyed the degree of freedom I had in terms of choice of what I intend to implement. Throughout the process, I received a lot of support from both of my mentors which helped me become not only a better software engineer, but also a scientist.

Image for post
Image for post
Active segmentation GUI

The first step in incorporating the project was creating the deep learning panel. It provides the possibility to select the intended model, choose suitable parameters (learning rate, number of epochs, epoch size) and also decide if one wants to use pre-trained weights as a part of transfer learning. One can decide about the data augmentation (flipping and rotation). I was hoping I would be able to incorporate the already implemented filters as possible preprocessing, however I lacked time and that is something that could be improved after the GSOC period.

Image for post
Image for post
Deep learning panel

The user is able to use deep learning panel in two ways — either by importing full ground truth (labels) or by creating partial truth — ROI masked used further on for ROI learning (still to be implemented, I would love to work on this after the GSOC period). In both cases, the user is able to see the labels via image overlay implemented beforehand.

Image for post
Image for post
Preview of imported labels
Image for post
Image for post
Sample results obtained by using U-Net architecture

An example of obtained prediction:

Image for post
Image for post
image
Image for post
Image for post
prediction

The results I obtained are not as good as intended, but I could only train U-Net specifically on a very limited dataset (4 images) with limited computational power (16GB of RAM in my computer). It would be really interesting to check the results using more powerful hardware.

In the future I would love to extend the models implemented to e.g. ROI -based learning — one of the papers I would really love to implement is here. This one, on the other hand is slightly more complex, yet it constitutes an improvement to the latter.

While researching on things I wanted to implement, I created a short presentation about ROI based learning:

In the future I would like to expand Active Segmentation with other models which would be helpful for its user, however it is also problematic because of limited access to hardware, such as GPU for training the models. This is also one of the reasons I decided to focus on learning based on using pretrained weights, making it more accessible for future potential users. Moreover, I would like to work on features I mentioned before (the use of already implemented filters as a part of data augmentation, implementing above-mentioned papers).

Other papers I used:

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store