Using Optical Flow for Cilia Segmentation
The goal of this research is to classify normal and abnormal cilia movement patterns. The data are microscopic grayscale videos of cells with cilia.
Over the course of the semester in the Quinn Research Group, I worked on a team that was developing a solution to detect ciliopathies. Cilia are tiny hair-like organelles that are present on the outside of eukaryotic cells. They are responsible for tasks such as sensing a cell’s surroundings, clearing dirt from the lungs, and signaling surrounding cells. A defect in the cilia can be very dangerous and lead to ciliopathies like liver disease and retinal degeneration. However, ciliopathies can often be identified through the movement of the cilia, which is where my work this semester began.
The first step in modeling cilia movement patterns is to find out where in a video or image the cilia are. Our approach was to use Optical Flow to estimate motion between two frames for each pair of frames in a video. Then a single distance value would be calculated from the optical flow magnitude vectors. The distance value would then be converted to a binary mask based on a specified threshold.
Optical flow is a term for the problem of determining where a pixel has moved between two frames. There are a number of algorithms that try to solve the problem, but different methods work better for different types of movement. Because of the varying methods and accuracy, we decided to bundle together four optical flow algorithms: Lucas Kanade, Farneback, TVL1, and PyFlow. Each of these methods have different approaches to the optical flow problem and the papers are linked below.
The optical flow algorithms take in two frames and output a frame with u,v directional vectors. We then take the u,v directional vectors of the frame and transform them into a distance value using sqrt((𝑥2−𝑥1)²+(𝑦2−𝑦1)²); where x and y are the max and min over the entire video. Then all of the frames with distance values n standard deviations above the mean of the frame are turned into a 1 and everything below the n standard deviations is set to 0. This creates a binary mask to be used for further analysis. After obtaining the binary masks, we used a gradient boosting machine to predict a segmentation mask.
The basis of a Gradient Boosting Machine learning model is a Gradient Boosting Decision Tree (GBDT). A GBDT combines predictions of many decision trees by adding the trees together. A gradient boosting machine will iteratively choose split points in the data to minimize loss error. The gradient boosting machine starts with a weak model then adds another model that best minimizes loss.
We used the lightGBM gradient boosting machine package. The downside of gradient boosting machines is the compute time to choose the best split for the data because the algorithm must scan all possible splits in order to choose the best split. To reduce the compute time, a gradient boosting machine will implement data sampling, drop sparse data, and bundle features. The lightGBM model uses a unique approach to reduce compute time called gradient based one sided sampling. This means that the model will concentrate on data with larger gradients. The larger the gradient of the data the quicker the error loss can be reduced.
The lightGBM model takes in a NxM array, called the feature array, is stored as X. The lightGBM needs data to train the model which is stored in a Nx1 array, Y. X will then be used to predict Y, as in a simple regression model. The features for the lightGBM were the 4 optical flow method binary outputs bundled together in a (H*W) x 4 vector, which are validated by a hand drawn binary mask (HxW) x 1.
The accuracy was evaluated using an intersection over union formula. This means the number of ‘1’ pixels in a predicted and ground truth image is divided by the number of ‘1’ pixels in a predicted or ground truth image. The baseline accuracy for the optical flow segmentation methods by themselves was at most 41% accuracy. After running accuracy tests with the k-folds algorithm the lightGBM model was able to segment cilia with up to 71% accuracy. The k-folds algorithm divides data into different train and test data sets within the original data set. For example, a data set with 10 points, using 7 training and 3 test points, will choose different combinations of the 7/3 train/test k times. The k-folds algorithm gives a more accurate accuracy of the model because it tests the model on many ‘new’ data sets.
This research semester has been a fantastic opportunity to work with a team to solve a real problem with real impact. The full solution to this problem still requires more work on both the segmentation of cilia and classification of the movement. Additional work on the segmentation would include spatial methods of recognizing cilia given a single image; then creating a pipeline that incorporates both the spatial and temporal approaches.
The semester has taught me a lot about data science. I learned that independent and critical thinking skills are necessary for research progress. Also, that many solutions in research use the findings of previous work in novel ways; as is the case in this approach. Lastly, I learned that cleaning and restructuring data is one of the biggest tasks in data science, but has significant impact on the output.
TVL1
‘https://www.ipol.im/pub/art/2013/26/article.pdf’Farneback
‘http://www.diva-portal.org/smash/get/diva2:273847/FULLTEXT01.pdf’Horn Schunck http://image.diku.dk/imagecanon/material/HornSchunckOptical_Flow.pdf
Lucas Kanade
http://robots.stanford.edu/cs223b04/algo_tracking.pdfGradient Boosting Machines
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3885826/
