As I recently began working in a lab, I became aware of the problem that biochemists encounter all too often. Namely, hunting under a microscope for crystallized proteins in a tray, each one with hundreds of wells. This rote action seemed to me like a perfect application for automation and machine learning, two of my favorite things! I thought to myself, what if I could build a contraption to take pictures of every well in the tray and have the computer simply report back to me which wells have crystals? This is my attempt to do just that (note, it is a work in progress):
I want to emphasize that there is actual utility to such an idea. These trays are used universally in scientific research and industry. With each tray taking a researcher 10+ minutes to analyze, reducing the overhead of such a tedious procedure could let our best and brightest get back to focusing on the actual science, not a game of eye-spy.
My initial idea was to create some sort of frame that would universally attach to microscopes and would move back and forth, using a standard microscope camera to snap pictures. From my previous experience with Raspberry pi controllers I knew I could use one to control the motors I would use.
I quickly discovered the smaller motor was tragically under powered so I switched to using two larger NEMA motors:
I’m fortunate in that I also am a 3D printing hobbyist and that I can fabricate my own parts, which is what I did to create the frame of CrystalBot:
I designed the first prototype in Fusion360:
It was at this point that I decided to not rely on the user’s microscope at all. Having a dedicated camera greatly simplifies the engineering. After some looking on Amazon, I found a handheld microscope camera perfect for this application:
Now I had my first prototype. The robot moves the tray and takes the 288 individual pictures:
So how do I get a computer to do all the hard work for me? This is where I get to use machine learning!
As all machine learning practitioners know, the basis of a good model is data. Fortunately, the MARCO (MAchine Recognition of Crystallization Outcomes) organization has already gathered the perfect dataset! (40,000+ training images)
In fact, MARCO has already trained a model and published their methods in this brilliant article.
The MARCO model is a convolutional neural network based on the popular Inception-v3 model. Using this model I can achieve 95% correct identification of crystals in my wells.
But can I do better? I have been playing around making my own CNN but with limited success. Currently I’m trying to build a model based on the NASnet architecture, but my results don’t come close to the accuracy of MARCO’s model.
Even though I haven't been able to make my own model as successful as MARCO’s, I have reduced the time of analyzing an entire crystal tray to just over 2 minutes!
Now that I have a working prototype, I am continuing to experiment with new neural network models for image classification. With the rapid advancement of CNN’s, I’m excited to keep learning and improving my machine learning techniques!
Additionally, the current prototype is just that, a prototype. I am now looking into making CrystalBot a more durable machine, potentially using milled aluminum instead of plastic and wood. I also want to make a better user interface, and solder a permanent circuit board instead of the bread board.
My intention is to release the designs and code and keep this project open source so that any researcher is free to try and replicate/improve my design. The current code, written in python, can be found in my github repository.