‘Snout’ Makes Mushroom Classification Fun & Easy with Augmented Intelligence
Snout is the winner of the summer 2019 Hal R. Varian MIDS Capstone Award.
With over 135 species and 66 genera of some of the most popularly-found wild mushrooms available, there is so much to explore in the world of mushrooms. To guide and foster this exploration, Master of Data Science students Andrew Larimer, Kathryn Papandrew, Daniel Rasband, and Rahul Vaswani created Snout, a mobile app that leverages augmented intelligence to guide users through the steps of responsibly assessing a mushroom they have found.
Snout prompts the user to take photo(s) of the mushroom for inference to narrow down the most likely genera and then involves the user in narrowing down possible species through intelligently-determined human-in-the-loop questions. Throughout this classification journey, users can learn more about parts of a mushroom and characteristics. The team’s goal is to utilize Augmented Intelligence in order to help people learn about the fascinating world of mushrooms.
What inspired your project?
Dan Rasband: We initially wanted to help hikers and backpackers easily identify edible species of plants in emergency situations. However, as we researched different target groups, we quickly realized that a phone app solution was inadequate for identifying the edibility of plants and decided to pivot. We found mushroom foragers to be a large and welcoming community, and fell in love with the wild world of mushroom identification.
What was the timeline or process like from concept to final project?
Rahul Vaswani: Early on in the project, we decided to use weekly sprints and to utilize Github project boards. This allowed us to create small, manageable tasks to be completed each week. Thus, our first few sprints were centered around data collection and acceptance. The next few were focused on modeling and fine tuning. In our final sprints, we focused our efforts on productionalizing our work and getting the app up and running.
How did you work as a team? How did you manage to work on your project as members of an online degree program?
Rahul: If you would have told me at the beginning of my MIDS career that I would be working on my capstone project with my friends from different states, I wouldn’t have believed you. We had all worked with each other at some point during our MIDS career, and our work ethic and passion meshed together very well. The online aspect was only tough with respect to us being in different time zones. However, by utilizing Slack and Zoom, we were able to reach out to each other daily.
How did your I School curriculum help prepare you for this project?
Andrew Larimer: A few of us had taken Data Science W251, Deep Learning in the Cloud and at the Edge, which gave us the foundation for building models that could be run on-device out in the world, which was important to us in an app that is designed to be used out in nature where wireless service may be spotty. That course also gave us a taste of computer vision models within MIDS, which was a great starting place for our work on Snout.
Do you have any future plans for the project?
Dan: We’d love to continue working on the project by developing new features and supporting a wider variety of mushrooms. Along with that, we hope to see continued use of the phone app.
How could this project make an impact, or, who will it serve?
Andrew: Many machine learning models can get pretty good results but either still fall short of human performance or else simply aren’t perfect enough to trust with very serious decisions.
While the domain we worked in was mushrooms, I think at its core this was a project about exploring the interaction between a helpful machine learning model and a human user in making a decision with potentially serious consequences (due to the dangers of misidentification in consuming mushrooms).
We also worked to make sure that doubt, appropriate to the accuracy of the model, was visually expressed in the results by prominently displaying an, “Or it could be…” section with other species below our top guess.
We strove to educate users about the consequences of their decisions, provide them rules that would keep the safe, grant them access to a model that would help with their decision-making, and guide them through the rest of the decision-making process in a way that empowered their decision-making faculties rather than replacing them.
We also hope the app will help people’s phones redirect them out into nature and help them explore and learn more on their own!
