Rasoee — A Picture is Worth a Thousand Ingredients
An attempt by a group of food-loving students to make the diverse world of different cuisines more navigable for our fellow eating enthusiasts.
Rasoee won First Runners-Up at the 2020 Global PyTorch Summer Hackathon in the mobile/web application category. Written by Arijit Gupta. Made with Ameya Laad, Anish Mulay, Dev Churiwala and Smitesh Patil, undergraduate students at BITS Goa
What is Rasoee?
Our main inspiration while making this was looking at the number of people sharing pictures of food they’ve made at home during the lockdown. We then started wondering if there was someway to make it easier for others to replicate the ones they like themselves. Rasoee is a web/mobile application that can be used to identify food items from images. The app can identify from 308 different types of dishes from over 5 different cuisines. After identifying the dish, it also provides(if possible) a list of key ingredients that are used to make this dish, and link to a recipe and some tutorial videos on YouTube.
How it works
Rasoee uses an image classification model trained on a dataset of 308 images. We chose to train an EfficientNet B2 model that we believed was small enough to be deployed as an edge model and had enough depth to classify images into such a large number of categories. The making of this involved many steps, starting from creation of the dataset and ending with the making of the mobile and web application along with integrating the model into it.
Scrape your way to a dataset
While analysing the datasets available online, we saw that the most valid dataset available was the Food-101 Dataset.
But, the Food-101 Dataset seemed a bit restricted in variety to us, and while the quality of the dataset is extremely high, we decided to go out on a limb and create our own dataset. To do this we used a bing image scraper and a lot of patience. What resulted was a terribly noisy dataset that we then cleaned to the best of our abilities, but we now had a dataset with 308 different categories of food.
Models, models, everywhere
Now came the moment where we had to make the most difficult decision of all, which model to use? PyTorch has such a great array of models that can be used even in the official documentation that we were spoiled for choice. After going through many models, and looking at their size v/s depth trade-off we settled on using one of the EfficientNet implementations for PyTorch, specifically the EfficientNet B2 model which has 9.2M parameters and a size of ~300Mb. Three of us were training different models and preprocessing, but we finally settled on the model that gave us the highest accuracy, and we achieved an accuracy of 91.31% using CrossEntropy loss. The training script is very basic and the training notebook can be found here.
User interface
The application has both a website and a mobile version. While our primary aim was the mobile application as it would be more convenient to users, the website can also be used on a mobile browser or any other edge device. It is very simple to use, just upload an image and wait for the results.
The mobile application has a very intuitive UI with a simple press to upload and a clean results screen.
The model was optimized for edge devices and implemented in the mobile app with a lot of ease, and a lot of credit for that goes to the handy TorchScript feature that is provided by PyTorch which allows you to convert your model into a script module.
Looking ahead
We really wanted to make a segmentation and recognition model for various ingredients so that we can tell the user whether they have all the ingredients required for a particular dish or not, or tell them what all dishes they can make with the ingredients available to them. We hope to be able to add some functionality like this in the future.
So that was all about our project! Feel free to check out:
- Our Website
- The Github Repository
- The Mobile Application
Finally, we would like to thank the PyTorch and Devpost teams for organizing the Global PyTorch Summer Hackathon, despite the trying times. The online Hackathon experience was great and all of us had a ton of fun working on our idea, and it was great to see it deemed worthy of a prize.
