In Vino VeritAI?
Can WineMe be your personal AI sommelier?
Picture this; after a long week, it is finally Friday and you have gotten together with a group of friends. Everyone has bought a bottle of wine and the first course has been served. The dilemma strikes — which bottle do you open? Pairing food with the correct wine shows a high level of sophistication, but also requires a level of knowledge and skill that you do not yet possess. Were you at a restaurant, this problem will be mitigated by the waiter who will have recommendations. But you are at home — there is no expert waiter to make a pick from your wine menu. What do you do?
Enter WineMe! WineMe searches to solve this in-the-moment lack of decisiveness by bringing the waiter from the restaurant right to your table as an in-your-pocket expert.
How does it work?
WineMe uses artificial intelligence (AI) to elevate your dining experience. Simply photograph your dinner plate, and the app will recommend the perfect bottle of wine from the ones you have entered beforehand. This way, you can enjoy your meal without needing to leave the company of your guests. If the wine recommended does not suit you, you can ask for a second recommendation. Furthermore, each recommendation is accompanied by an explanation of how and why the app made this particular choice. This includes the information on what it recognized your dinner as, what part of your plate it focused on and what flavors in the wine, that made it suit your dinner.
Utilizing a Domain Expert
To get a better understanding of what is important when pairing food and wine, we contacted a sommelier. Here we learned the basis of how they determine which wine to choose, like if the food has a strong taste, one should pair it with a strong wine. Furthermore, we learned about the key concepts of wine which included the notes of a wine, whether it was dry or sweet and the intensity of the taste. The sommelier provided us with their flavor chart, which is what they usually base their judgement on. By knowing these attributes of a wine, one would be able to determine if it would go well together with a given dish.
Implementation
The AI model behind the app is an image classification model trained by using the TensorFlow guide and implemented in tensorflow.js. The model is trained on 101 dishes from the Kaggle dataset Food-101, which contains more than 100,000 images. These dishes included both main courses, entrees and desserts, ensuring a great variety in the dishes recognized. The model is able to recognize whether a dish should be served with red, white or rosé and whether this wine should be sweet or dry. This is based on the interview with the sommelier.
As a proof-of-concept, the app currently utilizes a dummy database containing predefined wines and their respective categories, such as sweet red or dry white. For example, if the model suggests a dry red wine for a dish but none is available, it will explore alternative categories like dry white until a suitable wine is found. Although the wine categories are currently managed as arrays, a more robust implementation would involve using a dedicated database like MongoDB to store and manage wine information effectively.
To make the app available on the web, we followed Henrik Pedersen’s guide which allowed us to make the app accessible through GitHub Pages.
The app is built with HTML, with its functionality provided through JavaScript. All of the styling was done through CSS and based on a Figma made before the implementation started.
Human Concerns
Our focus when designing the app was on the human experience. We aimed to give users control over recommendations by offering them options. This was achieved in several ways. Firstly, if two wines met the recommendation criteria, both would be suggested to provide multiple choices. Additionally, we introduced the “Another Wine” function and button, which recommends a second-best option based on the given wines. We also prioritized transparency in explaining how the recommendation was made. To accomplish this, we created a page dedicated to providing further details on what the app observed in the picture and why it recommended a particular wine.
On this page (Fig. 1), we outline what the app identified in the meal, such as “dark meat” for a dry red wine, along with the wine type. Moreover, we planned to include a heatmap to illustrate where the app focused its attention when making its decision, reassuring users that it primarily considered the plate. By providing users with this information, we aim to enhance their understanding of how the AI model functions and enable them to identify any potential errors. Below the heatmap, one will find additional information about the recommended wine, such as its tasting notes and intensity. By offering a detailed explanation of how the model reached its decision and providing information about the wine, our goal is to improve users’ understanding of wines and their ability to pair them effectively in the future. At last, we want the app to be trustworthy with no hidden agendas. This is one of the reasons why the model only recommends wines that you’ve provided; to avoid any suspicion that a recommendation could be influenced by economic factors.
What’s Next for WineMe?
The current version of WineMe’s AI does not support the ability to create heatmaps using TensorFlow’s Grad-CAM implementation. Obviously, the AI-model would have to be restructured so that heatmaps will be shown on the page that explains the AI’s decisions.
We could also introduce a feedback feature, that allows the user to provide feedback on their dinner experience to support user agency. This would help the app learn and improve recommendations over time, using a method called reinforcement learning.
Currently, WineMe focuses on making on-the-spot decisions. However, it could be expanded to also support dinner planning by allowing the user to enter the name of their dinner in text form. This prototype has the button for this feature, though the feature is not implemented. Another way to allow this planning could be to train an AI model to also recognize ingredients, though this would require extensive training and data collection.
Conclusion
WineMe offers a practical solution to the dilemma of pairing wine and food, by utilizing technology. By using artificial intelligence and user-centric design, WineMe simplifies the wine selection process, providing users with tailored recommendations based on their meals. For future work, WineMe has the potential to evolve further, with enhancements such as heatmaps and feedback mechanisms to better support the user’s needs. With WineMe, the perfect dinner experience is just a tap away.
