AR-empowered Smart Kitchening
Cooking and managing large-scale Events with a Virtual Management Chef
This is joint work with Martin Hermannsen.
Introduction
The climate in catering kitchens is rough and tough. The spirit is dominated by hecticness. Tough time schedules, a great deal of staff and process managements, the dynamic shifting of resources — trained and untrained — is in the hands of the chief and demands much structure definition, stress tolerance, and a great management skills from a person whose actual passion is about creating a unique, delicious dish. To be successful, a contemporary chef needs both the magic of an outstanding cook and the management talents of a corporate CEO.
Pain
Let us look into the process of leveraging floater cooks. A floater is a cook hired to help out for a particular event. There exist a vast network of agencies to book a floater for a particular event, as those events typically significantly vary in size and menu scope.
Floater cooks have to adopt to their everchanging environments. In the time they are with the client they have little time to learn to cook the dishes fast enough. A company have often apprentices, but they can not cook all dishes. Employees need time to teach them. Due to the harsh climate and the stress in the kitchen, the staff can not teach the floater cooks and apprentices enough. Employees of a kitchen need training and a helper to make cooking easier. This would help employees in cooking and for the training and education of employees. This would also save employees time.
Gain
To solve this problem, we prototyped an augmented reality (AR) application to guide employees step by step through a cooking recipe without the need to control the glasses by using hands. In addition, each step will show which kitchen utensils and ingredients are need to use. These should be automatically recognized and highlighted by the glasses. The cooking handles are observed by machine learning, and if done incorrectly, the glasses will show the correct cut of the ingredients. At the end it will be shown how to drape the ingredients perfectly on the plate. So every cook can do any dish.
The new employees just have to put on smart glasses and follow the steps that are displayed. They will learn the processes in your kitchen. It will also tell you if you’re doing the job right or wrong, so you can revise any wrong handles and the dish will be perfect, even if your employees are not fully trained.
In addition, the glasses can be used as a central IOT-element in the kitchen, so that they can control all devices in the kitchen. For this we have to cooperated with kitchen appliances manufacturers and we would win a strong reference partner.
Our product can be used in many situations. Thereby many pain-points of our customers are addressed. First, we focus on the restaurant kitchen. Above all, the training costs of new employees and apprentices can be reduced there. The glasses can teach the employee all the cutting techniques he need. By learning the right cutting techniques, they can reduce waste.
The focus is of the augmented reality glasses is on learning cutting techniques. The glasses can also be used at education courses.
If the augmented reality glasses prevail in the private sector, then we can also offer the application to the society. The customers would have an enhanced cooking experience, whereby every dish would succeed.
When the private persons use the glasses, we gets knowledge about the customers of the supermarkets. To this we could sell the information which products they customers would buy next. So they could better plan the buys and sales of they products.
Related work
There are some related work in the augmented reality area. In a work Auxi of the “Schwäbische Hochschule Gmünd”, students have developed a concept for cook rookies with augmented reality. Speech is used to control a Microsoft Hololense.
In another project Holocooking, you use your phone as augmented reality glasses to cook a recipe. Here you can adjust the recipe to your own wishes and is guided step by step with a voice control through the recipe.
In a hackathon cocking learning videos were implemented in an augmented realit yenvironment.
The most similar project to ours is from HAW Hamburg. It was developed for the Foodboom App. Ingredients were recorded by camera. Based on the ingredients the App suggest recipes that can be cooked with the ingredients.
A big competitor is Ikea. The company has started a project similar to that of the HAW of Hamburg. Recipes were also suggested based on ingredients. These were detected by cameras, which were installed in the kitchen.
None of the related work tries to educate cooks. The most known cooking applications of the society are in the Appstore of mobile devices. But they do not all have to deal with augmented reality. The most famous in Germany are Chefkoch and KptCook. At KptnCook, the user is presented with three different dishes every day. There are also dishes that are exchanged monthly. When choosing the recipes is suggested in which supermakt the customer can buy suitable ingredients. Then there is a step by step explanation of how to cook the recipe. Probably the app works with supermarkets and sells them the data, which ingredients will be bought more often in the next few days. Chefkoch has an big recipe database which were always extended by the users. In addition, the recipes are rated by the community. The app also offers cook learning videos.
Technical solution
Selection of augmented reality glasses
At First the augmented reality glasses have to be selected. There are three main providers of augmented reality glasses: Google, Microsoft and Epson. Google has developed and launched Google Glass 2. The google glasses can only be bought in large numbers by Enterprise customers. Therefore, we can not use these for our prototype.
The latest augmented reality glasses from Microsoft is the Hololense 2. It costs 3115 € and uses a Windows operating system. Applications can be programmed with unity on the glasses. The display has a resolution of 2560x1440 pixels. It weighs 556 grams and the internal memory is 64 GB. In addition, the Hololense 2 has many sensors: “accelerometers, gyroscopes, mangnetometer, 5 microphones, 4 visible light cameras, 2 IR cameras, 1-MP time-of-flight depth sensor”. Connections can connect the glasses with USB, WLAN and Bluetooth.
Epson has two augmented reality glasses, the Moverio BT-300 and the Moverio BT-2000. The price of the BT-300 is 669 € the BT-2000 costs 2000 €. Both glasses use an Android operating system. The BT-300 has a display resolution of 1280x720 and the BT-2000 has a (960x 540) x 3. The weight of the BT-300 is 70 grams in addition, this still has a control unit. The BT-2000 weighs is 556 grams. Both glasses have the following sensors: A-GPS, light sensor, compass, gyroscope. In addition, the BT-2000 has a microphone. The glasses can connect to other devices via USB, WLAN and Bluetooth. The battery life of the BT-300 have a maximum of 6 hours, the BT-2000 have a maximum of 4 hours, but the battery can be changed during the operation. For long-term use, the BT-2000 is more suitable than the BT-300.
From our Hochschule Flensburg we got two Epson Moverio BT-300 for developing the prototype. Therefore, we have used these glasses to develope the prototype. Due the purchasing power of the restaurant industry is the Epson BT-300 a good choice, because the price of the glasses are low. The Microsoft Hololense 2 has the better technical equipment and the feeling is better, when you use the glasses a long time.
Develop the prototype
To develop an app for the Epson BT-300 we have used Android Studio. First, the cook can choose a dish. He can choose between an egg sunny side up and an egg well done (Figure 1). In the figure 1 is the background black, but on the glasses this is transparent. So the cook is not thrown out of reality and still can perceive everything. Subsequently, the needed ingredients are displayed. In the following steps is described step by step how to cook the dish perfectly.
For hands free control we have used the gyroscope of the glasses. With a quick head movement to the right and left can be navigated through the app. In the selection of the menu, you can move your head to the left to go to egg well done. If you move your head quickly to the right, you can choose the egg sunny side up. In all other activities you can navigate back with a head movement to the left and with a head movement to the right you can go to the next step.
Security in Android applications
With Application Sandbox android uses a Linux protection to identify and isolate apps. The isolation protect Apps from malicus apps. Android assigns for that an unique iser ID to each application.
The UID is used to set up a kernel Application Sandbox. The kernel guaranteed security between the apps and the system. So the apps can not communicate to each other and have a limited access to the operating system. The sandbox is based on the seperation between processes and file permissions.
Because the Application Sandbox is in the kernel, this security model extends to native code and to operating system applications. That’s why native code is as safe as interpreted code.
On some platforms, developers need a development framework, a set of APIs, or a language to enforce security. There are no restrictions in Android to write secure applications.
Unique Selling Proposition
A unique selling proposition makes a start-up unique.
Our glasses are the only AR glasses that teach the customer to cook. With the help of machine learning, the cooks will be shown how to cook. In addition, using machine learning to determine when is the best time to go to the next step. This would automate the navigation. None of the related work offers these functions. With this features is our augmented reality glasses unique.
