AR Menu

An Android Experiment

By Asad Mansoor, Moiz Muhammad and Justin J Carlson

Hi there, welcome to the behind-the-scenes of how AR Menu, an Android experiment for Google I/O Challenge 2017, was created. If you are here for the first time, check out the experiment on GitHub or Google Play store.

The Challenge

Back in February, Google announced the 2017 Google I/O Experiments Challenge to showcase creative projects that were created for Android, Chrome and AI applications. Not only these experiments are fun to interact with, but they include ways in which technology could be evolved in the near future. Android experiments, in particular, include projects that showcase how Android can be used in an unique and innovative way. This opportunity triggered a series of hackathon-style sprints that turned a simple cool idea into a product that I believe could provide a service to consumers. With the use of this blog post, we hope to shine the light on the importance of this experiment, the challenges we faced and how you can contribute or build something similar for yourself.

Menus are poorly design and obsolete

The way we dine and experience different cultures have significantly transformed in the past decade. Restaurants have seemed to evolve from a materialistic design point of view of hospitality and decorated designs to mesmerize the consumers within their environment. However, the main product that a restaurant sells is constrained to a paper menu, limiting the attributes of the items as a list view. Not only this prevents the product reaching its full potential in marketing the product, it can also allow the consumer to introduce bias on particular food items through previous experiences or unclear representation. We believe food is an art form and wanted to encapsulate the information being presented in a menu to a 3D graphical representation that would accurately showcase each item.

The vision for AR Menu is based on convenience. Suppose, you walk into a restaurant to grab a quick bite and find an empty table to be seated. You pull out your Android device and scan a target placed on the table, that will pop-up the items being served in an augmented reality view. Fancying all of the available choices, you select the item you wish to order and submit a request to the kitchen from your phone. After you have eaten, you can simply pay from your phone and be able to leave a review as you head out. In case you are an Instagram user, you do not forget to capture a quick picture of your food to share it with the world :P

Tap and order.

Demo

AR Menu showcasing menu items in a form of augmented reality.

The Implementation

Part I — Technologies

Part II — Image Target

Part III — Unity Model Generation

Part IV — Android Component

Part I — Technologies

AR Menu is created using

• Vuforia
• Unity
• Android Studio

Part II — Image Target

The image target is a resource that the AR camera will recognize and portray the augmented models. The types of target can be selected as a single image, cuboid, cylinder and 3D Object.

To set up your image target, you will need to access Vuforia Developer Portal and create an account. Then you will need to generate a license key for the application, which will be imported into Unity. Create a database and the corresponding image target from within the portal. Granted that you have chosen a single image as your image target, Vuforia will indicate a rating from 0 to 5 to illustrate how effective it is for the AR camera to pick up on the features.

The higher the rating, the easier it is for the AR camera to track the image target. To achieve optimal efficiency, select an image that has features like sharp, spiked and chiseled details. During the experiment, we tried two different images shown below with their corresponding ratings.

So why did we select the QR Code? Well, the features shown in the QR code provided a great rating for the performance of the application. More importantly, we wanted the users of the experiment to be able to interact with the image target using the AR Menu application, as well as using any standard QR Scanner application to be able to reach this blog post.

Menu image (left) rated as 3 and QR Code image (right) rated as 5.

Part III — Unity Model Generation

Unity is used to import Vuforia Unity package and a pathway to generate the 3D models. With the package, a AR camera is provided which can be inserted into the scene. To fully configure the AR camera, we will need to provide the license key which we received from the developer portal. After that, Unity is simply an open playground to customize our augmented model however we like. We considered two paths in delivering our experiment:

Unity Asset Store

The Unity Asset Store contains various free models which you can easily import into your project. Even if you are looking for more complex designs, they can be easily acquired with a small sum of price. If you cannot decide on an asset, we highly suggest the following links:

• 3D Bakery Object
• Red Samurai

Bootstrapping Unity Models

If you have experience in creating Unity models or would want to learn, there are plenty of online resources and video tutorial to assist you. Google is your best friend here.

The development of food models in Unity.

Part IV — Android Component

You can deploy to an Android device directly from Unity to test out your experiment. The AR Menu wanted to take it one step further. Defining the realm of Augmented Reality on Android devices, we wanted to showcase a business-minded solution that was more than just entertainment. The use of Augmented Reality could be integrated in our daily lives and business application to showcase real attributes of products. The Android applications wraps the augmented menu features with a series of activities that resembles the dining experience. We wanted to capture more than just an augmented view, we wanted to introduce the convenience of the experiment on how users can interact with restaurants of the future by visually inspecting the menu items, placing their order and paying from their phone.

Additional Android activities in companion with the augmented reality module.

Interested in contributing or using our code?

Find the project on GitHub where you can fork the repo and push us your work. The ARMenu directory contains the source code for the Android application and the ARMenuUnity contains Unity scripts.