Beginner’s Guide to Developing on Augmented Reality Smart Glass
A Walkthrough Tutorial & Review of Building Any Application on the lightweight Augmented Reality Smart Glass — Mad Gaze
Overview
In this tutorial, I will first introduce some background information on smart glasses. Then, by using a hackathon project example, I will walk you through the entire progress from setting up the smart glass to wirelessly mirroring the display screen, building, and installing any application on it.
Disclaimer: This article serves as a reference for you to have a picture of how it’s like to work with smart glasses. I am using Mad Gaze as the device because this is the only usable glass I can borrow from my lab. No commercial deal to promote any product here.
Motivation
I wasn’t able to find an end-to-end tutorial online for developing on smart glasses. And after I posted my hackathon project experiences, many friends became interested in also getting their hands dirty on smart glasses.
In addition, my experience is that building a smart glass application is not hard given sufficient guidance. I thus decided to write this tutorial :)
Hope you like it and happy coding!
Background
So what is a smart glass? A smart glass in this case is simply a normal glass with a lightweight and portal Android OS embedded on the right handle, and there is a additional small piece of glass for projecting the display screen.
Basic Control
- Navigate the screen via a mouse (like in a computer): Move your finger around the touchpad.
- Enter / Select: Press the touchpad button.
- Return to previous page: Double tap the touchpad.
- Scroll the screen: Double click (not tap) the touchpad and then scroll.
- Turn power on or off: The power button is on the internal side of the right handle. Long press the button to power-on or power-off the device.
Here’s how a screen looks like. I am mirroring my screen display of the glass in a computer (will show how to do this in the next section). The below screen shows a list of apps available to use. You can imagine it like using your phone through a virtual interface right in front of you. Just like in normal smartphones, you can browser the Internet, take pictures with camera, etc. Normally, it will also have a play store where you can download official apps. (although I’m going to teach you how to download any app soon.. 👾)
Tools that I used
Using the exact same tools is not a must. Feel free to use any alternative. :)
Hardware
- Windows computer (or any computer that can run the softwares here)
- Smart Glass with Android OS (Mad Gaze, Ares)
- USB cable that connects smart glass with the computer
Software
- Android Studio
- Droid@Screen (optional, for Android screen mirror)
Others
- Wifi / 3G / 4G cellular network that can be shared across multiple devices
Start Hacking!
Step 1 to 11 is about displaying your smart glass screen on the computer. Step 12 to 13 and above is about how to develop and run applications on the glass.
- Follow all the instructions here to install Droid@Screen
It requires you to also install Java and Android SDK. You also have to properly set environment variables such as ANDROID_HOME and appending to the PATH variable. After running the Droid@Screen jar file with Java, you will be prompted to enter the full path of your ADB executable.
What is ADB (Android Debugging Bridge)? It is a command line tool to communicate with your Android device. It is usually installed under the path android_sdk/platform-tools/adb.exe
2. Set the environment variables
(Feel free to skip this step if have no problem with this)
What do the environment variables do here? It tells your computer where to find the file when a command is specified. For example, before setting the environment variable, you have to enter the entire java executable file path
C:\Users\tina2\Downloads\jdk-13.0.1_windows-x64_bin\jdk-13.0.1\bin\javainstead of justjavato execute the java command.
In your Windows computer’s Control Panel, search for “Edit the system environment variables” and then click to open the “Environment Variables” tab. Click “New…” on the bottom right and then enter ANDROID_HOME as the Variable name, and C:\Users\<your_username>\AppData\Local\Android\Sdk as the Variable value.
To execute the command java directly in the terminal, you have to add the path of your Java’s bin folder to the PATH environment variable. For me, it is C:\Users\tina2\Downloads\jdk-13.0.1_windows-x64_bin\jdk-13.0.1\bin
Similarly, to execute the command adb directly in the terminal, you have to add the path where your adb executable is located in. For convenience for also executing other common commands, you can add all the following paths:
%ANDROID_HOME%\platform-tools%ANDROID_HOME%\tools%ANDROID_HOME%\tools\bin
You will see the below screen by simply double-clicking the line containing the PATH variable (the 4th line as shown above). Then press the right button “New” to add a new line to the PATH variable. We add a total of 4 lines here.
3. Connect your smart glass and computer to the same wifi / cellular data
4. Connect your smart glass and your computer using the USB cable
There are two purposes for this step. One is to mirror the screen via the cable. The other is to figure out the IP of the smart glass and to have the smart glass listening to the TCP/IP port in later steps.
5. Check if your smart glass devices has been successfully connected
Open the terminal, and type the command:
adb devicesYou will see the following screen if succeeded:
6. Mirror your smart glass screen on the computer with Droid@Screen
java -jar C:\Users\tina2\Downloads\droidAtScreen-1.1.jarYou will see the below screen if succeeded:
6. To mirror your smart glass screen wirelessly, run the following command to know the IP address of your smart glass. It is 172.20.10.12 here.
adb shell ip route
Why the smart glass IP is 172.20.10.12?
The line172.20.10.0/28 dev wlan0 photo kernel scope link src 172.20.10.12 metric 310means that packets sent to any IP address within the subnetwork 172.20.10.0/28 must be sent through the network interface wlan0 with 172.20.10.12 as the source IP address. Since our smart glass is the one sending the packet, we identify this as its IP address.
7. Set ADB to listen on TCP/IP on port 5555 (feel free to use other unoccupied port)
adb tcpip 55558. Unplug your USB cable between your smart glass and computer
You should see the “List of devices attached” is empty when running
adb devices
9. Connect smart glass wirelessly (replace the IP address with your own)
adb connect 172.20.10.12:5555You should see your smart glass device again in the “List of devices attached” when running
adb devices
10. Congratulations! You can now see the screen of your glass in your computer. Your friends can also feel the joy when you’re playing with your glass!
11. Adjust the size of your screen by pressing the magnifier button below
12. Open an Android application project in Android studio
As described previously, the smart glass is actually running on Android OS. Thus, any Android applications can be compiled and installed in the smart glass, given that the Android OS version is compatible with the application. After connecting your glass and the computer (either wired or wirelessly), you can select your glass and run the app on Android studio:
13. Run your app on smart glass
After successful installation, you will see your app appear as one of the app icons here. Click on the app and then you can use it as normal apps.
Bonus: How to Download any Apps in Android?
- Go to https://apkpure.com
- Search for your desired apk and download it
- After connecting ADB with your device, install the apk via ADB
adb install ~/Downloads/example.apk4. You are good to go! :D
Review
After a day of experimenting with the smart glass for the hackathon, I summarize my experience with MadGaze Ares as follows:
- Short battery life: Within 3 hours of my hackathon demo period, the battery went out for 4 times.
- Long recharging time: When the device is powered on and connected to wifi, it takes around 4 minutes to charge for 1% of battery.
- Not steep learning curve to control the glass: There are only two buttons (power / touchpad) and two types of gestures (tap / double-tap).
- Text input can be improved: Now you have to position your mouse on the exact location of the element on your keyboard. The size of the keyboard can be larger. Luckily, text input is rarely needed for most applications.
- The weight of the glass is the same as normal glass: This is a big plus for me. All other VR / AR headsets I’ve tried are too heavy for daily use.
- Eyes won’t get tired when looking at the screen: Although the screen is small, I still feel comfortable even when using it for a long time.
- Not overheating too seriously: This is compared to other similar weighted devices such as Google Glass. I only feel a little bit warm when running some applications.
- The overall performance is similar to Android phones: The speed and response time is not second to normal Android phones.
- Accurate voice recognition and voice control: By saying the name of the app, the smart glass will automatically open the app for you. The recognition accuracy is high and the response time is fast enough.
- Particularly delightful when taking photos with the glass: It feels really amazing the moment you capture the scene you’ve just seen in your glass.
- Fashion in design: I think the glass looks good on me.. In addition, wearing this glass, every one will stop by and look at you with curiosity.
Wrap Up
Now you have an overview of how a smart glass looks like, how to wirelessly mirror the smart glass screen, build and install any application you want on it. The smart glass technology has its limitations but definitely worths a try!
This is actaully my first tutorial on IoT / AR related applications. Before this, I wrote mostly about blockchain and cryptocurrencies as an enthusiast.
Hope you like it. I’m open to any suggestion. Thank you:)
Special thanks to my lab mentor Tristan BRAUD on providing lots of guidance!
