Augmented reality with Xamarin and Wikitude
Augmented Reality (AR) blends reality and virtual content together. In contrast to virtual reality, where the user completely immerses themselves in a virtual world, augmented reality focuses on overlaying reality with additional information. This often leads to more complex requirements for position determination and tracking. On mobile devices such as the iPhone, augmented reality has been available since 2009 and continues to evolve with expanding hardware capabilities. However, since each mobile operating system requires a different implementation for the same AR applications, this article explores the possibilities of implementing an AR application using the cross-platform development environment Xamarin.
Xamarin
Xamarin is a software development kit (SDK) based on Mono, which is built on the programming language C#. It enables the development of applications (apps) for Microsoft Windows Phone, Apple iOS, and Google Android platforms. Apps written with Xamarin can access all the native features of the respective platforms.
For iOS, the program code is converted and translated into native Objective-C code. In Android, the app is compiled and executed using the Mono runtime environment. Windows Phone utilizes C# as its native programming language and can directly execute the application. The significant advantage of using Xamarin is that the apps developed with are sharing the codebase an can run on the same level as natively developed programs.
Xamarin.Forms
To create consistent user interfaces with just one source code, Xamarin offers the option to use “Xamarin.Forms”. This allows an app to maintain the specific operating system layout while remaining portable across all platforms. This feature enables the creation of an app with a familiar layout for the user.
Xamarin.Forms is not as feature-rich as native programming for user interfaces. However, it is sufficient for creating all the basic elements of a user interface. The app can later be enhanced with platform-specific functionality. For this purpose, additional code paths, known as PageRenderers, are required for each platform.
Augmented Realtity Frameworks
There are numerous AR frameworks available. These software development kits (SDKs) simplify the integration of functionalities for developing an AR application. The SDKs handle tasks such as webcam or camera usage and spatial tracking. Some SDKs also provide additional tools for placing and manipulating custom content, such as 3D objects, in the virtual reality space.
The website “SocialCompare” lists more then 50 different frameworks in a comprehensive table. These frameworks offer varying levels of functionality and target different platforms. Some are designed for mobile operating systems like Android or iOS, while others are intended for Windows or OSX systems.
Wikitude
The Wikitude Framework is an Augmented Reality SDK developed by the Austrian company Wikitude GmbH. The framework allows for the display of location-based AR objects through the screen of a mobile device. The position is calculated using GPS, compass, and motion sensors. Additionally, the Wikitude AR SDK supports marker-based AR, where not only QR codes but also complex images can be used as markers.
Currently, Wikitude is the only framework that supports Xamarin. The SDK is offered under a commercial license, but a free trial version and a free version for students are available upon request.
Wikitude Project Files
The Wikitude Framework utilizes this format to implement AR functionality. The content of a project file includes the initialization HTML file, the corresponding JavaScript code, the objects that are to be inserted into the real world, and the marker in Wikitude’s own format. Once a project file is created, it does not require an internet connection to run.
The project files can be created using the web application called Wikitude Studio.
iBeacon
The iBeacon technology standard was established by Apple. An iBeacon is a small Bluetooth 4.0 Low Energy transmitter that sends a signal according to the iBeacon standard. The UUID allows for the identification of an iBeacon. Additionally, further information can be transmitted through the Major and Minor fields of the signal. The communication always occurs in one direction, from the iBeacon to the receiver.
To utilize this technology, a Bluetooth Low Energy-capable smartphone with iOS 7 or Android 4.3 or higher is required.
Beispielanwendung
Below, a multi-platform AR app developed using Xamarin.Forms and the Wikitude SDK is presented. The indoor positioning is achieved through the use of iBeacons.
Szenario
The program aims to enhance the experience of visitors at an art gallery by providing additional information about the exhibited photographs. Upon entering the gallery, visitors are informed that an app is available in their respective app stores. The app enriches the exhibited photographs with additional information. For example, the app allows users to augment reality through the display of their mobile devices by adding images, text descriptions, video content or 3D models. Each photograph is equipped with an iBeacon, which communicates the user’s proximity to the corresponding photograph to the app. When the user launches the app and stands in front of a photograph, the app loads the appropriate component within the program by combining the iBeacon with Wikitude AR content. The app uses the device’s camera to display additional information on the screen as a real-world overlay. Users can interact with the content by moving their mobile devices or touching the objects on the device’s display.
Program structure
The program, called ARArt, is structured as illustrated in the following diagram.
The bottom layer of ARArt is the Xamarin.Forms section, which is responsible for the user interface. This is followed by the implementations for the iBeacon functionality and the invocation of the Wikitude library for iOS and Android. Next are the native implementations of Wikitude for each platform. The final layer is the Wikitude Project File, which includes the marker and the objects to be virtually added to reality.
Components
The ARArt program, as described in the section on program structure, is divided into several components. The following section provides detailed explanations of the most important components.
Xamarin.Forms Component
In the Xamarin.Forms component, a unified user interface is defined for all platforms. Additionally, the Beacon class is created with its respective properties.
public class Beacon{
public string Name{ get; set; }
public string Uuid{ get; set; }
public int Major{ get; set; }
public int Minor{ get; set; }
public double Accuracy{ get; set; }
public int Rssi{ get; set; }
public string Proximity{ get; set; }
public Beacon(){}
}The layout consists of an activity indicator for searching for iBeacons or photographs. Additionally, the IBeaconLocater interface is defined with the methods: List GetAvailableBeacons(), void PauseTracking(), and void ResumeTracking(). The platform-specific implementations will later rely on IBeaconLocater.
Furthermore, a view is generated where the Wikitude component is launched. Since this differs between Android and iOS, an empty class called ARPage is created as a placeholder, which will later be replaced by native views.
Native Xamarin Components
In order to access native interfaces such as Bluetooth, a separate code path needs to be created in Xamarin for the IBeaconLocater interface defined in Xamarin.Forms for each platform. Additionally, different implementations are required for displaying the Wikitude component, which are covered by the ARPage class.
Xamarin.iOS
The class BeaconLocateriOS is created, which contains all the logic for receiving iBeacon signals. The class implements the interface IBeaconLocater.
In order to display the Wikitude component, a separate PageRenderer must be defined for the ARPage view. This creates an iOS-specific view and refers to the Wikitude component in it.
Xamarin.Android
Since the handling of iBeacons is somewhat more complicated under Android, the classes MonitorNotifer and RangeNotifer must be created in addition to the interface implementation BeaconLocaterAndroid. These inform the BeaconLocaterAndroid about changes in distance from the Android device to the iBeacons.
The Wikitude component must be called by its own PageRenderer for the ARPage view. The ARPageRenderer also needs a suitable Android layout XML file which must also be created.
Android security guidelines require that permissions are specified in the AndroidManifest.xml file when the app is created. The app needs access to Bluetooth, camera, compass, internet, location and accelerometer to function. In addition, the service for the detection of iBeacons must be entered within the field (see code listing).
<service android:enabled="true" android:name="com.radiusnetworks.ibeacon.IBeaconIntentProcessor">
<meta-data android:name="background" android:value="true"/>
<intent-filter android:priority="1">
<action android:name="com.PolyTJSearch.Android.DID_RANGING"/>
<action android:name="com.PolyTJSearch.Android.DID_MONITORING"/>
</intent-filter>
</service>Wikitude Component
The Wikitude component must be integrated into the component above via a reference. Once this has been done, the class Wikitude.Architect can be called in the iOS or Android component. This class is initialized and can then process Wikitude project files.
Evaluation
For the evaluation, a captured image from the 1930s should be expanded with AR content. For this the digital image was converted into a marker by Wikitude Studio.
An image, a 3D object, a button and a text were set as AR content.
The identifiable marker was reliably recognized under different conditions. Tests were conducted in the exhibition using the original image, tests were conducted using the digitally displayed image on a monitor, and tests were conducted using a simple black and white photocopy of the original image.
In addition, the app was tested with various devices (Samsung Galaxy Nexus, Acer Nexus 7, iPhone 6, and iPad mini were used). The app worked as intended on all devices. However, it was observed that a poor image quality of the smartphone camera has a negative impact on marker recognition.
Result
Not every image is suitable as a marker. Images with sharp edges are better suited than paintings with smooth transitions. As shown in Figure below, the ARArt application performs very stable. Even the black and white photocopy used, including the reflections, reliably displays the AR objects. However, it has been found that the camera quality plays a role. The AR application works more reliably on an Asus Nexus 7 and iPad mini compared to a Samsung Galaxy Nexus equipped with a lower-quality camera.
ARArt is compatible with Android and iOS devices and thanks to the use of Xamarin.Forms, it looks identical on both platforms.
It has been found that the Wikitude SDK is a comprehensive tool for augmented reality applications. The SDK can integrate texts, images, buttons, HTML widgets, 3D models, and videos. It is also capable of displaying multiple objects simultaneously. Limited interaction with the objects is possible.
The integration into Xamarin is currently not straight forward. It would be nice if Wikitude Project Files could be directly accessed within Xamarin.Forms. Additionally, Windows Phone is not yet supported by Wikitude within the Xamarin framework.
NOTE: Article was originally written in 2015 and you can find some sample code on GitHub.
