An overview of the programs that I study and use to develop augmented reality

by Maria Grishina, a UX/UI designer at Archy:

Here we work not only on the user experience in augmented reality but also the objects in it. Virtual and augmented reality programs were initially developed for special devices — helmets and glasses. The lack of powerful and compact equipment, as well as accessible and understandable software, was one of the factors that hindered the development of AR/VR technologies. The AR trend was revived again after the emersion of Apple’s and Android’s supported frameworks (ARKit and ARCore) aimed at developing mobile applications with AR that can be run on smartphones. In this article, I will briefly describe the programs I study and/or use for AR development.

Torch AR: rapid prototyping in augmented reality

In October 2017, Torch 3D raised $3.5 million to create free tools for rapid prototyping in augmented reality that can be done without programming skills. Now we have Torch AR app that can be downloaded by all users of iOS devices starting from version 11.3.

Mobile AR apps are a hybrid of regular mobile 2D apps and apps that are traditionally used in headsets or glasses. An AR mobile app can have three different interconnected layers:

  1. 2D UI elements that are bound to the screen,
  2. 3D and 2D elements located in three-dimensional space,
  3. images were taken by the camera.

Each layer can interact with the other. Any tool is not complete if it does not allow the developer to create and test the entire user interface for all three layers, or does not interact with the tools of other layers.

Torch AR users can create interactive prototypes, work collaboratively on projects in real-time, work with existing 3D and 2D files, tools, and workflows, as well as such features of the mobile system as payment, voice input, information exchange, social networking, and GPS navigation. Torch uses familiar gestures and tools to place objects, change their properties, add complex interactions, go through the entire prototype in Play mode, add collaborators; and all this without leaving the 3D mode. It is also possible to import 3D models from Sketchfab and Poly or 2D resources from Sketch and Adobe into your design and change their properties, for example:

- create high-quality prototypes in augmented reality without a line of code;

- use Torch interactive system to start the animation, make dynamic changes of objects’ properties, such as:

  • size
  • location and color
  • navigation between scenes
  • development of complex and personalized user interfaces
Torch AR application interface

Modeling in Blender

To start working with three-dimensional objects, I was looking for something free (or inexpensive), universal, functional, with a good base of lessons and tutorials. Blender happened to be the right choice. This open source program has been around for a long time. Behind its constant development is an army of dedicated artists, teachers, and enthusiasts. Blender boasts an impressive set of 3D modeling tools and is considered a viable alternative to paid modeling programs. It is capable of creating images and animations that match any other 3D modeling software on the market. Blender is also perfect for beginners because it supports many languages. It is not surprising that as the program develops, more and more professionals choose it for serious projects. In fact, features and functionality of this program are almost as good as those of more advanced 3D graphics packages, and it all comes for free. It seems like it gathered a little bit from every known program used for the creation of three-dimensional models. But at the same time, it is a completely independent, unique package of three-dimensional graphics.

Like in many other 3D editors, the user works in some kind of a scene or viewport. This is where a 3D model is created and edited, as well as rotated, moved, scaled, etc. It also displays all changes related to the animation, texturing, and visualization processes.

A comparison chart showing Blender features against other 3D editors (information from Wikipedia)

Program functions

  • 3D-modeling. Presented by almost all existing methods of creating and working with three-dimensional models. Object design is available based on:

- primitives,

- polygons,

- NURBS-curves,

- Bezier curves,

- metaballs,

- boolean operation,

- Subdivision Surface,

- basic tools for sculpting. As with 3Ds Max, the program offers a large number of different modifiers applied to the model;

  • Animation that comes in a package is really great. You can use the following tools:

- rigging (skeletal animation),

- inverse kinematics,

- grid deformation,

- limiters,

- keyframe animation,

- edit vertex weights, etc. The dynamics of solids and soft bodies, as well as particle animation is amazingly realized as well.

  • Texturing and shader sets. The program allows you to lay multiple textures on one object. It includes a range of tools for texturing, including UV-mapping and partial adjustment of the textures. A variety of customizable shaders gives you more flexibility when working with materials;
  • The ability of drawing. Yes, this 3D modeling program provides the ability to sketch using different types of brushes right in the application window. This function is there to help users create 2D animations. Its configuration is also flexible, especially when it comes to working with layers;
  • Visualization. The package is equipped with several built-in visualization tools and supports integration with various external renderers;
  • Basic video editor. Even many advanced users of the program are unaware of the fact that Blender has a built-in video editor. It is not as powerful as specialized software created for this purpose, but it is quite good;
  • Game engine. An extremely interesting feature of the program is a built-in game engine for creating interactive 3D applications. Python API program interface of the application allows you to make any adjustments to the game you are creating.

I went to Blender website, clicked on the “download” link, and chose the installers available for Mac, Windows, Linux, and even FreeBSD. As with many open source programs, the installation was easy and fast. After that, I ran the program. The splash page contained several links, including one to the official Blender online manual. The manual is a great resource, but it’s better to use it if you are already familiar with 3D and Blender. At first, Blender interface can be overwhelming. But it is not as difficult as it seems at first glance. Once you understand its logic, it becomes much less frightening; it is quite similar to Adobe programs such as Photoshop or InDesign. By default Blender interface consists of three areas:

1. toolbar — a set of tools that are used more frequently;

2. 3D viewport or work window — a vertical grouping of many management interfaces/preferences/properties, which is very similar to Adobe products;

3. time graph for animation (it reminded me of Adobe After Effects).

Blender interface

Knowing these basics makes Blender much easier-to-understand. But I wanted to explore it myself before looking at any instructions. However, it is still better if you start with training articles or videos. Blender did not turn out very intuitive, at least for me.

Unity and Vuforia

These tools are mostly used by the developers in our team, so I will not dwell much on them. There are many AR devices on the market, but the main aim of AR applications is the mobile space of platforms such as iOS and Android. That is why the combination of Unity and Vuforia is convenient — it is used in cross-platform development (you have to pay for Vuforia if you publish an app in the AppStore or Google Play). Another advantage is that an application can be tested directly in the editor; you just need a webcam to do this. By default, Vuforia works with pattern recognition. Pattern recognition (also called tracker recognition or image tracking) is the process by which a camera recognizes a pre-defined image and gives the location and orientation of a recognized 2D object. As a result, it allows you to render some content on top of it. This works best when the tracking image somehow matches the content. For example, floor plans for rendering buildings work well over a building image.

Unity interface

It is crucial to choose a good quality image for tracking. With Vuforia, you can upload a selected image to the developer portal to check the quality of its tracking, and you need to do this before you start developing. The developer portal rates the image, and more importantly, shows its “feature points.” For the tracker to be good, these feature points must be densely distributed over the image. They should not contain repetitive patterns. At runtime, the camera searches for these feature points to calculate its location relative to the image.

Modern mobile devices provide sufficient computing power to perform AR algorithms. Multiple sensors and constant connection to external infrastructure provide orientation in space and other additional data. Libraries create a platform for software development based on the AR interface. 3D modeling becomes more accessible for developers. The functionality of such programs is increasingly expanding. All this allows you to work with augmented reality more conveniently and think about new ideas of interaction.