ARKit in the Manufacturing World
At this year’s RWDevcon, I explored a new side of the iOS framework. This article is a recap of what I learned at Joey DeVilla’s workshop on augmented reality (AR) : the history behind it, a shallow dive in the framework and my take on the future of AR in the manufacturing world. Enjoy :)
Brief History of AR
Spoiler alert, AR was invented way before Pokemon Go! The first augmented reality apparatus was invented in 1968 by Ivan Sutherland. 3 years before before Woz put together the Apple I computer. Sutherland was a Harvard computer scientist who also pioneered the development of computer graphics and human computer interactions. The first prototype he built with two fellow Harvard students was so bulky, it had to be hung from the ceiling because of its sheer weight.
Tom Caudel (at Boeing in 1990) followed Sutherland’s footsteps. He set out to replace 20 feet long peg boards for assembling wiring looms with a system workers could wear on their head. Unfortunately, his system was not taken to production due to hardware limitations such as battery size, processing power and network constraints. The tracking system was insufficiently responsive when workers moved around.
Disney used the technology to celebrate Mickey’s 83rd birthday in Time Square in 2011. Visitors could dance with the characters and their interactions were projected on a big screen
The concept always failed to make it to the mainstream market until Pokemon Go came along, in 2016. While Google had its Project Tango since 2014, it is Apple who took the spotlight with ARKit shipped with the release of iOS 11 in 2017.
What is ARKit
ARKit is a Visual Inertial Odometry (VIO) framework tracking the correspondence between the real-world space and the virtual space. It combines real world images, virtual images and sensors to give the augmented reality experience. The framework abstracts the device’s sensors and all the mathematical calculations needed to create the illusion that your virtual content is part of the real world.
The requirements are pretty straight forward : any iPhone or iPad running iOS 11 with an A9 chip or newer.
Environment Detection
ARKit can detect notable features such as 2d images, faces and objects, horizontal and vertical planes. ARKit can automatically detect flat surfaces in a scene through feature points tracking and correlating those features point with a vector, gravity (y axis). The information detected throughout the AR session such as alignment, geometry, center of the plane and extent (width and length) are stored as anchors. The framework is made of different kind of anchors when detecting different kinds of objects.
Positioning in Space
The position is tracked by two different systems: Visual (camera) and the Inertial system (accelerometer, gyroscope). Calculations of these two systems are combined to give the most accurate position of the device in space. The distance traveled is tracked in 6 dimensions, more commonly called the 6 degrees of freedom (6DOF) : 3 translation axes and 3 rotation axes. The first 3 dimensions are used to calculate the difference in positions. X, Y and Z use a standard coordinate systems following a right-handed convention. The other 3 dimensions are used to calculate the device rotation. Pitch (x), Yaw (y) & Roll (z).
When combining these two systems, ARKit can compute depth information, planes and enhance the illusion of depth in the 2D images recorded. This is done by comparing multiple frames and correlating that with the distance traveled from the device’s Inertial system. It is called stereoscopic or stereo imaging.
Image Recognition
Image detection greatly improves the use cases for AR. It can be used as an input mechanism to trigger functionalities. For example, in Perficien Digital Labs’ video, bus schedules and routes are prompted to the user when detecting a corresponding stop.
To achieve optimal results, best practices include…
- Image detection works best on flat surfaces. For example, wine labels on bottles (non-planar image) won’t be detected as well as a paintings on walls.
- Physical size of the source image needs to be defined accurately. The information is used to calculate the distance between the reference image and the device.
- The scene captured needs to be lit :fire-emoji: (pun intended). Poor lighting conditions, gloss or reflection can disturb or prevent the detection.
Augmented Reality in the Manufacturing World
Businesses like Boeing, NASA and Bosch are investing great amounts of money in AR to increase manufacturing efficiency and improve training for their employees. For example, workers assembling boeing’s 747 & 767 jets are using handsfree glasses to get real-time interactive 3D wiring diagrams. Productivity is reported to be up 40%. This endeavour, called Project Juggernaut, started in 2013 with an engineer building a prototype using the first production unit of the Google Glass. The use case was getting the right information in real time without the technician’s input. Information displayed in the worker’s field of view reduces the time it takes to associate information with objects.
Bosch, the German multinational (full disclosure: who’s also an investor in Poka), developed its own AR framework and application : Common Augmented Reality Platform (CAP). CAP is an all-in-one solution to display written information video clips, pictures, safety instructions with audio clips, 3D data, circuit diagrams, technical schemas.
What’s next?
These examples of AR in the manufacturing industry only scratch the surface of what’s possible with AR, Image Recognition and the hardware built to support these new frameworks.
With Google and Apple, each releasing their AR framework on Android and iOS, it lowers the barrier of entry for developers wanting to build software using augmented reality. In the near future, with augmented reality glasses being commoditized, we will see mass adoption of this technology in small and medium businesses.
When asked “How could you possibly have done the first interactive graphics program, the first non-procedural programming language, the first object oriented software system, all in one year?” Sutherland replied: “Well, I didn’t know it was hard.”. As software engineers, we need to start prototyping, building, make mistakes, try over again and see what works. I believe this is what creates mass adoption of a new technology!
