Exclusive builds of tracking, image based tracking [ar.js]

9 min readAug 12, 2020

Ar.js currently (03.08.2020) has two different builds, both maintained, exclusive to each other.

AR.js with Marker Tracking + Location Based AR

Markers are an efficient way to bring people to the starting website but for a multi layered interaction experience it does not blend into the environment.

AR.js with Image Tracking + Location Based AR

Image tracking with ar.js happens with Natural Feature Tracking or NFT, a technology that enables the use of images instead of markers.

The software tracks interesting points in the image and using them, it estimates the position of the camera. These interesting points (aka “Image Descriptors”) are created using the NFT Marker Creator, a tool available for creating NFT markers. It comes in two versions: the Web version(recommended), and the node.js version.

NFT Marker Creator

You can find the web version here: https://carnaux.github.io/NFT-Marker-Creator/

Here you can find tips on what kind of images to choose for an image marker: https://github.com/Carnaux/NFT-Marker-Creator/wiki/Creating-good-markers

A good image marker

It`s all about the stars

The software looks at the complexity of the image. It looks at the contrast of neighboring pixels therefore an image with a lot of visual information will always perform better than a minimalist image.

Size matters

The software warns that images wider/higher than 1000px might take +20min to generate. With the original image even after +30min nothing was happening other than a message from Firefox that a webpage was slowing my browser down.

Started generating the smaller one at 14:46. Four minutes later there it is. Three files waiting to be downloaded.

Three files, three file extensions:

.fset
.fset3
.iset

In either cases, you will end up with three files as Image Descriptors, with .fset, .fset3, .iset. Each of them will have the same prefix before the file extension. That one will be the Image Descriptor name that you will use on the AR.js web app. For example: with files trex.fset, trex.fset3 and trex.iset, your Image Descriptors name will be trex.

In my case the name of my image is 20200723_144013_resize, which is what I`m linking in the url.

Make sure to upload the image and index file to a https server.

Best way to test if your custom image marker works is by using the official image tracking example as your work document and setting the gltf-model entity as a child to the a-ntf.

Hierarchy and working structure

Best way to start off is by using the official image tracking example as your work document.

Copy the code from the example into your text editor (for example Atom). Make sure not to forget to add <html> <head> ... </head> <body> ... </body> </html> to the example.

Change the <a-ntf url to your custom image markers location on your server. This example will place a 3D dinosaur over your image.

DPI

An important factor is the DPI of the image: a good dpi (300 or more) will give a very good stabilization, while low DPI (like 72) will require the user to stay very still and close to the image, otherwise tracking will lag.

DPI also influences the size of the elements that will be linked to the image.

In the first example, where the image has 72 dpi, the size of the dino is relatively inline with the size of the image.

The view throught the smartphone when pointed to the image marker (screenshot)

In this second example, where the image has 300dpi, the size of the dino is huge relatively to the size of the image. You can only see the nail of the dino.

Size of the marker

One of the attribues that can be given to the <a-nft\> is size which would specify the size of the marker in meters. In both of the above examples the image marker is a 2.5 meter tall wall.

I was expecting the augmented content — the dino — to change size based on the value of this attribute however it displayed the same view as with a non specified size attribute.

Content

My experiments follow the goal of augmenting the real world with content that is taken away from that exact environment and through certain transformations is brought back into it digitally.

The idea builds on the practice of tagging, some kind of information left behind in public space. These public spaces are an ongoing growing collection of tags. Once the architecture is created the collection grows organically.

Leaving traces in public space seems to be a global behavior. Its typically not done in the location when one lives.

Is the excitement caused by the mystery surrounding the identity of the person who left the tag? Leaving a timestamp on the trace signals a wish to be acknowledged as the discoverer of the place, the original foot stepper. Perhaps it is an individual wish to “own” something publicly and instead of occupying land it turned into micro-surface ownership.

Once a place is tagged there is usually an unspoken rule of ownership of that piece of “land”. Similar to graffiti on the walls the passing of time deteriorates the tag and after a while it can be re-tagged by someone else.

INTENTION: Creating a platform that allows for visual tags(in form of photography) to be placed over the public space with some sort of anonymous tracking and time-stamping system.

The platform is the tool by which participants in the tagging can document their views of the public space. The participation takes the shape of an attentionwalk: built up and guided similarly to a sound-walk but instead of bringing the focus to the sounds in the imminent present around the person the focus is directed to the visual landscape around the body. In the activity of according a hyper attention to the tiniest details of the streets and documenting it through the platform, the role of Augmented Reality becomes one of an archive.

Perhaps archiving the public space through an experiential space juxtaposed on reality can preserve the view of the people that participate in making it lively or leaving a glimpse from the past to some future generations.

TEST 1.1 & TEST 1.2

GOAL: Stabilizing the image to the same context it was taken away from EXPECTATION: The mixed-reality landscape through the phone creates an illusion of real vs digital elements.

TECHNICALITIES: Stabilizing tracking

test of different z depths, separate images related to the same marker image but on different z positions

Test 1.2 + human, screenshot from running test

CONCLUSIONS:

TEST 1.1 does not really add any interesting layers to the landscape

TEST 1.2 gets interesting when more depth is given to the elements — from a certain angle they look on the same z-axis but then from a side view they are actually displaced playing with their size to achieve the illusion.

DANGERS: Tracking purely based on image tracking does not seem stable. Need to fix either double tracking with gps or if that is not possible work only with gps.

Test 1.2 + human was a nice result. Occluding the original image marker on the wall did cause some flickering to the stability of the augmented content but the effect as seen on this screenshot is quite curious.

Real / Unreal

work by Studio Spass, Street Sculptures for the international art festival De Wereld van Witte de With→ https://studiospass.com/work/de-wereld-van-witte-de-with-12/

The experiment of having a person “in the augmented” led to questioning what is real physically in the space and what is only digitally there.

What if some element taken from the digital layer would be placed physically in the space as well, on top of the reality to cause confusion?

A real object that provides a glimpse into what you can see in the augmented layer.

This made me think of the work of Studio Spass when they overtook public spaces with a graphic layer that looked rather digital.

The starting point of the experience / attentionwalk could be the mural in front of the entrance to V2_ which could have some physical elements in front of it. These could be instructions on using the app and the link to the app as well.

Or it could be something reoccurring around the area of Witte de With to bring attention to the happening.

TEST 1.3

GOAL: Converting 2D image into 3D object

https://www.smoothie-3d.com/site/page_project.php

Creates an obj and jpeg texture file. This can be imported into blender.

3D software view of the cut-out elements

This is the view in blender from the exported obj file

the 3D chunks acting ass one 3D object

TEST 1.3 A. The cutouts acting as one element attached to the image marker

TEST 1.3 A.

In this test all the separate cutouts still behave as one grouped object. Even though there is quite a considerable difference in the depth between the elements it is not really using the spaciousness of the location.

TEST 1.3 B.

Leaving notes on the texture of the 3D elements, highlights the anamorphic aspect of making the cutouts 3D even better

TEST 1.3 C.

The blobs in this experiment are no longer grouped together which allows for a more spatial distribution

Architecture

Combining image based + location based tracking

I was hoping that I would be able to

nicolocarpignoli, jul 17 2020: Multiple markers are not supported. They do work (few, like 3–4 max) but bugs/glitch may happen. Also, it’s very heavy for the device to handle.

Location based tracking

gps-camera, the element that calculates rotation, position of user and the distance between the user and places.

Positioning Content

position = "x y z"

Rotating Content

rotation = "x y z"

A-Frame uses a right-handed coordinate system. When aligning our right hand’s thumb with a positive axis, our hand will curl in the positive direction of rotation.