Augmented Augmented Reality
Six designers from the EPFL+ECAL Lab rethink AR
The potential of AR
Constructing music with building blocks, being guided through a city by sound, linking memories with objects, creating compositions through dance, drinking up Tiki culture and blurring meditation with anxiety. This weird and wonderful jumble of activities are all explorations on the topic of Augmented Augmented Reality.
Most of us know what Augmented Reality is, but it’s still a relatively new technology that has only tentatively been integrated in the mainstream. Millennials might play with face filters on Instagram and Snapchat, or maybe we consult an application to name the mountains around us or play a fun little AR game. But surely Augmented Reality can do more than this?
In order to do explore this potential, we decided to take a step back and think about what Augmented Reality really means. We wanted to question what devices could be used, what could be recognised and what feedback could be given. Maintaining the idea that we will add a digital layer to the analogue world, we asked the questions; What can AR tell us? What can AR make us feel? How can AR change our understanding of the world?
We investigated these ideas over the course of one semester during the MAS in Design Research for Digital Innovation at the EPFL+ECAL Lab. The project was led by Romain Collaud and Lara Défayes and engineering support was given by Delphine Ribes and Yves Kalberer. We named the project Augmented Augmented Reality and each of us six students followed our own paths of exploration, eventually resulting in six diverse projects.
Urban drifts
Pierre-Xavier Puissant wanted to focus on how sound can be used as an information layer on top of sight, as a non-intrusive, hand-free way to augment one’s reality. He proposed DÉRIVE, an auditory AR experience, which unites urban exploration, community, poetry and serendipity.
Based on French philosopher Guy Debord’s Psychogeography, the experience invites the user to forget the paths they always follow, and to reinterpret and rediscover their everyday environment. “Dérive” literally means “drift”: it is the very core of this pseudo-science, a tool to actively question the everyday’s topography. Following divergent narratives, users are drawn away from known paths. By connecting creatives to urban wanderers, one given area can lead to dozens of different “driftings”, resulting in hundreds of interpretations of the same space.
A web interface allows creative users to make a “Dérive”: an aural interpretation and a set of possible paths to follow, based on a set of freely defined criteria. Each path is made up of a set of places, for which the user has to create original audio content. Complementing this, an AR app allows urban wanderers to experience these “Dérives”, based on their current location. The app proposes the available site-specific “Dérives” from which the user can choose. They can then define a number of contextual parameters. Finally, the user is invited to put the device back in their pocket and be guided by sounds.
The app is hands-free and non-constraining: all instructions are given through sound indicators, the absence of visual interaction trigger the user’s imagination and allow their sight to run free on their hic et nunc surroundings. The proposal is by nature site-specific, as it relies on the wanderer’s visual field as a canvas for their mind to project on, augmenting its own reality.
Graphical composition
Margaux Charvolin was interested in how music might be composed using visual blocks and how this might affect our perception of it. She was inspired by several, more avant-garde experimentations of visual musical scores, that allow the musician to impart their own interpretation through the music. The idea of synesthesia, the mixing of two senses from one stimulus, was also integral to the development of the project through associations between sounds and visuals.
Margaux tested a variety of geometric markers and visual systems to try and find the right graphical balance between meaning for humans and meaning for an AR detection algorithm. Her final outcome allows users to compose music with coloured building blocks that fit into a platform. Each tile holds a colour and an abstract form that corresponds to a beat or riff. Holding and moving an iPad above them launches the extra magical layer of music and the composition comes to life. The tiles can be arranged and rearranged in any formation generating a new song everytime, and allowing new possibilities for making music. For this final piece, she was inspired by the musical posters of Florent Berthaut and modular systems made by Ibàn Ramon+Didac Ballester.
Body painting
Elise Migraine performed a digital body painting experiment, called “Lines and Circles in Space”. She developed 3D software which used the movement of the body to make 2D images on a screen. The images created were abstract, composed of circles and lines of different colours. The floor was mapped in different areas, each corresponding to a specific drawing tool. Moving the body within the space creates a painting, a dance, and a unique experience.
Four artists inspired Elise’s work. Firstly, the paintings of Kasimir Malevitch which depict the world with abstract figures, as pure as they can be, trying to show a clean truth. This was an interesting contrast against AR technology which uses machine learning to interpret images, and tries to add meaning and detail to forms. In this work, Elise uses an algorithm to capture the movement of the body in space, but only to generate the abstract forms it draws. This is also a reference to the work of the contemporary choreographer William Forsythe who explores how the body can draw forms in space. In his performances this is revealed by a layer of drawing on the video of the dance, where he verbally describes his moves.
This relationship between the body, space and the machine was guided by the work of Oskar Schlemmer, teacher of the Bauhaus in the 1920s. With his students, they developed ballets with stunning Bauhaus scenography and costumes which questioned this relationship. Augmented reality brings the digital realm out of the screen and into 3D space. It becomes a matter that can nearly touch or sensed, reducing the distance between digital content and our bodies. In this way, AR technology could be a new way for us to explore this dynamic of physical, digital and our body. Finally, the style is influenced by the work Rafael Rozendaal who captures the sensitive experience of digital realms with bright color, interactivity and movement.
Bringing memories to life
Hélène Portier proposed a new way to access and experience our personal archives and memories with an augmented souvenir box. She wanted to create a new link between physical objects and the wealth of captured images we store on mobile devices.
The starting point was linked to unpacking the meaning of seeing through a screen with Augmented Reality, and indeed through a camera. A camera reproduces a real-world scene, allowing us to perceive things from a different moment in reality. It also lets us manipulate any of the content it records, altering our perception of that reality. In this way, using a camera to record a moment is closely linked to the theme of remembrance; both create new realities in the present.
Today, more than ever, we document every aspect of our lives in intimate detail. Digital cameras, connected objects and smartphones mean we are constantly creating our own new digital archives. Random Access Memories uses these records to create new realities. A simple wooden box links our experiences of the physical world to our digital heritage. A photo can unlock memories we haven’t thought about in years. A train ticket can tell the story of a whole day. A flyer can reveal forgotten moments.
Messing with meditation
The very point of meditation is to alter the mind to take a step back from reality while being deeply aware of it. Yoann Douillet’s project uses AR to critique the modern “wellbeing injunction” through meditation, which is manifested in the numerous experiences, apps and devices which designed to allegedly help the user to reach mindfulness.
In the experience, the user wears a one-of-a-kind AR helmet, through which they can see around them. They are also equipped with a breath monitoring device, and a belt that can tighten or loosen itself, acting as a haptic feedback. The goal of the experience is to focus on breathing. The user is invited to follow a rhythm imposed in the AR view. The more the user fails, the more the haptic belt tightens, and the blurier the vision of its surrounding. The user is forced to relax, otherwise they get crushed by the belt. But by knowing that they can be crushed by the belt, they panic, resulting in a vicious circle. By design, the experience can only increase in intensity, resulting in panic from the user.
The design of the helmet reflects the intentions of the experience with its medical-looking and anxiety-provoking appearance. A key inspiration were the pseudo-medical experiments of Hugo Gernsback. For example, the Isolator was an attempt to create a helmet that would restrict senses and distraction and allow its user to fully focus on a specific task, despite being extremely uncomfortable.
The work draws its inspiration from early XXth century Miracle Medicine shows. Rogue medics used to travel the country to sell “miracle” elixirs in fairs, supposed to cure all diseases, but which often did much more harm than good. Being showcased in fairs, it’s easy to imagine that these products pulled the same type of users as those who download wellbeing apps looking for miracle products. In addition, the fact that the setup of the device requires an expert brings back the doctor-patient relationship, and the users submission to the doctors orders.
Cocktails and culture
Emily Groves hosted an augmented cocktail party in which guests discover the origins of Tiki Culture, a fascinating movement from the 1950s and 60s which had a huge influence on Western cuisine, music, film and literature. The idea was to use real food and drinks as the catalysts for discovering more about the historical background of Tiki.
Guests ate food taken from original Tiki menus and drank two versions of the classic cocktail the Mai Tai. In order to tell the difference between the cocktails, guests had to place their glass on a special wooden “Mai Tai Machine” which would recognise the glass and reveal the recipe and information about the cocktail’s origins. In addition, authentic tiki music was playing and historical graphic material from books, films and menus was available to look at. Two digital tablets also unlocked an additional animated layer related to each of the graphic pieces.
Allowing the guests to taste, hear and discover Tiki Culture for themselves augmented their experience. The wooden Mai Tai Machine made AR about objects and added a magical touch. The extra content on limited tablets added a social element, encouraging people to share what they had seen with each other.
What’s next for AR?
With these proposals, we gave very diverse, and very personal interpretations of a new trending technology. But what does it mean for the medium itself?
The most outstanding insight that these experiments show is that we’ve only scratched the surface of what AR can do. While most AR experiences in the mainstream are focused on the “wow” effect, the tools to create AR experiences are becoming more and more accessible and open to experimentation. Hopefully, this will push AR to become a credible interaction medium.
Pushing the boundaries of a rising technology gave us the opportunity to see how unexplored the possibilities are. With the whole notion of reality being so open to interpretation, AR was a wonderful playground to begin our questioning. Unlike VR, which takes you away from the real world, the very principle of AR is based on interactions with reality. In this way, it can reveal an unprecedented perspective on our environment, ourselves and the interactions in between. Seeing, hearing, touching, smelling and tasting all define our perception of reality and are senses we can use to open up new relationships with our surroundings.
Once the concept of reality is unlocked, the ways to interpret and augment it become infinite. However, in order to acquire a status of medium — and not of pure message, as many AR “wow” effect experiences rely on — it needs one-of-a-kind prototypes, personal experiments and tailored experiences. It’s time to play with reality.
Article written collaboratively by Emily Groves, Pierre-Xavier Puissant and Elise Migraine
