Augmenting Your Inner SupARhero: a template for civic engagement
At the Center for Civic Media, we focus on creating or leveraging digital technology for social change. As an artist, a documentarian, and a creative coder with a do-good streak, I am exploring how to utilize augmented reality (AR) for civic engagement and activism. Beyond the dinosaurs now roaming our living rooms, the Pokemon Go craze, and ubiquitous information annotation, how do we best design AR to connect people, address communal needs, and deliver action?
To many here at the MIT Media Lab, new technology is iterative technology—tech built on earlier efforts that could span back to the early days of the Lab, or conceptually as far back as the ancient world. Mixed reality (XR) is no exception, especially if you are familiar with the “fairly recent” innovations of the 1960s and ‘70s, such as GPS navigation, the respective augmented systems by Ivan Sutherland and Myron Kruger, the cyborgian storytelling of Lynn Hershman, ELIZA’s purported “intelligence” and on back. As such, a critical eye is cast on such new iterations: the primary question being whether this iteration will be singularly impactful as opposed to derivative.
If you’ve ever had a conversation with me or heard me speak (here or here) about human expression and technology, you’ll know that I chafe against the sterile uniformity that often characterizes digital technology, whether it’s segmentation of bytes, geometric graphics, mailchimp templates or activist narratives. In my opinion, well-designed AR can bring back some of the mud and texture that’s missing in our sterile albeit interconnected digital world and lackluster public engagement scaffolds. Way-finding, embodied sense-making, the collage of immediacy: this is how AR might excel and how it might be able to elicit change.
It is true that the combination of AR toolkits with mobile devices is a real game changer. Never before have so many people had digital creation and augmentation capabilities at their literal reach throughout physical place and space. Conversely, it is also true that having people look at their mobile phones to better connect with the world is a perverse mediation. It doesn’t matter how fetching an AR story is if you turn into a zombie on the sidewalk and collide with oncoming pedestrians, cats, or telephone poles.
While we research better AR discovery alternatives beyond the mobile phone camera and speaker (some of my colleagues here at the Media Lab are looking into glasses and wearable haptic sensors), it is useful to iterate on theme. It is apparent that even the mobile AR platform can effect real impact, and this spring I have been begun exploring this in earnest. Across many sectors, especially non-profit and media, industries are exploring AR for social impact as well:
The use of enabling technologies such as AR could potentially be the best way to show stakeholders the state of a disaster area or the potential of an infrastructure project by overlaying images of before-and-after landscapes.
— Jimmy Vainstein, Sr. Project Manager at The World Bank
“RYOT [Media] began with a simple mission to link every news story to an action. We wanted to make the news empowering rather than depressing. To turn people on by what was happening around them rather than turn them off.”
This past May, I conducted a week-long AR storytelling workshop in Toronto in conjunction with documentarian David Tames’ six-week Alternative Realities dialogue and his Northeastern University undergrads. We structured the workshop to challenge the students to get out and into the world, and to identify real world needs that could be surfaced and even perhaps solved with an AR mobile experience. Most importantly — to brainstorm, collaborate and co-create with local Torontonians.
As such, we identified and connected four community partners with the four student groups. Their tasks were to
- identify the stakeholders within the community, the primary need or goal of the community partner,
- then determine an aspect of that need that could be best addressed through AR affordances (accessibility, site-specificity, GPS coordinates, information annotation, privacy, context-sensitive calls-to-action).
- With that context established, to design this experience and then
- to prototype that design using either a pre-existing AR app (in this case, Hoverlay), or to make one with ARKit.
As I mentioned earlier, storytelling with mobile AR is currently limited to overlays in the mobile phone’s camera video and audio feed, but compelling stories still they are. We worked with Nicolas Robbe’s Hoverlay to address our basic needs: creation of digital anchors (or overlays), persistent storage for these overlays along with social networking and sharing (ARKit 2.0 will provide persistent storage FYI). This gave the students an ability to create experiences so that others could experience them at any point in the future in either a site-specific or a remote way. Focusing on the 5 W narrative elements (through an excellent hands-on workshop led by MIT Media Lab researcher Natalie Gyenes) and on the design techniques of entry, engagement and extension (covered in Pete Zak’s excellent presentation, thanks Pete!), the students shot interviews, recorded narrations, implemented 3D graphics and designed the ideal experience to share these persistent digital assets across place and space.
One of the most rewarding experiences teaching this workshop was how the students pushed beyond their comfort zone, meeting with the community partners, and co-creating a story that could help address the needs of that partner. These college students are the digital generation, teethed on a inter-networked world. Getting them off their devices in the classroom was hard enough. Getting them out in the world and engaging with strangers was quite rewarding, and their shift from hesitation to enthusiasm was really inspiring.
Wandering without a map, experiencing the texture of one’s surroundings is a pleasure I myself only began to appreciate in my late 30’s. Before, unless I was on vacation, I buzzed from one location to another without much thought about the in-between. It made great sense that one of David’s requirements for the course was requiring the students to often take local transit (versus Lyft or Uber) and to “walk about,” reporting back on their “psychogeographic” experience. Paying attention to the visceral and embodied senses — the grit of the city street, the faces of its people — during a jaunt down the sidewalk is certainly a more powerful experience than following directions in one’s map app (for a deeper dive into walkabouts, dérives or flaneurism, explore Walter Benjamin and Baudelaire).
Indeed, before the predominance of Alberti’s perspective, navigation through a city was much more of a ground level experience. In my graduate thesis, I discussed this difference:
Once upon a time, Robert Romanyashyn explains, art was more body-based. Paintings of an urban settlement would translate the sense of the city, how it was to be within its walls, rather than looking at it from the outside. He cites two paintings that convey each perspective of Florence, a fresco in Loggia Del Bigallo compared to the Carta Della Catena (1436) He notes that the emergence of linear perspective has caused a disconnect in the way the modern person goes about living his life. An immobile eye rules our world, having transformed us into spectators, observing spectacles:“Ensconced behind the window, the self becomes an observing subject, a spectator, as against a world which becomes a spectacle, an object of vision.” The cause, he argues, is the scientific mindset that has determined how we interact with the world, and each other. Furthermore, this visual tyranny is anti-body: “In the space of linear perspective vision the body is progressively abandoned.” This is problematic as “regardless of the distance which we practice and achieve, we remain in our everyday living situations bodily creates with a carnal knowledge of the world.” (Romanyshyn, 1989, pp. 38).
As such, it seems likely that a layer of content well-integrated into a visceral way-finding experience is a natural outgrowth for AR technology — provided it is the kind of peripheral or “calm technology” Amber Case promotes. Both the students and I were excited to imagine an accessible, non-invasive documentary-style story format that helps humans be better humans exactly where they already are.
Prototyping the experience
Over the week we explored desired AR affordances: anchor location and persistence, layout schematics so the creators could control how the user would experience the digital story elements, and then additional interactivity with those elements — such as providing feedback or answering a call-to-action. Should the reader like to follow along, download the Hoverlay app (iPhone only for now, apologies), and under Trending, locate the individual projects.
~~ Community Partner: Building A Better Bloor Dufferin
Hoverlay Account: BloorDufferin
As one would imagine, the week’s activities exposed some design challenges. One, that layout of content is of critical importance in site-specific, contextual storytelling. Because the mobile camera can be utilized from any angle in any orientation, curating an experience is challenging, similar in some ways to theatre in the round. For example, the undergrads and the community partner, Building a Better Bloor Dufferin, wanted to give voice to the high-school students whose school was planned for demolition. They wanted the green-screened videos of current high-schoolers to be digitally placed exactly at the latitude and longitude and facing out from the school itself. We asked what would happen if the user is coming from the left or the right of the school — how would they know any content was there? Should it billboard the flat 2D videos or keep them oriented as designed by the creator and if so, risk missing the video entirely? What is the sweet spot between obtrusive versus peripheral notification?
Invitation and entry into AR is another design challenge. One must download a mobile app to discover content, such as Hoverlay or another. Furthermore, the act of downloading an app requires intention, interest and some know-how. Still, barrier to entry doesn’t appear too limiting as The New York Times and the Huffington Post have both created VR and AR apps for their readers to download. In the case of BBBD, the group decided to install a large QR poster directly across from the subway and bus stop. That way, the concerned neighbors and high-school students could learn of the AR story experience just beyond a free app download.
Below: a great example of a “double-dip” augmented experience: the “hacked” posted sign of the projected development IRL (in real life) and the digital Hoverlay anchor in AR land.
~~ Community Partner: Newcomer Kitchen
Hoverlay account: NewcomerKitchen
In a nearby neighborhood, The Depanneur, a wonderful culinary venue that offers drop-in dinners, also hosts Newcomer Kitchen, an organization that hires Syrian refugee women to cook take-out meals on Wednesdays. The Depanneur, although wildly successful as far as public support and media coverage (Prime Minister Justin Trudeau came to visit), struggles towards a scalable business model.
Students and Newcomer Kitchen founder Cara Benjamin Pace discussed the problem that these Syrian cooks and the neighborhood consumers do not interact; as such the exchange is merely transactional rather than transformational. The identified goal was to strengthen the connection between the creators of delicious home-cooked meals and the consumers in hope it would create better and more business. They designed an AR experience based on the “to go” paper bag.
In this use case, the layout schematics were important for remote users — outreach was externally and remotely focused. Subscribers click on the paper bag to meet the creator of their meal, learn about the ingredients, and respond — all in the comfort at their own home, even at their own dinner table. The ability to provide a default layout and then to allow users to change that default was therefore very important.
In the prototype through the Hoverlay app, under the NewcomerKitchen public feed, we watch one of the chefs, Farud, introduce herself and tell us about the “most delicious salad in the world” and its ingredients. Any user, regardless of location, can virtually meet Farud, stop and start the videos in order, and examine the ingredients. It was also fun to be able to place the food onto a table and Farud at the back of it talking to the camera.
“It was less about providing food and more about creating a direct connection between the refugees and the city, fostering a mutual connection and sense of hospitality.”—Emily F., Northeastern University sophomore
Immigrant and refugee populations have much to offer, and communal kitchens and sharing of food is an excellent and meaningful way to bridge cultural divides. As Len Senater, owner of The Depanneur said to me, the feasibility and effectiveness is already proven, it’s the financial support that needs to be activated.
Whether it is the green-screened image of Farud layered over our own camera feed in our own kitchen or her friendly face and kind voice, there is something compelling here that bodes well for both AR visual storytelling and civic engagement. Why AR? A regular mobile app may deliver information along with transaction capability, but it will likely not deliver a semi-immersive visual story. There is also something about Farud’s green-screened video in our camera feed that is a little more compelling than the usual rectangle. A properly designed feedback/response platform through AR could invigorate fundraising and greater civic involvement.
As such, future collaborations between Hoverlay and Newcomer Kitchen will give the women agency to broadcast their own creations and recipes and clients can comment back. It is hoped that this two-way visual conversation, co-authored by the stakeholders themselves, will garner more attention and financial resources to the kitchen and its greater community.
~~ Community Partner: Bentway Skateboard/Build For Bokma
Hoverlay Account: BuildForBokma
The Bentway Skateboard/Build For Bokma foundation group created an AR experience to reincarnate a skateboard park that was torn down by the government. The 3D model was superimposed at scale in the camera feed with audio narrations placed around the scaled model. The goal of the group:
“We want to give a voice to the kids that want to preserve this space and have say in the construction of the skatepark. They will be able to go through this AR experience and then have a call to action at the end to sign a petition or something of that nature so the community leaders have a understanding of how important it is to include the children’s voice.”—Liam C., Northeastern U freshman
Signing petitions perhaps isn’t the best way to garner action, but as far as giving a voice to primary and secondary school-aged kids, I find this a very touching and worthwhile project. An easy-to-use AR app like Hoverlay can provide children a degree of autonomy to express how they feel about a particular place. Certainly city and school officials may be inclined to make different decisions due to a popular cluster of AR stories that young people have organized and “dropped” at a desired skateboard park area.
“We think we have achieved something very special here, by combining AR technology with activism for a cause we truly believe in. We hope Build For Bokma only goes up from here, and accomplishes what it has set out to do.”
~~ Community Partner: Convenience Stories
Hoverlay account: ConvenienceStories
Mom and Pop convenience stores could be a thing of the past. This is why Convenience Stories was founded, to expose the loss of neighborhood stores that were once a real part of the community versus the chain stores replacing them. Our student group identified a number of stores along a well-known city street and at first showed us an Alberti-like topographical map:
Over the week, as we asked them to show us why we should care about these points on line, they iterated towards a way-finding, embodied experience in which the texture of the street and the stories of the shop owners were accessible and shareable through the AR app. The physicality of the place and the people were the most compelling, and again, I think this is where the ROI can meet the road in terms of civic engagement. A community-sponsored augmented reality channel focused on these locally owned stores could help local residents realize the value — and the vulnerability — of their mom-and-pop shops.
~~
In conclusion, perhaps designing AR for civic engagement is kind of like designing a micro-documentary — the narrative arc requires some key story elements to get a target audience to care about the underlying cause. There has to be an identified need, and an explanation as to why that need hasn’t been answered. The key is surfacing that story and that need in a non-intrusive but targeted way to people who might care. This is why a design like the Newcomer Kitchen AR experience could really work— the providers and consumers already share an interest in food and community; they need to activate and “convert” the attention when the food is being ordered, delivered or eaten. This is also why an AR experience is a better choice than a regular mobile app — you can co-locate a visceral experience — to the kitchen, to the store where the ingredients are bought, to the delivery, to eating of the meal itself. The adage of accessing the heart via the stomach is certainly true of many — my family especially!
What we do know is you can’t just float a “listen to me!” AR button or IRL (in real life) soap-box solipsist on a street corner and expect someone to take the time to find out why they want to listen or contribute. The Bloor Dufferin group struggled with this challenge — why would a busy commuter take the time to download the app and walk around the neighborhood? Yet still, with the stimuli planted in a fairly tight-knit neighborhood, the development project’s deleterious impact on the high-school students and their families, and that real-estate development in greater Toronto is running rampant, the odds are that a daily commuter might make the effort to become more informed.
I think that AR can and will vascularize lackluster calls-to-action especially if these calls are connected to a place and space — if we solve for the problem of the right place at the right time and as quick and as painlessly as possible. The way-finding and sense-making I mentioned above — how can we connect the audience to the authenticity of people, place and problem/puzzle at a time in which they are willing and able to pay attention. How do we then convert that moment of attention to something sustainable and longer lasting?
Perhaps we need something like an AR app for Superheroes? Does it run behind the scenes sending you notifications as you explore a neighborhood? Do you set up (god-forbid) another profile and password to discover the needs of a local community? Is there a concept like an “impulse give” that could work into our daily habits or would that be detrimental to community-building in general? Can we organize AR for Good scavenger-hunt flash-mobs? What are your thoughts? Let’s explore and get smart about smart technology.
Thanks for reading! If you found this article useful, help me share it with the larger XR community!
