Augmenting Network Sculptures
This article is introducing an ongoing collaboration between Moholy-Nagy University of Art and Design (Budapest, H) and Barabasi Lab (Boston, US). The aim of the project is to bridge physical, 3D printed spatial networks with their corresponding background information layers using augmented reality. Since a virtual model is aligned to the actual, printed network sculpture, the audience can reveal layers of information by navigating over different layers of reality: interacting with a touch based mobile device on its screen, but also, navigating in the actual space by moving the device around. Using the built in motion sensors, gyroscope and the realtime camera image, the result is a mixed reality experience, where layered digital information is connected to the actual, physical network sculpture.
A workshop
After an initial brainstorming with network scientist László Albert Barabási, we started the project with a workshop at MOME campus with Mihály Minkó, where designers, artists, engineers and people from different fields of humanities were invited. We were investigating ways of how we can mix the language of augmented reality with network visualisation problems. Two theoretical lectures were given as a starting point, dealing with the taxonomies of augmented reality (and its corresponding cognitive models of the different modalities of space) and network topology: visualizing, understanding the structures of scale-free and complex networks.
Collaborative thinking, tinkering and DIY culture is also a crucial part of our workshop methodology, so we got some plastic components that consist of simple building blocks for assembling different molecular systems, and the participants had to build simplified networks by their own. The instructions were based on some extremely filtered and limited data they got in their hands, sitting around a table, constructing the object together, regardless of their professional background. This way of collaborative design opens up meaningful conversations, invokes several questions on the topic that is being investigated by the group. As a result of the workshop, we got some interesting ideas on the navigational concepts within augmented reality, we also had a very first-hand experience on how networks, nodes and their edges can be created.
Since networks are abstract structures in their own, visualizing them in another abstract model (augmented reality) is even more challenging, we are building something that none of us experienced before.
So far, so good. It seems that
The methods of understanding complex networks and relationships can be learnt and understood better if we have hands-on experience regarding to a structure (building, combining, restructuring parts and pieces) besides reading theories and abstract concepts only.
Augmented reality is an emerging field of visual communication, that has a lot of potential, which means it also needs a lot of hands-on experimentation. As the computational paradigm where melting actual space and algorithms started with nautical navigation, our time is also dealing with the same paradigm with new methods. Today, people have to combine actual physical space with layers of virtual content: these are the basics of space-aware computing, where we have to build the language for the next era of computational design.
Telling meaningful stories within this hybrid space is a very novel concept for both content creators, designers and engineers. Understanding the context of these complex scenarios involve people from different background professions, including engineers, humanities (literature, semiotics, etc), cognitive scientists, visualization experts, and most importantly, experts who are dealing with the ethical, biological consequences of technology, such as artificial intelligence and different layers of reality.
A prototype
The goal of the collaboration between the two institutes is to find and build meaningful visualizations for these conceptual interrelations between space, data and its representation. It is an interesting journey in order to create new aesthetics based on data & networks, where scientific concepts are easier to grasp for people who are outside of the academic circles of design or scientific communities.
We started to build an iOS prototype in C++ (OpenFrameworks) that is based on Apple’s augmented reality platform, ARKit. We are using feature points from the device’s camera feed to construct, load and save world coordinates that are connected to the physical network sculpture. We also use the same app to render virtual content on the screen, using OpenGL combined with Apple’s Metal as the graphical renderer.
The UI and the application lifecycle is based on custom components, there are no OS-specific built in elements and modules that are used, which leads to easier portability and future-proof, cross-platform development strategy. The prototype later can be ported to Android’s ARCore platform or other operating systems of the upcoming head mounted reality industry.
The usage of the application is intended to be “space aware”, which means it is using some special concepts beyond conventional navigational routines (such as touch, swipe, pinch, etc). The distance from the actual, physical object plays an extremely important role in the experience.
The application concept, data transformation, design and development is made together with Mihály Minkó. The app is about to be released in 2020 as an integrated part of the forthcoming exhibitions of different networks and generative sculptures made by Barabasi Lab, this post will be updated, accordingly.
