Computer as a Medium for Human-Environment Interaction: The “Color of Nature” Tangible Interactive Art
An HCI written justification with a proposed concept of interactive art that enhances interactions between human, computer, and environment
The world has come to the point when technology becomes a part of our lives and exists everywhere. To use and design technology, human-computer interaction (HCI) is a field of study that focuses on our interaction with computer technology. As implied by its name, the concepts of HCI mainly focus on humans and computers; however, having a focal point toward specific stakeholders may result in the exclusion of other things around them. To shift the focus away from “humans” and “computers” toward things around us, the “Color of Nature” tangible interactive art is proposed. This interactive art will develop a new focus on human-environment interaction, using technology as a medium.
In this post, I will address the background, inspirations, resources, and methods used in the production of the “Color of Nature” tangible interactive art referenced from various research papers. I will also discuss Winograd’s paper on interaction design and expand Winograd’s vision with the concept of posthumanism, while introducing the concepts of tangible design, interactive art, and art as a tool for social practice. Then, the process of the proposed art will be demonstrated with an analysis on future vision of HCI that the “Color of Nature” suggests.
Back-to-Nature Movement and Posthumanism
Winograd (1997) argued the idea of shifting a focus from computers to the lives of the people using them. Mentioning three important disciplines for the growth of interaction design, Winograd demonstrated the importance of social discipline along with engineering and design disciplines, which implies that the future of technology not only leans toward technical aspects of computing but also links to social aspects of society. With technology being ubiquitous and continuously evolving, the back-to-nature movement was introduced as a question toward “our physical embodiment in the world,” and it also illustrates how physical interaction within the physical world could potentially be the future of computing.
While the concepts of HCI are centered around interactions between humans and computers, there could be other excluded stakeholders being involved in these interactions. Forlano (2017) argued the concept of posthumanism that shifts the focus from human to nonhuman stakeholders, including but not limited to animals, natural environment, and machines, which have been neglected when the term human-centered design is mainly used. Forlano discussed several issues that occurred when designing without considerations of nonhumans and suggested the idea of posthuman design that “resists binary categories and, instead, integrates the human and the nonhuman” (p. 20). As an increase in technology development comes with a lack of environmental and ecological awareness, expanding the scope of HCI to cover things around humans and computers would still benefit in the future of technology and at the same time not leave anything behind. Connecting to the “back-to-nature” movement, the concept of posthumanism demonstrates how people could interact more with the environment to reemphasize our existence and connection to the world we live in.
Rahm-Skågeby and Rahm (2021) also expressed concerns toward long-term geological processes with material developments and impacts from the exclusion of nonhumans in the current concepts of HCI. As we are entering the geological epoch of the Anthropocene, it is known that humans significantly and negatively impact the environment, ecology, and ecosystems. While it is impossible to turn back time, our responsibility for these environmental and ecological issues still remains. Rahm-Skågeby and Rahm suggested that we should “acknowledg[e] that humans are inseparable from other planet-shaping powers” (p. 8) and move toward design thinking with depth. They argued that another challenge for HCI is to look for new ways to explore the interactions between humans, technologies, and geological temporalities since it is the time to move beyond human-centeredness and the exclusion of stakeholders.
Posthumanism Interaction Design
With a goal toward sustainability, inclusivity, and resilient futures, Liu (2020) discussed HCI theories and methods with a concept of posthumanism interaction design in her dissertation. Elaborating on the term Anthropocene, human activities have big consequences to other living and nonliving beings. Posthumanism interaction design (PID) employs posthuman epistemologies to account for the needs of both human and nonhuman stakeholders, and it starts with a shift from human-centered thinking to posthuman thinking. In Liu’s dissertation, the reasons for complementing HCI with PID are elevating user experience for all, attending to and challenging power, designing for the real world, and making other worlds possible. These reasons involve reorienting attention to the environment and nonhumans, promoting equal power toward all stakeholders, being mindful of design consequences and impacts, and looking for alternative futures through PID lens. Liu referred to the “arts of noticing” concept proposed by anthropologist Anna Tsing as an interdisciplinary approach toward multispecies stakeholders.
The arts of noticing take root in response to the current pressing ecological, political, and social concerns that we are facing, encouraging the re-examination of research assumptions, as means of pursuing alternative pathways towards preferable futures. (p. 39)
The “arts of noticing” also include adjusting perspective in order to notice differently, and it is an inspiration to create this tangible interactive art as another medium to raise awareness of environmental and social conflicts. According to Liu, the ability to notice differently is to “acknowledge, engage, respond, and cultivate relationality between multiple frames of reference” (p. 63) no matter what type of stakeholders they are. Based on the ability to “notice differently”, I decided to focus on engagement and response, which then became the main concept of the “Color of Nature” tangible interactive art — to feel and get in touch with nature.
Tangible Design in Physical Environment
According to Ishii and Ullmer (1997), “the use of graspable objects… will lead us to a much richer multi-sensory experience of digital information” (p. 8). Considering tangible design interconnects with the back-to-nature movement through physical interaction, it suggests the possibility of elevating connections between humans and the environment. Since the goal of tangible design is to bridge the gap between digital and physical environments, Ishii and Ullmer outlined three key concepts regarding tangible embodiments of digital information within the real world. The key concepts include interactive surfaces, coupling of bits and atoms, and ambient media. Interactive surfaces require a transformation from a physical surface to an active interface between the physical and digital environments. Coupling of bits and atoms uses graspable objects as media to get information from the physical world, transform the information into bits, and transfer it to cyberspace. Ambient media such as “ambient light, shadow, sound, airflow, [and] water flow [could be used] for communicating information at the periphery of human perception” (p. 5). These three concepts suggest different approaches toward tangible design, answer different problems, and employ different techniques.
Shaer (2009) also stated that tangible design puts together engineering, art, and social sciences. Apart from well-designed software, it is necessary to consider physical and aesthetic aspects as well as social settings of the design. Shaer suggested the use of conceptual foundations in tangible interaction to describe the concept of design, scenarios, interaction techniques, and implementation details. The conceptual foundation provides a framework for the process of creating tangible interaction, which requires “not only applying existing concepts and practices but also exploring new ideas, domains, and techniques” (p. 252). Shaer broadened the scope of tangible design and expressed the need for further exploration. Tying back to Ishii and Ullmer’s work, Shaer’s paper implies that the concept of tangible design is flexible and moving forward with our creativity and ability to make it work.
Art and Interaction Design
The main differences between art and design are purpose and restriction. While good design needs to achieve its purpose and sometimes has restrictions, art has no boundaries. Nevertheless, when it comes to the meaning, understanding design is usually simpler than interpreting art. For this reason, it is challenging to create art to convey a meaningful message while also expecting most of the audience to understand.
Tomás (2017) argued that artworks can convey critical messages regarding social and political situations, but they are usually not following any methodology toward HCI. Arts can be a source of inspiration, but to have an epistemic value, artists need to consider artistic research methodologies to make an artwork become artistic research.
[T]he type of knowledge created by artistic research projects is sensual and physical; it is purely an embodied knowledge. This knowledge must be acquired through sensory and emotional perception — precisely, through artistic experience. (p. 5)
With the concept of experience playing an important role in artistic research, Tomás discussed several theories of art for understanding experience and summarized that “the goal of artistic research is to open up new visions and interpretations into some proposed questions and artistic experiences” (p. 6).
Tomás included the analysis model for artistic research to help artists create artworks that have clear structure and focus. The model is based on three concepts, including problematization, methodological conceptualization, and artistic inquiry. Problematization is the first step to help “transform an artistic process into a process of research” (p. 9), followed by methodological conceptualization, when the artists had to decide which methodology they would employ. Lastly, artistic inquiry covers the conceptual element that could create new theoretical questions or lead to new approaches. When an artistic experience comes with knowledge and values, the goal of art extends beyond its visual aesthetics. Employing this analysis model for artistic research would help organize the messages and experiences of the “Color of Nature” tangible interactive art in order to achieve its goal.
Art and Social Practice
Since the purpose of creating this interactive art is to strengthen connections between humans and the environment as well as raise awareness about environmental and ecological issues, understanding how art influences social change is a critical approach. According to Esche and Bradley (2007), many artists used art to suggest and generate “alternative social models” for a long time. The production of their art is often related to social practice, whether it served the need of cultural, political, or economic movements. The techniques and forms of art also varied from street art to performing art, depending on the artists’ messages and the situations. As time goes, the production of art is also developed, resulting in a more variety of techniques and methods that artists can employ in present days. The timeline of art history indicates artistic trends within each period of time. As art is one form of storytelling, many artists use art to portray the events or situations that they encounter during the times they live in. In this day and age, it is the time for conceptual art, or in another word, an art for movement.
Proposed Tangible Interactive Art: The Color of Nature
The “Color of Nature” is a tangible interactive art that enhances interactions between humans, computers, and the environment. Noticing the differences in natural colors and synthetic colors, I created this art with a goal to raise people’s awareness of the more-than-human world and provide a feeling of getting in touch with nature.
While brainstorming ideas for the “Color of Nature”, I considered three concepts from the analysis model for artistic research and applied them along the process. I started with defining a problem that connects to the concepts in HCI,
“how can art enhance interactions between human, computer, and environment?”
After that, I chose the Art-Practice and Critical Discourse as methodologies to serve the purpose of using art to convey messages and raising questions about human interaction and social practices. Then, I employed the aesthetic strategy inspired by color palettes, focusing on differences between synthetic and natural colors.
From Ishii and Ullmer’s key concepts of tangible design, the “Color of Nature” is inspired by the concept of coupling of bits and atoms that uses everyday graspable objects from the physical world to transform data into cyberspace. However, as I mentioned that the main idea of this art is to let people feel and get in touch with nature, I decided not to have any graspable object as a medium but let people directly interact with the environment using their own hands. This approach left me with a challenge of how to connect people’s hands and computers, which then led to the idea of using a hand glove to make wearable art that can transform data into the digital environment and enhance the ability to touch and grasp natural objects, as shown in Fig. 1.
My intention was to employ as few materials as possible, to make the wearable art simple to use and wear. For this reason, the “Color of Nature” only requires a hand glove, a color sensor (Adafruit TCS34725 RGB light sensor), and the Arduino UNO with a port to connect with a computer.
The process of making this interactive art started from designing the schematic of the circuit to connect the color sensor with a computer, using Arduino UNO (Fig. 2) and attached the sensor on a hand glove (Fig. 3).
As shown in Fig. 4, I wrote a Python program using Python’s Graphics Library to display the color that is received from the color sensor in real time, with a color palette in the right to record colors that have been detected. The final step is to go outside, feel the surrounding, and explore this tangible interactive art with natural objects.
Future Vision of HCI
From various research papers used as references for this interactive art, there are several approaches that I found interesting for the future of HCI. Firstly, the concept of tangible computing as a medium could be explored in more directions. The “Color of Nature” portrays the use of tangible computing in art and social awareness, and it also guides a path toward environmental sustainability. Offering a richer multi-sensory experience, tangible computing assists humans in interacting with physical objects and the environment. Considering the ability of technology to enhance interactions, computers can be a tool to connect humans with other stakeholders thus expanding the scope of future HCI.
Secondly, to further develop Winograd’s vision, the concept of posthumanism could support the back-to-nature movement. As the back-to-nature movement raises a question about human’s physical embodiment and experiences within the world, it would not be the real world if only humans exist. Therefore, if we were to think about going back to nature, it is also necessary to take nonhuman stakeholders like animals, the natural environment, and machines into consideration.
Lastly, there could be more variety of forms that apply the concepts in HCI in the future, other than technology and design. The tangible interactive art proposed in this post is also another form of HCI that draws on creativity and aesthetics. To link with the first vision, if we can use computers to connect humans with other stakeholders, there will also be more topics to include and present through HCI concepts.
References
Esche, C., & Bradley, W. (Eds.). (2007). Art and social change: A critical reader. Tate Publishing.
Forlano, L. (2017). Posthumanism and design. She Ji: The Journal of Design, Economics, and Innovation, 3(1), 16–29.
Ishii, H., & Ullmer, B. (1997, March). Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems (pp. 234–241).
Liu, S. Y. (2020). Posthuman Interaction Design: Designing with, through, and for Human-Nature Interaction (Doctoral dissertation, Indiana University).
Rahm-Skågeby, J., & Rahm, L. (2021). HCI and deep time: toward deep time design thinking. Human–Computer Interaction, 1–14.
Shaer, O., Horn, M. S., & Jacob, R. J. (2009). Tangible user interface laboratory: Teaching tangible interaction design in practice. Artificial Intelligence for Engineering Design, Analysis and Manufacturing: AI EDAM, 23(3), 251.
Tomás, E. (2017, September). How The Arts Can Help Tangible Interaction Design: A Critical Re-Orientation. In Informatics (Vol. 4, №3, p. 31). Multidisciplinary Digital Publishing Institute.
Winograd, T. (1997). From computing machinery to interaction design. New York.
