Draw a story: waving your fan
Jisoo Shon | Carnegie Mellon University
HyperSense: Augmenting Human Experiences in Environments
Abstract
In our world, there’s a thing, creature, and space. And the act of those components proceeded by the passage of time. As human beings, we perceive and react to our environment based on innate recognition ability. However, if we think a little more, what we see or feel implies the possibility of being subjective and intolerant because we are humans. And here the concept of HyperSense reframe this human sensory experience by augmenting some receptive sense with the spatial context. This pictorial describes computation, embodiment, phenomenology, and temporality of interactions by conducting experiential dance performance.
Keyword
Interactive Performance, Projection-Mapping, Creative Drawing, Augmented Dance Experience, Interaction Design, Multi-Sensory Dancing
Introduction
This project was started by looking at some experiments relating human body movements to music or sound in a generative way and showing the idea of “body as an instrument”. Specifically in this paper, I’m looking at dancing, and among some various types of dance, I choose a traditional one, Korean Fan Dance. This dance requires a tool (a fan) for the performance and that induces unique postures of dancing. Since every piece of music, dance, and painting possesses its own inherent story, and the investigation brings this narrative more vivid and relatable for the artist. I explored some ways to let the dancer leads the story of dance and music and enjoys dancing creatively.
Related work
In this area of work, there were a lot of inspirational dance projects or reactive stage setting that was helpful to elaborate on designing the experience. Firstly, tracing the body movement and translate it to another medium like graphic visualization and the audio effect was something that captures the moment in our life to amplify the Temporality[1] in physical space. E-trace: Memories of dance[2] performance was a good example of this approach. The dancer moves freely in the empty space and her movements draw a certain line in the air or screen.
Secondly, I tried to consider the immersiveness of the experience and the method of 3D projection seems useful. It does not only expand the visual of physical space but also adds a series of interactive layers to a performance. In a Floating projection by Joanie Lemercier[3], the technique he used for creating volumetric projection enables looking at the structure of the universe at various scales that nicely modifies our perception of the real world.
From this insight, lastly, I was looking at other supplementary mediums like light and sound that bringing into human visibility. For instance, Hiroaki Umeda, a Japanese choreographer[4] who claimed that choreography is not only limited to human bodies but to anything that is capable of movement shows his longstanding belief that a human body is an intrinsic part of nature with a blurred distinction between living things and human. This approach was easily found in some interactive dance projects[5] and a dancer’s movements on the light path illustrate that connection with the surrounded environment and the period we are standing upon.
Methodology
Along with the exploratory research part, I tried to implement my concept in reality. And I followed this outline of “explore, synthesize, iterate, and design” to develop the interaction that I envisioned. The study was started by exploring computational visualization tools like Processing and I looked at codes behind fascinating interactions. Because I wanted to try body movement detection I did some experiments with a platform like RunwayML and ML5 for using Posenet and gradually it evolved into a semi-prototyping process and from there I incorporated other senses like sound and lighting.
- You can find the reference of the platform that I tried and the project code here.
As fan dance interaction evolved, I defined the system diagram to clearly understand the input of information and output of interactions. Firstly, the fan’s movement generates the output of pixels in p5.js software and depends on the fan’s surface color, it projects different colors of particles. In addition, I tried the posture estimation feature that detects the different types of posture and controls background music.
- Semi Prototyping process and code here.
The Draw a story ended up with four different experiences. The first type shows a drawing with single and multi-colors so that the dancer creates some light traces behind their body movements and completes the piece of generative painting. In the second phase, it showed the action of decreasing the volume and moving to the next song by using posture prediction. I applied the posture of stretching and raising arms for music volume control and covering the face behind a fan for changing the background music.
Reflection
My design process helped me to practice the meaning of HyperSense that brings a new kind of sense that hasn’t noticeable in our natural status. This approach opened up an opportunity to create a medium and projected outcome at the same time. For my final prototype, it was difficult to keep a balance between the concept exploration and execution part, and next time, setting a clear direction and action list in the earlier step might be helpful to deal with it. Lastly, I learned the uncertainty of outcome because whenever I tried to execute something and designing the experience, the anticipated outcome was not always satisfied. For example, for using projection mapping, the dark lighting setting is an important factor to clearly project the graphic on the wall, but at the same time, to create pixel visualization by detecting color, a bright environment was essential. Therefore, I tried multiple lights settings to fulfill both variables. Of course, because I conducted the performance by myself, the quality of dance was under expectation and it was hard to record the performance and see the output of motion detection instantly. But overall, I was able to enjoy the moment of swing the fan and felt the power of controlling the environment which I think one of my achievements.
Conclusion
To conclude, the Draw a story project asks me some meaningful questions about the multi-sensory experience for performance. There is always an audience for this type of production and considering the significance of visual and aural argumentation for them is important to properly offer a joyful experience instead of excessively increase distraction. Go back to my initial research question of “How a dancer leads the story of dance and enjoys dancing in a different way?” this still focuses on the dancer as a subject. But while implementing the final prototype, I realize it was hard to recognize the projected lightings on my body unless putting an extra screen in front of me. Therefore, I want to improve this sense of control by enabling a dancer to look at what they are producing synchronize with the action in the next prototype. And right now the dancer moves without regard for the specific visual output, and I guess anchoring the visual outcome first and choose dance choreography changes the overall experience and this is something I can take a look at. Besides, the fan dance itself was an interesting topic because, in this project, I was utilizing a tool as a medium to control the environment for the dancer. Also, the moments where the fan caught the projection on its surface captured the attention of the audience and It would be great to explore the potential of this tool more.
References
- (2016, June 16). The Temporality of the Landscape Revisited. Retrieved December 20, 2020, from https://www.tandfonline.com/doi/full/10.1080/00293652.2016.115145
- (2014, October 6). E-TRACES, memories of dance on Vimeo. Retrieved December 20, 2020, from https://vimeo.com/108109673
- (n.d.). No-logram — Studio Joanie Lemercier. Retrieved December 20, 2020, from https://joanielemercier.com/no-logram/
- (n.d.). Collaboration Project of Asia Dance Companay — Hiroaki Umeda. Retrieved December 20, 2020, from http://hiroakiumeda.com/project.html
- (n.d.). The word — Interactive Dance — Ji Young Chun. Retrieved December 20, 2020, from https://www.jiyoungc.com/the-word
Additional resources
(n.d.). Transient — Impermanent paintings — In Kepler’s Gardens. Retrieved December 20, 2020, from https://ars.electronica.art/keplersgardens/en/transient-impermanent-paintings/
(n.d.). Quayola | Sedition | Digital Art for Sale & Exhibitions. Retrieved December 20, 2020, from https://www.seditionart.com/quayola
(n.d.). KNOSPEIII-Nowhere — Caroline Reize. Retrieved December 20, 2020, from http://carolinereize.com/KNOSPEIII.html
(n.d.). Ghostcatching (1999 ESHKAR KAISER) — OpenEndedGroup. Retrieved December 20, 2020, from http://openendedgroup.com/artworks/gc.html
(n.d.). 2020 Conference on Movement and Computing. Retrieved December 20, 2020, from https://moco20.movementcomputing.org/
