A renowned dancer performed with an AI model — Can AI stimulate the dancer’s creativity?

Nao Tokui (Qosmo)
Jun 1, 2020 · 11 min read

It is a story of one of my projects that tries to expand a dancer’s creativity by artificial intelligence (AI).

“Israel & イスラエル — Israel Galvan + YCAM”

Israel & イスラエル Photo by Yuki Moriya / Courtesy of Yamaguchi Center for Arts and Media [YCAM]

Since the end of 2017, I have been involved in a project with one prominent flamenco dancer. Why flamenco? It’s an attempt to see how AI can be used to extend flamenco as a form of dance. This project is led by YCAM (Yamaguchi Center for Arts and Media), Japan’s leading museum and research institute for media art. (Please note that this article is a review of the project from the author’s personal point of view. I do not represent YCAM.)

Flamenco dancer Israel Galván has been called a revolutionary in the world of flamenco. He is a choreographer, a dancer with style with minimalism and physicality that is very different from the traditional flamenco. (I’m sure I’m not the only one who has a stereotype of flamenco where a beautiful woman dances with a rose in her hand while clicking castanets…) You may find more similarity between his flamenco and modern dance or modern ballet than with conventional flamenco. You can see it well from the interspersed use of the word “controversial” in the Wikipedia article on him.

Solo by Israel Galván

At first, I had no clue, “Why, flamenco? Why would I do that?” But when I met him for the first time, he told me that “flamenco is music,” and that changed my perspective. I had a strong impression of the elegant upper body movements in flamenco. According to Israel, the key to flamenco is the Zapateado (foot rhythm like tap dance). If the Zapateado and Palma (handclaps) are the rhythms, then the hand movements are the “visual melody.” It is flamenco that conveys emotions as multi-sensorial music that integrates the rhythm of Zapateade and Palma, the visual melody of the body movement, guitars, and songs. Once I realized it, my direction became more explicit: I could think of this project as a music generation project with AI.

Israel Galvan (Wikipedia)

He is also known for rarely dancing with other flamenco dancers. According to him, the reason for this is that when you dance with other dancers, you “tend to stay in the existing conventions.” Instead, he dances with a giant rocking chair as a fighting bull (above) or with an unstable chair with one leg cut off. In one of his pieces, “La Casa(2006)”, he incorporated the movement of the shelves in the house into his dance (video below). In order to break down the stereotype of gender, he even disguises himself as a woman.

“La casa”, Israel Galván (2006)

While these objects physically respond to his movement in some way, of course, these objects don’t have any notion of flamenco rhythm patterns. Nor is it just exact echos of Israel’s rhythms. These physical objects react to Israel’s movement in somewhat predictable ways with some randomness and contingency.

What is important is the delicate balance between predictability and unpredictability, and the ambiguity that leaves open room for Israel to reinterpret.

It was in 2017 that YCAM began to approach him. YCAM didn’t have the idea of using AI from the beginning, but in the course of discussions, the use of AI was proposed in response to Israel’s desire to “dance with his alter ego” and “have a buddy like Sancho Panza for Don Quijote.”

AI DJ Project / YCAM (2017) Photo by Yasuhiro Tani / Courtesy of Yamaguchi Center for Arts and Media [YCAM]

Around that time, I was working on my AI DJ project at YCAM. It was very natural for me to join this dance project. Needless to say, there was a lot of overlap with AI DJ project, which started out with the idea of “playing DJ alongside with my alter ego DJ.”

The delicate balance between predictability and unpredictability, and the ambiguity that leaves open room for the dancer to reinterpret.

Nevertheless, once the project actually started, we faced a lot of different problems. Unlike common music, in this case, there is no digital data at all available for the training of AI. The first challenge for us was to make a custom flamenco shoes with sensors. YCAM InterLab, a research branch within YCAM, managed to build the shoes with piezo, gyro, and pressure sensors, and start collecting data on Israel’s Zapateado. The shoes can collect data at three points in his foot: the heel, toe, and belly of the foot.

Custom flamenco shoes for training data collection — Production: YCAM InterLab

I won’t go into detail here, but the creation of the shoes for this data collection was way more challenging than we initially thought. The first shoe we made broke in three seconds when we asked Israel to try because the pressure of Israel’s step was too intense (graph below). The second technical challenge was how to increase the temporal resolution (sampling frequency) to record his zapateado. It turned out that he taps more than 10 steps per second at its fastest. We also had to make the shoe entirely wireless to use them in the performance.

An example of failed data. Too much pressure is applied to the step, causing the sensor to stick to the upper limit.

In flamenco, a compás, a 12-beat-long loop, is a basic unit of musical development. We had a hard time getting Israel to put on the data collection shoes and dance different compás over and over again.

Recorded sound in a data collection session. You can hear how fast he goes.
An audio sample generated based on recorded sensor data. I used a VST plugin used by foley artists to convert sensor data to audio.

We implement a model that generates zapateados by using the data collected over these painstaking efforts as training data. Using the wireless shoes to take Israel’s steps as input data, our zapateado generation model will predict and generate the next development leading to the input.

As for the generation model, we tried various models used in music generation, including one that uses RNN/LSTM to estimate the next step from the previous sequence of steps. Finally, however, we adopted an architecture called the Variational Autoencoder to generate two-compás-long steps(24 beats). The architecture of the model is similar to the one of MusicVAE of Google Magenta.

The model learns whether or not there is a hit in each grid of quantized beats (onsets), the strength of the hit (velocity), and the deviation of the hit timing from the grid (timing offsets).

Roberts, A., Engel, J., Raffel, C., Hawthorne, C., & Eck, D. (2018). A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music. http://arxiv.org/abs/1803.05428
Tokui, N. (2020). Towards democratizing music production with AI-Design of Variational Autoencoder-based Rhythm Generator as a DAW plugin. http://arxiv.org/abs/2004.01525

In the performance, data from Israel’s last 2 compás is read by the shoes and fed to the model to generate the next coming comás. To make generated step patterns into audible sounds, we made simple machines that use solenoids to hit the wooden floor. We could use software synthesizers and samplers for this purpose, but we decided not to. We wanted to put physically moving machines on stage, so that the audience might feel the existence of invisible AI, without “anthropomorphizing” AI too much. (The solenoid machines were created by my old friend, Kanta Horio)

The solenoid machines used in the performance — Photo by Yuki Moriya / Courtesy of Yamaguchi Center for Arts and Media [YCAM]

In the creation of this project, I went through the process of collecting data, training the Zapateado generation model, and validating the results over and over again. However, when I let Israel hear the results of the generation of the learned model, he always told me:

“It’s too flamenco-y.”

“I don’t need a degraded copy of myself.”

His comments had me tearing my hair! I trained the model with data of Israel, but at the same time, I needed to deviate from them. It was the biggest challenge throughout this project.

I tried various things: to mix traditional flamenco rhythms (e.g., Bleria and Sigiriya) and Israel’s freestyle rhythms in training / to take a generated pattern as a probability distribution and randomly sample from it / to emphasize or invalidate the deviations from the grid (time offset).

How to deviate moderately from flamenco-ness while learning the steps of flamenco

As for random sampling, you can think of an irregular roulette with big holes and small holes. If you roll roulette and throw a ball in, you’re naturally more likely to fall into a big hole. If you repeat it over and over again, the probability of a ball falling into that hole should be proportional to the size of the hole.

Examples of the output of the generative model

What the rhythm generation model does is similar to creating this uneven roulette. For example, the first beat of the compass has a very high probability of a hit, so it corresponds to a huge hole. If there are a total of 10 hits in the compás, then if you throw 10 balls, they should be distributed in the compás according to the size of the holes, i.e., the probability that there are hits as predicted by the generative model.

Here, what happens if we try to make the larger holes bigger and the smaller ones smaller? This should lead to a less surprising result. On the other hand, if we even out the difference between large and small, we can produce a highly unexpected result while following the prediction moderately. Completely eliminating the variations in the hole size will result in totally random outcomes. The parameter that controls this randomness is called temperature and is often used when using these generative models. We manipulated the temperature parameter as the scene unfolds during the performance.

Israel & イスラエル — video and lighting by Satoru Higa and Ryo Kanda, two up-and-coming visual artists — Photo by Yuki Moriya / Courtesy of Yamaguchi Center for Arts and Media [YCAM]
Israel & イスラエル Photo by Yuki Moriya / Courtesy of Yamaguchi Center for Arts and Media [YCAM]

Finally, our performance, “Israel & イスラエル” had its premiere on February 2, 2019 (イスラエル is Israel in Japanese). The title hints at the existence of the “two” Israels, Israel himself (Israel) and the AI alter ego of him trained and made in Japan (イスラエル).

Filming: Atsushi Tanabe, Rumi Tanabe, Kosuke Shiomi Film Editing: Atsushi Tanabe

After the first performance, Israel came to me with a satisfied look on his face and said to me:

“I felt like there was an unknown creature on stage, not a musician, not a flamenco dancer, not a human being in the first place. It was very inspiring.”

“I sometimes forgot I was dancing alone.”

初演直後のイスラエルと

It was the best compliment for the team and me, as the project was intended to stimulate his own creativity.

I imagine that he found the balance between predictability and unpredictability of our AI system unique and exciting, and it stimulated his instincts as a dancer.

I felt like there was an unknown creature on stage, not a musician, not a flamenco dancer, not a human being in the first place. It was very inspiring.

“I felt like there was an unknown creature on stage, not a musician, not a flamenco dancer, not a human being in the first place. It was very inspiring.”

During the rehearsal sessions, he was reluctant to dance with AI, which made the production team quite nervous. Now I understand why so. He wanted AI to “surprise” himself in the production, in front of the audience. I’m sure that he is confident enough to properly respond to anything coming out from the AI model.

I don’t care if I fail. I want you to surprise me.

In his studio in Seville.

Six months back from production, in the summer of 2018, I flew to his hometown of Seville with the staff of YCAM InterLab to visit his studio. The main objectives were to deliver the shoes to him for data collection and to facilitate discussions about the direction of the performance.

During our time off, we walked the streets of Seville. We even went to a show to get a better understanding of the flamenco culture. The show held at a special venue called” tablao” was great by itself, but what impressed me the most was a scene in a bar we came across by chance. A circle of singing and dancing flamenco spontaneously arose there. In the center of the circle, there was an old lady singing as she has been doing for decades. When the lady stopped singing, to my surprise, a 20-something girl in front of me stepped forward and started dancing.

The dance and song continued without any particular arrangement in advance. I witnessed a world that is only possible because of the unwritten rules shared by those born and raised in flamenco culture.

Daily flamenco life in Seville

How difficult it is to bring a whole new set of values to the traditional flamenco world that is so firmly rooted in. It was the first moment I realized that, even though I was and am an outsider. I believe I caught a glimpse of Israel’s mindset that made him want to try dancing with machines.

This project is currently in progress: after the first performance in February 2019, we performed at the Maison de la Culture du Japon in Paris for three days in October. (Shout out to the generous support of the Maison.) I personally would like to continue to develop and update the generation model to reflect the current state of AI technology.

A group photo of the team after the Paris performance. The hand sign represents “Yamaguchi,” where YCAM is located.

I hope that the current situation caused by the new coronavirus will be over as soon as possible.

I hope we can soon show new performance somewhere in the world with Israel, the members of his company, and all YCAM staff.

Credits:

Israel & イスラエル

Direction, Choreography & Dance
Israel Galván

Cast
Richi Owaki (YCAM), Rie Okada

R&D Direction
Takayuki Ito (YCAM), Richi Owaki (YCAM)

Technical Direction
Richi Owaki (YCAM), Junji Nakaue (YCAM), Pablo Pujol (Israel Galván Company)

Production Management
Clarence Ng (YCAM)

Machine Learning & Artificial Intelligence System Development
Nao Tokui (Qosmo, Inc.), Miyu Hosoi (Qosmo, Inc.)

Data Capturing System Design & Development
Mitsuhito Ando (YCAM)

Visual Development
Satoru Higa, Ryo Kanda

Device Design & Development
Kanta Horio, Alberto Boem, Keina Konno & Richi Owaki & Hoshiro Ando (YCAM), Pablo Pujol (Israel Galván Company)

Sound Engineering
Junji Nakaue (YCAM), Pedro Leon (Israel Galván Company), Mitsuhito Ando (YCAM)

Additional Compositions Extracts
Ikue Mori

Video Engineering
Richi Owaki (YCAM), Keina Konno (YCAM)

Stage Set
Hoshiro Ando (YCAM)

Lighting
Fumie Takahara (YCAM)

Lighting Program & Network Construction
Yohei Miura (YCAM)

Assistant Director & Script
Miguel Álvarez-Fernández

Hair Makeup
Sayano Tani

Coordination
Rie Okada, Asumi Kitahori (YCAM)

Interpreter & Translation
Rie Okada

Delegate Producer
Akiko Takeshita (YCAM)

Co-producer
Pilar Lopez (Israel Galván Company)

※The project was made collectively among the members above regardless of the credits.

Technical Support
Yano butai

Education Program
Daichi Yamaoka & Soma Ishii (YCAM)

Public Relations
Momoko Aoyagi & Nanami Hashimoto & Satomi Okazaki (YCAM), Tomoko Yotsumoto (Sankai Production, LLC)

Documentation
Tomoya Watanabe (YCAM)

Legal Consultant
Tasuku Mizuno (CITY LIGHTS LAW)

Administration
Kiyomi Arifuku & Yuriko Asamoto (YCAM), Rosario Gallardo (Israel Galván Company)

Co-production Coordination
Pilar Lopez (Israel Galván Company)

Organizer
Yamaguchi City Foundation for Cultural Promotion

Co-developed
YCAM InterLab

Association
Yamaguchi City, Yamaguchi City Board of Education

Support
the Agency for Cultural Affairs, Government of Japan, Japan Arts Council, Embassy of Spain in Japan

Produced by
Yamaguchi Center for Arts and Media [YCAM] & Israel Galván Company

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store