Last summer, I had been so tired of my client works indeed. So I decided to go back my hometown and devote the whole summer to the music video for my friend producer, imai. I’ve made the video for ‘Eye’ from his electronic duo group_inou, which used StreetView and other lots of creative coding techniques. Also in this video, I introduced openFrameworks and ViveTracker to realize the smooth and free camera movement in a stop motion. I would like to write the behind-the-scene down shortly.
Japanese Confectionery — Cultural Context
Some people in non-Asian countries might think “what are these objects dancing on table?”. The answer is this emoji:🍡. These are various types of rice cakes, which especially popular for old people in Japan. I personally love their taste, minimalistic appearance, and the softness. These have completely nothing to do with the lyrics — maybe I unconsciously wanted a kind of therapy by moving them haha
The title of the track ‘Fly’ is often misheard as ‘fried’ by Japanese. The scene three rice cakes avoiding from fried-shrimp and then deep-fried was a kind of wordplay. In contrast, they finally flew in the air at the last scene.
Also, all of the video had shot at my grandparents’ house that I have nostalgic memories.
Visualizing Camera Trajectory with ViveTracker
I was wondering how to make the video look something new while using traditional stop-motion technique. My attempt in that video was to introduce a shaking camera movement. I built a special shooting system like below to realize it.
The basic idea is:
- Keep tracking camera position and angle with ViveTracker.
- Every time a frame is captured, record a transformation of camera at the moment.
- Visualize a camera trajectory in 3D space using openFrameworks. (CamGraph.app)
- With looking at the visualization, move the tripod by hand.
Although it’s hard to explain even for people in video industry, it’s NOT a motion control system. I’ve felt most of camera works using some sliders or rigs are too smooth compared to the animation by hand. The key point of this system is that it doesn’t move a camera automatically but just displays ‘how the camera has moved’. Looking at the information, an animator has to move the tripod by themselves as same method as other objects in a frame.
This system worked successfully beyond my expectation. I think the smooth but bit wiggled camera shaking really matched with a groove of the track. I open-sourced the very tangled codes just in case. (GitHub)
For more informations:
Syncing with Music
I’m basically a motion designer fighting with fucking AfterEffects everyday, but not a good stop-motion animator. So I somehow tried to integrate my way of thinking for motion with this physical technique. I never wanted to make a pre-vis in my computer because it forces me to behave as if ‘human-powered renderer’ in a real world. I’ve tried the approach in this video with Hu Yu and Laurent Gray two years ago. However, it was so painful and we were about to fall depression haha
In this project, I just drew a kind of graphic notation on my iPad Pro. The I used it only as a hint for timings of motion so that I can left enough spaces to decide how to move them just like the straight-ahead method.
Some professionals use ‘X-sheet’ feature on Dragonframe to do this, but I’m used to horizontal timeline so displayed it on my custom app.
In this several years, one of my theme as a video director is how I can extend the video production technique — like a lot of visual artists such as Tony Hill, Rybczynski and Michel Gondry has done.
The point is — they merely show their unique and up-to-date processes inside their videos. Even if we know nothing about the behind-the-scene, their videos still feel something new, yet emotional.
When I look at the recent art & tech scene, I occasionally feel there’s so many projects ended up with just a demo of cutting-edge technology. Such a video tend to begin with sequences such like calibrating projectors’ grid, someones wiring lots of stepper motors with distracting technical descriptions.
In this project, I also wanted to show my attitude for video production — designing new workflow to create something new with whatever I can use, but no need to show its process inside the video. I hope people to just enjoy the 🍡s’ dance along with imai’s emotional track, and only some geeks like you would understand what made the video feel something new technically.
I don’t make sure whether it worked, though, I would be grad if someone feel empathy for my stance.