Lessons on how to create a multi-sensory integrated VR experience :
Wearing an expanding and contracting universe.
How much do we need of each stimulation to create the sweetest tasting sensorial virtual soup ?
The thing about reality is that it’s part interpretation and part physics and mostly unmeasurable. It’s no wonder the bit that we can pin down is hard to reconstruct.
It is much easier to reconstruct the unknown just because we can’t untrust it. This was our driver for explaining Dark Matter Dark Energy experientially. It also creates a level playing field for us tinkerers and hackers with little budget compared to the big players with all the fancy equipment and huge teams. We only have our passion and imagination to push the boundaries.
I had been pondering about the hive mind for a while and wondering how we could mobilise a group through individual egalitarian communication and environmental data to make something happen (banked as a future project). I needed a collaborator to help me. I am autonomous by nature but can’t do anything on my own.
Luckily I have this one talent. In fact, it is the only talent I do have : bumping into the right person at the right time … and this time it was electronics engineer and film-maker Coriolan.
Coriolan : Do you want to do an artistic collaboration ?
Bushra : Yeah funny you should ask, let’s do the hive mind.
Coriolan : No let’s do it about something simple …. like dark matter.
Bushra : Er….I am not sure that’s simple ….. a VR experience…at least nobody will be able to contest it …. OK. There’s a competition to present at the New Scientist but we need to make a VR film and we could win money. I don’t know how to do vision and sound but I can do body stuff. Maybe we can hang the VR film off the haptic experience?
Coriolan : Don’t worry I know how to make films.
Bushra : OK let’s do it.
We checked NASA and The Hubble Telescope and other reputable sites for references. I knew a bunch of physicists (hell I had even dated one for 7 years) who I had every intention of fact checking with but wanted to get it to a level where I wasn’t wasting their time before approaching them. First mistake.
Coriolan and I brainstormed some crazy ideas on how to represent the universe expanding visually and haptically. We developed a crafty styled piece of black painted cardboard with LEDS on each end spun on a motor with a 360 camera in the middle (Cori actually made a working demo) to create galaxy light trails in the VR film but in reality a room filled with balloons hitting you randomly. Such a charming idea, it makes me smile. Unfortunately totally unviable.
Making it happen
I went back to the drawing board and found the free space simulator called Space Engine based on real data created by a techie astronomer hobbyist. After spending half a day messing about with it, we realised our Macs were woefully slow. Unfazed I managed to rent a high-spec PC from Bedford though I did have to collect it which involved dragging this trunk of computer equipment which I could fit myself in and probably as heavy as me back to my London studio on the train on the same weekend as I was moving house.
Now we could progress but I was so tired. I had cobbled together a script but Coriolan, to my relief, sensed I was flagging and took the initiative to very politely rewrite it. We decided because of his very french accent I would be the one to narrate it. I tried to channel Hal (2001) and Holly (Red Dwarf) and I hoped to God our sound engineer would make me sound like a different person.
Though we worked on the first draft together, Coriolan used his expertise in filmmaking to make a coherent edit, with me in the background insisting on particular scenes.
Coriolan : I am not sure the black hole scene is totally relevant.
Bushra : Are you telling me you are going to take out the black hole scene ? Don’t tell me you are going to take out the black hole ? But I love the black hole. It’s so cool. Please don’t take out the black hole.*
In parallel, we discussed how we could enhance the immersion by simulating a feeling we had never felt.
Designing The Experience
I left him to the film and I progressed the garment design. I wanted the electronics to be as portable and compact as possible in an un/expanding garment pressing against the body. This led me to look at fabrics which were airtight used in kites, waterproof and inflatable life saving jackets. We also experimented with hacking air pumps to control the inflation and deflation. They were all too bulky and noisy which led Coriolan to propose using blood pressure machines. Genius !
The aesthetics needed to be show-stopping, after all this is the universe we were talking about. Recently I had been working on an interactive book and my head was revelling in paper manipulation. Metallic paper in fact. I had also spent some time in Japan on an art residency earlier on in the year and been influenced by a Miyake retrospective, especially the origami dresses. I mood-boarded ideas and sketches into pinterest and started ordering reflective ripstop and other interesting materials and draping silhouettes on the stand and started to apply the paper manipulation techniques onto fabric. I enlisted the very talented Wura from my studios to sew everything up.
It was very important to me to soften the dehumanising utilitarian, dare I say, patriarchal look of the VR headset, so we customised that to match the couture-theatrical styled garment. After all, in a museum, sometimes waiting in the queue for a VR experience can be longer than the actual VR experience. As soon as the person puts on the VR garb, they become the protagonist and performative to the rest of the group. Also I felt this was a necessary step towards bridging the anti-social gap of virtual and reality.
Meanwhile, a friend’s experienced sound producer husband volunteered to help us. Somehow the film and garment were coming together and we had ordered the electronics to embed for the haptic element. We were on track.
I even managed to borrow a DSLR, clear some space in my studios where there was a white wall backdrop and pressure a fellow artist Claire to model. Claire was the most reluctant model in the world. No I didn’t need to art direct her in the style of ID magazine; Claire was actually that annoyed. Great photo though.
All was well with the world.
The First Mistake
Remember that first mistake. You know the one in the beginning. Well as is the universal law of first mistakes, it chose to rear it’s ugly head now near the finishing line.
Just as I was going away on holiday, my Physics PhD doctorate friend reverted back to me with a last minute fact check:
“I love the visuals but I am not happy with the script”.
It transpired I had to only re-narrate 3 sentences…. Didn’t seem like much but let’s recap the sound critical path. Rewrite the script -> me narrate the new script -> Cori to edit and make sure it fits into the film -> Cori send to the sound engineer -> Sound engineer to sort out the sound and send back to us -> Cori to overlay it into the film and reupload.
However Coriolan had started full-time employment and with all the will in the world had no time, we had pretty much used up our favour with the sound engineer and I was on holiday and had a really bad cold so not the best for voice work. Needless to say we totally missed our initial deadline.
How to pull it back and push it further
But by this stage, I had lined up other important meetings to demo at. So we carried on regardless and managed to source through a facebook friend a very talented sound guy Cameron in LA who sprinkled his spatialising sound magic.
This 4minute 7second VR film, we naively threw into the mix as something to hang off the haptic experience had nearly killed us was finally ready. We had learnt a hell of a lot of lessons ready to apply a lot more efficiently to the next project.
And what of the haptic experience ? With 4 days to go before the first important meeting, the electronics looked like spaghetti arduinase on my desk.
Slightly panicking, I reached out to a PhD student electronics engineer friend Hugh who I had worked with in the past. He really is a like a fairy godbrother who sort of just appears when I need him, all casual,unassuming and open to my crazy ideas, rudimentary knowledge of electronics and tight deadlines and just waves his magic wand and makes everything work.
Alongside him, an ex coder/now tinkerer/excellent maker friend from my studios Mark and his dog Captain Pugwash came to see what I was up to and ended up fixing the buzzer neck piece which actually freed up Hugh to create a 3rd haptic piece.
So finally it was done. We had a VR film with spatial sound, a show stopping garment with a pneumatic waistband and a separate buzzing neck piece synchronised with the film with a customised matching headset; and an additional haptic demo just for good measure. Oh and a moody photo of Claire wearing it all.
I was now ready for the ball, I mean meeting.
*Sorry for being such a princess about the black hole, Cori. You were right of course.
Some pictures and movies about the project available for demos and exhibitions : http://www.bushraburge.com/vrdark1/
Bushra Burge is Consultant with a team looking for creative innovation opportunities or interesting collaborations with nice people.