Why quantity matters for creativity

Stating the obvious: some ideas are definitely worse than others. Like buying the Brooklyn Bridge or choosing plastic surgeon on price alone (no pics). And then there are the great ones, the wheel, the Negg, the abolition of slavery just to name a few.

Also quite obvious: we want more of the good ones and less of the bad. And some people seem to have figured it out: the Jobs, Gates, Bezos of the world. Is it just that they were lucky in the gene lottery? Probably, but hey, we all have to play the hand we’re dealt! So what to do?

Unfortunately it is not possible to decide beforehand to come up with a great idea. It is not until after the fact the we can know if it was good or bad, sometimes it can take decades.

What we can do is increase the chances, the probability for great ideas to appear. It turns out that people with great ideas have the same share of bad ideas as the rest of us. The difference is that they generate a lot more ideas. Quality is a result of quantity!

Increasing the number of ideas is the topic for this issue of Battery. Not by producing ten times more ideas on the same silly problem, but by increasing the number of “problems” that scream for attention.

And there is a ton of opportunities hiding in plain sight! We just have to start seeing them, and this is a matter of mindset.

The first article is about what I call the innovation trigger. Whenever something feels complicated we should take a step back and realize that this is an opportunity to be creative, to innovate!

If something feels complicated... - Dinahmoe Battery

The second is about why we should nurture a contrarian mindset. There will be no great ideas if we just accept reality as it is. So by constantly questioning everything we might annoy the crap out of our friends but we will have an infinite source of material to generate ideas from.

Be a rebel, question everything! - Dinahmoe Battery

The last is about a single word: “obvious”. Whenever we scratch the surface of something that seems obvious we realize that it was much more complex than we thought. Discarding something as obvious is a lost opportunity ot be creative and innovate.So this is the first step in the process: to increase the number of ideas just by generating more “problems” that needs to be solved. The next step is to come up with ideas for solutions. This will be a topic for a future issue!

In defense of the "obvious" - Dinahmoe Battery

Before you go…

Agree or disagree, ask questions or suggest improvements,
the comments section below is all yours!

To get in touch, please connect with me on LinkedIn.

If you want to get notified about new articles and projects, please subscribe to Dinahmoe’s newsletter by clicking here

Battery003: Where do great ideas come from? was originally published in Dinahmoe on Medium, where people are continuing the conversation by highlighting and responding to this story.

Six projects, six interaction models

This article was originally published at Dinahmoe Battery as a part of Battery002: Smart content, shoppable videos and Bandersnatch!

Interactive film and video can be a million things. It could be interactive fiction, documentaries, music videos, games, instructinal videos. All types of videos can be made to benefit of interactivity, if it is done right.

Here is a selection of our favorite interactive film and video projects.

ONLY — The Liberation is not only a music video but a fashion lookbook where you can buy the items in the video. Shoppable video before the term existed!

Cisco — Internet of Everything is an interactive interview where you can have a two way conversation with the video. Interaction is obviously voice!

Canada Goose — Out There is a cinematic short film where bonus material was integrated in the experience in a seamless way. .

Outcast — The Possession Begins is an interactive trailer for the TV series Outcast. You control the experience with your eyes.

Nissan Infiniti — Deja View is what can be called interactive fiction, pretty much where Bandersnatch would end up. What is unique though is that the viewer doesn’t control the film directly. The characters in the film are actually calling your real life phone, what you say in those conversations affects how the story evolves.

Netflix Sense8 Fanvid Generator allowed fans to create their own fanvideo by chatting with a chatbot impersonating one of the characters in the series. A unique video was then rendered based on the conversation.

I hope these projects will inspire some new creative ideas on how to make a boring static film project into an interactive masterpiece! Here we go!

ONLY — The Liberation

The Liberation is an interactive music video telling the story of the ONLY girls who makes a visit to an unexpecting sleepy town. At four points in the story the user has to interact to make the story move along. The interaction is super simple and at exactly the right level for the viewer’s curiosity to kick in.

All clothes in the video are tracked and clickable which allows the viewer to expore and even buy the clothes. During this time the user move into a kind of frozen universe where you can explore the clothes and accessories that are currently showing.

The music is a specially adapted version of a track by Lune: Let Go. We worked together with Lune’s producer, Carl-Michael Herlöfsson, to adapt it to interactive use. Our mission was to make it work in the interactive experience while still syncing perfectly to the linear parts.

We split up the music in several layers so that we had individual control over them. One layer was the vocals which was split into separate phrases with delay and reverb tails. This way Lune would always finish a phrase even if the user stopped the video in the middle. We created a separate layer for the “frozen world” when the user pauses the video that always plays in sync with the main track. This makes the transition between the worlds totally seamless. For the interactive parts we created music loops that could go on until the user interacts.

When they do, the music waits until the next bar before moving on to the next musical section. There is even a drum fill to bind the sections together as in the linear track.

This and many other small details makes the music totally seamless, almost like a band was playing it in realtime while you watch. The user then forgets the technology and just experience, making this a truly unique interactive film.

The Liberation has been richly awarded: Cannes Lions Gold, 2x Cannes Lions Silver, CLIO Awards Gold (Digital/Sound Design), FWA Site Of The Day & Site Of The Month and more.

Cisco — Internet of Everything

We and GSPSF invited then Google Chief Technology Advocate Michael T Jones to explain the concept Internet of Everything . “Everything” is a pretty big topic to cover in an interview, so we wanted to give the user the possibility to navigate the interview to what interested them most. An interactive interview! Or as Mr Jones puts it, a real two-way conversation.

In this project the whole experience was in the browser, no 4th wall to break, but to present content in a novel and useful way. The user activated their microphone in their browser and could then both navigate the topics and ask questions. Menu options were available for those not using voice.

To enhance the experience we used the user’s mobile phone as a second screen. This content was not only available during the interview but after, as a list of interesting topics to explore deeper.

Using voice made the experience much more personal and human.

Canada Goose — Out There

Out There is a great example of how interactivity can be used to make the experience much richer. It is the first ever global campaign released by luxury outdoor brand Canada Goose. It features the real, incredible stories of the brand’s most inspirational brand ambassadors.

The interactive short film was shot in New Zealand by an impressive veteran film crew. Directed by Academy Award-winning filmmaker of “Crash” and “Million Dollar Baby” Paul Haggis, cinematography by “12 Years a Slave” Sean Bobbitt, music by Oscar-nominated composer Josh Ralph and sound engineering by Oscar-nominated Tom Myers/Skywalker Sound.

The backstory

It is not every day you get the chance to destroy the cinematic art of a two time Academy Award winner with some interactive disruption but that was clearly a possible outcome here.

The film was surely going to be spectacular, no doubt. But it was the idea behind the script that made it truly special. Each of the scenes in the film is based on real events in the life of the brand ambassadors.

They were interviewed about their memories of the specific event, and they told this story and several more in a very personal way. They also allowed access to their private photo albums with previously unpublished images. All ingredients were there to add emotional depth to the scenes in the film.

Normally this kind of material is tagged on in the end but then we would miss the opportunity to give the scenes that extra depth. Instead we wanted to integrate it and connect it to the actual scenes. When the user reaches a scene they should get the option to go down the rabbit hole to hear the real backstory and see the images.

It was essential to not break the movie magic so the integration had to feel like it was a part of the film. The transition must be totally seamless, the music score should continue uninterrupted, the visual transition and the presentation of the personal photos must feel cinematic and in the style of the film.

Here are some early drafts for the photo albums

How it turned out

For the user it starts like any normal video. When moving the mouse/touching the screen a timeline appears at the top which allows the user to navigate to any position in the film. The rabbit holes are marked on the timeline so that the user understands there is something special happening there.

When the user reaches a scene a subtle indicator shows that they can experience the true story behind the scene. By interacting the user is seamlessly taken into a photo contact sheet filled with personal photos describing the scene. The person starts telling their remarkable memories with their own voice, while photos from their personal collections visualises the story. In the site capture below you can hear Marilyn Hofman-Smith tell the story of her close encounter with death.

There are two additional memories that the person shares. The contact sheet is interactive so the user can explore it and find additional photos by moving it around. When the user is done they are transported back to the film at the same position where they left off.

It doesn’t feel like the main film is interrupted, it feels like the story is enhanced. From this point the user looks at all the fictional scenes with totally new eyes, knowing that there is a real story behind the scene.

Outcast — The Possession Begins

This is an interactive trailer for the TV horror show Outcast that we produced for our client Campfire. The series is created by Walking Dead Robert Kirkman and is all about demons and possession, kids chewing off their own fingers and worse, great fun!

The experience allows you how it feels to get possesed. When you close your eyes you allow the demon to possess you, when you open your eyes you realize that the possession is all around you, there is no escape. It is all about immersion, to bring the user into a narrative and lose themselves. It is hard to find an example more suitable of an experience that responds to you!

Some backstory

Budget and time did not allow to even think about developing a custom solution for detecting opened and closed eyes, so we looked at which existing solution would produce the most reliable result. Visage was by far the best at the time. However, the library was huuuuge (13MB) so using it on mobile devices felt like a bad idea. This was not a showstopper since mobile is used in situations where closing your eyes might seem suspicious to people around you.

While keeping your eyes closed you could hear a demon walking around you (3D binaural sound, headphones highly recommended). The visuals weren’t that important (eyes closed, duh!), we made a simple video loop with the black goo that is so persistent in the series.

The demon script was a linear story which you followed to the end, even if you did not want to. You could try to your eyes to stop it but the moment you close them you were back exactly were you tried to escape.

For the open eyes world we needed to create visual material which required access to original material from the series. This turned out to be a challenge. The fear for leaks before launch of the series was palpable (the general paranoia for leaks has gotten far worse since then 😓). We were required to work on a computer not connected to the Internet, not a simple thing interactive production.

It was solved through a combination of physical security and that we only got access to one second long clips that did not reveal ANYTHING about the plot. In Swedish we have an expression which translates to “cook a soup on a nail” which describes the situation perfectly. The soup actually ended up quite tasty in the end.

Depending on where you opened your eyes in the demon storyline you were served clips that were connected to what the demon was talking about. The clip ended in a loop (seamless of course) until you closed your eyes again.

The interactive soundtrack was a very important part in making the experience immersive and as intense as it was. I will cover the importance of sound in interactive applications in a future article.

From a technical POV

The experience was made to have the widest technical support possible. It supports desktop, tablet, mobile, and can be controlled by eye tracking, keyboard control, touch. The full experience is supported on all major browsers including Internet Explorer. iPhone before iOS10 doesn’t support interactive film since it opens video in its own player, but thanks to a hack found by our friends at Earth People we were able to support that too.

And how did it turn out?

Pretty good I think. It won three Clios, a gold for Technique, a silver for Innovation and a GRAND in Digital, and a Bronze Lion in Cannes for User Experience Design. It is not an Oscar but close enough (genetically Lions and Oscars are 99% the same, I don’t know about Clios though).

Nissan Infiniti — Deja View

Deja View is a great example of choose-your-own-adventure. Itis an interactive film where you interact with the characters in the film by talking to them. During the experience the characters will pick up their phones and call your real world phone. What you say to them determines what will happen next and how the story evolves.

The backstory

When I read the brief from our client Campfire for Deja View it was immediately clear that this was something out of the ordinary. It described an interactive film experience where the characters in the film calls you on your mobile phone and you have a conversation with them. It had to respond immediately to user interaction. The goal was to create a motion pticture quality experience. At the time I did not know the scope of the script but in every aspect it felt huge, lots of scenes, alternative takes, clips.

Super cool but could it be done? Or rather, could it be done with the quality and user experience that the idea deserved? I jumped into the deep end to define the project components. A custom phone system, speech recognition, natural language processing, dialogue flows, the list goes on. And the icing on the cake: realtime edited interactive video with a level of control never done before. It was a fantastic challenge!

The producer expected a Flash solution but I did not want to do that. For me it was already dead as a technology (this was 2013, ancient times). I also wanted the experience to run on iOS and Android. This was not a requirement from the client, but the mobile trend was clear to me and it felt like a good investment for the future.

The traditional method of preloading clips would not be feasible. At all decision points there would be several clips to preload since all had to be ready for playback. The logic for this would become very complex and brittle. Adding tablet support and it was clear that this was not the way to go.

Without gettting too technical we developed a method tthat allowed us to use a single video element and that supported the unique complexity of the project.

But that was only one of many challenges. The one that took the longest to find the optimal solution for was the phone system, the speech recognition and natural language processing. It was not until a couple of days before the deadline for the proposal that we knew we had a solution that could deliver with quality. One day that story might get its own article.

How does it work?

The user calls a number and gets a code, enters this in the browser and the experience starts. The first scene show the two main characters sitting in a car, waking up with no memory about who they are or where they come from. Looking at their mobile phones they discover that someone has called them several times, the number on the screen is YOUR phone number. The man decides to call you and your mobile rings in real life. Here is a screen capture:

The characters calls you at several points in the story to ask you for help to solve the mystery. During the calls you don’t see the actors (alhtough that would have been possible too), what you say during the calls affects how the story evolves.

In the screen capture above the user mentioned “dry cleaner” (makes sense if you have done the experience 🙂). After the call we weave what the user said into the conversation by playing a corresponding clip.

The video is very responsive allowing us to change a single sentence as above whithout any latency, buffering and loading. In several other decision points there are up to four alternative clips, sometimes within a second from the user interaction, which are played back as if it was edited that way.

There was over 35 minutes of video and a total of over 300 clips that were controlled interactively. Below is a simplified flowchart for the video part.

The flowchart does not include flows for the interactive audio track. To cover all these edit points the audio could not be on the video itself since that would cause ugly cuts in the audio. Creating an interactive sound track is an essential part of making the experience feel like motion picture quality. I will cover this part in a separate article.

The flow charts for the interactive phone calls are also not there, each of them similar to the one for video, a potential topic for another article (so much to write!).

Netflix Sense8

Sense8 is a Netflix series about 8 people that are emotionally connected in a supernatural way. The fan culture for the series was very strong and users produced fanvideos with their favorite characters from the series. Campfire approached us with the idea to enable users to create fan videos without having to know anything about video editing.

The solution was a chatbot driven experience where users communicated with one of the main characters from the series, Nomi Marks. During the conversation we detected what characters in the series that the user liked and what emotional content they preferred. Based on this we rendered a personalized video that the user could share on social media.

Behind the scenes

We already knew how the fans wanted a fanvid to look like since there already was a ton of them on YouTube. The raw video material was 15 hours from season one and a bonus Christmas episode. The first challenge was: how condense all this material down to basic building blocks that can be combined into an infinite number of fanvids that really feel personal.

We started with editing the raw material into separate clips. We did not use the audio from the clips since we wanted to add a super emotional music score on the final video. This excluded all clips where you could see the characters talking.

The first tests assembling the clips into videos showed that using separate clips was the wrong approach, we lost all story telling and the final video just felt random.

We needed to keep related clips together to make sense so we experimented in combining the individual clips from a specific scene into one or more mini stories. The editing had to be like a movie trailer so every single clips was cut down to its minimum. Except the love scenes which does not work well in action editing style.

We were now down to around 1000 mini stories, 1,054 to be exact. The length for the final video should be between 35 and 45 seconds. Each final video would then be a combination of around 5 mini stories, each between 5 and 8 seconds long. This would create enough number of combinations to blow my tiny math brain.

All mini stories were categorized for character and feeling in a custom CMS that allowed easy testing and rendering of videos. New tests started to look good but still felt a little random.

We added a logic layer to handle the obvious story telling errors:

• some clips had to be in order, e.g. a house that is blown up cannot resurrect in the next clip.
• some were different versions of the same mini story and should not be in the same video,
• some only worked as the last clip etc.

Now we had all pieces in place. The server application rendered a title card with the user’s name, combined a selection of clips based on the user’s preferences and the clip logic, added a custom music score, rendered the final video in less than 2 seconds, returned it to the user so they could share it in their social channels.

The final result has made people cry, literally. I am not kidding.

During the first week with a completion rate of 90% over 8,000 users went through the 3 minute long chatbot conversation, got their personal video and shared it in their social channels.

Two approaches: in-browser or server side

There are two approaches to generate dynamic personalized video. Either the personalization is done in the browser or it is rendered on the server side and then delivered. Both methods have its pros and cons so which to choose depends on the project. Below are the main differences:

In the case of Sense8 we used server side rendering of two reasons:

• the high number of mini stories
• we wanted the user to be able to share a native video, not just a link

These are the two most common reasons for any project to use server side rendering. The main benefits to do the personalization in the browser are

• scalability. In-browser uses CDN for assets and can be scaled to any number of simultaneous users. Server-side rendering requires dedicated hardware that has to be scaled linearly with the number of users.
• immediate playback, server side rendering takes time for the rendering which makes it unsuitable for realtime delivery.
• interactivity, e.g. added functionality, user selections. Dynamic personalzed video in the brwoser can be combined with any of the functions described in the article about interactive video.

Conclusion

This was just a selection, and only a brief introduction to where interactivity makes sense. As I said in the introduction fiction, documentaries, music videos, games, instructinal videos, all can be made to benefit of interactivity if it is done right. Eventually many of what today is static videos will become interactive.

This article was originally published at Dinahmoe Battery as a part of Battery002: Smart content, shoppable videos and Bandersnatch!

The UX of interactive film was originally published in Dinahmoe on Medium, where people are continuing the conversation by highlighting and responding to this story.

Caring for the details pays off

This article was originally published at Dinahmoe Battery as a part of Battery002: Smart content, shoppable videos and Bandersnatch!

It has always been our highest priority to create experiences where the technology doesn’t get in the way. Every obstacle and disturbance breaks the magic. You are thrown out of experiencing mode and into observing mode, or in worst case complaining mode.

The obvious examples are loaders, bad UX, glitches, bugs, but here I will write about what breaks the experience in the medium itself.

The goal is to create what I call a seamless experience. The viewer should not even notice or think about the smoothness, they should just be totally absorbed by experiencing.

If awards are anything to judge from it seems like caring for the details pays off pretty well. The three productions below have collected a total of

• 4 Cannes Lions: 1 gold, 2 silver and 1 bronze
• 4 Clios: 1 GRAND, 2 gold and 1 silver
• FWAs Site Of The Month, Site Of The Days
• and more

So cheers for craftsmanship, here is a selection of our contribution in the seamless category. If this article is too specialized for your taste and you just want one takeaway, just remember the awards 😜.

Our forgiving eyes

The mind has an amazing ability to accept the weirdest cuts in film.Jumps in positions, angles, time, space and we’re just “ok that makes sense”. Cuts is actually how we tell stories visually. Even visual errors are passing the quality test, freeze frames are fine as long as they are not too long and frequent.

Sound on the other hand immediately is noticed as an error. A single drop out in the audio and everyone, not only the media pro, noticed that something was wrong. We expect sound to be continuous, and we immediately notice when it is not.

We are even more sensitive to music, jump out of sync and the whole dance floor gets off balance. For a music professional we are talking single digit milliseconds.

Seamlessness in a nutshell

If you jump on the timeline on a YouTube video you don’t expect a continuous experience, quite the opposite actually, but in interactive film creating a cut that you notice is never a positive thing. At best it just feels a little sloppy, at worst it breaks the experience. So it makes perfect sense to put some work into avoiding interruptions of all kinds.

There is another side of the coin though. Sound can actually be used to bind interactive film together, smoothing out the glitches and allowing the viewer to experience the film uninterrupted.

Below are three different projects that showcases how we use the sound track to create a seamless experience. If you are not familiar with the projects, please have a read here since I will skip the general presentation and instead jump start with a screen capture and then breaking it down from there.

Please note, you really have to look at the screen captures to get the whole story, so don’t be lazy and try skimming!

Outcast — Possession Begins

In Possession Begins we are moving between our inner world where a demon wants to possess us, and the outer world where we can see how possession is all over the place. As you can notice the transitions between the two worlds are totally seamless.

The inner world

Starting with the inner world, i.e. having your eyes closed. Visually we show a short video loop of black goo. iOS doesn’t support multiple video elements playing at the same time, so a video loop would not allow a seamless transition between the two worlds. To solve this we did the loop as an image sprite instead.

The black goo does not have any sound in itself, instead we are in the world of the demon. The sound track is binaural to make you feel that you are in the middle of it all. The inner world is a linear story that you cannot skip or jump around in. You can try to escape by opening your eyes, but when you close them again you are back at the same place you left, no escape.

We separated the sync sounds from the demon voice. The timing of the demon phrases was time locked to the sync sound, so if you would run the experience from start to end with your eyes closed the separation would not have made any difference.

It is in the transitions between the two worlds that it matters. We could now let the demon complete his current phrase uninterrupted instead of doing a fade out. There are several benefits to this:

• it makes the experience feel more seamless and crafted.
• we know that the user got the whole story
• the demon becomes more real, he is not just faded away like some silly audio file, he doesn’t repeat himself when returning as if he was just on pause.

Here is a simplified timeline to ponder:

The outer world

Over to the outside world. When you open your eyes we make a visual transition to a nightmarish scene showing someone “enjoying” the company of the demon. The scene is selected based on what the demon just talked about so depending on when you open your eyes you will get a different scene.

There are ten different scenes, all using the same form: a buildup that lands in a video loop that runs until the viewer closes their eyes again.

The sound for the buildup and for the loop were controlled separately. Adding the sounds to the video clips would not have been possible since this would produce disturbing cuts in the audio.

ONLY — The Liberation

The Liberation is an interactive music video for the clothing brand ONLY. There are four different positions where the viewer has to interact to make the story move along. All clothes in the video are tracked and clickable which allows the viewer to expore and even buy the clothes. During this time the music video is paused.

This is one of my favorite projects. The interaction is super simple and at exactly the right level for the viewer’s curiosity to kick in. It is shoppable video before the term existed. The story, casting and direction is great!

It is also a perfect project to show the importance of seamlessness and a fun challenge to solve. Here is a simplified timeline of a part of the video:

Since the video is paused when exploring the clothes we could not have any sound on the video itself. We wanted the sound to fade out after the video was already paused, otherwise there would be an ugly cut in the audio. Therefore all sounds were played and controlled separately.

Starting with the sync sound. The sound design is not too busy since the music is the hero, but there are sync sounds all through the experience. There are sections in the film where the music is played in loop, during those there is instead an ambence loop playing.

The music itself was mixed without lead vocals. Each phrase of voice over and vocals where separate files and played in sync with the music track. This way we could let the current phrase play uninterrupted to the end even if the user paused in the middle of it. Seamlessness! The lead vocal phrases where bounced with delay and reverb to make them end in an uninterrupted way, as if the vocalist wanted to make a pause.

When the user pauses the video the music mix is replaced by a pad playing the same section and in perfect sync with the mix, once again allowing for a seamless transition. We also play a transition sound designed for the visual transition effect that makes the transition even more seamless.

At some points in the video the music track loops until the viewer has completed a task, e.g. at 2:04 in the screen capture above. When the viewer is done with the interaction we wait for the next bar, play a little drum fill and move to the next musical section, all in perfect sync.

Nissan Infiniti — Deja View

Deja View is a more “traditional” interactive film. However, there was nothing “traditional” with the size and scope of the project: over 35 minutes of mastered film, 300 video clips, 200 audio clips, a custom built phone system, speech control, NLP, fake AI etc.

From an interaction standpoint it is also special because of the way the user controls the experience. There is no direct interaction with the film, instead the story is changed by what the user says in the phone conversations.

That did not reduce the requirements of responsiveness. At one point the last words in a conversation on the phone affected which one of six available clips that should immediately play. Seamlessly.

The technical solution we developed allowed for this complexity in a highly effective and responsive way. I have described this more in detail here.

But this is about seamlessness! So let’s have a look at the timeline:

One of the benefits with our technical solution is that it allows very responsive and precise control over even very short video clips. At 1:27 in the screen capture one character mentions a dry cleaners. It is one of five alternative clips that could be played at that position, you can see the clip in the chart as a dynamic edit. We use this responsiveness all through the experience.

This level of fine grained control over the video clips makes seamlessness in the sound track more of a challenge. We used the sound track on the video file itself for dialog and spot sounds that would not be interrupted by the dynamic selection of clips.

In some cases this was not feasible, e.g. dialog or spot sounds that overlaps an edit point between two dynamically edited clips. In those cases we made the sounds separate and triggered them in sync with the video.

The ambiences were separate sounds controlled dynamically depending on clip. This way the ambience helps to bind clips together that are selected dynamically, making sure that all dynamic clip transitions are seamless. The music score also had the same function, to bind dynamically selected clips together, covering all the eidt points that otherwise had become obvious.

That was about it! And if you think this wasn’t too impressive, let me remind you of the video flow chart:

And then the flow charts for the interactive phone calls on top of that, but I think there are enough timelines and charts for a day.

To summarize

To make productions seamless is all about the user experience. It reduces friction, removes obstacles and creates a premium feeling. It pays off to care about the details!

This article was originally published at Dinahmoe Battery as a part of Battery002: Smart content, shoppable videos and Bandersnatch!

Interactive film under the hood: the Art of Seamlessness was originally published in Dinahmoe on Medium, where people are continuing the conversation by highlighting and responding to this story.

…my phone told me this morning. It could not have been more true and a better timing

.This morning my iPhone showed me a message that I have not seen before.

“Battery is sufficiently charged”

It could not have been more true and better timing.

Today we are going live with Dinahmoe Battery, battery.dinahmoe.com. If you want to be boring you could say it is a blog, but who wants to be that guy?

So it is more of a “magazine” with “issues” containing “articles” sharing the same “theme”. It will be in sync with Dinahmoe’s Newsletter, which always have had a theme of sorts.

I feel super excited about this format! This way I can explore topics more in depth while giving you the reader the option to pick and choose what interests you the most. The articles here on Medium has felt like islands in an ocean (lot only my stuff). It will also force me to complete some series of articles that is hanging, e.g. How to write bug free code, Design Doing, Value Marketing and more.

But it is not only where I am going, it is also what I leave behind, medium.com. Medium is on the wrong path, at least for me.
Here is a selection:

• since I want to reach as many as possible I want my articles to be free, but they no longer curate non premium articles. There is no way to reach a wider audience than I already have. The main reason of being on a platform is gone.
• since only premium articles are curated, that means that close to all articles presented on medium.com are premium. If you visit the site as a non member, there is nothing to read which means there is no reason to visit the site at all. How are they planning to grow the audience?
• if a user clicks on a Medium link to a premium article they lose one of three free reads without any warning. This is the case for links both on medium.com and links anywhere else. Clicking a medium link is a mistake that users only do once, after that they will never click on a medium link unless they are already members. How are they planning to grow?
• There is a way for the author to share free links to premium articles, and apparently links shared on Twitter is not affecting the free reads, but have you unintentionally lost a free read once then you will be reluctant to click any Medium link.

There are so much more, but I think the above is more than enough.

I will not stop publishing here on Medium, but will be less consistent. Well, consistency, writing is just for fun and heavily affected by other projects, so as consistent as it can be.

So please come join us at battery.dinahmoe.com, it will be fun! New content will be added every sometimes, our newsletter will keep you up to date.

Cheers, Johan

Battery is sufficiently charged was originally published in Dinahmoe on Medium, where people are continuing the conversation by highlighting and responding to this story.

Bandersnatch was a one-off, here’s how to make interactive film mainstream

This article was originally published at Dinahmoe Battery as a part of Battery002: Smart content, shoppable videos and Bandersnatch!

Netflix’ Bandersnatch episode of Black Mirror has made for example Amazon, YouTube, Walmart, BBC bet on interactive content. Netflix are doubling down with more interactive content and I would assume there is a lot of activity behind the scenes. Unfortunately…

Bandersnatch is NOT the future of interactive entertainment and
choose-your-own-adventure is NOT the right way to sell it to the audience.

Don’t get me wrong. Bandersnatch is truly impressive! But it is a one-off. Imagine the drop in viewers if the whole season was done the same way.
Read the user reviews at imdb.com and you will know I am right.

As TechCrunch puts it:

This collective rush to interactive, personalized programming may lead some to believe this is indeed the next big thing in media and entertainment. But the reality is that these shows are costly to produce and difficult to scale compared with traditional programming. Plus, viewer reaction has been mixed so far.

But do not despair dear reader, interactive film has a very bright future
if done right, here is how.

It is viewers, not gamers

Interactive film is not new, the game world have been at it for the last 30 years. What is new is the medium, a mainstream TV and movie platform, and a totally new audience with other expectations on a viewing experience.

Bandersnatch could very well prove to be the starting point to bring interactive film to a mass market. But for this to happen we have to stop thinking about the audience as “gamers”. Netflix’s audience is not looking for a game experience, they are there to get a good story, they are “viewers”.

Key takeaway: viewers are as mainstream as they come, go get them!

Focus on the 80%

Whatever we do we cannot write a script that everybody loves. We should not even try. As a rule of thumb we should aim for:

• making the experience amazing for the 80%,
• making it good enough for the next 15%,
• and don’t spend a minute on the last 5%.

Sounds reasonable? Getting a five star review from 80% of the viewers would be a massive success! We will get back to the 5% shortly.

Key takeaway: focus on making the 80% happy, everything else is a waste of time 😜.

There is only one story

Interactive film is often described as non-linear. What most forget is that the human experience is always linear. The interactive flow can be the most advanced and intricate ever but the viewer will still just experience a linear flow where one scene follows another.

It doesn’t matter if there are “millions of permutations and five different endings”, in the end the viewer has only experienced exactly one (1) permutation with one (1) ending.

If we were able to predict exactly what choices the viewer would do then we only had to produce those specific scenes and skip the rest, the viewer would still think that they made the choices and affected how the story evolved. In theory we could cut film production time for e.g. Bandersnatch to a third without changing the viewer experience one bit. In theory.

Key takeaway: always start with the ideal flow, the path we want the user to take through the experience. Forget the permutations, the bells and whistles for now and focus on the story.

Permutations for whom?

The only way to know that the film could be different is to play it again. You would really have to love the first playthrough to even consider it, then you have to be curious enough about the alternatives to invest another couple of hours. Congratulations, you are one of the 5%!

Remember that we target the TV and movie audience. How many would watch a linear film again immediately after the first view? 0.1%?

At the moment the number of permutations and endings has a PR value, but if interactive film catches on then that value will fade pretty quickly. Then we are back at judging things the way we always do, whether it was a great experience or not.

For those playing it again the story and ending better be different and at least as good and rewarding as the first time, otherwise the word of mouth will be a bad one. As if making one great film wasn’t enough of a challenge.

Key takeaway: think long and hard about why you want all the different paths and variations. Is it really adding to the experience? Or is it to show off your interactive skills for your peers?

Who wants different endings anyway?

It is movie night, what do we want to watch? Let’s watch a love movie!

In most cases we choose a movie that suits our mood, and when we have seen it we want to feel that we got what we wanted. If it is a love story then we want the characters to live happily ever after, if it is a tragedy, not so much, if it is a drama then we want the character to sort out something or evolve as a human being, if it is a hero story then we want them to save the world.

Most movies don’t change genre in the middle. They end pretty much as we expected, and we are fine with that! That was the reason we picked it in the first place. We want a single ending, the best!!!

Key takeaway: creating a love story with an alternative ending where they die unhappy is a sure way to piss off half of the audience.

The illusion of choice

Netflix and others call Bandersnatch a choose-your-own-adventure. That is not only a trademark infringement that brings you a lawsuit, but also quite misleading.

The term implies that you get a couple of characters, an initial setting, some goals and then it is up to you to do whatever you want, i.e. choose your own adventure (duh!). But this is not true even in the gaming world, and definitely not the case for live action film. If you think you can choose your own adventure you will be wildly disappointed.

Luckily the 80% of Netflix viewers do not have that expectation. Permutations and alternative endings are not a priority, a good story is. This allows us to shift resources from the 5% to the 80%. It not only saves money and time, but also opens up for true creativity and innovation in interactive storytelling.

Key takeaway: the audience wants a great story and isn’t longing for an open world where they can do (almost) everything, if they were they would be playing Red Dead Redemption right now.

All great stories are the same (almost)

A picture says more than a thousand words, but please check out Christina Wodtke’s post The Shape of Story, there is more both pics and words, a great read! My favorite part is #5: “Resolution is boring, keep it short”.

The part in the middle, the struggle, is where the action is! This is where we should develop the interactive storyline. The viewer wants to realize the outcome, be the one that makes it happen, make the love couple meet, put the hero to the test, kill the bad guys.

Although not all decisions go as the viewer wants, every step will eventually lead to the resolution (which is boring).

Key takeaway: everything interesting happens before the end.

The goal is the compass

Back to the interactive love story. The obvious ending is that the characters find each other and live happily ever after. There could be some variations to that but anything totally different would surely be a disappointment. For each genre it is pretty clear what the goal is and what would be a rewarding ending.

Agreeing with the viewer about what the goal is might sound like a spoiler, but it is not about giving away all the details, it is about giving the direction. The goal works as a compass, helping the viewer to make meaningful choices. The user will feel motivated to make the choices that they think leads to the goal. In the end the user will feel like they are the real reason for the characters to have found each other, have succeeded, especially if it was hard to get there.

Key takeaway: make sure that the viewer and the film share the same goal. That makes all interaction filled with purpose, and reaching the goal a personal achievement.

Fuck love, what about insanity, cruelty, death?

To tell a story of utter hopelessness and despair we need to make the choices meaningful. By doing the “right” choices for the character we could actually feel stronger for them than if the director just put them into the situation.
We did this, we are responsible for the outcome.

Why not have one ending that is positive? It seems like a nice thing to do to the character. The answer is expectations. When you choose to engage you do that because you expect an outcome, is it love, hero, tragedy, mystery, mayhem. Stay true to the story arc.

Key takeaway: meaningful choices is what makes the journey worth the effort.

Do the twist!

A single ending doesn’t necessarily mean that it is one that the viewer expects, as long as it doesn’t break their expectations in a disappointing way. So don’t hesitate, do the twist!

Best Plot Twists Ever - IMDb

Key takeaway: just do the twist!

Summary

For interactive film to become mainstream we must look at it from a viewer’s perspective, not a gamer. This will not only improve the user experience for the 80% dramatically, it will reduce production costs to a level where it is actually feasible long term to produce.

Looking forward to get going!

This article was originally published at Dinahmoe Battery as a part of Battery002: Smart content, shoppable videos and Bandersnatch!

Before you go…

Agree or disagree, ask questions or suggest improvements,
the comments section below is all yours!

To get in touch, please connect with me on LinkedIn.

If you want to get notified about new articles and projects, please subscribe to Dinahmoe’s newsletter by clicking here

How to write an amazing interactive film script (while cutting production costs in half) was originally published in Dinahmoe on Medium, where people are continuing the conversation by highlighting and responding to this story.

Not the label but the stuff we find when we scratch the surface

Words are weird. Take any word, focus on it, think about how it sounds, looks, how it is spelled, and of course what it means. After a while it starts to dissolve and questions arise: is it really spelled that way? And it looks and sounds kind of corny doesn’t it? Even the meaning becomes slightly obscure, ambiguous. Why haven’t I noticed this before?

I have been thinking about the word “obvious” lately. And sure enough it quickly lost its simplistic meaning and exploded into a world of opportunities and challenges.

It turns out that under the hood there are treasures to be found. What seemed “obvious” at a glance turned out to be brilliant or important when you stopped for a second and reflected.

So here is my defense for the “obvious”, not the word itself, the label, but the stuff you find when you scratch the surface.

The “obvious” is in the eye of the beholder

Before the dissolution of meaning begins, let’s get the benchmark definition from Oxford Dictionaries:

Easily perceived or understood; clear, self-evident, or apparent.

and derogatory

Predictable and lacking in subtlety.

It is “obviously” a word open for interpretation. It is not like “red” or “herring” where it seems likely that we can reach a consensus. Each and every word in the definition oozes subjectivity. “Easily perceived” by who? “Easily understood” by who? You get the idea.

The weird thing is that despite its inherent subjectivity it is used to make something seem more objective. Labelling something as “obvious” relieves it from scrutiny, no need to think about it anymore, it almost becomes a fact. Compare that to something that is “not obvious”, which forces you to think.

How do we know?

So when something seems obvious to us, what does that actually mean?
Well, there are two main options:

It seems “obvious” because we are brilliant! Yay!

If something is truly obvious for us, it could be because we are brilliant and that we immediately recognize its “obviousness”. Or

It seems “obvious” because we are not [brilliant]…

The other person said something truly ground breaking but for us it sounded more like they were “just stating the obvious”. Or even worse, it was just a load of crap but it sounded true enough and we swallowed it, hook, line and sinker. In both cases, the obvious wasn’t that obvious at all.

Can we be 100% sure which one it is?

Of course not! We are only humans, we only know what we know and there is a bunch of biases and social factors that severely obscures our thinking.

Luckily there is a simple way out of the problem, we will come to that shortly.

Between brilliance and ignorance

Let’s switch the roles, that we are saying something that we think is truly ground breaking and the other person responds with that it is “obvious”.

I am stating the obvious all the time. Or so I am told. Well, not all the time, sometimes I just get a blank stare back. I can never be sure, am I just silly or does the other person not “get it”?

But silliness is far from the only reason that we or someone else choose to discard something as “obvious”. Here is an incomplete list of real world examples:

It is obvious because I am way smarter than you!
Powerplay! Whatever you say I have already preemptively thought, I don’t even have to listen.

It is obvious because I don’t have a clue what you are talking about
The confused look in my eyes is just a distraction, let’s move on to something else.

It is obvious because I just don’t want to talk about it
I resist without informing you. Changing my opinions is just not my thang.

It is obvious because I own you, biyatch!
Powerplay again, but no need for intellectual superiority, just go for the nuts! Bam!

It is obvious because I wish I would have come up with that first
Simple solutions feel obvious when said out loud. I will not give you credit for something that is that obvious! The fact that you thought about it and I didn’t doesn’t count!

It is obvious, but…
Whatever you say I will “but” you til the end of time. “You need to change your diet”. Yes, but… “You need to break up with your abusive partner”. Yes, but… “You need to quit crack cocaine”. Yes, but… Ha! You loose!

It is obviously true for anyone else
We are all unique snowflakes! The problem that I have is so complex and uniquely mine that the solution must be equally complex and personal. I refuse that silly, obvious solution that you offer me. I actually feel offended!

Personally I have learned to never accept the “obvious” as an answer. I just repeat my silly statements until I either cut through the fog or realize that I was actually the unlucky one.

The case for the obvious

Why do I care so much about the obvious? Because in many cases it isn’t. Discarding something as obvious is a lost opportunity. The moment something goes from the obscure and becomes truly obvious for us is pure joy and true learning, a Eureka moment!

Labelling something as obvious makes us stop thinking. We assume that we have already understood everything there is to understand about this object/question/person. It is an intellectual and creative dead end.

The “obvious” makes it impossible to notice the finer details, in the same way as the pounding music at a club makes any intellectual exchange impossible. We are left judging something from how it looks on the surface.

The “obvious” is just a starting point!

Regardless if we are the ones that think something is obvious, or that someone thinks that we are “just stating the obvious”, the respons should be the same: ask.

Let’s not assume that we understand.

Very little in life is obvious when you scratch the surface, especially when it comes to human beings. We all have different wiring, experiences and viewpoints. By asking we open up for value on so many levels.

Maybe what the person said was truly brilliant and we did not get it, maybe it actually was obvious but the reasons for saying it or the conclusions they draw from it wasn’t.

And maybe we get a new friend by not coming off as an arrogant asshole 😜.

Before you go…

I write to find likeminded and would love to hear from you!
Agree or disagree, ask questions or suggest improvements,
the comments section below is all yours!

To get in touch, please connect with me on LinkedIn.

If you want to get notified about new articles and projects, please subscribe to Dinahmoe’s newsletter by clicking here

In defense of the “obvious” was originally published in Dinahmoe on Medium, where people are continuing the conversation by highlighting and responding to this story.

Bye bye Siri/Alexa/Duplex! Hello Beautiful!

This article was originally published at Dinahmoe Battery as a part of Battery001: An issue dedicated to Voice!

This is the second part about the voice assistant of the future. If you have not read part 1 then you probably should. Or not, depending where you come from. Anyway here it is!

The future of voice assistants: a personal digital clone??

A conversation with my future assistant

If Duplex can mimic a human conversation, why can’t we make our stupid voice assistants truly useful? Let’s extrapolate! Here is a transcript of a conversation with my digital assistant from a not very distant future, anonymized of course:

Hey [beep], let’s do some catching up!
Sure! How can I be of help?
Remember the conference in Portland?
Yes?
There were two guys I met at one of the evenings, I think they were from Canada, do we have their contact info?
Was this on Tuesday or Wednesday night?
Let me think… [pause] …which was the last day of the conference?
It was Friday.
Then it must have been Tuesday.
There were two…
[interrupts] No wait, it was actually Wednesday!
I have a note about two “guys” which you discussed opportunities in the media space, but none of them seem to be Canadian.
Never mind, that’s them! What was their names again?
[beep] and [beep]
Cool! Could you please send a standard mail: nice meeting you blah blah let’s keep in touch, or something like that?
I found their emails at [beep.com]. Should I send separate email?
What do you mean?
Do you want me to send separate mail to both, or include them in one?
Ahh, good point, let’s do them separate.
Ok. Do you want me to read it before sending?
Nah, I trust you!
Ok, sent!
Thanks!
You’re welcome!

That is an assistant that fits the description Conversational AI. So is it intelligent? Do we have to wait for Artificial General Intelligence before this can become reality?

Not at all! It is just math 😜! Here is what might be happening under the hood.

Hey [beep], let’s do some catching up!

Activated! Intent: “to catch up”, context unknown, ask for clarification

Sure! How can I be of help?
Remember the conference in Portland?

Context: References to Conference Portland April 14–17 found in several applications, setting as context, verify

Yes?
There were two guys I met at one of the evenings, I think they were from Canada, do we have their contact info?

Searching notes from the conference for conversations with two persons, found two conversations Tuesday and one Wednesday, a wide and shallow person search finds none related to Canada, ignoring this since “think” indicates that country is not a required entity. Ask for clarification on day.

Was this on Tuesday or Wednesday night?
Let me think… [pause]

“Let me think” equals a pause

…which was the last day of the conference?

Retrieving conference schedule.
Potential secondary contexts: conference schedule / Friday

It was Friday.
Then it must have been Tuesday.

Two partial hits, ask for clarification

There were two…
[interrupts] No wait, it was actually Wednesday!

Retrieve one partial hit, deliver a summary of meeting notes, comment on nationality

I have a note about two “guys” which you discussed opportunities in the media space, but none of them seem to be Canadian.
Never mind, that’s them! What was their names again?

Deliver participants’ names, drop secondary context

[beep] and [beep]
Cool! Could you please send a standard mail: nice meeting you blah blah let’s keep in touch, or something like that?

Search for email addresses at [beep.com], ask for complementary information

I found their emails at [beep.com]. Should I send separate emails?
What do you mean?

Question not understood by user, rephrase and clarify

Do you want me to send separate mail to both persons, or include them in one?
Ahh, good point, let’s do them separate.

Populate mail template. Ask for send approval

Ok. Do you want me to read it before sending?
Nah, I trust you!

Send mail

Ok, sent!
Thanks!
You’re welcome!

Done! That wasn’t too hard, was it 😜? Obviously not an implementation specification, but no challenges that are not possible to overcome. There is no need for any real intelligence (whatever that is) to execute any of the underlying functions. It requires a tight integration with all the functionality, notes, search etc, but that is just work.

And for the understanding and the speech generation, we already know that it is feasible through Duplex. So we’re almost there then? Well…

Data, we need more data!

As the Google team states Duplex has to be “deeply” trained in a “closed domain” that is “narrow enough to explore extensively”. It has to cover the absolute majority of use cases to be useful. For this it needs a lot of data, in this case transcripts of calls to restaurants and hair salons. A lot of them.

To realize the assistant of the future in the same way as Duplex does we must multiply the amount of data needed with the number of domains that we want the assistant to handle. “A lot” times “a lot” equals “will not happen”.

Do the math. It is just unmanageable.

…unless the assistant can learn by itself

Maybe you have heard about AlphaGo that in 2016 won over the world champion in Go. Things have moved on since then and the latest iteration AlphaGo Zero was able to beat all previous versions after just 40 days of training. Without any human involvement.

In the case of Go it is quite easy to know if the training works or not. Did it win the game? Then the black box did something good, if not… That makes it possible to let it play against different versions of itself and learn.

Compare that to a conversation. If you think it is about winning then you probably don’t have many friends. In many cases there is no clear way to say if a decision was good or bad, even the outcome can be impossible to judge other than on a subjective, human level.

Duplex solves this with what they call real-time supervised training. It is essentially humans that interact with Duplex and reinforces the “right” decisions, the human “teaches” Duplex how it should react. This reduces ambiguity and the time needed for training.

So who is going to teach our assistant right from wrong?

Customization vs. generalization

A customized solution is created to solve a specific problem, and will always generate a higher quality result than a general solution. The problem is the scaling, there is no way to develop customized solutions for every use case.

Duplex is a customized system, it does what it is supposed to do amazingly well, but will utterly fail on everything else. Our existing voice assistants on the other hand are generalized, they are supposed to handle many different tasks, which they do in a generally crappy way.

There is something oddly backwards with the process of training our black boxes. First we take data from as large number of individuals as possible, classify and analyze a specific aspect of their behavior, use this data to train the assistant which calibrates a statistical application to be “good enough” to be applied to anybody, but not really be perfect for anyone.

The future assistant should not be general, trying to do everything for everyone. It should be customized for you, it should be trained to solve the tasks that are unique to you.

It should be YOUR assistant.

The customization is you!

A large data set is needed to set an average ground truth because of the individual variation. If the data set is you, then the variation will approach zero. Your favorite color is always blue, your favorite food is sushi but sometimes a beer and burger is the obvious choice, you enjoy talking with the same people, you complain about the same things, choose the same clothes except for rainy days.

In short, you are predictable.

If the data set is just you, then the amount of data needed is infinitely smaller, like 8 billion times smaller. Maybe small enough for your assistant to live on your mobile phone. It is no longer a general assistant shared by everyone, it is your personal assistant. To make the distinction even clearer:

it is your private assistant.

Private meaning that only you have access to data and functionality. Seriously. As not in the cloud.

So how does the Assistant of the Future work?

The assistant collects as much data as it can get from our activity in both the physical and digital world. Everything we say or write, the responses we get, every meal we eat and flu we have. Everything. The more data points the better.

From that unstructured data it uses unsupervised learning to identify correlations and patterns that will become useful in decision making in the future.

The real “training” and “learning” will happen in the daily interaction with the assistant. In the same way the Duplex team uses real-time supervised training our behavior and direct guidance will “educate” the assistant and reinforce the right decisions.

The assistant will have to be “curious”. Whenever the situation is unfamiliar or actions cannot be performed it should ask questions. “What are you doing now?”, “Can you please rephrase?”, “Why is that person screaming?”.

You are the teacher!

Not only that…

The Assistant of the Future is … you

So where will all this lead us? It looks to me that it all points in the same direction. So let’s extrapolate the future!

Your future assistant consists of every experience you have had, it has been around since your birth, it learned everything together with you, how to walk and talk. It increased your learning speed and capacity by providing instant access to infinite knowledge and processing power. It has always been there for you, through all the ups and downs that a physical body is blessed with.

Voice is no longer needed. It is possible already today to reconstruct images based on brain waves so most of the data collection will be retrieved directly from the brain, detecting intent before there are words. And it will work in the other direction too, the assistant can imprint patterns in your brain directly. The physical and digital merges.

The assistant could live its own parallel life, be active far beyond booking haircuts and dinners. Then during your sleep it integrates its experiences and memories with yours, they become indistinguishable from your own.

The assistant would from all practical and legal aspects be you.

You have since long forgotten that your assistant is just a statistical application, a complex but deterministic system of probabilities.

And when we get there, what will we talk about?

This article was originally published at Dinahmoe Battery as a part of Battery001: An issue dedicated to Voice!

Before you go…

I write to find likeminded and would love to hear from you!
Agree or disagree, ask questions or suggest improvements,
the comments section below is all yours!

To get in touch, please connect with me on LinkedIn.

If you want to get notified about new articles and projects, please subscribe to Dinahmoe’s newsletter by clicking here

The future of voice assistants: a personal digital clone?? Part 2 was originally published in Dinahmoe on Medium, where people are continuing the conversation by highlighting and responding to this story.

What Google Duplex tells us about the future of voice

This article was originally published at Dinahmoe Battery as a part of Battery001: An issue dedicated to Voice!

Recently I published an article about what is called Conversational AI. The term refers to our beloved voice assistants: Alexa, Siri, Google Assistant, none of them famous for their conversational skills.

From the article:

We will get better and better at mimicking human behavior and suddenly we have created something that passes not only the Turing test but also get our emotional and intellectual approval to become a part of the conversation.

And when we get there, what will we talk about?

It turns out to be haircuts and dinner bookings!!

The demo of Duplex at Google I/O 2018 left me with two very contrasting feelings. The first was, wow, super impressive, the second was, is this really solving the right problem?

I get the idea that some businesses don’t have any online booking. Another solution to that problem would be to give them that for free, but whatevs.

Now Google are rolling out the Duplex functionality and we get the taste of how it will be to use. This is how to make a reservation:

and here is what happens on the other side:

Two videos and again two different feelings. The last one, super impressed! The first, well…

Making Google Assistant do the booking took 75 seconds, felt awkward and quite far from a natural conversation, requires 8 screenshots and a lot of text to describe. The actual booking was very smooth and took 35 seconds.

As a tech demo it is very impressive but there are far better use cases. Swapping roles, the human is calling a business and Duplex books appointments seems like a simple change which would solve real world problems for a lot of small and medium sized companies. Then replacing all those extremely annoying IVR phone systems, then replacing support, then sales, then…

While self driving cars are not around the corner as far as I can see, this will start to replace humans pretty soon. The implications are profound.

Looking under the Duplex hood (the layman version)

There are three components involved in the process: converting the user’s speech to text, analyzing that text to find a proper response, and generating speech from that response.

The speech recognition part seems to work flawlessly! The generated speech blows my mind! It is expressive and has a natural flow and intonation. But it is the component in the middle where the real magic happens: that takes what the user said and finds a proper response.

Duplex works autonomously, i.e. without any human involvement, and has been trained for this task, and this task only. From the Duplex post on Gooogle’s AI blog:

One of the key research insights was to constrain Duplex to closed domains, which are narrow enough to explore extensively. Duplex can only carry out natural conversations after being deeply trained in such domains. It cannot carry out general conversations.

Booking a restaurant or a haircut are both clearly defined in scope, the number of paths the conversation can take and the number of data types, e.g. how many, what time, is limited. The scope is much bigger than it seems, but not infinite. So all in all, it seems doable, and obviously is, since they did it!

But what does “training” mean?

I never thought I would write about artificial neural networks but here we go!

An artificial neural network is like a magical black box (yes, really!), each with its own specific function. As an example, let’s pick one with a soft spot for cats. Show an image of a cat to the black box and it confirms, this is a cat ❤️. Show a hot rod racer and the black box gives the verdict: no cat 👎.

In order to do its magic the black box has to be “trained”. In the beginning it knows nothing, we show a cat (obviously not a real cat, just the pixels!), it cannot even spell cat but gives an answer anyway. Each time it gives an answer it gets feedback on whether it was right or wrong. This way it “learns” and gradually gets better at recognizing cats. At some point we decide that it does the job good enough and we end the “training”.

“Training” and “learning” sounds, well, human, like there is some living thing in the black box. There isn’t, it really seems magical but it is “just” math, a statistical application, and it doesn’t “learn”, it calibrates itself based on feedback on its performance.

The really weird thing is that we do not fully understand what is happening inside the box. Put in other words:

Artificial neural networks are simply deterministic algorithms that statistically approximate functions. It’s just not possible to exactly say which function they approximate.

We have created the application but we cannot peek inside.
It performs its magic anyhow. A magical cat black box.

“Training” Duplex

Over to Duplex. Instead of showing cat and hot rod car images, they use real conversations when humans are booking a dinner or haircut. The black box is trained in a similar way but now it is text in and text out. If the user says a specific sequence of words, then the black box should reply with another specific sequence of words.

To begin with the responses are totally off, but by comparing them with the answer in the real conversation the black box “learns”. At some point it has become good enough, a magical haircut booking black box, hello Duplex!!

This is obviously not the whole truth, some of it I knowingly omit, most I am blissfully unaware of. But that is how deep (!) we get here and now.

So is Duplex the future?

In one way it is, it shows that it is possible to mimic human conversations in a way that is almost indistiguishable from the real thing. Duplex has so far a very limited scope, haircuts and dinners, but the method is applicable on many use cases. I am sure we will see a large number of new applications in the coming year.

So is the assistant of the future just the sum of all these applications? The data and processing needs will approach infinity, should we just hope that Moore’s law applies?

There are other concerns, such as privacy. The more useful an assistant becomes the more personal it gets. Do we want Google, Apple, Amazon to listen in to our most private conversations and apply their generalized models that has been “improved” by supervised training?

This is the topic of part 2, don’t miss out, it will get super futuristic!

The future of voice assistants: a personal digital clone?? Part 2

This article was originally published at Dinahmoe Battery as a part of Battery001: An issue dedicated to Voice!

Before you go…

I write to find likeminded and would love to hear from you!
Agree or disagree, ask questions or suggest improvements,
the comments section below is all yours!

To get in touch, please connect with me on LinkedIn.

If you want to get notified about new articles and projects, please subscribe to Dinahmoe’s newsletter by clicking here

The future of voice assistants: a personal digital clone?? Part 1 was originally published in Dinahmoe on Medium, where people are continuing the conversation by highlighting and responding to this story.

Talking with machines that talk back

This article was originally published at Dinahmoe Battery as a part of Battery001: An issue dedicated to Voice!

Nothing beats a good conversation, to explore topics of shared interest, the exchange of ideas and experiences. And as much as I love that, I loathe the opposite, where no topics trigger any interest and all questions just drops dead.

That is how it feels to talk to Siri, Apple’s digital assistant. Nothing interests her, most things she doesn’t know anything about, and when she does her response is usually not that exciting. Her colleagues seem to be slightly ahead, but as conversation partners they all suck.

Our dead boring assistants are examples of what is called Conversational UI and Conversational AI (oh the irony!). There seems to be some work to be done here. What is actually required to get the conversation going?

What is a conversation?

I’ll let Oxford Dictionaries chime in:

A talk, especially an informal one, between two or more people, in which news and ideas are exchanged.

Makes sense if we ignore the “people” part. Here is a typical Alexa user interaction:

Alexa, what time is it?
The time is twenty to five

Practical, but a conversation? I doubt that anyone would describe this “I just had a conversation with Alexa about time”. Which on the other had sounds really intriguing, what did she say?

How about this then:

Alexa, do you know what time is it?
It is twenty to five. Why do you ask?
…?!

Whoa, a conversation starter! That’s much better! I initiated the conversation, I wanted to learn the time, Alexa obliged but also asked a follow up question. As if she had a will of her own…

A real conversation is a dance

Human conversations are genuinely fascinating when you start picking them apart. Take this short excerpt from a phone call between two friends that haven’t talked for a while, catching up:

B: Um, have you talked to… what’s her name…Sheila?
A: Sheila who?
B: Sheila…what’s her name…Nadine and them’s sister
A: White
B: You see her still?
A: The twins getting ready to graduate girl
B: Really?
A: Yes. I know we getting old

There is so much going on here, it is like a dance, they are taking turns, mixing questions, answers, prompts, implicit knowledge, assumptions, clarifications, acknowledgements, all within 8 lines. A dance.

The example is from this paper which provides a detailed breakdown and analysis of the same lines and more, great stuff!

The state of virtual conversations

The breadth, depth and complexity of human conversation is immense and is very hard to replicate in a virtual personality. All behavior has to be replicated in a system of rules, i.e. logic. Whether that is even possible is a topic for a nice, long, late night philosophical conversation, probably not with a machine. Regardless of your point of view the challenge is massive.

Most conversations don’t even have a clear goal, they move from topic to topic, and at some point both parts decide that the conversation is over, that it fulfilled its purpose. Good luck with figuring that one out.

There has been some amazing progress lately. Google Duplex’ ability to mimic a human is super impressive but it is strictly limited to a specific domain. Try saying something outside script, like offering a harcut when it asks for a dinner and see what happens.

Chatbots are generally much more conversational than assistants. The four time winner of the Loebner prize Mitsuku is able to respond meaningful to almost everything you write, an incredible achievement! Still, it is hard not to notice that I, the human, am driving the interaction. Mitsuku has no will of her own.

Intention as a driver

Intentions or intents is a core concept in platforms for creating conversations. It assumes that the user wants to do something, e.g. order a beverage, more specifically a double espresso macchiato. Ordering a beverage is then defined as an intent, and each configuration option — size, type, add-on — is defined as entities.

If you are able to cram all this into a single sentence then you’ll just get your coffee. If you leave some entity out, then the bot asks for the missing entities. And that is as fun as it gets.

Machines can have intentions too

The terminology of conversational platforms is built on that it is the user that has intent, that drives the interaction. Google Duplex puts this on its head. Now it is the machine that seems to drive the conversation, that has intent.
The human fulfills the need of the machine…

However, it is an illusion. Duplex is still just a much more fancy way to fill in the blanks, and when they are filled then the interaction is over.
No conversation.

It takes two to tango

Think about reconstructing the human-to-human conversation quoted above in a chatbot. The driving part in the conversation is constantly switching, intents are running in parallel, questions are answered by the part that asked them. A beautiful mess.

In order for a conversation to happen it is not enough that one of the parts is driving interaction, we need at least two parts with a will of their own, with their own goals and intentions.

Fake it till you make it

I think we have to wait a bit more for the truly conversational assistant. But I am a fan of fake it till you make it. We will get better and better at mimicking human behavior and at some point we will surprise ourselves.

Before getting to some conclusion, we have had the fortune to work with several projects where the goal of the conversation wasn’t a utility but to give the user the impression of talking to a real person. Below are some examples of our experiments into the fake conversational space.

Nissan Infiniti — Deja View

Deja View is an interactive film where you interact with the characters in the film by talking to them. During the experience the characters will pick up their phones and call your real world phone. What you say to them during those calls determines what will happen next and how the story evolves.

Here is how it works. The user arrives at the site, calls a phone number on the screen and gets a code, enters this in the browser and the experience starts. The first scene in the film shows the two main characters sitting in a car, waking up with no memory about who they are or where they come from. Looking at their mobile phones they discover that someone has called them several times, the number on the screen is YOUR phone number. They decide to call the number and your mobile rings in real life.

Here is a screen capture of a conversation.

The core concept of Deja View was to break the fourth wall. This was achieved in several ways:

• the characters in the film that is shown on your computer called you on your mobile phone. The physical separation makes it feel a little magical even if you can figure out how it is done.
• talking on the mobile is something associated with “real” humans, not with characters in a film.
• what you say outside the film affects how the story in the film evolves. The characters even reference what you said during the call in the film, using your own words.
• every conversation is based on what the user had said earlier, giving the feeling that the characters remembered you.

Everything sums up and the experience of talking to the characters becomes much more believable and emotionally engaging.

In the call the characters are driving the conversation, i.e. have intent, which makes it easier to make sure the conversation leads to something worth experiencing for the user, and also gives us enough data to drive the story in a meaningful way.

There is a lot to say about this project, it was exceptional in many ways. I have described the project on a general level here.

Slaves for Santánico

Slaves for Santánico was created to generate awareness and excitement for the premiere of From Dusk Till Dawn: The Series, a TV show based on Robert Rodriguez’ film of the same name. We were invited to bring the show’s most infamous character, Santánico Pandemonium, to life as a virtual character.

The channel for the conversation was “Santánico’s Party Line”. Talking urinal cakes and burlesque posters were placed in dive bars promoting Santánico’s 1–800 number. When they called they had a conversation with Santánico, where she demanded they prove their devotion. She responded to callers based on their originality, creativity and depravity. Her favorites were immortalized on the Tumblr: SlavesForSantanico.com.

We developed the conversation logic that made her come to life, and bring out the true depraved creativity of her fans. We set up the phone system including a custom phone server, speech recognition, natural language processing, dialogue logic etc.

Compared to an online campaign the activation in the physical world made the call feel more real. But we wanted to take the experience of the conversation to the next level. We added background noise and sound effects of the room where Santánico took your call, disturbances to the phone line etc. This served two functions. First, it made the call feel like a real phone call, not a fake one. Second, it uses the power of misdirection, making the user slightly disoriented, no longer focusing on whether Santánico is real or not.

In the same way as for Deja View it all sums up. Each detail might not be that important in itself but the sum is larger than its parts, the conversation becomes more real and the user gets a much more immersive experience.

Cisco — Internet of Everything

We and GSPSF invited then Google Chief Technology Advocate Michael T Jones to explain the concept Internet of Everything . “Everything” is a pretty big topic to cover in an interview, so we wanted to give the user the possibility to navigate the interview to what interested them most. An interactive interview! Or as Mr Jones puts it, a real two-way conversation.

In this project the whole experience was in the browser, no 4th wall to break, but to present content in a novel and useful way. The user activated their microphone in their browser and could then both navigate the topics and ask questions. Menu options were available for those not using voice.

To enhance the experience we used the user’s mobile phone as a second screen. This content was not only available during the interview but after, as a list of interesting topics to explore deeper.

While Deja View and Santánico were immersive and emotional experiences, this was more function and utility. Still, using voice made the experience much more personal and human.

Conclusion

We have to wait for a truly conversational assistant a bit more. The data needed for training to get beyond dinners and haircuts quickly becomes unmanageable and it is an infinitely long way to reach the open world of human conversations.

The current tools for creating dialogs are not made for real conversations. The user is the only one driving the conversation and the machine is only filling the slots. Furthermore, in a real conversation, not only does the intent move between the parts, it is constantly in flux. The model of intents and sub-intents are simply not working for creating the organic flow of a conversation.

That doesn’t mean that we should stop trying! We will get better and better at mimicking human behavior and suddenly we have created something that passes not only the Turing test but also get our emotional and intellectual approval to become a part of the conversation.

And when we get there, what will we talk about?

That is the topic of the next article:

The future of voice assistants: a personal digital clone??

This article was originally published at Dinahmoe Battery as a part of Battery001: An issue dedicated to Voice!

Before you go…

I write to find likeminded and would love to hear from you!
Agree or disagree, ask questions or suggest improvements,
the comments section below is all yours!

To get in touch, please connect with me on LinkedIn.

If you want to get notified about new articles and projects, please subscribe to Dinahmoe’s newsletter by clicking here

The state of Conversational AI was originally published in Dinahmoe on Medium, where people are continuing the conversation by highlighting and responding to this story.

Voice is the future — here’s what to do while waiting

This article was originally published at Dinahmoe Battery as a part of Battery001: An issue dedicated to Voice!

Speaking to computers have been a part of sci-fi since forever. HAL 9000 gave us an idea about what to expect both from computer’s conversational skills and their mental disorders. Now voice assistants are in everyone’s hands so this must be the future, right?

If you saw the Google Duplex demo last year then it actually seems like it is finally here, the future! Natural conversation, tone of voice, intonation, impressive! I will cover Duplex in another article, due shortly.

And Google is rolling it out now. In selected cities. English only. For booking restaurants (which I don’t do that often). And haircuts (which I don’t need). And probably a list of other stuff before getting to something that makes sense to me. It seems like at least I have to wait for the future a bit more.

But while waiting there is a lot we can do! Here is my top priorities for building smart and useful voice apps today!

Before starting, let me define what I mean with voice apps. On one hand there are simple one-shot request/response skill/actions, on the other there are the AI assistants that are supposed to handle everything. A voice app is what lives in the huge but still quite unpopulated space in between. The voice equivalent of a chatbot.

Starting with skipping the “obvious”

If we have already decided to create a voice app then we also have figured out the reason why, what specific problems we want to solve, why voice is the natural choice compared to a standard website or app, and more, so I will not bore you with such trivialities 😜.

Some of the advice below might still feel like they fall under the “obvious” label, but if they were that “obvious” then everyone would do them, and that is “obviously” not the case.

Setting expectations, ours and the user’s

Building anything beyond a simple single-shot request/response skill/action can get complicated pretty quickly. That is why we need to have a very clear idea about what the objective of the voice app is and how to achieve just that.

This is also essential from the user’s perspective. An app with a clear focus is easy to explain and easy to use. Make sure that the user understands what the app does and doesn’t do from the very start. Setting the right expectations is the first step to avoid frustration and disappointment, both for the user and ourselves.

First make it work, then make it great

It is easy to get carried away with all the creative opportunities. Put all those amazing ideas in a box, mark it “ideas for later”, start with an MVP and build from there.

Focus on basic functionality, test with real users (see below) and verify that the experience is on the right track before getting too clever. Do the simple solutions, avoid guesses and assumptions about user behavior, let the dialog flow evolve based on real world testing. Never solve a problem before knowing it is real, “just to be safe”, just put a note in the book and let reality be the judge. In proper user testing of course 😜 (see below)

Focus on the 95%

Define what is the most common user behavior and build the flow from there. Focus on giving the best user experience for the absolute majority, not the outliers, there is no way to make everyone happy anyway.

Unless being edgy is the goal of the app, don’t spend any time on designing for all the edge cases, e.g:

“What if the user says “f**k you”?? We need to cover for that too!”

No, we don’t, and actually we shouldn’t. The conversation should be designed to be fun for the 95% that wants to play along with the rules, not the rebels that want to break them. People can come up with more weird shit than we will ever be able to cover for, just trying will make them even more determined.

Let’s just give up from the start, make it boooooring to break the rules, that will get most of the trolls back in line. This way we reduce overall complexity and lower probability for unintentional errors, win-win-win!!

If you still want to go that route, here is how Mitsuku handles provocateurs.

The Curse of the Chatbot Users

Test, test and test

Proper user testing is essential for creating an amazing user experience. Sounds obvious, but it turns out that end user testing on voice platforms is a real pain, especially if you want to support several platforms.
Don’t use that as an excuse!

The more we know about a project, the less useful we are as a test subject. The worst testers are developers (sorry guys!), they know exactly what works and not, so they simply don’t do anything that breaks the experience. A schmuck on the other hand breaks apps within minutes.

Both Alexa and Google Assistant provides tools for testing. However, the tools are not constructed for testing by “real” users. They are developer tools and are a part of the dev environment. The level of damage the aforementioned schmuck could cause is massive.

So to do proper end user tests on Alexa/Google Home the skill/action has to be deployed to the platforms and must be accessed by supported hardware. Don’t let that discourage you, just plan it well and you will be greatly rewarded.

Logs are the real treasure troves!

Logging user interaction is not only for development, but for the live application too. There is so much to learn from real user interaction. Make sure that every interaction is logged in a structured and searchable form so that you can squeeze all the value out of it.

The logs help us catch the cases where the voice app wasn’t able to understand the user’s intent. What did the user say, whether we should add training data or add there a response missing. Or do they deserve to get the fallback option?

The logs will not only help us to fix what is broken, it will also tell us about the user’s expectations, and how they would like to use the app. Keeping an eye on the logs will guide you how the app should evolve over time.

Voice is not text

For someone coming from chatbots the difference looks small, they might even use the same tools. But voice is not text. In speech we use simpler words but longer sentences, so the user input will be quite different.

The responses from the app must be short and to the point. Points in the flow where the user can make selections must be simplified. Buttons are no option. Large number of options and long speech chunks are not good user experiences. KISS!

If you want to cover both voice and text, think voice-first. It is easier to adapt a voice flow to text than the opposite.

A script that feels great in our heads might be totally awkward when read aloud. And the users never say what we assume they will. Test, test and test. And then test.

Context is King

Context is what makes human interactions flow and feel natural. In voice apps context is what makes it possible for the user to handle more complex tasks, such as making selections between multiple options. Keeping track of where the user is in the process makes your voice app feel smart and give meaningful responses. Context awareness rules!

What is the fallback?

“I’m sorry, I don’t know how to help with that yet.”

Occasionally our voice app will not understand what the user says. Maybe the user has a strong accent or doesn’t understand what they are supposed to do. It is not uncommon that apps keep on repeating “I don’t understand” or variations on the same theme until the user gives up in frustration.

Instead, think about why the user gets stuck and what they need to get back on track. An example on how to handle this:

Do you want option A or B?
[mumble mumble]
I did not understand, do you want option A or B?
[blah blah blah]
I am sorry, I still don’t understand. Let me know if you want option A or B, or if you want to get back to the menu.
[doobee doobee do]
Here is the menu…

Choose a 2 or 3 strike strategy, make sure the user understands that their input has not been understood, clarify the options if needed, and if all else fails get out of the gridlock by going back to an earlier step.

Be careful with “personality”

One of the most common advice for making a chatbot great is to give it a personality. For custom voices, see below, that is a part of the package. But if we plan to use the platforms’ standard voices, maybe a personality doesn’t even make sense?

In all cases, make sure your app understands what the user says, otherwise we have a sure way to annoy the hell out of our users. For example responding

“Sorry, dozed off for a second. What were you saying?”

when it doesn’t understand…grrrrrrr. A stupid bot with a “personality”, not a good idea.

Use a custom voice

Personality in a custom voice is a completely different thing. Instead of using the default voices for the platforms you could use a voice actor and record the responses for your app. There are several reasons why this could make perfect sense:

• it makes the app unique and personal, and stand out from the generic Alexa/Google Assistant apps
• it can express any emotion and state of mind, the tone of voice will be made to fit the content and context
• it can be made to fit a brand
• it makes the user experience consistent over platforms, e.g. Alexa, Google Assistant, web based.

There are some downsides though. Where a digital voice can say anything, a pre-recorded voice is limited to what has been recorded (duh!). It increases production costs, requires more planning ahead, might be hard or even impossible to update at a later time.

An option is to create a custom digital voice, but that will not solve the challenges with tone of voice and expressiveness.

Wrapping it up

Scratching the surface as always. Some final thoughts that I did not cover:

• Speech will replace many of our click/touch interactions, but will still benefit from supporting visuals, i.e. smart displays. To solve both, think voice-first.
• Smart displays (which now are pretty dumb) will get the same capabilities as mobile phones because why not?
• Skills and actions are now hardware dependent, but the functionality we want to access is general. In that way the home devices are like browsers and all services should be developed platform independent.

That was all for now. In the next article: why Siri is such a lousy conversation partner:

In order for a conversation to happen it is not enough that one of the parts is driving interaction, we need at least two parts with a will of their own, with their own goals and intentions.

The state of Conversational AI

Until next time!

This article was originally published at Dinahmoe Battery as a part of Battery001: An issue dedicated to Voice!

Before you go…

I write to find likeminded and would love to hear from you!
Agree or disagree, ask questions or suggest improvements,
the comments section below is all yours!

To get in touch, please connect with me on LinkedIn.

If you want to get notified about new articles and projects, please subscribe to Dinahmoe’s newsletter by clicking here

How to build conversation friendly voice apps that won’t kill you was originally published in Dinahmoe on Medium, where people are continuing the conversation by highlighting and responding to this story.