Microsoft Design Expo- Spring 2016
Project Brief:
“Design a product, service or solution that demonstrates the value and differentiation of the CUI. Your creation should demonstrate the best qualities of a symbiotic human-computer experience, which features an interface designed to interpret human language and intent. Your creation should clearly demonstrate foundational elements the CUI calls upon in order to delight people. It should meet a clear need and be extensible to wider applications. It may be near-term practical or blue sky, but the idea must be innovative, technically feasible, and have a realistic chance of adoption if instantiated. Of course, to deliver an optimal experience, much is implied — from data and identity permissions to cross-app agent and/or bot cooperation and coordination (first and third party); your design should minimally show awareness of these barriers or explore solutions to them.”
Week 1: (Explorative Research)
Defining CUI :
Conversational user interface is different from voice interactions because there is an assumption that there is some sort of AI sitting behind the interface, where as voice interactions is more directions, like Siri. Conversation implies that something can actually talks back in a way to move the conversation forward.
Conversation is also just listening and responding, to me a conversation doesn’t have to be verbal, it could be body language or thought, as long as there are provocations and some sort of response. There is a mutual understanding of the context and each others intentions.
Defining Symbiosis :
Licklider’s Symbiosis:
- The AI helps you understand what you are asking for
- The computer can decide what is best
- Man and the AI cooperate in decision making.
- The computer thinks as you think, there is no predefined program.
Types of Symbiosis:
“The cooperative “living together… of two dissimilar organisms” is called symbiosis.” — Licklider
Symbiosis is a “Co-operative interaction”, an “extension of yourself” and the “competence mirrors your own”.
Scoping- the beginnings of a Territory map:
I wanted our project to be extensible. In that our end result allowed for people to look upon something differently, something small but significant. Group members saw CUI as being beneficial to users that are situationally or temporarily disabled.
To start finding a territory, we used the mapping exercise below to brainstorm potential interesting areas and understand interests of other members in the group.
x axis: potential areas of interest (education, health, family, etc.)
y axis: areas of effect (personal, group, community, etc.)
We assessed each potential area by looking at access, competition, and jumping off point. From this we created the beginnings of a territory map:
which was then refined to this:
There were three main areas of interest that came out of this: Something to assist in learning, something to help with family life, and something to do with working life. While these seem vague, our focus was one of every day life, not something that would help someone in extreme circumstances, such as a firefighter etc.
Week 2:
What is CUI/context-aware system good at?
- Efficiency: facilitates communication between man and computer to efficiently manage tasks at hand
- Reducing cognitive load: understands one’s intent & purpose to operate simultaneously
- Adaptability: knows one’s context and can quickly adapt to the new environment / situation
Territory Map Refined:
Young parents encapsulated the interests of the group
After some research on first-time, young parents, all the products we came across were directed towards the child, as a way of caring for the child. Most products ignored the needs of the parent and put the child’s needs first and foremost. If we design for the parent first and the child second, it differentiates us from other products.
Interviews:
Peter: Confirmed some hypotheses we had around young, first-time parents. He talked about the identity shift parents go through, peer pressure from family/ society and the idea of what constituted being a ‘good parent’.
Suzanne: Talked about the support and affirmation needed by new families, specifically the mom. There is also this emphasis placed on the new child rather than on the new parents, she stated that she felt it should be reversed that the parents need as much attention as the new baby.
Kakee: Not wanting children, she talked about the tension between her and her observations of her friends who have settled down. How priorities shift, and the idea that it takes a village to raise a child. Her friends life choices are not her life choices, and it is isolating.
Week 3:
Understanding Loops:
Single: Set the temperature at 68 degrees.
Double: the thermostat asks ‘why should I set the temperature at 68 degrees’
Triple: The thermostat starts questioning the frame and stating how we assess the frames “why are we measuring in degrees at all”
Under what circumstances would we accept a machine who questions our choices?
Symbiosis:
“It’s like a dance, you push the other person accepts the feedback and then pushes back”- Gordon Pask
Is gathering information and orchestrating actions symbiotic?
Trying to breakdown the problem even further:
If a problem is too large, what is a simpler problem, what is a problem related to the one you want to solve, how can you relate the solution back to the larger one?
Stakeholder maps:
Trying to understand who are potential stakeholders in the first-time parent ‘system’. Mapping with actions to stakeholders made the map seemed flawed because actions are not duplicated between stay at home parents and working parents.
How does the interaction change between different family members when using CUI? Different personas to each member?
Interviews:
Jason: (working new parent) Schedule was centered around baby so that responsibility was balanced. His wife texts him throughout the day to keep him in the loop about the baby. His transition into parenthood seemed natural, his new role was accepted.
Tammy: (single mother) She relied heavily on the community to help her with her child. Such as when she got coffee the people at the counter took care of the baby until she was done. She also described the troubles of traveling alone with the baby.
Carla: (ran a day care) The teachers recorded everything, when they ate, played etc, as well as reflective recordings.
Hannah: (on maternity leave) She also recorded everything the baby did, as it gave her an idea of a pattern. She was exhausted but didn’t want to hire help until the baby was old enough.
Overall, I think it is interesting how most of the parents we have talked to are just making it up as they go, and could really be provided with more just in time guidance.
Taking a look at all the interviews thus far:
We wrote down highlights from each interview and went through each interview to make sure we captured everything.
We took the quotes and clustered them into themes, We then voted on themes we thought were interesting would like to explore a little bit more.
We then took the ‘winning’ themes and tried to find the gaps in our knowledge. This step guided our further interviews.
Week 4:
Our goal this week was to really narrow down our territory to a specific direction. We decided to stop looking for ‘problems’ and started looking for ‘opportunities’ or moments that we could enhance.
Contextual Inquiries:
Deepa: Deepa is a working parent who is working both in industry and towards her PhD, she takes care of her four year old mostly by herself as husband lives elsewhere. Deepa talked about baby center and conversation as a away of affirming her actions as a parent. Since her husband is away, she talked about how they communicated and how talking to him served also as a venting session about how her son, Veer acted that day.
Likewise, while we were doing a touchstone tour of Deepa’s apartment she told us this story about a time when Veer was misbehaving, and he brought her to the refrigerator where a picture of them were and asked Deepa, “Why can’t you smile like this”.
Cyert Center: Jiyoung was able to go back to the Cyert Center and observe the infant wing watching the hand off of kids. One thing she found interesting was that even though the babies do not understand what the teachers or parents were saying, they would still have a conversation with the infants, trying to interpret what their child wanted, or thought.
Interview:
Annie: (On maternity leave) She stated that the most challenging part of being a new parent was accepting her new role and changing her life to accommodate this new role. She goes to a government run baby center to meet parents in a similar socio-economic group. She gets the bulk of her information from people she meets there, where she can see how they parent first as a way of seeing if she wants to take their advice or not.
ELITO method:
We broke down our interviews into parents and professionals, and applied three categories born from the clusters above to each group. These categories were:
- Knowledge of What Works
- Attachment
- Conversation
From this, values of Bonding, Curation and Affirmation emerged, to which we wrote some ‘how can we statements’ to guide idea generation.
‘How Can We’ Statements:
- How can we strengthen the bond between parents by using the baby as an agent for humor?
- How can we improve the richness of “did you know what your child did today” stories being exchanged when a child is passed between parents, or from daycare to parents?
- How can we provide community/mentorship to isolated parents to reduce mental anxiety?
- How do we curate information from anonymous people by only connecting parents to similar caretakers to reduce information overload?
- How do we make short bursts of mundane time more memorable for the working parent to encourage parental involvement?
Week 5: (Generative Research)
Interviews:
4moms: Being local to Pittsburgh, we set up an appointment to talk about their thought process when designing new products for new parents. Some key take-aways were that Parents are thinking about the child, not the task, and that time was precious for first time parents. So their products prioritized, ease and efficiency.
Where to Focus:
After presenting our Exploratory Research results to Microsoft, we decided to focus on Rituals and to choose a moment to focus on.
From this list, we picked playtime, feeding, and waking as starting points for further investigation.
Week 6:
TEI:
Dixon and I went to TEI in Eindhoven to present another project. While there, presentations and demos provided a refreshing step back/ inspiration and insight into the project. In that conversation can be had through form, and that form doesn’t have to be static.
Workshop:
Jiyoung started experimenting with potential workshops. Taking a mental model diagram as inspiration, she did a dry run of a workshop that focused on identifying pain-points in tasks and solving these pain-points through magical devices. This allowed focus on emotions and reasoning behind each action to emerge.
Week 7:
Looking at past interviews, The moments identified above are moments don’t really hold ‘pain-points’ per say, they are just the moments that parents like to talk about, because they are amusing or they are proud of solving the moment.
Guests also suggested using more abstract methods for creation of the ‘magical device’ done in the previous workshop. So instead of drawing this device we used play blocks to build the device, while having the parent walk us through the device step by step, through this we learned a lot about their routine, and values.
Generative Workshop Questions:
- What are moments or rituals that your family goes through regularly that you enjoy.
- Design a magical machine or think of a superpower that would make this moment more enjoyable.
- What are regular moments that you don’t enjoy.
- Design a magical machine that will help you enjoy the moment.
Findings:
We held four other workshops this week, with a wide range of parents (socio-economic, age, etc). There was a common theme of conversation, or connection with their child, in trying to learn about their child.
From these workshops, the following were our key findings:
“Happy Crabby Funny”: A family ritual where daughters had to describe a happy, crabby, and funny moment in their day, if their response to “what did you do today?” was “nothing much”.
“Ok, Rainy, Sunny”: A family ritual, but with both a positive, neutral and negative slant. As opposed to the one before where there was a more positive slant. The negative prompt was there to understand how hard the parent had to work to make their child’s day better before they went to sleep.
“Car time as connection time”: The only time when the kids are present and cannot run away.
Week 8:
Generative Workshop Findings:
Group workshop: Parents made machines that gave insight into how their child was truly feeling, put their baby to sleep and told you how long they were going to nap for, or was able rewind time to experience precious moments again and again.
Individual workshops: Participants built a ‘more time machine’ that would stop time so the parent could have more time with their child, another machine gave tips on how to handle the terrible twos
Interviews:
Instead of asking them to built a machine, we asked them if they had a version of Jarvis what would it be and what would they use it for?
Bimpe: She would use Jarvis to help with homework, to individualize her child’s learning needs. When asked where else she would use Jarvis, she felt like technology didn’t have a place in the time she views as quality time with her kids. She did say that she would use Jarvis to help with scheduling so she can spend more time with her children. Loves when her husband takes over, as it gives her time to herself.
Mary: Would use Jarvis to do chores, so she can spend more time with her daughter and husband. She worries about spending enough time on everything, and finding a balance.
Sense Making:
Wrote down interesting points from the research done up to this point and clustered them into three directions.
All focused around some sort of conversation, where Parents either had part of the story or none of the story and needed to be filled in.
Potential directions:
- Call of Duty: Encourage parents to empathize and offer support to each other
(Parent to Parent- gives a piece of the puzzle)
2. Lens of Truth: Understanding the needs of children who are not yet able to vocalize their needs.
(Parent to Parent- Parent has no information from Child)
3. Marginal Telepathy: Ask more engaging questions for a richer conversation.
(Parent to Child- gives a piece off the puzzle)
Week 9:
(Spring Break)
Week 10: (Evaluative Research)
If our goal is heading towards information into ones child’s day, it is not just what the child did, it is also interactions with other children and people. We also feel that there joy is in the solving of this ‘how was your day’ puzzle. We didn’t feel comfortable providing 1:1 report of what a child did, because it would be robbing the parent of the discovery and conversation with their child.
So how can we provide information about the kids day, perhaps using the interactions of other people to provide information in an abstracted way?
How can we provide abstracted/vague information to allow parents to understand the environment one’s child is in.
What is Human Computer Symbiosis?
We are giving parents an ability they currently do not have, which is a more directed intuition, or a piece of the puzzle about their child’s day. Is augmenting the human symbiotic? It benefits the human, but how is the machine benefitting? is it just the data being collect? do we want to collect data?
How do these two actors reach consensus on what is beneficial for the family?
What is CUI?
We are circling back to these two questions. To me, CUI has always been a conversation. And a conversation is just two actors interacting, as would a software interface. I think hearing ‘conversation’ has thrown other members of the group off in that conversation could be synonymous with ai.
Speed Dating
After running generative workshops, we decided to use speed dating as a way to narrow down and refine our concept.
Talking to the same participants in the generative phase, we walked each person through differing ideas, and had them ‘fund’ ideas by giving them 6 quarters.
Dimeji: If technology needs to be in the picture, the parents should be the mediator. Technology should also be seamless, and you should be unaware that technology is present.
Dylan: Creativeness lies in not having something to do, I think we need to be careful around this idea as I think creativity is extremely important when children are young. We don’t want to rob children of their intrinsic need to be creative.
Surveillance is weird, and we need to be careful in what we give the parent information about their child’s day. As children need space to work through her day by herself. However, the concepts shown where the output is abstracted, the reaction was more positive.
Speed Dating Reflections
Participants say they want different things than what they choose to ‘fund’. Likewise, when a parent is presented with differing versions of the same concept, they begin to like the concept when they previously had a negative reaction towards it.
Personalities of the CUI
We are starting to think about the personality of the CUI. If the main interaction is through the child, do we want to make it act like a Grand-mom? something magical? something strict?
What is the voice? CUI’s currently are all female (Siri, Alexa, Google). They preform a servile role, and are there to support the user. Do we want that? if it is a child is the voice more of a cartoon character? are there two voices? one that interacts with the child, and another that interacts with grownups?
To do this, we went around in a circle where one of us was the CUI and the other two as the human actors trying to test the boundaries of what seemed acceptable for the personality/limitations of the CUI.
What would the system look like?
The purpose of stories:
Stories are told to teach, and prepare children for life events such as a new sister or the death of a pet. Stories allow children to process events in an abstracted way, where it is not about them, but they can see themselves in the characters.
Week 11:
In the past two speed dating sessions, we grouped all of our concepts into four groups, and went through each, bringing out different concepts based on conversation.
These groups were:
- Learning Activity- Concepts meant to enrich school activities at home.
- Storytelling- Concepts related to using story as a proxy to create opportunities for conversation.
- Prompt- Concepts that used word or image to help recall of daily events during conversation.
- Cooperation- Concepts dealing with sharing of responsibilities between parents. (Eliminated in this week)
Each concept included various types of input and outputs.
Parents were most excited about the prompts and story telling ideas. When asked what each participant would invest in, storytelling was the most interesting, but prompts were the most applicable, and therefor made a better investment.
Speed Dating Findings
Opening Convenient possibilities for Creativity: Parents didn’t want to be told what to do, they wanted a choice.
Information source: Parents liked the idea of the child gathering or the teacher providing information. Information needed to come from a trusted source in a regulated manner.
Week 12:
Based on the speed-dating session, we narrowed down our concept to a specific experience flow. The concepts below are what informed out overall flow.
System Flow:
Thought exercise to see where conversation between human and computer would occur.
- Setup
Once the product is initiated, parents can set values, interests, etc. - Input
(a) Passive input: email, SMS, parenting resources, calendar, 3rd party apps, IoT products at home, etc.
(b) Active input: kid’s device (collects things about kid’s day) - Planning/Negotiation
Parents can review the content of the book, and collaboratively decide on which story to tell with the computer based on input - Delivery/Output
The contents (about kid’s day, about a moral parent wants to teach) is delivered as a story through a figure or a book.
System 1 and System 2
Two systems started emerging, instead of one overarching system.
System 1: A collection of data that each individual in the future will have. Each person’s system can be connected to another to create different relationship modules.
System 2 (i.e. our Product) : System 2 allows access to information that is tailored to parenting and to the child’s day. This information leads to parent-child activities and bonding.
Conceptual Map:
Each person’s System 1 can be connected to another’s and utilized for different purposes.
Degree of Storytelling:
Along side figuring out the system, we wanted to understand how storytelling would work symbiotically. We broke it down to three possible scenarios:
- Computer tells the entire story (Computer tells story and asks questions for the child to participate)
“I feel like I can walk away and do something else, what is my role here?” - Parent creating a story with the computer supporting (Computer helps when parent gets stuck)
“It’s hard to make a story on the spot. I feel like I need to do too much work here. What if I had a really long day? However, I like the flexibility.” - Computer Providing a basic story structure to the story (Parents can read the story and interject when wanted)
“This makes it easier for me. I want to know a little bit more about the story before hand.”
from this, we mocked up some story boards
Week 13:
Pilot Testing:
We visited one of our participants in her house and asked her to tell stories to her child. We tested a prototype that included features we wanted to include in our eventual design:
- Input from the child (parent has no prior knowledge of)
-We asked the child to take photos of various objects around the house that he wanted to see included in the story book later. - Input from the parent
-We asked for a moral they wanted to teach through story telling. We also had them decide what genre the story would be.
Reading vs making up stories.
We asked parents to tell a story to their child without the use of a book. The child seemed the most engaged with this tactic, as they were part of the story, and could contribute to the story line. The story lines were definitely non-linear and completely off the wall, which to me was great and really allowed for creativity. Depending on the parent, this tactic seemed either natural, or taxing and not relaxing.
In this type of story I did agree that if our purpose is to provide stories with morals, parents probably needed more support/guidance.
Child as a Hero.
Children seemed to be more engaged in stories that included themselves.
Length of Story.
Parents choose story books that they are familiar with and know the amount of time it takes to read them. The parent should be able to set the time limit to each story, if the story is taking longer to get through, the computer can shorten the book to stay within the time limit.
Prototyping an augmented book.
We used an ipad that was connected to a computer to simulated augmented content. We placed the ipad in a book with clear pages, so that the ipad could be seen, and the pages could also be turned.
Week 14:
Remote Testing Findings.
- Multiple levels of freedom in Storytelling is preferred
- If my child is lying, I prefer the story to tell the story to tell the moral.
- It is good to build the story together so you can learn what the child thinks in realtime.
- Have the story build over time.
- Parents would like to read the summary of the book before hand.
In Person Testing:
- When we asked the child to bring his favorite toy, be brought a camera.
- Mother wanted the things that her child collected to play a more active role in the story, such as a character
- Saw an opportunity for books to be stylized by different author and illustration styles.
System Map.
The system is equal parts computer, child, and parents. The computer has access to the parent’s, child’s and other databases that can provide evidence based parenting data.
Symbiosis:
Symbiosis is achieved by having three actors tell the story instead of just one.
Magnifying Glass Map.
Photos taken from the magnifying glass are placed into the story and stay around for two weeks after being talked about. If not talked about they don’t stay around.
Narrative Structure.
Lengthy conversations were had about what agency the human actors would have over the plot. At what point would the computer have to comply or course correct to stay with the chosen moral/ general story line.
This conversation led us to the chart below, which then led to us focusing on the ‘Partial Freedom’ for the purposes of this project.
The choice of ‘Partial freedom’ solved the dilemma of where and when the computer would need to course correct to keep the story’s plot intact, but also retained the quality that allowed ones imagination to run free. I think if this project was to continue, I would like to explore the complete freeform section, but done in a way that supports the parent so they do not find it tiresome.
So with this course correction problem out of the way we decided to utilize The ‘Hero’s Journey’ a three act story structure, seen below.
Week 15:
Vision Video
This week was planning and creating the vision video, creating storyboards, dialogue and filming.
We shot the same thing with two different families, one had an older child (slightly outside of out 4yo target age) and the other with a 4yo.
Validation:
The family with the 4yo, didn’t work out filming wise, but provided us with the validation that our product matched parent improvisational storytelling techniques.
When watching parents make up stories, there were a lot of instances where the child would have to acknowledge correct behavior. An example being a parent asking “what will happen if the child doesn’t listen to the parents” to which the child responded “the zombie will come”.
Week 16/Microsoft Expo
Final Vision Video:
Presentation at Microsoft:
Page created after Project, and based on my weekly updates, other team members weekly updates as well as team process blog.
Notes to self:
- Need to add images of my book and Jiyoung’s book.
- Add picture of the three of us testing out the CUI’s personalities.
