Phase 4: Refinement and Evaluative Research

Team 16: Amrita Khoshoo, Diana Minji Chun, Hannah Koenig, Shambhavi Deshpande

This post chronicles our team’s progress as it happens for the fourth phase of our Interaction Design Studio 2 Project, taught by Peter Scupelli in Carnegie Mellon University’s School of Design. You can find the full process publication here.

3/25 Reflecting and determining next steps

Presentation 3 Debrief

During the class session on Wednesday, we reflected upon our presentation for phase 3 on Monday and discussed the feedback we received from Liza, Arnold, and Peter. Between the two ideas we conceived in the last phase, Mosaic and Learning (401)k, we had received a lot more praise for Mosaic.

We were excited about both of these ideas, and we evaluated our next steps, the scope of the project, and the 6-week time horizon of the rest of this semester. We considered one option as building both of them out and evaluating which worked better. We also thought more about choosing Mosaic as the primary concept and possibly merging Learning (401)k into that.

All of us were inclined towards picking one concept so that we can make better use of time, build it out, flesh it out, and move forward into stage 4 of the project, excited to work on prototyping and evaluation of our ideas.

Next Steps

We talked about what form this intervention should take — a digital platform or a service. We determined that it would take the form of a digital platform but also have an emphasis on interpersonal interactions. We thought of utilizing real-time interaction perhaps in the form of video conferencing. We questioned if it is even possible to build any service without in-person components.

We briefly brainstormed upon potential audiences and thought that we could define a team or target audience now and define personas, or we could define personas and then figure out a team. We can start with “team personas”, think of scenarios, and possibly determine individual personas.

As we look forward to walking on our newly sketched path, a few things that we are collectively excited about are policy design, service design, building service blueprints, and building a digital platform from ground-up.

Timeline

Hannah quickly created for us a calendar for this phase, which is very helpful to be backcasting our progress from the next presentation.

Guidance

We then had a feedback session with Peter, where we talked about more ideas related to Mosaic, potential audiences, ways to prototype and evaluate, and the broad goals and expected deliverables of stage 4 of this project.

Peter described how he liked the community aspect of this concept, and he said that in Mosaic, on one hand, we are bringing people who are working together, and also bringing people who aren’t working together. We might be building up a social network around work. It can be helping people with their next job, or it can be helping someone who is building a team to figure out people who have the right skills. We talked about similar interventions of LinkedIn learning and ConflictU and questioned if Mosaic would be something that plugs into other platforms like Linkedin, or if it would be a new platform.

Peter emphasized on storyboarding the ideas in this concept Mosaic and trying to understand how people receive different parts of it. He encouraged us to start thinking about features at low fidelity and start to test them, while also thinking about the big system. He mentioned what Richard Buchanan has described — that as humans, we can only experience a path through the system and not the whole, and that we need to be picking some pathways and testing those, as we cannot test the whole system. In terms of the expectations for deliverables of stage 4 of this project, Peter encouraged us to focus on having multiple rounds and feedback, rather than a single polished product.

We then talked about audiences, thinking about who we have access to, and who would benefit most from the intervention. The current situation of social distancing has its pros and cons. All of our evaluation rounds will be performed remotely now, which is unfortunate. On the bright side, it opens up more windows to who we can be testing with, since nobody is more remote than anybody else.

We also discussed whether we need to be doing an IRB review for this project, and determined that since we are not working with vulnerable populations, we are just doing research with adults, and not to publish a paper, we probably don’t need to do a review.

3/26: Mosaic Concept Development

We met on Thursday to further develop our concept of Mosaic, we began to do so with a Theory of Change Activity. We also took the time to formally check-in about how everyone is feeling about our progress and teamwork.

Checking In 😌

All of us thought that phase 3 worked well, despite our transition to remote work due to the covid-19 pandemic, which everyone thought went very smoothly. Even though WFHing, our interactions feel normal, and we are having fun with remote working in our own way — be it our neon colors and extra-large fonts in draft presentations, or funny virtual background experiments in Zoom, or amazing social media compilations. 🃏

We also matched each other’s thoughts on why this is working — because all of us are excited about the concepts that we have come up with, and everyone feels that there is an element of their thoughts in the concept! ❤️

Study Grouping 📖

Before the meeting, we collectively read through the papers that Sofia had shared with us: “Soft Skills”: A Phrase in Search of Meaning (Matteson, Anderson, Boyden) and The Importance of Soft Skills: Education beyond academic knowledge (Schulz).

“Soft Skills”: A Phrase in Search of Meaning (Matteson, Anderson, Boyden)

Through this paper, we found a highly informative and critical lens to look at soft skills. The authors have outlined some key prerequisites to research or interventions that focus on soft skills. They have argued that first of all, we need to define what soft skills are. Then we need to figure out how to measure them. Then we can look at whether people can develop them, and then investigate potential outcomes of soft skill application in practice. They have also elaborated upon related terms that are generally used interchangeably: skills, dispositions, attitudes, beliefs, values.

Skills: The ability to access knowledge from a domain-specific knowledge base and use that knowledge to perform an action or carry out a task.
Dispositions: Individual qualities, relatively stable over time, that influence behavior and actions performed as part of an individual’s skill set.
Attitudes: A positive or negative judgment, based in part on emotion, about an outside entity.
Beliefs: An acceptance that certain factual evidence is true, informed by an individual’s own values.
Values: General standards or principles that guide behaviors among varying situations and to which individuals feel a strong commitment.

Their argument that “including dispositions in soft skills shifts the focus from what someone can do to how someone is, which can lead to problems with accurately evaluating an employee’s performance.” struck out to us. Interestingly, the authors of this paper advocate for ideas that very closely match our concept of Mosaic:

“Moving away from research to application, we identify a strong need for instructional materials to develop soft skills both in LIS coursework and in continuing education. One option could be to create a modular course that would introduce, diagnose, and provide training on developing soft skills in librarians. Modules could be delivered across a variety of platforms, such as through a course management system, by a series of webinars, or as printed material delivered through face-to-face meetings. The structure of each module could include lessons that introduce and describe discrete soft skills, materials for participants to self-assess their own abilities, case narratives of the skills as they play out in a library setting, exercises to practice building each ability, and opportunities to reflect on the skill development process.”

We asked ourselves: How should we address the research gaps on soft skills? How should we select an appropriate combination of the target audience and soft skills? Perhaps we need to define the skills, and perhaps it would be best to choose a group, and then focus on them, such as librarians who we worked with in the last phase. We could also use existing frameworks in the article about categories and related skills, or look at the future of work reports.

The Importance of Soft Skills: Education beyond academic knowledge (Schulz)

With this paper, we learned about working with soft skills and hard skills in conjunction with each other. This paper suggests that embedding soft skills training to training hard skills is a great way to teach both.

It describes some of the lacking qualities of graduates that employers complain about as: communication skills, team working, business or project management knowledge, verbal and numerical reasoning, lack of rhetoric training.

The authors of this paper try to answer what are the most important soft skills, and describe them as such:

Communication skills: This is more than language proficiency, it is about self-esteem reflected on body language, adequate discussion skills, good presentation skills in order to market oneself and one’s idea. Adequate communication skills are a prerequisite for a range of other soft skills like moderating discussions or conflict management.
Critical and structured thinking: Critically filter the endless stream of incoming data, analyze it, and make informed decisions based on it.
Creativity: “Thinking outside of the box”, finding innovative approaches to problem-solving, brainstorming and mind mapping which is already common applications in the business world.

It further questions whether personal traits and habits be changed, and determines that a person’s ability to recognize and acknowledge certain behavioral shortfalls or plain bad habits is important in this aspect.

This paper looks where soft skills can be acquired to give a few examples of formal training (https://horton.com/), student-centered learnings (DeBono or Neuro-Linguistic Programming, Rob Krueger upside-down lecture) and self-training based on books, or methods such as presenting in class, working in groups to talk about ideas, particularly during technical courses.

From this paper, our key takeaways were to think about including in our intervention something about self-reflection or self-awareness. We determined that we should agree on the definition of soft skills that we will use, perhaps starting with their umbrella of most important soft skills (critical thinking, communication, creativity). We also thought that in the intervention we design, we could integrate group activities as a way to practice soft skills.

Theory of Change Activity 🚀

As we began to detail out our concept Mosaic, we employed the framework of the Theory of Change. Thanks to Arnold and Peter for sharing it with us, and also to Hannah for leading us through it from her experience using it earlier.

What is the Theory of Change? (Center for Theory of Change)
Theory of Change is essentially a comprehensive description and illustration of how and why a desired change is expected to happen in a particular context. It is focused in particular on mapping out or “filling in” what has been described as the “missing middle” between what a program or change initiative does (its activities or interventions) and how these lead to desired goals being achieved.

We looked at a few examples of the Theory of Change in use:

We drew a framework to build our Theory of Change with these steps:

First of all, each of us silently jotted down our thoughts for 5 minutes on the first step of the Theory of Change:

Then each of us talked through our thoughts to the group:

Diana’s thoughts were centered on a better workplace with trust; transparency which could lead to autonomy; pathways for reskilling for people who are seeking a career change, promotion, or job change related to automation; Seamless integration of remote work for teams facing a crisis such as covid-19 or other world changes; Equal opportunity, equal voice, equitable power distribution for people from different backgrounds — thinking about how might we even the playing field for those coming from affluent backgrounds and those who are not?

Shambhavi’s thoughts were centered on better communication helping decrease resentment in organizations, which could lead to long term service, which could then lead to longlasting and truly sustainable organizations; Appropriate channelization of negative emotion leading to more openness about general resentment as well as act as for more serious matters such as assault, or hostility; Understanding people’s stories and backgrounds which could lead to better communities within and outside of work.

Amrita’s thoughts were centered on creating a workforce that is successful in their careers and also during times of transition, which connects to opportunities for reskilling to become highly successful in the long term; Being able to combat harsh conditions; A workforce that is happy; Teams with stronger communication skills which connects to adaptability; Balance between technical skills and soft skills; Collaboration as the key to better functioning teams; Providing people with personal as well as professional opportunities to grow; Reframing what conflict is because conflict is not always bad and can be healthy; Employees feeling safe; People being able to advocate better for their positions/needs/desires; People being able to curate what they need to learn.

Hannah’s thoughts were centered on a transition design perspective and the question of what is the point of people being more satisfied at work? Just being fulfilled individuals is not enough — It is necessary to have meaningful communities, which connected to nurturing a planet, and that maybe work is about a higher goal, and not about generating money and economic growth; Individuals being self-aware and connected to themselves; Productive individuals and productive teams — where the term “productive” needs to be defined; The remote work crisis as an opportunity to slow down and redefine what it means to work, and an opportunity for workers to think about their rights; Connecting the future of work to the future of life; A future of work that is decent for all; Recognition of the fact that the future of work hinges around the future of life on earth, and that if we continue to burn through resources on the planet, it won’t matter how productive we are, hence a redefining of what it means to work.

From the discussion of our thoughts on goals, we saw these themes emerging:

We were able to fully acknowledge the wickedness of the “future of work” landscape, and how it cannot be separated from the idea of a “future of life”. Even in our generative research workshops, quite a few participants mentioned that they want a separation between work and life. Sometimes, people are not proud of what they do, or if they want to be in a place where their work doesn’t define them. However, as we look at the broad landscape of “work”, it is apparent that separation of work from life is a myth. This also gives us a window of opportunity to build healthy communities at workplaces. It is becoming more and more evident to us, that healthy work is inherently tied to healthy communities that relate to work.

On a “lighter” note, virtual stickies don’t have to be rectangles! ☁️ 😋

Just being more expressive about our thoughts 💭

And here is an exclusive backstage scene from our work session 👩‍🚀

Hannah in space!

3/29 Cont. Theory of Change

We relooked at our Phase 4 calendar and worked out a clearer timeline of tasks until the next presentation.

Then we continued working on the second step of the Theory of Change:

We rearranged ideas and organized our thoughts on a logical flow of how Mosaic would lead to Decent work for all.

Intermediate state of the logical flow of how Mosaic would lead to Decent work for all.

Meanwhile, we have kicked off prototyping with cute little kitties!

3/30 Cont. Theory of Change

At the beginning of class, Peter shared the updated rubric for Phase 4. The areas we were asked to consider: Setting up the problem space, Context, Strategic, Coherence, Research grounding, Human/Biosphere-centered design, Project, technology and scale, and Presentation. We were glad to see that within strategic considerations, he particularly asked How we understand the theory of change, where the project rests on, the impact we hope to have in the world.

Motivated, we continued to refine our logic of the Theory of Change.

During this process, we talked about some of the questions, assumptions, and ideas we had:

• How Mosaic can be used for a team or individual? Could someone come into the platform free of organization or team association and would that take away from ConflictU functionalities of team learning and working together?
• Is this platform open source or vehicle for other companies to load their own modules? Who will set the tone and/or learning modules? (Mosaic or user companies) Is our higher goal of Decent Work for All important to protect?
• What if this is participatory learning? Teams can upload their own stories and modules to share and people can learn and modify.
• Onboarding of Mosaic: Is there a general module about soft skills for onboarding? We need to unpack “Individuals go through coursework to learn new soft skills about communication, conflict, and collaboration” What do people do on our platform?
• Are we being radical enough? In regards to Sofia’s questions: why is it that we are trying to translate face-to-face productivity into the digital sphere, should Mosaic actually be an opportunity to talk about workers rights and slowing down a bit? Is effective collaboration actually linked to other ways of being/living that include a redefinition of work objectives, are face-to-face soft skills the same ones needed in a digital environment? Maybe this can come in terms of audience, functions

To our last question, Shambhavi shared with us a chart we can formulate our thoughts around.

We finished our diagram and reflecting on Peter’s feedback during group check-in, we decided to read the speed dating paper he shared and move onto the storyboarding for our next meeting. The two scenarios we are building our storyboards on are;

1. Teams transitioning to remote work
2. Individuals going back onto the job market/upskilling (Formerly incarcerated, freelancer, gig workers)

4/1 Storyboarding

During class, everyone came back with some storyboards based on the discussion we had Monday. Seeing each other’s storyboard was a great way to talk about our assumptions, similarities, and differences among the ideas.

Scenarios that came up varied from new graduates going through onboarding with Mosaic, workers seeking specific skills for a promotion or growing freelancer business, to imagining Mosaic as a virtual assistant helping workers set up the healthy working habit in WFH situation.

The themes and questions we talked about:

• Format of the learning- it should be flexible and bitesize learning modules for busy professionals who are juggling existing responsibilities.
• Urgency- Do we want to focus on the remote transition or overall work?

Source: https://www.catalysts.cc/en/the-catalysts-way/9-4-situational-leadership/

The discussion of feedback was crucial in thinking about what type of intervention we want to design. He asked us to utilize the stage of behavior change framework; different interventions will be effective on different people on different stages. This reminded Hannah about the Leadership model which is a matrix based on two axes; high to low supportive behavior and high to low directive behavior.

With this discussion in mind, we crafted our storyboard themes that are based on remote team transition. After we decided which scenario we want to tackle, we reached out to people who are in our target group and tasked each other to individually come up with another round of storyboards according to the selected topics below.

4/3 Storyboard Speed dating

Friday evening, we put together the storyboards together as a presentation deck. since we discussed our layout and fidelity, all of our storyboards looked cohesive and uniform.

We interviewed a wide range of knowledge workers who have been working remotely or are transitioning into remote work due to the current situation; 2 lawyers and 4 engineers.

The 2 lawyers are our team members’ significant others. They fit the target group profile and easy to recruit.

We scheduled all the speed ratings back to back, giving 20 minutes for each party. The engineers were interviewed individually due to their locations and the lawyers were interviewed together. We gave guided the participants use a hot-warm-cool-cold judging scale which they can say while we are going through the storyboards and we also asked them to rank the storyboards in the end.

We tried a couple of different approaches to getting feedback. First few, we did 3 at a time and ask them to give us feedback. Quickly we found out that doing it this way was faster but also, make the participant forget the first two scenarios. Later, we went through one by one and asked for feedback after each storyboard. This yield more in-depth feedback but also took much longer.

We appreciate everyone taking the time to talk to us and give us great feedback!

4/5 Storyboard Speed dating Synthesis

After Saturday break, we came together to synthesize our session. Some of the important things we noticed through speed dating

• People found quick/actionable modules such as facilitation 101 more helpful vs. behavior change such as work-life balance.
• In-person element is an important part of work training
• People do not like conventional HR training due to its lack of context and it’s nature of individual-based tasks.
• People wanted clarification on a few storyboards

For the last point, many participants asked who is providing the service and what the mission of it are. Many assumed this was company affiliated product. This made us wonder if that would be most ideal for what we are trying to achieve.

For the next round, we are trying to catch two birds with one stone- refine our idea as well as testing out the possibility of Mosaic as an independent consumer product.

4/6 Planning for Concept v2

Today, we caught up with Peter and started planning for our second prototyping round.

We’re currently at a crossroads. Do we increase design fidelity? Or do we continue exploring aspects of our concept? Mosaic has the potential to be an enterprise, an independent, or a hybrid platform. We want to understand the value of each, which would mean another round of storyboarding and user feedback. After speaking with Peter, we decided that it’s better to understand how Mosaic might fit into someone’s life, the scenarios in which is might be used, the needs it can address, and the story behind the product. We decided our second round of testing would be about the independent-version of Mosaic, using potentially a mix of storyboards and lofi wireframes.

v2 protoyping synthesis + plan

One great discussion we had was around how progressive should Mosaic be. Should it go beyond traditional modes of work and arm people with skills useful for everyday life? Should it help with political advocacy or social movements?

Really, what we’re thinking through is what does it mean to work and what perspective should Mosaic have on the future of work?

More to come tomorrow…

4/7 Concept v2, continued

We accomplished a lot today! We gained more clarity about our second prototype/method of evaluation and worked on our presentation outline. Can you believe our next presentation is less than a week away?! Lots to do!

GROUP SHOT + hannah’s candle + diana’s mug + the wizarding world of harry potter

We decided that we’ll aim for two more iterations and evaluative tests:

• Round 2: will be geared towards developing and evaluating independent Moasic concepts with storyboards and initial wireframes.
• Round 3: will be geared towards fleshing out our chosen Mosaic concept and increasing design fidelity (wireframes). If we’re unable to get feedback on this iteration before our presentation, that’s ok. We’ll hit the ground running with testing in the next phase.

We hope to run Round 2 evaluations in two parts:

1. Storyboards of independent Mosaic.
2. Low fidelity wireframes of Mosaic’s landing page with relevant content (content/copy + hierarchy will evaluation measures).

To help us think about our objectives for this round of testing, we brainstormed a number of questions (image below).

questions and things we hope to learn moving forward!

Key questions include:

• Concept: What concept resonates most with people? Enterprise vs. individual?
• Role as Independent Platform: What would the role of Mosaic as an independent platform be in someone’s life? What needs could/should it solve for?
• Political Aspect: How political/progressive should the platform be?
• Community Aspect: How geared towards community learning or community networking should this platform be?
• Individual/Sharing Aspect: What information would people want to learn about themselves and share with others?

At the end of our meeting, we outlined our presentation.

We ended our meeting as all meetings should, with a photoshoot!

hannah’s candle: gollum + library edition

4/8 v2 + Presentation

Today we planned for our second round of testing by narrowing our questions, brainstorming storyboards, and recruiting participants. First, an obligatory team shot!

on wednesdays we wear pink (i forgot the memo)

We decided to focus in on four specific questions from our question brainstorm:

1. Role as an independent platform:

What would the role of Mosaic as an independent platform be in someone’s life? What needs could/should it solve for?

We will address this question through storyboards. We started brainstorming scenarios around this question, discussed, and prioritized. We wanted to focus on a breadth of ideas, each showcasing a different role/need. Our scenarios will reflect Mosaic as an independent platform in volunteer, friend, family, individual work, and teamwork scenarios.

scenario brainstorm + chosen concepts to storyboard

2. Political Aspect:

How progressive should it be / how much of a redefinition of work should we adopt (mission statements?)

We will address this question through lofi wireframes. Will will write various mission statements/slogans that will span a scale from neutral/mainstream to more progressive for a landing page wireframe.

brainstorm

3. Individual Sharing Aspect:

Would you want your information to help personalize your experience, like give you recommended learning modules?

Lofi wireframes show different ways users can personalize their experience. For this test, we’ll focus on onboarding.

Would you be comfortable with or have reservations about info to be shared with others (like your working style or what you’re learning right now)?

Lofi wireframes that address public/private information sharing.

Because of the rapid timeline, we’ll be testing tomorrow. We then reached out to our network to recruit! Stay tuned…

4/9 v2 evaluations

Round 2 evaluations! We spoke with 7 people and got feedback on 8 concepts, 3 mission statements, and 5 ways to handle data privacy. It was whirlwind.

testing!

We started by testing 8 different concepts. We were evaluating Mosaic’s potential role as an independent platform and how far we might be able to push the boundaries.

test 2

8 storyboards

mission statements + data

Next, we tested our mission statement wireframes, followed by our first set of data privacy wireframes, ending with our final set of data privacy wireframes.

Overall, it great to test the independent platform Mosaic. We plan to meet on Saturday to discuss our key takeaways. Stay tuned!

4/11 v2 evaluation synthesis

We met to share key takeaways/surprises from 2nd round testing, discuss mission statements, agree on a single concept direction, and start working on a list of to-dos for our presentation.

Round 2 Insights

Key insights from testing include:

1. Participants were unsure or concerned about the term soft skills. It was not clear to them or they thought it might not be clear for others.
2. Storyboard that dealt with soft skills that directly related to work (career, professional skill) resonated better with people. They were uncomfortable about using Mosaic in personal or intimate settings.
3. People talked about motivation of using Mosaic and sharing of their info in relation to the legitimacy and credibility of the platform.
4. All and all, we felt that people had positive feelings about what Mosaic could be and wanted to find out more. We believed that we had feedback to refine and detail out what mosaic could be.

Mission Statements

We workshopped four possible mission statements.

1. HR and career coach for the new economy! For a lifetime! We will help you do side gig, get your promotion, get your next job, your twentieth job, and will be always on your side helping you see the bigger picture (you should be on everyone’s side). It’s about thriving with people (communication, facilitation, leadership, negotiation), HR for Social Justice (workers’ rights, advocating for cause), and process (self assessment- curated curriculum projects).
2. Mosaic is a platform for teams, in and out of the workplace, to work more collaboratively together. Using the platform, new/existing teams can build strong team dynamics through individual/group facilitated learning around effective communication, conflict management, inclusivity, and facilitation. Because team collaboration is the future of work/ensures resiliency in the future of work. Individual modules would complement team-building skills.
3. New Mosaic could be a digital platform to help individuals and teams build better careers for themselves, through online modules and mentorship about life skills, such as communication, conflict, collaboration, organization, resilience, volunteering.
4. Mosaic is an independent digital platform where people can learn and practice soft skills like conflict resolution, communication, and collaboration. It provides interactive, asynchronous learning modules for individuals and groups in professional contexts, designed to provide self-awareness and resources to facilitate conversations that improve working relationships. Mosaic: work with purpose. For workers, volunteers, and teams anywhere. Collaborate with purpose?

Defining Mosaic

We then discussed and deliberated on various aspects of Mosaic and how we imagined each aspect working.

We ended our meeting by listing out all assets needed for our presentation tomorrow. In true T16 fashion, tomorrow will be a marathon day. Let’s get it!

4/12 Getting Presentation Ready

Today was a marathon day! We normally spend around 8 hours getting ready for a presentation. Today, we spent around 11. The day went by fast! We started by reworking our Theory of Change to be more in line with our new version of Mosaic.

simplifying and connecting our research to our ToC

We talked about our business model, our data model, and our various personas. We then moved into our presentation dec to start compiling our assets.

non-profit model / transparent, user-first data model

About Mosaic: Mosaic is a non-profit learning platform for individuals and teams in and out of the workplace.Through interactive modules, people can learn and practice collaboration, facilitation, and communication skills.

Mosaic’s new mission statement: Mosaic’s mission is to build a more resilient workforce through access to soft skills education and actionable information about workplace rights. Our vision is to make the future of work more participatory and decent for all.

More to come tomorrow!

4/13 Evaluative Research Presentation

Phase 4 Presentation Slides

Scenes from Phase 4 Presentations

Presentation Feedback

We got some great feedback from our Index critics, professors, and classmates. Key points included:

• Given that there isn’t a single definition of soft skills, how could Mosaic be the standard?
• One of your biggest stakeholders is companies, and it would be worth thinking about how you can incorporate them into your business model and value flow.
• Your theory of change is clear. Have you thought about your theory of action, or how you go from initial concept to launch and adoption?
• You’ve introduced a series of learning content on collaboration. What’s the detail there? What happens during those modules?
• What will Mosaic look like when you prototype it in more detail?

Phase 5 here we come!