Design Research: Understanding Portfolio Making at KAS
Authors: AnnaB, with GyuEun Park, Miaojun Xu and Yu Zhao
Where are we in the design process?
The design research phase is an iretative process of research and synthesis. We conducted three user studies — insights gained from one research will inform the planning of the next research.
What is the outcome of the phase?
The outcome of design research phase is a narrowed down problem space and three design principles that will guide the ideation and prototyping phase.
User Study 1: Understanding the Landscape of Portfolio Making
Goal: The goal of this research was to understand the landscape of expert portfolio making. The good practices of expert portfolio making can be translated to a middle school context; while the challenges expert still face can help us leverage what is important and what is achievable. Meanwhile, the results could be of potential interest to the teachers, who might be interested in using the results to guide students to better build a portfolio.
Method Selected: We used contextual inquiry as our research method. It involves observing users and asking users questions as if they are the masters and we are the apprentice. We asked the participants to walk us through how they made their portfolio. We paid special attention to their motivation, tool in use, documentation process, creation process and evaluation. We also asked questions about their challenges in the process.
Participants: We conducted four contextual inquiry sessions with experts in the field of design. They have experiences ranging from 2 to 18 years. They are all students at CMU — one is an undergraduate, two are master’s students, and one is a PhD candidate.
Findings: We found that experts create portfolios because to portray their thinking processes, abilities, and final products in one place. The main reason they create the portfolios is to get jobs. A good portfolio should lure people in, get people interested in you, and finally get people to call you for an interview. Expert portfolio makers keep this purpose in mind during the entire portfolio creation process.
Experts also try to align the choice of platform with their purpose. Most experts choose a digital medium to host their portfolio for its accessibility and their job-hunting purposes. Experts select the projects to be shown based on levels of relevance, completeness, skills included, contribution and strength, but also based on whether they like the projects or not.
“I think the big thing is you have to consider what the purpose of your portfolio do you want it to be” — P4
Experts agree on documenting during making. The most common method of documentation is taking pictures. They take pictures of whiteboards with sticky notes, collaboration moments, and sketches with varying levels of fidelity. They also ask others to take pictures of themselves, partially because they are focused during the process and would rather not be interrupted. They emphasize that documenting should not take up too much time, because it distracts from the making.
While editing their portfolios, experts say that condensing is an important phase. It allows portfolios to be simple, quickly consumed, and easily understood. The text should make sense to an audience even if the author isn’t there to explain it. Experts also seek feedback, both formally and informally, for their portfolios.
Implications: The experts have a similar motivation for creating portfolios, which is acquiring a job. This might not be particularly relevant for middle school students, but perhaps we can encourage them to consider why being good at making portfolios now would benefit them at a later date. We may also try to create a motivation that is similar to getting a job within the classroom.
In this day and age, it seems like most experts use digital portfolios, so that is the format we should encourage in students. Students don’t have to worry about picking and choosing projects as much, because it’s likely that the teacher will choose what projects they have to include. However, it could be useful to encourage the students to think about why the teacher wants to include those projects.
Documentation seems very important, but it shouldn’t take up too much time. Perhaps we can encourage the students to help each other document, so the burden isn’t on one student to document their project by themselves.
Finally, students should be able to make their profiles easily understood by outside sources, such as their parents. We should pay special attention to making sure students don’t leave out certain information because they think it’s implied or understood. On the other hand, we should encourage students to be concise and not include extraneous information. Similarly to how experts ask for feedback, we may be able to get the students to perform a peer review and provide feedback to each other.
Synthesis Artifacts:
Our synthesis artifacts include four user profile boards of the four experts we interviewed and a work model of how a typical professional creates and manages their portfolio:
User Study 2: Understanding the Learning Context
Goal: The goal of this research was threefold. First, we wanted to gain familiarity with the educators and stakeholders at our site, Kentucky Avenue School. Second, we wanted to understand their teaching and learning values and organizational goals. Third, we wanted to investigate the underlying causes and consequences of their struggles with portfolio making. With these three goals in mind, we believed we could start exploring a solution space that would be both effective and respectful to the cultural environment of the school.
Method Selected: We started with two directed storytelling interviews. Directed storytelling is a method of ethnographic research, where the researchers understand stakeholder experiences from the personal stories they tell. We sat down with two participants and asked questions about successful and unsuccessful documentation in the past, the challenges they struggle with, the consequences of the problems, and the solutions they have tried. Every question contained sub questions, which we used to prompt the participant and coax more insights from their stories.
Participants: We conducted the directed storytelling interviews with two employees at Kentucky Avenue School. The first interviewee was the school principal, who is also the teacher for middle school maker classes. She has been teaching at KAS since 2006. The second participant was the fourth and fifth grade teacher at KAS. While she does not work in the school’s maker-space, her classes are project-based. She started teaching at KAS fairly recently, in the fall of 2018.
Findings: We found that teachers don’t have a lot of time for documentation and may prioritize other goals. They forget to pause and document, often preoccupied with teaching and helping students. As a result, not enough documentation is produced and the documentation that is created is delayed or incomplete.
“You know things are going well. It’s just hard to like stop what you’re doing, especially because it’s like lessons that are really Hands-On usually require a lot of prep work.” -IP2
We also found that students do not understand the importance of documentation and lack the mindset and attitudes necessary for deep reflection. This is because students don’t receive grades, favor simple tasks, are ashamed or afraid of judgement, and don’t have practice analyzing themselves. It’s also difficult for teachers to ask the students the right questions to reflect. As a result, students don’t learn enough from their process and mistakes.
“It’s hard because we don’t give a grade for maker here. So that’s been an issue.” -IP1
Finally, we found that there are challenges with the amount of space available and the limitations of the tools, such as the bulb digital portfolio application. It is difficult for young students to log into bulb because they have to type in long emails. Additionally, teachers can’t edit student websites and are not using the tool to its maximum potential. As a result, there is less interaction, engagement, and improvement in the portfolio making process. In terms of the room itself, there is not enough wall space to showcase student work, and students tend to move from room to room. This makes it difficult to keep track of student work and properly display it.
Design Implications: Any solution involving documentation has to primarily focus on the student and teacher focus and flow. We may want to explore the possibilities of automated documentation, or a method that reminds the teacher regularly.
Students believe writing long paragraphs is difficult, so the solution should provide enough scaffolding to make it easy and less intimidating. They are also ashamed of failure, so we may want to encourage talking about what went wrong, so we can make them view it as an important part of the journey instead of something that should be hidden.
The problem of space is not something we have control over, so our solution should be primarily screen-based in order to preserve the room that KAS does have. Additionally, the problems with bulb might mean it’s not a satisfactory platform. We may be able to create our own platform that borrows the part of bulb that work well and avoids its problems.
Synthesis Artifacts:
Profiles of two professionals at Kentucky Avenue school were made. We also created a diagnostic mapping the synthesized the problem, cause and consequence identified.
A higher-level of major problems facing the teachers at Kentucky Avenue School is synthesized as the Venn Diagram:
User Study 3: Understanding Students
Goal: The goal of this research was to understand the learners in context. In other words, to gain insight into the student experience in project-based learning at Kentucky Avenue School.
Method Selected: We decided to conduct both an interview and a cultural probe during this user study. First, we conducted a 15-minute interview that contained questions related to the organization of bulb digital portfolios, their strengths and limitations, purpose, and documentation methods such as photographing and writing. The interview was followed by a 5-minute design activity that required the student to draw and write about a tool that would facilitate their documentation process.
Participants: We interviewed four seventh grade students at Kentucky Avenue school. They were chosen by the Principle and taken out of class to participate in our study.
Findings: Students believe that bulb is not intuitive. It lacks personalization and is rather “school-ish.” They also display indifferent or passive attitudes towards reflection. It is perceived as “another version of homework,” resulting in low motivation and attempts to get it over with as quickly as possible.
Students say that writing is not the easiest, most enjoyable step in the documentation process. In fact, students find it either stressful or tedious. They often don’t know what to write, and like when the teacher provides templates or prompts to guide them.
Design Implications: In order to encourage students to reflect properly, we should make our solution fun and interesting. It should incorporate components that are not typically found in a homework assignment. Since students showed positive reactions to the personalization part of bulb, we should incorporate personalization into our final solution.
Students struggle with knowing what to write, so our solution should provide prompts or templates to guide students during this process. We can pull the scaffolding away over time, but we should definitely start with some sort of guidance. Additionally, since students find writing stressful, we may want to consider alternate forms of reflection. It’s possible that a visual or auditory component might make the process easier for students.
Synthesis Artifacts:
A persona of a hypothetical student:
An affinity diagram of the major problems facing students at Kentucky Avenue school:
Area of Focus
Based on these three user studies, we narrowed down our problem space to one area of focus.
We wanted to concentrate on supporting reflection and writing in more depth through scaffolding and customizing goals, incorporating documentation smoothly, designing rewarding experiences, and utilizing peer cultures.
We believe that building this integrated, natural, and rewarding experience for both teachers and students will achieve a quick, collaborative, customized, and engaging reflection process. Through these higher quality reflections, students will be able to better practice their metacognitive thinking, deepen their understanding of what they learned, and transfer related knowledge, skills, and dispositions.
