Teaching Visualization with Design Studies for Social Good

This post is about our paper at ACM CHI 2020 (Best Paper Award) — Design Study “Lite” Methodology: Expediting Design Studies and Enabling the Synergy of Visualization Pedagogy and Social Good by Uzma Haque Syeda, Prasanth Murali, Lisa Roe, Becca Berkey, and Michelle A. Borkin. To learn more, visit our project website.

The picture shows examples of how students developed novel visualizations using the Design Study “Lite” Methodology. Task abstractions, ideation process design sketches, and final visualizations are shown in the piles from left to right respectively.

TL;DR: Design studies are an integral form of research in the field of visualization where researchers work with domain experts to solve a problem through visualizations. This process can take several months to even years to complete. For this reason, visualization educators typically do not conduct full design studies as part of their curriculum. This exclusion creates a gap between the knowledge acquired by visualization students and the skills to apply the knowledge as researchers. So, is there a way to teach and implement design studies with novice students in just a semester? In our paper, we introduce an accelerated design study framework called the Design Study “Lite” Methodology (DSLM) that can be completed in just 14 weeks.

A design study is a research process in which visualization researchers create data visualizations to solve a problem faced by real-world stakeholders. For example, visualization researchers can collaborate with organizations to visualize the energy performance of large building portfolios like universities and commercial businesses just like Brehmer and colleagues did in their design study paper. They can even collaborate with medical imaging experts to build a novel visualization to help with disease diagnostics as Pandey and others did in their paper. Design studies are an iterative process where the researcher first needs to identify a collaborator to work with, understand and familiarize themselves with the domain problem, and abstract the problem into a form that the visualization researchers can understand and solve. After this, they iterate on the design with their stakeholders, build the visualizations, and finally carry out some form of evaluation to validate the design and usability before handing it over to the stakeholders. Design studies are a time-intensive process that can take months to even years to complete. It can take a long time to execute if, for example, the visualization expert needs to learn essential domain knowledge to understand the problem, collaborators are not able to frequently meet or give feedback, or if technical challenges are encountered during implementation.

As such, design studies are not typically included in visualization course curricula as a semester only lasts ~3–4 months. Our research showed that, out of 29 university-affiliated visualization courses with openly accessible websites, 89% did not have a practical implementation of design studies in their curriculum. These students miss out on an essential component of the visualization discipline where they get to work with real-world data and real collaborators and ultimately contribute to their professional and personal growth.

The Design Study “Lite” Methodology framework.

We contribute in our paper a novel framework Design Study “Lite” Methodology (DSLM) that can accomplish a complete design study in only 14 weeks. It is an adapted and expedited version of the seminal design study process proposed by Sedlmair and colleagues. Using DSLM, students can learn the essential components of a design study and implement these skills in a real-world setting involving stakeholders outside the classroom.

How we implemented DSLM within a semester

DSLM includes seven design study steps spanning an accelerated timeline of 14 weeks. Whereas typical design studies involve identifying a collaborator as well as writing a design study paper as a part of the timeline, these steps are purposefully left outside of the DSLM timeline. This is because, in DSLM, the domain collaborator is chosen by the instructor of the course before the start of the semester. Likewise, publishing a research paper is left optional for the students to work outside the semester since the primary goal is to achieve the objectives of the stakeholders.

Here is a brief breakdown of the steps of DSLM:

The timeline of the Design Study “Lite” Methodology demonstrating how it can be implemented in 14 weeks. The weeks indicate the end of the week, i.e. when the corresponding work is due. For example, by the end of week 5, the final digital sketch should be ready and then from the end of week 5 to the end of week 9, the final visualizations and the report draft are due.

Before the semester

• Step 1 (Choosing a collaborator): This can be tricky because not all problems can be answered through visualizations or be well suited to the course’s learning objectives. This selection requires careful winnowing and can be done best by visualization experts. Since DSLM is about teaching novice students design studies, it makes sense to not overwhelm them with a selection that they are not well equipped to perform.

During the semester

• Step 2 (Abstract): Students spend the first 3 weeks to understand the domain problem. They interview the stakeholders and translate the problems to tasks that their visualizations will answer.
• Step 3 (Design): Once the tasks are identified, the students spend the next 2 weeks iterating on different hand-drawn design sketches for their visualizations. During this time, they also get feedback on their designs from the instructors and the stakeholders. Using the feedback, they finalize one of their designs.
• Step 4 (Build): After finalizing a design sketch for the visualization, the students spend one month (4 weeks) to build it. By this time, the students have already learned the necessary technical skills to create an effective visualization in their class. To demonstrate mastery of the course concepts, the visualizations have technical requirements (in our courses the visualization had to be web-based and interactive). During this time, the students also write the first draft of the outline of their project report.
• Step 5 (Evaluate): In this step, the students get 2 more weeks to keep working on their visualizations and prepare it for a usability test. A full-scale evaluation is not practical to complete within the expedited timeline, so a qualitative evaluation is carried out in the class with fellow students as participants.
• Step 6 (Disseminate): In the final 3 weeks, the students complete their final project report and make modifications (if any) to their visualizations based on the usability test. At the end of the second week, they present their final project to the class, deliver the visualization to their stakeholders, and submit their project report. During the last week, they solicit feedback from their stakeholders to make sure that they have no concerns with the final product.

After the semester (optional)

• Step 7 (Write): The paper or the final design study project report is for the purpose of completing the course project requirements. However, students can still choose to write the design study paper for publication as it is included as an optional step in DSLM.

And there you have it. Design Study in just 14 weeks!

Adopting DSLM into visualization curricula with Service-Learning

Although DSLM can be applied on its own, we implemented it in conjunction with a pedagogical model called Service-Learning (S-L).

The picture shows how S-L meets both learning and community goals.

Service-Learning (S-L) is an experiential learning model in which classroom learning objectives are aligned with community service to fulfill both pedagogical and community goals. Community service, volunteerism, internships, and field education are all different types of S-L. A central element of S-L is reflecting either through written prose or group discussion to enable students to develop critical thinking skills as well as reflect on how their learning and service relate.

We advocate for the use of Service-Learning with DSLM as it provides additional benefits to the whole process by not only helping the students learn the course material but also helping the community. It also greatly simplifies some steps of the DSLM process. During the first step of DSLM, the instructor can choose a collaborator through his/her own connections or through the support of a Service-Learning facility within the institution.

The process that the S-L department of Northeastern University in Boston, Massachusetts follows to identify a community collaborator for the visualization instructors.

For example, the S-L department at Northeastern University in Boston, Massachusetts has partnerships with hundreds of local non-profit organizations. They identify and recommend suitable projects to instructors after a three-step filtering process where they take 3 passes to eliminate and narrow down the list of collaborators for a particular course. Instructors then choose a collaborator from this list for students to work with during the course of the semester. This process starts at least a couple of months before the start of each semester.

S-L also requires the students to volunteer certain hours in their partner organization and reflect on it through written prose. This volunteerism and reflection component of S-L also helps the students to relate deeply with the partner organization and help them better understand the domain problem. This, in turn, increases the success of the projects.

The picture shows the benefits of integrating Service-Learning to Visualization education.

As shared through course feedback surveys, the students liked the Service-Learning component of the technical course. Their comments echoed the following themes:

• It provides real-world data science experience.

“In the classroom, we are often given clean, groomed data to work with, but through service learning, I was able to learn how to work with messy, real data that serves a real purpose.”

• It provides professional experience to students.

“I had a chance to develop professionally and collaborate on a project on a team.”

“I understand how to communicate with people in my community more.”

• It Impacts students on a personal level.

“This experience definitely made the data feel a lot more personal.”

“The power of being connected to an individual’s needs was powerful.”

• Students make a positive impact on the community.

“This makes accurately portraying their data even more important to me, because I now know how important they are to the community and how much these visualizations can help them understand where they can improve.”

To learn more about Service-Learning and for resources on how to implement it into your teaching, please visit our website. More information and thoughts on Visualization for Social Good can be found in this medium post by Michael Correll.

How does DSLM handle typical design study pitfalls?

A typical design study can fall into many pitfalls as mentioned by Sedlmair and colleagues in their paper. However, the systematic layout and the supervision of a visualization expert in DSLM greatly reduces the chances of many of these pitfalls from occurring.

Pitfalls identified by Sedlmair and colleagues in their paper. The red ticks represent the pitfalls that DSLM handles through its systematic structure.

For example, DSLM prevents pitfall #1 (premature advance: jumping forward over stages) from happening in the first place as students cannot jump to another step without completing the previous ones and running it by their instructors.

Likewise, in the winnowing stage, the process of identifying the domain partners by the S-L organizers and the instructor goes through several winnowing stages, so the chances of having a premature collaboration are reduced to a great extent (pitfall #3). The systematic structure and regular feedback sessions from the instructors and the collaborators ensure that the students are on the right track, taking care of pitfalls 18 (learning the domain language too little/too much), 19 (too little abstraction), 20 (premature design commitment), and 22 (non-rapid prototyping).

For a more detailed discussion about how DSLM reduces the chances of regular design study pitfalls from occurring, we refer you to our paper.

Evaluation of DSLM with S-L:

We implemented DSLM with S-L across 5 semesters of visualization courses (3 undergraduate and 2 graduate courses). During this time, students worked with 7 different partner organizations, many of whom were recurring partners across different semesters.

Every semester, students and partners were asked open response questions on their learning experience with DSLM and S-L through a voluntary online survey form. We also interviewed three partners about their thoughts on the whole process. All of these responses were open-coded and 4 high-level themes were identified in both the student and partner responses using thematic analysis techniques. We list the themes with supporting quotes below:

Feedback from Students

• Better Learning of the curriculum

“The service helped in applying course concepts in a real-world setting. We were solving real problems using what we learned in class.”

“The experience and course concepts were directly linked.”

• Impact on the community

“My service has made me interested in the idea of using data science skills to promote social Good.”

“It helped me see the greater picture of what my skills can be used for.”

• Efficiency in project completion

“This model not only gave me good grades, but I never got flustered…”

Amongst the positive comments, we also received some constructive criticisms. One such theme was as follows:

• Disconnect between S-L Goals and Personal Goals

“A more technical project would have been in sync with what we learned in the class and hence a better application of the concepts.”

Feedback from Community Partners

• Effectiveness of the course

“The students bring great technologies and fresh eyes to our challenges.”,

“…it brought out trends in our data in ways we had not seen before.”

• Success of the projects

“The quality matches the standards we display on our organization website”

“It helped us understand the data we have.”

• Learning experience for the partners

“I would change the goals if the students feel they can do a better job of something else and present a case for it.”

Overall, both students and partners expressed positive thoughts on the DSLM implementation with S-L. We also received a few comments for future improvements. For more details and results of our framework’s evaluation, please see our paper.

If you are interested in adopting DSLM in your course, please take note of the following!

Over the duration of the 5 semesters of DSLM implementation with S-L, we learned some valuable insights and reflected on what we could have done better. Therefore, if DSLM seems like something you want to try out, here are a few suggestions from us:

• Implement DSLM with S-L

As we mentioned above, S-L brings bonus benefits to the process! So, why miss out?

• Choose a domain problem that is easy to understand.

Remember that you have a tight timeline, so choose a domain problem that will not require students to do extensive background reading or training with domain experts. In our case, this was relatively easy as most non-profit community organizations focus on relatable topics such as education, volunteerism, neighborhood improvement, etc.

• Work with pre-curated and “clean” data.

Along the lines of the previous suggestion, it is best to choose a domain partner who can provide “relatively clean data” that will not be a time sink for students to refine. It is important to note that, even “relatively clean data” will need a lot of processing as well but aim for a balance where this does not become excessive.

• Structure and limit the amount of iteration

Design studies are iterative and lengthy. To provide iteration, but limit its duration, a pre-set number of feedback sessions with the collaborators should be determined at the beginning of the semester. This might hamper the full potential of the design study, but is a necessary compromise for an accelerated timeline.

• Setting the stage right and plan ahead

If you are not using a Service-Learning facility to identify domain partners, make sure you are doing it way ahead of the semester. This will provide ample time to make sure that both you and the domain collaborator are on the same page.

• Adjust pre-requisites to adapt to different semester lengths.

If you have a long semester, then students can have fewer prerequisites (prior skills) as there will be enough time for the instructor to teach and students develop skills. If you have a shorter semester, then increase the prerequisites so that students are adequately prepared and technically well equipped to tackle the problems.

• Set realistic expectations and have signed contracts

It is very important for partners to understand that the student projects might end up not meeting the level of sophistication they are looking for. Likewise, it is also imperative for students to understand that these are real stakeholders and they need to have a quality deliverable by the end of the semester. For this purpose, a signed contract agreed upon between the students, instructor, and organization is essential.

• Assign two groups to the same organization if possible

Another strategy to help alleviate the possible failure of a project is to assign at least two project groups to each organization.

• Have more tightly defined guidelines.

This will ensure a more streamlined project execution. Some partners expressed that they prefer more straightforward deliverables and that graduate students tend to make more exploratory visualizations. We attribute this to the fact that graduate students were less strictly guided in each step owing to their expertise compared to the undergraduates. However, in future implementations, we will make sure that all projects are strictly guided irrespective of their levels.

• Define clear Intellectual Property (“IP”) and data use guidelines.

Each partner organization in our implementation was required to prepare necessary privacy and/or data use consent forms for their collaborating student groups. This is very important to make sure that the final visualizations created by the students are used under fair policy.

• Maintain effective communication

Although a no-brainer, we feel that a reminder to maintain effective communication both with students and community partners is absolutely essential for a successful implementation of DSLM.

Give your students a taste of real visualization research!

Design studies are a canonical form of research in the visualization field. The goal of DSLM is to empower students to step into the shoes of visualization researchers, and with S-L create a positive impact in the community. Even though it is a simplified version of an actual design study, it exposes them to the process rather than a lack of design study experience as is the case in current visualization courses.

We also look forward to future work implementing DSLM in large (50+ students) classrooms, as well as apply DSLM for expedited successful research projects (e.g., semester-long internships).

Data visualization has the power to produce social impact and positive change, and students have great potential to contribute to it as budding visualization researchers. Don’t you agree?

This post is about our paper at ACM CHI 2020 (Best Paper Award!) — Design Study “Lite” Methodology: Expediting Design Studies and Enabling the Synergy of Visualization Pedagogy and Social Good by Uzma Haque Syeda, Prasanth Murali, Lisa Roe, Becca Berkey, and Michelle A. Borkin. To learn more please visit our project website. The website has links to our paper and supplemental materials and more information on Service-Learning. Also, keep an eye on the Youtube channel of ACM SIGCHI for our upcoming video presentation.

Thanks to Prasanth Murali, Michelle Borkin, Enrico Bertini, and Jessica Hullman