Designing Your Educational Design Work #4 — I used to think…now I think…

Stanford Graduate School of Education Professor Roy Pea once told me that “learning is a reflective conversation with oneself.” It’s a great statement, one I often refer back to and remind myself of.

What makes reflection so powerful, is that it prompts a process of self-explanation in a learner. But, often that means that the learning is happening inside and we as teachers can’t always see it. So how to make that thinking visible? Well as it happens, Harvard’s Graduate School of Education wrote an entire book called “Making Thinking Visible” and in it published a series of Thinking Routines that educators can use to help see (and get a pulse on) the learning and thinking happening in their classroom.

One of these Thinking Routines is a fan favorite at the d.school: I used to think…now I think…

This particular thinking routine helps students explore and reflect on how and why their thinking has or has not changed. It helps them develop a reflective muscle and is great when used in situations where learners are likely to have had their traditional beliefs, experiences, thoughts, or opinions challenged.

In Rich, Colin and my class, Spring Quarter Design Thinking Studio, students stand in a circle at the end of each week and share an “I used to think..now I think” statement. We capture them. For example: “ I used to think, finding the solution is very important to address a problem. Now I think it’s actually more important to find the problem.”

When we read these, it’s pretty easy to get a sense that some pretty powerful learning happened right? Students worldviews changed, their assumptions were challenged, and they developed a new point of view. But how can we measure that?

Enter sentiment analysis.

A super cool emerging technology that uses machine learning and natural language processing to understand and then categorize the sentiment of written language.

Great idea, but boy to build our own NLP model don’t we need like 5000 data points? Turns out, Yes! And we will get them (since the Fall quarter of 2019, the d.school has been collecting them at the end of each quarter). But before we properly commit to this long journey we decided to prototype what this might look like using IBM Watson’s Tonal Analysis tool. This tool uses Watsons comprehensive NLP corpus to understand sentiment, emotions, and tones in text. One of the great reasons for using Watson is that its corpus includes many different versions and vernaculars of English — so can better account for our diverse global society than many other NLP tools.

So what does that mean for a class at the d.school?

Let’s look at a human-centered example from a real student in our class. Their statement was “I used to think Ambiguity is bad. Now I think it is where the magic happens.” When capturing these statements we only want to measure the changes in sentiment to the student input component; the statement that follows the predefined aspect of the statements.

In this example, we can see that the analysis of the ‘I used to think’ student input identified strong sadness and tentative tones. And, the analysis of the ‘Now I think’ student input statement did not identify either of those tones but identified joyful tones. This suggests that this learner’s thinking and self-reflection about the subject matter changed from sad and tentative, to joyful.

Now what if we look at it at the level of a single class, but across the whole group of students?

April 16th was about two weeks into class. In this class, students were deep into their design project 1 and starting to think about how they would decide what move they needed to make next to get their DP1 to the right level of fidelity. Following the same approach as above, but collating together the inputs from all of the students, the analysis suggests that student thinking decreased in anger and fear (both strongly negative emotions that get in the way of student learning and collaboration) over the course of the class.

And what about across the whole course?

From our class so far (we still have two weeks to go) 99% of “I used to think… now I think” statements collected in our class, show a measurable change in sentiment and tone. 42% of which showed a very significant change (meaning of .6 or more). No matter what the changes in tones are, the fact that there is a measurable change in tones suggests that individual student’s inner persuasive discourse changed over the duration of the course. That is just old school educational scholar Bakhtin’s way of saying, students learned something. Which isn’t as much of a guarantee as you would think.

What do you think? Cool? Interesting? Concerns? Ideas?

This is a prototype; we would love your thoughts.