Reflecting with AI: A Tool to Develop Human Intelligence
Reflection is a key component of experiential learning at the d.school, and developing reflective skills is an important learning outcome in itself. I’ve created an AI tool — called Riff — to help people get better at reflection (try it here). This tool is part of a broader — not new — question of how technologies can transform the ways we teach and learn.
In d.school classes and workshops, learners from Stanford and beyond reflect at different moments during an activity (not just at the end). Reflection helps them notice their first-hand experience and contrast it with that of their peers and with theories they might pull from different disciplines. Reflection comes in many forms:
– Individual reflection allows each learner to connect the activity to their past experiences and knowledge, and to process it at their own pace
– Reflection in pairs or small groups further helps learners process personal experiences, and also introduces others’ experiences for contrast. Constructivist listening can be used here to amplify the value of this interactive reflection.
– Reflection as a whole group (which may be called discussion or debrief) enables social learning and allows the teacher to situate the activity and learners’ reflections in the context of goals and frameworks they set out to explore with the design of the activity. Additional learning outcomes beyond the teacher’s original intentions may be (and often are) discovered during this layer of reflection.
Experimenting with reflection in learning
For the past few years, I’ve been trying different approaches to help students develop reflection skills and habits, within my courses and, more recently, in asynchronous, self-directed ways. For that, I’ve used rule-based conversational agents (aka chatbots) to guide students through reflection exercises. (You can read about this in this interview with Stanford’s Digital Education. Many of these exercises also appear in my upcoming book Experiments in Reflection).
The recent advent and wide accessibility of more powerful AI technologies for natural language understanding (NLU) allowed me to use conversational agents to build on my previous reflection experiments.
Riff — an AI chatbot — is designed to augment individual reflection with questions that invite the learner to go deeper in the initial exploration of their experience.
These questions are dynamically generated by a GPT 3.5 Large Language Model (LLM) starting with whatever a learner chooses to input.
As an example, as part of a recent workshop I did with a group of college students visiting the d.school for the first time, I invited them to use Riff for about 5 minutes (anonymously) after a tour of the space and a design activity in teams, and before we dove into a group discussion of the experience. Below is the conversation one student had with Riff:
Student: I like how the space provides support for innovation. It’s helpful because it boosts productivity. I also enjoyed how hands-on the activity was. This activity helped me really dive deep into ideas and being able to see them helped me expand my thoughts. I also liked how the sticky notes were easily rearrangeable. I enjoyed the space.
Riff: What was it about the hands-on nature of the activity that helped you to dive deep into your ideas?
Student: I liked being able to see them and write them out. Whenever I write things out it makes me remember them rather than using a computer or device.
Riff: That’s interesting. How do you think your reflective process might be different if you used a computer or device to record your ideas instead of writing them out on paper?
Student: I think I would’ve forgotten some key insights. Computers and devices remind me of assignments rather than research I care about.
Riff: That’s a great insight! It seems like you’ve discovered that the physical act of writing helps you remember and feel a deeper connection to your ideas. Are there other tactile or sensory explorations that you find useful in your reflective process?
Student: I like the personalization of this bot. I didn’t think it would ask questions related to my responses.
Riff: Thank you for your feedback. However, as a reflection assistant, it’s important for me to be responsive to your needs and perspectives. What other aspects of this reflective experience have been helpful for you?
Student: I think being able to work in a team was helpful. I liked sharing my ideas with my partner and hearing his ideas. I like diverse perspectives.
Riff: That’s great to hear! Working with a partner who has different perspectives and ideas can help you think more broadly and creatively about a problem or challenge. How do you think you can continue to incorporate this type of collaborative thinking into your future reflective processes?
Student: I think I would like to work with more diverse groups to solve certain problems or issues. Specifically in a college environment it’s important to hear what others have to say and it’s also important to be heard. I think the visual aid component of this exercise can also be used
Riff: That’s a great idea! Working with diverse groups can help you gain a more nuanced understanding of complex issues and can help you develop empathy for others. You mentioned that the visual aid component of this exercise can be useful in future reflective processes. How do you think the use of visual aids can aid in reflection?
Student: It helps me remember what others said. I’m a visual learner and also have photographic memory so it stays in my brain longer when it’s visual.
[At this point I asked the group to wrap up their conversations and move to discuss as a group, building on their individual Riff-mediated reflections]
From the conversation you can see that Riff is asking the learner to:
– Elaborate and be more specific in describing what they noticed from their experience (“What was it about the hands-on nature of the activity that helped you to dive deep into your ideas?”)
– Contrast the current experience with other past or hypothetical experiences (“How do you think your reflective process might be different if you used a computer or device to record your ideas instead of writing them out on paper?” and “Are there other tactile or sensory explorations that you find useful in your reflective process?”)
– Go beyond the first thing they noticed (“What other aspects of this reflective experience have been helpful for you?”)
– Think of how their future actions might be different (“How do you think you can continue to incorporate this type of collaborative thinking into your future reflective processes?”)
While I have yet to conduct a controlled experiment, my hypothesis is that, all other things being equal (time, context), these personalized and conversational nudges result in a deeper reflection than a response to a single reflection prompt (even if this prompt included all the nudges above).
How technology impacts learning
The study of the effects of technology in learning and the evolution of intelligence is not new. Developmental psychologist and learning scientist Roy Pea from Stanford’s Graduate School of Education outlined a model of distributed intelligence. In a book chapter from 1993 (see PDF here), Pea posits that “the mind rarely works alone” but often acts in concert with other minds as well as environments and situations. He proposes that we “should reorient the educational emphasis from individual, tool-free cognition to facilitating individuals’ responsive and novel uses of resources for creative and intelligent activity alone and in collaboration” (emphasis in italics is mine). In that sense, my hope is that learners’ use of Riff results in deeper reflections when using it, as well as developing their ability to reflect on their own, with or without the aid of tools and frameworks (including Riff), as well as with other humans. We are gathering data that will allow us to see if this is happening and how.
2022 was the year in which “generative AI breaks into the public consciousness” (1), with the release of text-to-image models like DALL-E 2 and Stable Diffusion, and chatbots like ChatGPT. ChatGPT in particular, which was made available to individuals at no cost, is the fastest technology to date to reach 100 million users (it took it two months, while TikTok achieved that milestone in 9 months and Instagram in 2.5 years). Many students were quick to realize that they could outsource homework to ChatGPT, which generated panic among educators, calls to ban the tool, and efforts to develop methods to detect its use similar to plagiarism detectors. While passing something composed by someone else –be it a human or an AI- as your own is unethical, there is an opportunity here that educators should not miss: we can bring the students along as we figure out how to take advantage of these new technologies to produce better outputs and, importantly, to evolve our human intelligence.
Even as the education system insists in assessing the performance of students as individuals, in the real world activities are more often than not collaborative and depend on the use of tools and consulting resources that we can’t (and shouldn’t) memorize. AI systems are now part of that set of tools, and we have an opportunity to invite students to use them mindfully and with the goal of figuring out how they might contribute to their own growth as intelligent beings. In this post, Todd Mitchell, Director of the Beginning Creative Writing Teaching Program at Colorado State University, describes how he explored the possible uses and limitations of ChatGPT with his creative writing students. Going back to Roy Pea’s distributed intelligence framework, we should strive to build an education system that nurtures “the learners’ attitudes and talents in designing distributed intelligence for their use and that of others, not only to participate in the designs of distributed intelligence provided by others” (2).
Riff turns the tables on the human-bot interaction
So, how is Riff similar to or different from the star of the moment, ChatGPT? Like ChatGPT, Riff is a conversational agent (or chatbot). Both are also powered by a LLM from OpenAI (currently Riff uses GPT 3.5 turbo, although that could very well change by the time you read this). But Riff turns the tables on the human-bot interaction when compared to ChatGPT. People ask ChatGPT questions (to which it might give plausible sounding but factually inaccurate answers) or ask it to perform tasks (like composing an essay or writing a block of code). Riff asks you questions, riffing off your initial reflection, and continuing to ask follow up questions as you respond, as you can see in the example I offered above.
The goal of using Riff is not to replace conversation and reflection with other humans but to augment it. In the example, Riff was used to promote individual processing before a teacher-guided group debrief. Another teacher-in-the-loop use is as homework after a class, which gives the learner the opportunity to reflect at their own pace. One feature of the tool that should be noted is that it captures the text conversations (anonymously) and the teacher can access them (this is discussed with the students in advance). The reflections provide the teacher with valuable feedback on what the students took away from a given activity. A recent event of my program, the University Innovation Fellows, allowed me to test this use case. We asked the close to 300 participants to reflect with Riff on a memorable learning experience from a day packed with experiential learning workshops. The reflections were then sorted according to the experience that was mentioned and shared with the facilitators as anonymous feedback.
These fast-evolving AI technologies are orders of magnitude more powerful than any technologies we have used to date to augment cognitive processes. Consequently, it is important that we proceed with care and establish the proper guardrails to keep learners safe.
In line with the philosophy of transparent pedagogy, the collection of reflections gathered as part of an activity can be brought back to the class for students to analyze, synthesize, and reflect further. In addition, we are currently working on a feature that would add a layer of AI-mediated synthesis of the reflections. This would take place across a group of learners for a given activity/class, as well as longitudinally over a series of activities/classes, and can be used by the teacher to assess how learning goals are being met (or to discover new ones).
Don’t take my word for it. Try it yourself!
Stay tuned for additional articles that follow the development of Riff and its use in different learning scenarios. But before you go, I invite you to reflect with Riff on a recent experience or a favorite topic of yours (you could also reflect on what you are taking away from this essay: did it spark any new questions, insights or ideas?) Use this link (be mindful that I’ll be able to read your reflections, but they are anonymous).
Finally, educators may request early access to the platform here. This includes the ability to customize your Riff bot with your own introduction, initial reflection prompt, and the final message/call to action your learners will get when they wrap up the reflection. In addition, you will get real-time access to the reflections contributed by your learners and the ability to generate a summary of all the reflections for each bot session, which allows you to quickly identify themes and insights to make your students’ learning visible and improve your teaching. Since there is still a lot we don’t know about how humans interact with conversational generative AI, we are limiting the use of Riff to learners 18+ years old to mitigate risks.
Read the next article in this series: Human-AI Collaboration: Establishing ‘Rules of Engagement’.
(1) Quote from the 2023 AI Index recently released by Stanford’s Human-Centered Artificial Intelligence.
(2) Pea, R. D. (1993). Practices of distributed intelligence and designs for education. In G. Salomon (1993). (Ed.), Distributed cognitions: Psychological and educational considerations, 47–87. New York: Cambridge University Press.
About the author
Leticia Britos Cavagnaro, PhD, is a scientist-turned designer with a knack for defying assumptions about what education looks like and what it can achieve. She uses emerging technologies to create learning adventures that transcend the walls of the classroom and empower learners to be self-directed, action-oriented, and reflective. Leticia teaches Advanced Reflective Practice to graduate students from Stanford’s Design Impact MS program. She is founder and co-Director of the University Innovation Fellows, a program of the Hasso Plattner Institute of Design (d.school) that empowers students to be co-designers of their education. Leticia works with educators from hundreds of schools and across disciplines in transforming their teaching practices by applying design abilities and pedagogical levers.
Riff is being developed in collaboration with Stergios Tegos from Enchatted.
Leticia’s book Experiments in Reflection is coming out November 28 2023 (👉🏼Pre-order here).
Chat with Riff here: riffbot.ai/demo/
Request early access to Riff customization platform here: riffbot.ai
