Week 11
Formatting our evaluative research survey and making it live
Survey outline and refinement
In developing the outline for our survey, we spent significant time considering the structure and what it was we were comparing. Was it two different problem framings? Two different approaches to an ARC Institute concept? How could we make the scenarios tangible—that is, show that our concepts help educators adapt and be resilient to a particular issue—without diverting participants’ focus to the issue we used as an example?
We decided to test systems of concepts, since the ones we had developed so far (ARC Hackathon and ARC Council) did not make sense when isolated from one another. This would give us an opportunity to provide some structure to the individual components while also maintaining focus on the overall system shifts we were proposing.
Working off of the vision plan description for ARC and the design principles/goals we had laid forth from generative research, there were three main outcomes we wanted to enable with each concept system: schools and districts are able to collaborate, teachers have the space to experiment, fail, and try again, and teachers have the means to weather external pressures and circumstances.
We wanted to focus on qualitative rather than quantitative responses to these concepts, but knowing that our audience was online and could thus opt out of the survey at any time, we also wanted to keep the survey short. We decided to use the Tripetto platform so that participants could make a selection in the concept system that seemed more appealing at first glance, leave qualitative feedback on those concepts as well as the system as a whole, and then proceed to the next concept or end the survey, depending on their inclination.
Even so, it was challenging to keep the concepts high level! We knew we were not at a stage to be testing minute details (like tablecloth colors at the ARC Summit, for example) but in developing storyboards it felt impossible not to at least consider some variations on the details of the concepts. We made a concerted effort to keep our storyboards short, however, especially since testing concepts online meant we would not have a captive audience.
Storyboards
We honed storyboards for two concept systems, each with three components. One was a “reflective” future in which daily reflection prompts, cross-school educator partnerships, and distributed breaks with built-in teacher work time prompt ARC through self-awareness and increased capacity to innovate.
The other was a “collaborative” future — one in which an ARC council at the district level researches and identifies areas for innovation, leading to ARC Summits during which groups can work on proposals for moving the needle on these challenges, and subsequently ARC Achievement awards that provide space, time, and resources to implement winning solutions.
It was challenging at first to separate the overview of the concept system from its individual parts; we wound up including a three-panel summary of the concepts drawn from their individual storyboards before diving into the individual concepts, as well as with a brief, written review of the system at the end before participants responded to questions about it.
We wanted our storyboards to look low-fidelity in order to indicate the unfinished nature of the concepts, so we made black and white drawings in procreate to accompany our storyboard text. We embedded them individually into the survey so that participants on mobile phones could still see them full-width.
Launching the survey online
Overall we’ve had some of our past participants contribute feedback, as well as 10+ participants from various online sources.
We knew that Reddit has been a platform in the past where students have been able to source a sizable amount of feedback, and so we decided to post to education related forums on Reddit. We initially hit a snag (r/teachers doesn’t allow direct links to surveys; the moderator still has not clicked to approve our post to be shown sans-link, despite their written approval in direct messages with our team), but were able to find several education forums without rules about surveys, which has led to more than 10 participants so far filling out the survey.
Facebook: Local “Buy Nothing” Groups
Chris and I also posted to our local Buy Nothing groups, where an ask of time/expertise is perfectly valid. For some reason it seemed my post was getting removed, but Chris got some traction in his group as well, and one educator said she’d also share it with her grad program.
Survey Swap and SurveyTandem
We signed up for two platforms that allow you to post surveys in exchange for taking surveys. From what I can see in the analytics, however, these have not been very fruitful; at most we’ve had one response total from both platforms. Many surveys seem to be posted by other students; I wonder if the qualifications about working in education, specifically, have limited the number of applicable participants on these platforms.
Next up: concept refinement and prototyping
We’ve begun to receive feedback on our concepts, and it is very mixed. We will no doubt start by affinity mapping what we have heard to determine the most important concept areas to address and improve. We’re feeling grateful to the educators who took the time to fill out our survey — most went above and beyond to review both concepts and leave their email addresses, as well, giving us an opportunity to follow up as we test and refine. We will have some significant work to do to refine the elements of an ARC system, ensure we are incorporating PPS’s equity lens, and also prototype the details of a single artifact from within that system, but we are up for the challenge.
