Explaining the Grade: Auto Essay Scoring and CRASE+
Podcast Episode 6: CRASE+ (Constructed Response Automated Scoring Engine)
In August 2019, Motherboard (vice.com’s Tech blog) published a critical article about automated essay scoring, “Flawed Algorithms Are Grading Millions of Students’ Essays.” The story notes that: “research from psychometricians — professionals who study testing — and AI experts, as well as documents obtained by Motherboard, show that these tools are susceptible to a flaw that has repeatedly sprung up in the AI world: bias against certain demographic groups.”
The bias problem is not restricted to essay scoring and arises from the tendency of artificial intelligence to amplify patterns from data, in this case the human graders that train the essay scoring engines.
[Read a transcript of the show at https://actnext.org/research-and-projects/navigator-podcast-ep-6-crase-plus/]
In this episode of ACTNext Navigator podcast, we’ll go under the hood of ACT’s automated essay scoring engine, CRASE+ (Constructed Response Automated Scoring Engine). Our guests are Erin Yao and Scott Wood. They’ve been working for many years on CRASE+, a product acquired in 2014 when ACT purchased Pacific Metrics. CRASE+ is a writing assessment tool that begins with human graders to develop a rubric. Data from human graders is used to train the automatic grading on a large scale.
We discuss CRASE+, automated scoring, the challenges of automated scoring using natural language processing, how to address the biases in human graders and the “flawed algorithms.”
Scott has a book chapter coming out in February 2020, “Public perception and communication around automated essay scoring,” in the Handbook of Automated Scoring: Theory into Practice.
Want to learn more about CRASE+? Download the product overview here.
The views and opinions expressed in this podcast are those of the authors only and do not necessarily reflect the official policy or position of ACT, Inc.
