Grammarly vs. Hairston
I’ve been meaning to test online “grammar checker” Grammarly for some time. As I’ve said before, everyone wants data, but no one wants to pay for research. But I so often refer to Maxine Hairston’s 1981 College English article on error that I thought it would be fun to see how sentences from the QUESTIONNAIRE ON CONVENTIONS OF GRAMMAR from that article rate, according to Grammarly.
(NB: Grammarly — which I was forced to install in my browser in order to access its free service — has flagged the word “error” in the previous sentence. It suggests I add an article, such as “the” or “an,” before the noun.)
The tl;dr version of Hairston’s article is that, if you give 84 readers from a variety of backgrounds 65 sentences containing some form of “error,” you will find that some errors bother some people more than others. Not all of the sentences in Hairston’s survey contain what everyone would call an “error.” The use of singular they, for example, is far less controversial now than it was in 1981, even though some readers still object to it. But Hairston was gathering data on readers’ affective response to error, so her survey is a good quick measure for a tool like Grammarly, which, according to the Forbes blurb it reproduces on its results page, is worth paying money for because it will help you “avoid looking like a fool.”
I don’t have time for comprehensive analysis (and I didn’t specialize in anything grammar-y like linguistics, so my analysis wouldn’t be technical anyway) but I’m going to post screenshots of all 65 sentences from Hairston’s article, as assessed by Grammarly, in case others might want to analyze them. Even without systematic analysis, it’s an interesting set of data.
I’ll start with Sentence 55, because it is my hands-down favorite:
“Good policemen require three qualities: courage, tolerance, and dedicated.” A fairly egregious error in parallel construction which Grammarly misses entirely, instead warning us that something we’ve said in this sentence, extolling the virtues of good policemen, is “politically incorrect.” Presumably what they would advise us to do, if we paid for their Premium Alert service, is change “policeman” to “police officer.” That’s not bad advice, but it would still leave you looking like a politically correct writer who hasn’t mastered parallelism.
OK, here are the rest of the sentences, in order. Sentence One:
I’m fuzzy on the specific error here; possibly “whomever” is preferred to “whoever”? Grammarly, however, seems to have keyed in on the passive voice, flagging a supposed “misuse,” but coyly hiding the details behind a paywall. As we’ll see below, Grammarly’s “premium” paid service is required if one wants more detail about everything from comma splices to cryptic “word choice” problems.
Next up in Sentence Two, it looks to me like we have an interrupting phrase here, “not anger,” which I would probably enclose in commas. Grammarly says it “ran hundreds of checks” on this sentence and Nah bruh, no big deal:
In the next entry, I detect a fused sentence — a common form of run-on. Grammarly has no problem with the sentence boundaries but instead furrows its brow over the possessive pronoun his:
Notice that Grammarly is again recommending substituting an article for some other kind of word. It would be interesting to know, in a world where research is funded, how often Grammarly does this, and what the net effect is on specificity of language when that recommendation aggregates over thousands of instances.
Oh look, in the previous sentence, Grammarly doesn’t like the use of “specificity” without an article:
In Sentence Four, Grammarly overlooks some missing quotation marks and instead tells us the comma before the quote is unnecessary:
In Sentence Five, Grammarly avoids (wisely, IMO) entering into the “between/among” fray:
In Sentence Six, Grammarly overlooks a comma splice. To be fair, it’s not one I find especially bothersome:
Hairston’s survey respondents apparently felt much the same; 23 of them said the sentence didn’t bother them, while 31 said it bothered them a little, and 20 reported that it bothered them a lot. Compare that to the fused sentence from Sentence Three, which bothered 61 people a lot, 13 a little, and only left five people unbothered.
Sentence Seven employs a singular they, which Grammarly may or may not want to talk to us about. We won’t know unless we pony up some cold hard cash:
Hairston had to discard her eighth sentence due to a typing error, bringing us to Sentence Nine, which seems to correctly identify the use of a modifier with “unique.” I’m not sure why “wordy sentences” is also listed among the “alerts.” That’s a pretty short sentence. But they’re probably going to counsel the big spenders who pay for Premium to cut the “most,” so I guess that’s a way to reduce wordiness:
Sentence Ten: No problems here! Unless you care about parallel constructions, which 43 of Hairston’s respondents cared about a lot:
No idea what the problem is supposed to be in Sentence Eleven; hardly any of Hairston’s original respondents were bothered by it:
In Sentence Twelve, Grammarly shows that for all its obsession with word choice, it is capable of missing some obvious boners:
Next, Grammarly correctly flags a sentence fragment, but want us to pay for the SECRET KNOWLEDGE required to fix it:
In Sentence 14, Grammarly latches onto a dangling modifier and diagnoses it as two discrete “issues,” which is fine, I suppose, if you’re trying to sell solutions:
In Sentence Fifteen, we see that Grammarly is pretty bad at detecting homophones:
I assume the problem in Sentence 16 is supposed to be “than” for “from”:
Situations like this make me realize that when we bring up “word choice,” students are likely to assume we’re talking about synonyms and shades of meaning (especially college students, who are learning lots of new discipline-specific vocabulary and also discovering additional nuances in words). Yet very often, “word choice” problems are more idiomatic, especially in regard to the prepositions that certain words are expected to take.
Maybe Grammarly covers all of that behind their paywall.
Next, a simple matter of capitalization, which Grammarly handles ably, and for free!
Then, we’re back to pay-go correction for sorting out a fairly simple run-on:
Here’s an easy win for Grammarly in Sentence 19:
Followed by an embarrassing whiff in Sentence 20, where Grammarly misses a clause calling for a comma and instead recommends a comma to indicate . . . direct address???
Another easy point on Sentence Twenty-one, which highlights the kind of error Hairston referred to as a “status marker”:
An easy call to lose the colon here:
Another singular they, which I’m happy to see Grammarly sanctioning (unless it’s just missing it):
Here, I’m not entirely sure we have a clause issue. I think this is just a list in a modifying phrase. Unless we’re directly addressing a prosperous and sports-minded reader, and telling him we direct our advertising to the young:
Sentence 25 is another mystery. Is it wrong to equate a “situation” with a patient”? Can a situation not be described by “when”? I’m inclined to agree with Grammarly here:
Subjective/objective pronoun confusion! Admittedly, that’s a pretty easy one to spot, even for humans. Hairston herself was surprised how little these errors bothered her readers, but noted that she was sure they noticed them:
Another sentence fragment, another polite reminder that only the elites are granted help with their sentence fragments:
I’ll admit I laughed at this one. Grammarly had been doing pretty well on the ten sentences or so before this one. But come on:
Another blooper. Shouldn’t detecting an interrogative be a fairly simple task for a computer? Humans have little trouble with it.
Here, Grammarly apparently misreads a short sentence with a long introductory clause as an incomplete sentence:
While here, Grammarly is simply recognizing that most people now use “data” as a singular noun, and no one outside the sciences cares much.
I don’t think Hairston included simple use of the passive as an “error,” so I’m not sure what the error here is supposed to be. Grammarly thinks it knows, but I left my wallet in my other pants so I guess I’ll never know for sure.
Here’s a pronoun reference problem that Grammarly fails to notice:
…and another successful diagnosis of objective/subjective pronoun confusion:
I simple double negative — another of the status markers Hairston singled out in her analysis. Status markers are errors that cause a particularly violent negative reaction in the reader. Since status markers tend not to be errors that impede comprehension, they are more often an indicator of how the reader perceives the writer’s socioeconomic status:
In Sentence 36, Grammarly correctly spots a problem with agreement in number:
. . . but then gives the all-clear to a sentence with only half the necessary quotation marks:
Another bad miss on unnecessary verb tense shifts:
And this, which to my eye seems to be a problem with the subjunctive, comes out of Grammarly’s black box labeled “inappropriate colloquialism.” It’s a real shame they want me to pay to find out more because I’m dying to know what’s colloquial in that sentence:
Finally, an easy catch in Sentence 40:
. . . and another easy catch in 41:
Wrong verb form in Sentence 42 — another status marker:
Here, as with the missing close quotes above. Grammarly seems unaware that parentheses need to appear in pairs:
Pretty obvious sentence fragment — Grammarly has been doing pretty well on these in the sample:
Sentence 45 is reminiscent of the “situation” with the noncomplying patient, above. Here though, Grammarly thinks there’s a problem with word choice. The only change that suggests itself to me is to make “refused” to “refuses,” to keep the whole thing in present tense. Not sure why that’s considered “word choice,” apart from the obvious fact that changing anything in any sentence beyond punctuation is going to mean choosing different words.
Another double negative; Grammarly does fine with these:
Grammarly gives Sentence 47 the all-clear, and I probably would too, though I assume Hairston would want “our” instead of “us”:
Another introductory clause that seems to confuse Grammarly into seeing a compound sentence:
I’m all for authorial discretion when it comes to commas, but I’d probably suggest this author set off “no matter how good his record” with commas. Read it aloud, you can see they belong there. Grammarly disagrees:
Here, Grammarly catches the obvious capitalization error in “texas” but doesn’t seem to grok the rest of the title:
Pretty obvious modifying phrase problem here, but Grammarly flags it as “word choice”:
Here, Grammarly catches a singular verb for a compound subject:
. . . but in the next sentence, Grammarly fails to see the need for a singular verb when a compound subject isn’t plural:
Plural they again, which Grammarly has been ignoring, but here decides that we really must talk about that passive construction, please put your money in the slot:
I covered Sentence Fifty-fiveback at the beginning. Fifty-six resulted in a good catch for Grammarly, though I think you could just set up an alert to flag this word any time anyone uses it and half the time you’d be right that it’s the wrong choice:
Finally, a simple comma splice that Grammarly seems to have pegged as a comma splice:
An easy capitalization fix in Sentence Fifty-eight:
In Fifty-nine, Grammarly catches the everyone/everyone confusion and also puts a red line under are, though it doesn’t say anything explicitly about singular/plural verb forms:
Sentence Sixty contains a subjective/objective pronoun that Grammarly apparently doesn’t care about. Neither do I, but it’s an odd miss given that the software caught others elsewhere:
Extraneous commas are one of the simplest things writers can look for and remove to declutter their writing. Grammarly is apparently fine with clutter:
Another status marker, correctly identified.
Its/it’s again — Grammarly is ON IT:
Definitely a fragment here, though what the word choice issue is I confess I can’t tell:
Verb form error, correctly diagnosed:
And we end on a comma splice. I’ll be charitable and assume that after Grammarly explains your third or fourth comma splice, it gives you a reduced Premium rate:
Given what we know about patterns of error and writers’ ability to reduce error over time with guided practice, I’m curious about the pricing model for tools like Grammarly. A good grammar checker, to my mind, is one that would help writers learn to avoid mistakes in the first place, and so become less dependent on editorial assistance. But if a tool like Grammarly does help people learn grammar, they won’t need Grammarly anymore. Giving people usable, transferable skills to improve their grammar would seem to undercut their business model.
It’s entirely possible (likely, in fact, given what I’ve found looking at similar products) that Grammarly simply isn’t very good because good writing rests on human interpretations of meaning, which machines are notably bad at. But, we should never assume that any software purporting to assist writers is truly “educational” in its intent — especially if it is a for-profit product. Such a tool generates income when it creates dependency, not autonomy.