AI Grading: Can Machines Really Evaluate Student Writing?

Gradescope claims an 80% reduction in grading time. Turnitin's Clarity suite made TIME's Best Inventions of 2025. GPTZero pivoted from AI detection into AI-powered grading. CoGrader offers rubric-based essay evaluation. The market is booming, and for teachers buried under 150 essays every weekend, the pitch lands.

But the research tells a more complicated story than the marketing does. It involves score compression, racial bias, and a persistent inability to recognize the best (and worst) student writing.

Adoption Is Outrunning Guidance

A 2025 Gallup/Walton Family Foundation survey of 2,232 K-12 teachers found that 60% had used AI tools during the 2024-25 school year. Among the 32% who use AI weekly, teachers reported saving an average of 5.9 hours per week, the equivalent of roughly six full work weeks per year.

Those numbers sound transformative. But 68% of these teachers received no AI training whatsoever, and only 19% work at schools with any kind of AI policy in place. Teachers are adopting AI grading tools largely on their own, without institutional guidance on where these tools work and where they don't.

How Accurate Is It?

The answer depends on what you mean by "accurate" and what kind of writing you're evaluating.

The Good Numbers

A 2024 study from ETS (the organization behind the GRE and TOEFL) by researchers Zhang and Johnson analyzed 13,121 essays and found no significant population-level difference between human and AI scores. On one essay set, AI scores fell within 1 point of the human score 89% of the time.

But that number dropped to 76% on other essay sets, a meaningful decline when the difference between a B and a C can change a student's GPA, scholarship eligibility, or self-confidence.

The Less Good Numbers

A 2025 study by Floden, published in the British Educational Research Journal, examined 463 exam responses graded by both ChatGPT and teachers. Seventy percent of ChatGPT's gradings fell within 10% of the teacher's grading. Only 31% were within 5%.

That means for roughly two-thirds of student work, AI grading was in the right neighborhood but not precise enough to use without human review.

But Humans Aren't Great at This Either

Before judging AI too harshly: human raters achieve only about 50% exact agreement when scoring the same essays independently. Grading has always been more subjective than we like to admit.

So AI doesn't need to be perfect. It needs to be good enough, and unbiased enough, to be useful. That second part is where things get difficult.

The Score Compression Problem

One of the most concerning findings comes from the Learning Agency's 2024 analysis of approximately 24,000 student essays using the ASAP 2.0 dataset. When ChatGPT graded these essays, it compressed scores heavily toward the middle of the scale. It handed out a disproportionate number of Cs, rarely gave the lowest scores, and never awarded the highest score (6) on the rubric.

This compression does two things. It penalizes excellent students whose best work gets rated as merely good. And it inflates weak writing by pulling low scores toward the middle. In practice, AI grading can't distinguish great writing from poor writing. It sees everything as roughly average.

For formative, low-stakes feedback, this compression might be acceptable. For summative assessments that affect grades, transcripts, or college applications, it is a serious problem.

Who Gets Scored Lower

The bias findings from a 2024 ETS study are worth looking at closely.

GPT-4o scored students from all racial groups an average of 0.9 points lower than human raters on a 1-6 scale. But the gap was not uniform. Asian American students experienced the largest disparity, receiving scores of 3.2 from AI compared to 4.3 from human raters, a gap of 1.1 points.

Black students also received lower scores from ChatGPT, though researchers noted a complicating factor: the same scoring disparities appeared in human scores as well. AI may not be introducing new bias so much as replicating and scaling existing human bias, applying it with mechanical consistency across thousands of essays without the possibility of a second thought.

AI systems also show scoring disparities against English Language Learners (ELL), compounding the bias problem for students who are already navigating educational systems in a non-native language.

As reporting from the Hechinger Report, EdSource, and KQED has documented, these disparities raise fundamental fairness questions about deploying AI grading at scale.

Mechanics vs. Meaning

A 2024 analysis published by Springer clarified the dividing line between what AI does well and what it does poorly:

AI excels at evaluating the mechanical aspects of writing, the things that follow rules. It struggles with the human aspects, the things that require judgment, cultural knowledge, and an understanding of what makes writing genuinely good rather than merely correct.

This maps directly onto the score compression problem. A well-structured essay with proper grammar but no original ideas looks the same to AI as a well-structured essay with a brilliant thesis. Both check the boxes the machine can see.

Teachers Aren't Buying It

Even among educators who see AI's potential, trust remains low.

A 2025 Carnegie Learning survey of approximately 800 educators found that while 77% said AI could be helpful, only 50% actually use it. The primary concern? Academic integrity: 81% worried that AI tools could undermine it rather than support it.

The numbers from other surveys reinforce this skepticism. According to EdWeek research, 52% of teachers say AI is unreliable and 53% lack confidence in using AI tools effectively. A survey by aiEDU found that only 5% of educators are confident their students use AI responsibly. And only 25% of administrators have provided any AI training to their staff, according to Carnegie Learning.

Teachers are caught between the promise of saving six hours a week and the fear of automating errors at scale. We think that skepticism is healthy.

This Argument Is Older Than ChatGPT

The tension around AI grading is not new. As far back as 2012-2013, MIT professor Les Perelman publicly challenged the automated essay scoring (AES) research of Mark Shermis, demonstrating methodological flaws in the studies that claimed AES was as reliable as human grading. Perelman showed that AES systems could be fooled by nonsensical essays that followed structural patterns, and that the research supporting AES overstated its capabilities.

More than a decade later, the fundamental criticism holds: machines that evaluate form without understanding meaning will always hit a ceiling.

Making It Work: Humans + Machines

The research consensus points not to AI grading or human grading, but to a thoughtful combination of both.

"Human in the loop" is non-negotiable. Mo Zhang, a researcher at ETS, emphasizes that AI scores should always be reviewed by a teacher. The machine can do a first pass; the human makes the final call.

Start with low-stakes work. Use AI grading for formative assessments (practice essays, drafts, homework) where the purpose is feedback, not evaluation. Reserve human grading for summative assessments that affect grades.

Let AI handle mechanics; let humans evaluate thinking. A hybrid model where AI flags grammar, spelling, and structural issues while teachers evaluate argument quality, creativity, and critical thinking plays to each evaluator's strengths.

De-identify student work before AI processing. Under FERPA, student data must be protected. Removing identifying information before running work through AI tools is both a privacy best practice and a way to reduce potential bias from names or demographic signals.

Never use AI grading as the sole assessment. AI should be one data point among several, alongside class participation, in-person conferences, process documentation, and teacher observation.

Where We Land

AI grading tools are fast, consistent, and increasingly capable at evaluating the mechanical dimensions of student writing. For teachers processing hundreds of assignments, they offer real relief on the routine stuff.

But speed and consistency are not the same as accuracy and fairness. Score compression means AI can't tell excellent work from mediocre work. Racial and linguistic bias means some students are systematically disadvantaged. And the inability to evaluate creativity, critical thinking, and voice means the qualities we most want to develop in student writers are exactly the ones machines can't assess.

AI will play a growing role in grading. It already does. The question is whether we use it wisely: handling mechanical work so teachers can focus on the human judgment that no algorithm replaces.

Limudi focuses on what happens before the grade, tracking the writing process through keystroke dynamics, revision history, and cadence analysis to verify authentic student work. Instead of replacing teacher judgment, we give teachers the evidence they need to assess with confidence. See how it works.