When AIs agree, two wrongs don't make a right

Richard Davis

9 November 2026 · 5 min read

Some AI marking tools check their own work by asking a second AI. It sounds reassuring. Our research says it catches far less than you'd hope, and here's what works better.

Every marker is sometimes out of step

No marker is perfect. Not a tired teacher at 11pm, not an experienced examiner, and not an AI.

Most of the time a good AI marker lands close to where examiners land. But every AI marker, ours included, occasionally gives a mark that lands far from where a panel of experienced examiners agreed, sometimes by a grade or more. We call these marks out of step with the examiners. Some are too generous, some too harsh, and sometimes they happen even on essays the examiners found easy to agree on.

So the useful question isn't "is the AI accurate on average?", something we already know to be true. It's "which marks should a teacher look at again?" That's what a confidence check is for.

The obvious fix: ask a second AI

A common answer is to have two AIs mark every essay. If they agree, the mark stands. If they disagree by more than a set amount, the essay goes to a teacher for review.

Some providers, for example, describe their process this way: a primary model marks each response, and an independent validator marks it again without seeing the first mark. Where they disagree beyond a tolerance, the response is flagged for the teacher to review.

This is an attractive approach on the face of it. It works on a brand-new question without requiring a lot of additional setup. And it keeps teacher oversight front and centre. We agree with both.

To their credit, some platforms do acknowledge the catch that two models can agree and still be wrong. We wanted to know how big that catch really is.

Why two AIs agreeing isn't proof

Think of two pupils who revised from the same textbook, checking each other's homework. Where the textbook is right, they both get it right. Where the textbook is wrong, they both get it wrong. Their agreement feels reassuring, but it tells you nothing about the textbook's mistakes.

AI models are a bit like that. They learn from much of the same writing, so they tend to be fooled by the same essays. We tested this on examiner-marked scripts with AI models from several different makers. When one model over- or under-marked an essay, the others very often erred in the same direction.

So the essays that fool one AI tend to fool the second one too. The two agree, the essay sails through unflagged, and the out-of-step mark reaches the student.

What disagreement actually catches

When two AIs do disagree, why is it? In our tests, mostly because one of them is a generous marker.

Imagine two bathroom scales, one of which always reads 2kg heavy. Weigh yourself on both and they'll disagree, and you'll know something is off. But the fix isn't to weigh everything twice forever. It's to adjust the faulty scale once.

Marking is the same. Adjusting a marker that runs consistently generous or harsh is called calibration, and any serious AI marker should do it before a mark reaches a teacher.

So we did exactly that, then asked again: once the generosity was corrected, did the essays two AIs disagreed about turn out to be the ones out of step with the examiners?

Barely. Sending those essays to a teacher fixed hardly more than sending the same number of essays chosen at random. That was true with each of the three AI models we tried as the second marker.

What a real check looks like

A second opinion is only worth having if it can see what the first one missed. That means it has to come from somewhere genuinely different, a fresh point of view, not another AI reading the same essay the same way.

Think of an experienced head of department glancing down a pile of marked scripts. They don't re-mark each one. They know what an essay like this usually gets, and they spot the one that looks like a 24 but has been given a 33. That's the kind of check we built. It doesn't mark the essay again. It knows how essays like this one have actually been marked on this question, across hundreds or thousands of real scripts, and it flags any mark that's out of line.

We tested it the hard way. We fixed the rules in advance, then ran it once on 696 real GCSE scripts, each marked by a panel of examiners, that nobody on our team had seen.

Picture a big pile of marked scripts. Hidden in it are the marks that are out of step with the examiners. A teacher only has time to re-check part of the pile, so which essays should they pick?

Same effort, three times the chance of catching a mark that's out of step.

Where we draw the line

We'd rather tell you our limits than have you find them.

• It's proven on one question so far. The 696-script test was AQA GCSE English Language Paper 1, Question 5. We're extending it question by question.
• It switches on one tool at a time. On every other question, the check runs quietly behind the scenes until it passes its own tests on that question's real marking. Only then can a teacher see it.
• It needs history. It learns from how a question has been marked, so a brand-new question takes time to reach that point. This is where the two-AI approach has a real advantage.
• It points to marks that look out of step; it won't catch every one. It's a pointer for a teacher's time, not a guarantee.
• Teachers stay in charge. A flag says "worth a second look". It never changes a mark on its own.

Three questions to ask any AI marking provider

Whoever you choose, including us, ask:

• What does your confidence check compare against? If the answer is "another AI", ask how often the two make the same mistake.
• Has the check been tested on scripts nobody had seen, with the rules fixed in advance? Ask how many of the marks that are out of step with examiners it actually finds, compared with picking essays at random.
• What are your accuracy figures measured against? The exam board's own standardisation essays and the chief examiner's marks are the gold standard. That's what our accuracy figures are measured against.

A confidence flag is only as good as the evidence behind it. If you'd like to see how ours works on your subjects, book a demo.

Richard Davis

Founder & CEO, Top Marks AI

Richard read English at UCL and Cambridge before founding Accolade Press, a boutique academic publisher. A lifelong educator and the author of four bestselling thriller novels, he founded Top Marks AI to bring rigorous, exam-board-calibrated marking to every school in the UK.