AI Marking for Teachers Achieves 0.85 Mark Average Error for Eduqas's Component 2: Question 2

This 10-mark ‘How does the writer…?’ question asks students to analyse how the writer uses language to achieve effects. Below we share the accuracy of Top Marks AI across 40 Eduqas exemplars.

For questions with a limited marking range like this 10-mark question, we focus on a particularly important metric: Mean Absolute Error (MAE). MAE tells us, on average, how many marks our AI differs from the exam board's marks. A low MAE means high accuracy.

As such, these results demonstrate just how accurate the Top Marks' GCSE English Literature AI marking tools really are. We think you’ll be impressed.

For this analysis we examined scripts from Eduqas English Language — specifically, the Component 2: Question 2 - How does the writer…? question.

Eduqas makes available numerous exemplar essays for their exam papers and we've put our tool to the test using 40 of those very same exam board approved standardisation materials. These exemplars showcase a broad spectrum of answer quality. They span lower to higher bands, providing a realistic spread for evaluation.

We took 40 of these essays and ran them through our dedicated marking tool. Then we measured the difference between the official marks the board awarded each essay, and the marks Top Marks AI assigned to those same essays.

For context, how do humans perform?

What level of accuracy do experienced human markers achieve when marking essays already marked by a lead examiner?

Cambridge Assessment conducted a rigorous study to measure precisely this. 200 GCSE English scripts — which had already been marked by a chief examiner — were sent to a team of experienced human markers. These experienced markers were not told what the chief examiner had given these scripts. Nor were they shown any annotations.

The Mean Absolute Error (average difference) between the experienced markers and the chief examiner was 5.64 marks on a 40-mark question — that's an average difference of 14.1%. If you are interested, you can find the study here.

How did Top Marks AI perform?

Across those 40 scripts, our system achieved a Mean Absolute Error of 0.85 marks. On average, the AI differed from the board by just 0.85 marks on this 10-mark question. As a percentage, that's an average of 8.5% difference — significantly better than the 14.1% human marker difference in the Cambridge study.

Moreover, 77.50% of the marks we gave were within 1 mark of the grade given by the chief examiner.

As an additional measure of accuracy, we also calculated the Pearson correlation coefficient, which was 0.87. This indicates a strong positive relationship between our marks and the exam board's marks. For context, the Cambridge study reports experienced human markers at just under r = 0.70 against the chief examiner on a 40-mark question, so r = 0.87 is materially higher.

We don't claim that Top Marks is infallible, but when it does get things wrong, just how bad is it? Well, let's turn to the Root Mean Square Error to find out. Root Mean Square Error (RMSE) is a measure of the severity of large errors. When you square the number 1, you still get 1, and when you square 2, you still only make a small jump to 4. But square 5, and you're suddenly all the way up at 25. That's how RMSE works — it (essentially!) highlights large errors by squaring them.

Top Marks AI's Root Mean Square Error was 1.08, meaning even when larger errors occur, they remain remarkably small relative to the 10-mark scale.

You can see the full side-by-side human and AI scores below.

Can I see a graph to help me visualise this?

Absolutely.

First, here's a scatter graph to show you what a theoretical perfect correlation of 1 would look like:

Now, let's look at the real-life graph, drawn from the data above:

On the horizontal axis, we have the grade given by the exam board. On the vertical, the grade given by Top Marks AI. The individual dots are the essays — their position tells us both the mark given by the exam board and by Top Marks AI. You can see how closely it resembles the theoretical graph depicting perfect correlation.

Discover how Top Marks AI can revolutionise assessment in education. Contact us at info@topmarks.ai.