Is Yacapaca’s GCSE 9-1 grading accurate?


My thanks to Bobby Grewal at Four Dwellings Academy for the awkward question.

Let’s start by asking what it is we are trying to measure. At Yacapaca, we have opted for “current” grading, as that’s what the majority of our users want. This means that we are attempting to answer the question:

  • If the student were to take the GCSE exam today,
  • and they had studied all the content to the same level as the content of this quiz
  • then what grade might they be expected to achieve?

Without actually throwing students as young as 11 into a real GCSE, how can we test whether our answers are accurate? Not only is that impractical; we can’t even (yet) back-calculate from real GCSE results. So we have to calibrate against some existing scheme for which we have data that can be validated.

Fortunately, there are two sources, NC Levels and GCSE A*-G. There are published conversion tables between each of these and the GCSE 9-1 grades. My preferred version is the QEGS Points Conversion Wheel.


The next question becomes “how do we know that our original NC and GCSE gradings were accurate. The answer is that we crowdsourced the calibration from thousands of teachers. As well as quizzes, Yacapaca allows you to add other grades from teacher-graded assignments. We correlate these to difficulty-adjusted quiz scores using linear regression. Thus, the accuracy of our quiz scores is based on the consensus in the teaching profession, rather than one arbitrary authority.

Now, we have to ask ourselves how accurate the result of one single quiz might be anyway. There are fundamental limits: consider the hypothetical case of a quiz that contains only one question, which may be got right or wrong. Does a correct answer equate to a Grade 9? An incorrect to a Grade 1? Of course not. A single answer tells you almost nothing. In practice, the absolute minimum you should rely on is 50 questions spread across at least three separate quizzes.

Finally, what accuracy can you expect from a single session with any assessment tool? Everyone has good and bad days; teenagers take it to extremes. If you really want Yacapaca to give you grades that you can rely on, my recommendation is that you use it little and often, building up a picture over time. Use the Progress Chart and the Parents’ Report to discover averages and trends in the data.

So is Yacapaca accurate? Yes, provided you use it intelligently.

The dog ate my homework

dogThe single biggest concern that I hear from teachers taking their first steps into elearning is that “computers are unreliable”. They fear lost work, broken connections, crashed servers – all things that can and do happen. Whilst I take full responsibility for problems that happen on Yacapaca, and work very hard to prevent them, I’ll confess to feeling a tad frustrated that the same standards are not applied to paper learning materials.

Lost or even destroyed books and folders are an everyday reality in schools, as are textbooks stuck in locked cupboards and worksheets jammed in photocopiers. Analysing these problems we find two types of problem; either of which is enough to give the average digital security consultant the screaming habdabs.

Only one copy of the data exists

All data is fragile stuff and it’s easily lost or destroyed. At Paperless School, we lock it all away in a secure facility with a controlled environment. We also take a complete copy every night and it in a separate, safe place. Exercise books, by contrast, are left on buses, burnt or dropped into ponds whilst looking for frogspawn.

The data is only accessible in one physical location

If you’ve left a book at home, you’ve had your chips. You just can’t access your data until the evening. Not that there aren’t restrictions on digital data; typically you need access to a browser. But digital access gets easier every year; paper access does not.

The real benefit that paper has is that we have become inured to its failings. We’ve simply resigned ourselves to the fact that someone in the class is going to be homework-less or textbook-less. It is high time that we realised that this state of affairs is not inevitable and that we can and should demand reliability of access to data, whether digital or on paper.