Yacapaca

The ‘exam paper’ has been around for the last 14 centuries and it is still going strong. The way things are going, the exam-paper looks likely to outlast even the news-paper.

But it shouldn’t. Paper-based exams are a hideously-unreliable method of assessing competence. Here are some of the major reliability problems of paper-based exams:

No control for state of mind
I have put this first because it bugs me the most. Student A got a good sleep, had a good breakfast, got a great pep talk from his mum. Student B is on her period and had a row with her boyfriend last night. Guess which one is going to Oxford. Even if you control for long-term biases such as socioeconomic status, making a high-stakes decision based on a sample taken at a single point of time is criminally negligent.

Demand for consistency results in testing only knowledge
Exam boards place tremendous emphasis on ensuring that any two arbitrary moderators would return the same mark for the same piece of work. This results in mark schemes that specify the exact words that should written to attain a given mark. No matter the intent behind the question, these words can always be rote-learned. In fact, rote learning seems to be the most efficient route to a high grade in most exams. Exam boards are perfectly aware of this, but achieving consistency is an existential issue for them. They are trapped.

A single modality: writing
Whilst alternatives such as oral exams do exist, 90% of the questions in 90% of exams require a written response. Understanding of an issue might well be better demonstrated in some other way, but that is just hard luck because the exam system is built around the (hand)written word and any alternatives are tacked on the side in a way that makes them even more expensive.

Restricted sampling of content
Teachers who get good results spend considerable time weighing the probability that each given piece of content will be in this year’s exam. No need to teach something that won’t be. The exam tail wags the learning dog.

Slow and expensive
The only time a typical adult handwrites a communication and has it physically delivered to its recipient is at Christmas. The world has moved on simply because pen and paper makes everything so damned expensive. If we got rid of that expense, we could spend the money on a better system.

The solution(s)

New technology gives us an enormous range of possible solutions to the high-stakes assessment problem. The objection that has so far prevented them from even being tried so far is “but they require technology that not everybody has”. Yes they do, but that technology is now so cheap and ubiquitous that it is time to make the commitment and demand that the laggards catch up. The size of the problem justifies some sacrifice.

So what should replace paper-based exams?
We don’t need to search for one perfect solution. To the contrary, we should be embracing a rich mix of assessment methods to fit particular subjects and circumstances. They should in general have the following characteristics:

1. Start assessing from Day 1 of a course. Sample briefly, frequently and regularly to build an accurate picture of each student over time.
2. Use methods that can test understanding and the ability to evaluate. These might for example include paired comparisons, chatbots or observed conversations.
3. Routinely include non-written components such as speech or video. Use monitored peer assessment so as not to soak moderator time.
4. Assess formatively. Summative testing eats into learning time, but formative assessment becomes part of the learning process itself and is essentially zero-footprint.
5. Do it all online. Cut costs, shorten timeframes. Use the savings to improve quality.

I have seen and used (in some cases built) systems that achieve each of the above, and that have successfully resolved every objection that I could come up with. They exist, they work routinely in other assessment contexts, and they are ready to be applied to high-stakes academic assessments.

Resolving the big objection
“Schools don’t have enough computers and they can’t trust their connectivity.” Well shoot, everyone else has solved this; why haven’t schools? There is no valid excuse. I have enumerated elsewhere how much money schools are wasting on manual marking and laid out a strategy by which they can acquire all the hardware they need with a payback of one year.

What we still need is for an exam board to to step up and start offering qualifications that are assessed exclusively online and that are structured to offer clear benefits over traditional paper exams. It will only take one to break the logjam. The question now is, who wants first-mover advantage?