It is long past time to abandon paper-based examinations

chinese exam g edit

The ‘exam paper’ has been around for the last 14 centuries and it is still going strong. The way things are going, the exam-paper looks likely to outlast even the news-paper.

But it shouldn’t. Paper-based exams are a hideously-unreliable method of assessing competence. Here are some of the major reliability problems of paper-based exams:

No control for state of mind
I have put this first because it bugs me the most. Student A got a good sleep, had a good breakfast, got a great pep talk from his mum. Student B is on her period and had a row with her boyfriend last night. Guess which one is going to Oxford. Even if you control for long-term biases such as socioeconomic status, making a high-stakes decision based on a sample taken at a single point of time is criminally negligent.

Demand for consistency results in testing only knowledge
Exam boards place tremendous emphasis on ensuring that any two arbitrary moderators would return the same mark for the same piece of work. This results in mark schemes that specify the exact words that should written to attain a given mark. No matter the intent behind the question, these words can always be rote-learned. In fact, rote learning seems to be the most efficient route to a high grade in most exams. Exam boards are perfectly aware of this, but achieving consistency is an existential issue for them. They are trapped.

A single modality: writing
Whilst alternatives such as oral exams do exist, 90% of the questions in 90% of exams require a written response. Understanding of an issue might well be better demonstrated in some other way, but that is just hard luck because the exam system is built around the (hand)written word and any alternatives are tacked on the side in a way that makes them even more expensive.

Restricted sampling of content
Teachers who get good results spend considerable time weighing the probability that each given piece of content will be in this year’s exam. No need to teach something that won’t be. The exam tail wags the learning dog.

Slow and expensive
The only time a typical adult handwrites a communication and has it physically delivered to its recipient is at Christmas. The world has moved on simply because pen and paper makes everything so damned expensive. If we got rid of that expense, we could spend the money on a better system.

The solution(s)

New technology gives us an enormous range of possible solutions to the high-stakes assessment problem. The objection that has so far prevented them from even being tried so far is “but they require technology that not everybody has”. Yes they do, but that technology is now so cheap and ubiquitous that it is time to make the commitment and demand that the laggards catch up. The size of the problem justifies some sacrifice.

50s exam

So what should replace paper-based exams?
We don’t need to search for one perfect solution. To the contrary, we should be embracing a rich mix of assessment methods to fit particular subjects and circumstances. They should in general have the following characteristics:

  1. Start assessing from Day 1 of a course. Sample briefly, frequently and regularly to build an accurate picture of each student over time.
  2. Use methods that can test understanding and the ability to evaluate. These might for example include paired comparisons, chatbots or observed conversations.
  3. Routinely include non-written components such as speech or video. Use monitored peer assessment so as not to soak moderator time.
  4. Assess formatively. Summative testing eats into learning time, but formative assessment becomes part of the learning process itself and is essentially zero-footprint.
  5. Do it all online. Cut costs, shorten timeframes. Use the savings to improve quality.

I have seen and used (in some cases built) systems that achieve each of the above, and that have successfully resolved every objection that I could come up with. They exist, they work routinely in other assessment contexts, and they are ready to be applied to high-stakes academic assessments.

Resolving the big objection
“Schools don’t have enough computers and they can’t trust their connectivity.” Well shoot, everyone else has solved this; why haven’t schools? There is no valid excuse. I have enumerated elsewhere how much money schools are wasting on manual marking and laid out a strategy by which they can acquire all the hardware they need with a payback of one year.

What we still need is for an exam board to to step up and start offering qualifications that are assessed exclusively online and that are structured to offer clear benefits over traditional paper exams. It will only take one to break the logjam. The question now is, who wants first-mover advantage?

 

6 thoughts on “It is long past time to abandon paper-based examinations

  1. I teach A Level and am also a senior assessor for a leading exam board yet I agree with most of what you say. Universities ar moving away from exams and essays yet A Levels seem to be more and more focused on this. Definitely time for a change. The obsession with exams is driven by government ideology and ministers who seem to want to drive education backwards to the 1950s rather than embrace the oppotunities offered by new technology.

    Reply
  2. I remember sitting in a meeting hearing a colleague inform us that the maths department had ‘cracked the code’. They had come up with a ranking list so that pupils could be placed in groups according to ability in our 8-form school. Our eager deputy head immediately declared that future ‘setting’ in our science groups will follow maths ranking lists.

    Great, said my key stage 3 co-ordinator, do you want to give you our ranking list based on our eci of module tests that our students take every half term. I have results in a spreadsheet! One look at that spreadsheet showed that we had come pretty close to formative assessment!

    It was a pity that they never took up our recommendation that they should ditch the maths ranking lists and use science data in stead. That was in 1988 I think.

    Good to see that things have not moved on much!

    Reply
  3. Pingback: A Riposte to the Traditional Written Exam | The Learning Renaissance

  4. Pingback: Yacapaca

  5. Pingback: The Game of Education | Yacapaca

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s