Randomised vs. linear question sequences in tests

randomisationIf you author quizzes or multiple-choice surveys in Yacapaca you will know that the system automatically randomises both the sequence of questions and the sequence of options within those questions. I get a steady trickle of requests to enable teachers to control the sequence both of questions and of options. In my opinion, this would be a massively retrograde step.

That is not to say that linear presentation has no benefits. The two most-often cited are that you can chain questions so that one logically follows the next, and that you can use phrases like ‘none of the above’ in the certain knowledge that the other options will indeed appear above, and not below. So, what is so bad about it that I would take such an uncompromisingly opposed stance?

Two things, actually.

The first objection is that a linear sequence presents a far-too-tempting opportunity to students to collude. I’ve seen this myself so many times in classroom observation. The students you are not watching are either copying their friends’ answers, or telling their friends what to answer, depending on their place in the social hierarchy. Even in a survey, with no right or wrong answers, most students are highly motivated to be seen to think the same as their friends. By ensuring that each student sees a different question at any given time (and keeping the timing sufficiently tight), you can minimise opportunities for collusion.

The second is slightly more subtle. A popular strategy is to set a test twice. This may be to gather more data for a more accurate result, to show progression, or to build students’ motivation. But what do the students learn if the sequence is fixed? They learn the sequence itself. More-auditory learners, with their stronger grasp of sequence, will be particularly susceptible to learning that the answer “respiration” always follows the answer “haemoglobin” and quickly start to screen out the actual questions entirely. When they come across these concepts in other contexts, they will be utterly flummoxed.

A similar argument applies to the randomisation of the options. Students will learn the location of the correct response, even on the first attempt, when what you want them to learn is the logic underlying the question.

So why not give authors the choice? Continuing the theme of twos, the reasons are first that it would introduce an unwelcome new level of complexity to the software, and second that it serves the wider Yacapaca community best if new authors learn good authoring habits right from the start.

There are some simple rules you can follow to give yourself all the benefits of a sequential approach, with none of the drawbacks.

  • To steadily escalate the level of challenge throughout a test, break it up into several quizzes, and keep them all in the same course. An advanced authoring technique is to set higher weightings for the more difficult quizzes, then use the course grade grid to calculate the final level.
  • To deal with questions that lose their logic when phrases like “none of the above” appear inappropriately as the first option, simply rephrase. “None of the others”, for example, does not make assumptions about position.
  • Finally, make sure that each question stands alone and does not presume that any other question will come before/after it.

If you are struggling with a truly randomised quiz engine for the first time, you do face something of a learning curve. However, perseverance pays. Whenever I have been able to engage with teachers who have asked for linear sequences, they have always been able to achieve their goals with randomised questions after a little thought, and several have converted to become the strongest supporters of the concept.

3 thoughts on “Randomised vs. linear question sequences in tests

  1. I’ve just discovered this issue. I am attempting to make an anti bullying survey but have come up against a couple of issues. These particular questions DO need to be linear and it would be useful to be able to mix questions types. Some questions are very definately one answer types whereas others should allow the kids to answer more than one option. eg Have you ever been bullied? as opposed to where sdid the bullying occur (could be in several differnt places) Any thoughts on this one?

    Reply
  2. Nicola, I wish I had time to look at your particular questions so that I could make concrete suggestions. As it is, I’m going to have to be rather general, sorry. I have never seen a quiz or survey that really, really, had to be linear. There is a knack to editing your questions so that they don’t assume any particular order, and it is well worth developing for the reasons I have given above. Bullying surveys are the most extreme example I can think of, of the way that peer pressure distorts results and obscures the very problems you are trying to reveal. Your other issue resolves to a feature request “can I have Checkbox Survey questions?”. This sounds perfectly reasonable, and I will put it on the list of things to do. It will be several months before we can get to it, I’m afraid.

    Reply
  3. I’d like to use Yacapaca so that exam classes can attempt multiple choice papers and I’d have both evidence of their attemps and automatic marking. I can’t post the material in the questions (as it is copyright the exam board). Wouldn’t this could be a very convenient use of Yacapaca?

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s