The why and how of timed questions

ImageWay back in the mists of time, on a hot Friday afternoon, I was doing classroom observation with a generally low-attaining Year 10 group. This was in the days before Yacapaca, and the students were using a previous system of ours called Paperless School. Like most systems then and now, this built quizzes straight into the main web page and offered no constraint over the time allowed.

I remember in particular two students lolling back in their chars, talking about snooker and making occasional half-hearted attempts to click answers when the teacher came by. Typical teenage boys, they got more social credit from projecting a “don’t care” attitude than they ever would from diligent study.

It was this episode, more than anything, that led to the creation of Yacapaca. I realised that we needed something that would hold their attention. In any given moment, there had to be a better reason to think hard about the answer to a question than to talk about snooker.

A crucial component (not the only one) was to insist that the time to think is now. Rather than rely on the teacher’s patrol and control, the software itself had to present the student with a simple choice: decide now, or lose.

First iteration: author-controlled timing

When Yacapaca launched, I directly controlled the very little content that was available. I set the quiz timing through direct observation of students, and this worked pretty well. When we introduced open authoring, it seemed logical to allow teachers to set the timing for quizzes they authored.

This did not work nearly so well. At least half of those who overrode the defaults clearly had no idea of what they were doing. Even those who did, were thinking only about their own students. Core to Yacapaca is the idea that your work benefits everybody; if the timing does not suit them, this is simply not the case. So, that was the wrong place to put that control.

Second iteration: automated, student-controlled timing

A little thought revealed an obvious truth: even within a given class, students’ preferred timings vary considerably. What’s more, they are not linked to ability. Many of the highest-scoring students are quite slow and methodical.

To compound the problem, question difficulty varies enormously too. To some extent this correlates with the amount of text a students must read, but that’s not the whole story. A highly-conceptual question may be very short, but require a great deal of thought.

The eventual solution was to do a vast correlation exercise across all 144,000 questions and 2.75M student accounts, to come up with ‘speed factors’ for each question and each student. We multiply these together to get the optimum timing each time a question is presented.

The aim is to keep the student in ‘flow’; a state of exclusive focus on the task at hand. To achieve this, each question must be timed so that the student will neither panic nor get bored.

This works well in most cases, but there are still some niches where it is not working as well as I would like. In particular, two categories have vexed me:

  • some students treat a question as a memory test and others expect to calculate or derive the answer.
  • some students start from such a strong belief that faster is better that they ignore the timer and challenge themselves to lightning-fast responding. I started to understand these kids a little better recently when getting trounced by a friend’s 14 year-old son in an ‘first person shooter’ video game.

Because of the way the system crowdsources speed metadata, these students’ problems cause knock-on effects that can undermine the correct timing of quizzes for other users. So on we go to…

Third iteration: bias to thinking not remembering

Another aspect of the Yacapaca quiz system that we have been steadily developing is the idea of reward points. Kids have been trained by computer games to value points, levels and so forth. We have taken advantage of this to build a motivational system in which students can move through levels defined by different avatars, purchasing these with points as they go along. The motivation points do not interfere with the level grading that is the most important output for teachers.

We are now devising a system (it’s not launched yet) that identifies the optimum time required for any given student to answer any given question, and explicitly gives them the maximum reward for using all of that time. In tests, it has worked well to slow down the fastest students. The system is called “Tortoise and Hare”. We are still tweaking the timing algorithm for it, and I am not quite ready to launch yet.

When we do launch it will still take a while for students to re-train their own expectations. As they do, I’m hoping you will find that more students from the margins will be able to start really enjoying Yacapaca. Your willingness to publicly buy into the importance of levels (as avatars, in our case) will make a huge difference in how well this works in your school.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s