Question quality measurement

The summer is upon those of us in the Northern Hemisphere, and I know you are just bursting to get down to some serious quiz authoring. So just in time, we have launched a new tool to help you raise your game – question quality metrics. Here’s the essential FAQ:

Where do I see the quality of my questions?

Look under the STAT button for any of your quizzes (also see screencast)

How does this help authors?

It will be much easier to write good questions if you have direct, clear, measured feedback about the quality of your current questions. You can also take action to improve your existing quizzes, by replacing the low-quality questions and recompiling the quiz.

How does this help teachers?

When choosing quizzes to assign, teachers can see a quality value for the quiz. This is an average of the quality of each question in the quiz. By choosing higher-quality quizzes, they give their students more accurate assessments.

How is quality defined here?

The best questions are those that allow a teacher to most clearly discriminate between students who ‘get it’ and those who don’t. Questions that are either too easy nor too hard fail this test, so they are low quality. The optimum percentage varies according to question type and number of possible answers.

How is question quality measured?

We take an average of all student responses, and compare it to the optimum response rate for each question type. For example, a four-option multiple choice has an optimum response rate of 62.5%. The closer to the optimum, the higher the quality score.

Why do ‘location’ questions all appear to be low quality?

‘Locate’ questions take longer to answer than other types, and this has historically led to a lot of wrong answers. In fact, we have just fixed the timing issue, but it will take time for the quality score to sort itself out.

What does the current question quality measurement not cover?

The biggest gap is the quality of the feedback. The best questions have strong, formative feedback. So far, we have no way to measure the quality of feedback.

What are the future plans for question quality?

  1. include feedback quality
  2. measure the quality of the distractors
  3. include this data into the course quality metric, and use it to adjust course ranking.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s