Chatango is an embedable instant messaging (IM) client, written in Flash. As I’ve written before, I’m convinced that IM is of enormous, but as yet unexplored, significance to schools. Until now it’s required either standalone application or a dodgy Java applet. I’ve been dreaming for months of creating one in Flash, but whilst I’ve been dreaming, Chatango have been doing. I’m wildly excited by the result, or rather by its potential, and I’ve just got to play with it.

Where better to play than here? Below, I’ve embedded a Chatango client that links direct to me. If I’m online, why not say ‘hello’? (Thanks Joi Ito for the link).

Assessing Evaluation with multiple-choice questions

In a previous post I covered the KUE model and bemoaned the infrequent application of U and E in multiple-choice questions. Assessing evaluative thinking in multiple choice may seem the hardest of the lot, because it’s the most abstracted form of learning. In fact, it’s relatively easy provided you are very clear about exactly what you are testing for with each question.

Continuing the cycling theme, I’d like you to imagine you are talking one-to-one with a student, and you want to know if he can evaluate why one should not ride on the pavement. What might he say, spontaneously in conversation, that would convince you he really had evaluated the issue?

“You shouldn’t ride a bike on the pavement”

He’s stated a rule, but failed to justify it.

“You shouldn’t ride a bike on the pavement because it’s bad”

Linguistically, he’s stated a reason, but it’s a tautology.

“You shouldn’t ride a bike on the pavement because it’s dangerous to pedestrians”

This we like. He’s stated a consequence.

“You shouldn’t ride a bike on the pavement because it’s against the Highway Code”

We like this too, because he’s cited authority. If he’d have just said ‘the rules’ it would have been tautological again.

“Riding on the road is better, because the road is smoother”

A completely different argument, but equally evaluative. He’s compared and contrasted.

“You shouldn’t ride a bike on the pavement, my mate did it and hit a pedestrian”

Also good; he’s cited evidence. He’s also generalised – not always a good thing, but certainly evaluative.

This last brings me to the sting in the tail of evaluation. What if he’d said “You should ride on the pavement because it’s safer than the road.” Evaluative thinking doesn’t lend itself to single right answers and this can conflict sometimes with the right/wrong nature of multiple choice.

Nonetheless, with a clear aim statement we can do a lot. My aim statement will be “I want to know whether students can evaluate the dangers of riding on the pavement from an ethical standpoint”. Note that just by saying that, I’ve leant towards an evaluation of consequences. The foils will all be written as evaluative statements.

You should not ride a bicycle on the pavement, because

  • you might hit, and hurt, pedestrians (correct)
  • pedestrians might yell at you
  • you might hit a lamp-post and hurt yourself
  • only young children would do it, so you’d look uncool

Once again I’ve constructed the distractors to be as close as possible to the key. The only significant difference is the one thing I am testing for. Typically for a question testing evaluation, all the foils are correct, but the one we are after demonstrates a perception of the ethical issues involved in pavement-cycling.

So, to summarise, to demonstrate evaluation, a student might
use modal verbs (could, should, ought, might etc)

  • enumerate consequences (or causes)
  • cite authority (moral, legal, spiritual)
  • compare or contrast
  • cite evidence
  • generalise from examples

Finally one thing I can’t emphasise enough. Personally, I am completely unable to construct decent U or E questions until I have written a properly specific aim statement. Once I have written one, they seem to come really quite quickly. This may be individual to me, but I certainly recommend you try it.

Assessing Understanding with multiple-choice questions

The National Curriculum couches learning achievement in terms of knowledge, understanding and evaluation (KUE), a cut-down version of Bloom’s Taxonomy. So if we are going to use technology for meaningful and useful assessments, we must address all three of these elements.

Now the easiest way to set up an assessment is via multiple-choice questions. Building the technology to serve and mark multiple choice tests is easy; most MLEs support it and there are plenty of free websites out there that will let you create and administer your own.

But what I’ve noticed looking at them, is that nearly always the authors have confined themselves to testing only knowledge (K). U and E are getting ignored, I suspect because teachers don’t know how to write for these higher-order forms of learning attainment; perhaps because they believe it’s impossible.

So, how do you do it? Go back to your experience of the most informal form of assessment possible.

Imagine you are talking one-to-one with a young lad, and you want to know if he understands how a bicycle works. What might he say, spontaneously in conversation, that would convince you he really did understand?

How about

“it’s got two wheels, pedals, and a chain”

Perfectly correct, but scarcely indicative of understanding.


“The pedals make it go”

Hmm…linguistically, yes, that’s understanding. But it’s on a six year-old’s level; I wouldn’t accept it from a teenager.


“The pedals drive the wheel through the chain”

Yes! Clearly this person has a mental model of the functioning of a bicycle. Contrast the verb ‘to drive’ with ‘to go’ in the previous example. It’s more specific. The preposition ‘through’ indicates that he understands how two concepts are connected.

How do we translate this into multiple-choice format? Start by getting clear exactly what it is you want to assess. My aim statement will be “I want to know whether students correctly understand how the drive train of bicycle works” The foils should all be written as statements of understanding about the drive train.

Which of these correctly describes the drive train of a bicycle?

  • The pedals drive the wheel through the chain (correct)
  • The pedals drive the back wheel, and the front wheel steers
  • The wheel drives the pedals through the chain
  • The chain makes the pedals and the wheel go round

Note in passing that my aim statement contains the question “how?”, surely the key word for a U question, even though the stem of the question does not.

So, to summarise, to demonstrate understanding, a student should be able to

  • answer ‘how’ questions
  • correctly describe relationships
  • use, or correctly recognise, specific verbs rather than general ones
  • use or, correctly recognise, appropriate prepositions and conjunctions

Next post:
Evaluation questions