Yacapaca

Computer-adaptive Testing (CAT) has actually been around since the 1970s but it is still little-known and less understood.

How CAT works

Any given CAT assessment is based on a large bank of questions of varying, but known, difficulty on a given topic. The first question is served at random. If the student answers it correctly, a more difficult question is served. If not, an easier one. This process continues until the system can see where the threshold of difficulty lies. And that is the student’s grade.

There are two problems implementing CAT that inhibit its wider use:

• You need a large bank of questions, much larger than for a traditional test.
• You need to know the level of each question.

As of this morning, Yacapaca has 181,934 questions. We know the difficulty level of each question because we monitor the success rate of students who have also been graded by their teachers.

CAT in Yacapaca

To use CAT in Yacapaca, you must be teaching one of the syllabi that already has defined topics. CAT is then available in three places:

• Topic homeworks. Select a subject, syllabus and topic, then assign that topic as homework for the class, with a time limit you set. Each student gets a ‘stream of algorithmically-selected questions on that topic, that is at approximately the right level for them.
• Revision. At the top of each student’s to-do list is an item “Revise [subject] for extra badge points”. Students can select their own topics within the syllabus, and set their own timer. This feature is on by default, though you can switch it off via the student set edit page.
• Keyword self-test. Students access this via the Revision Planner tab at the top of their to-do list page. A 5-minute CAT stream is constructed on-the-fly around a single key concept, to help the student revise. This is immensely popular with students, but we’ve not been very good at notifying teachers that it is happening. We’re fixing that now.

Limitations

There are limitations to CAT which mean it is not appropriate in all situations.

• Teachers often want complete control over the exact questions to be presented. CAT presents different questions to different students, and it selects them from a large bank that it is not practical to manually check in advance.
• Sometimes there simply are not enough questions at the right level of a given topic.
• The topic and keyword matching algorithms are not perfect. They will serve up the occasional ringer – a question that is just not relevant to the needs of the student. The wise teacher does some expectation management with the students before they start using CAT.

Overall, the trade-off with CAT is that you gain personalisation of learning at the expense of control. If you think that’s worthwhile, try it with your students. This table will give you a quick decision-guide for CAT vs. non-CAT quiz types.