Item analysis: what to do when a class averages 68% on a quiz

An average hides two very different classes

A class average of 68% can mean that everyone is a little shaky across the unit, or that most students are fine and one idea broke for half the class. The next lesson for those two classes is completely different, and the average cannot tell them apart. Item analysis can, usually in under ten minutes.

What to record for each question

For every question you need five things: the skill it tests, the share of students who got it right, the wrong answer chosen most often, the share who chose it, and how many answers it is based on. The last one matters more than it looks: 40% correct from five answers is a hint, from a hundred it is a finding.

A real example from a history course

One row from the table above, in detail. The question asked what was true of the principality of Kyiv during the age of fragmentation. The right answer, that the Orthodox metropolitan had his seat there, got 53.6% of 56 answers. The wrong answer "the Liubech congress of princes was held" got 44.6%: almost as many.

That wrong answer attaches a famous event to the wrong place and the wrong period. The congress was held at Liubech, in the Chernihiv land, in 1097, before the period the question asks about. It is one idea, which events belong to which place and time, that a short reteach with a map and a timeline can fix for the whole group.

Two rows in the table are even clearer: on the pottery question and the herding question, one wrong answer was chosen more often than the right one. That is the strongest signal item analysis gives.

How to read it: four patterns

Sort by share correct, lowest first, and look only at the bottom few rows. Usually two or three questions account for most of the lost marks. Each hard question then falls into one of four patterns.

From questions to students

Item analysis tells you what the class gets wrong. The next step is who. For any skill where wrong answers were scattered, list the students who missed it on two or more questions. That list is a small group for ten minutes while the rest of the class practises.

Common mistakes

• Reteaching the hardest question without reading the wrong answer, so the reteach misses the actual misconception.
• Throwing out a hard question as "bad" when the wrong answers show a shared misconception.
• Drawing conclusions from a handful of students.
• Checking that the reteach worked on the same question. Use a new question on the same skill a week later.

Check again on new questions

A week later, give a short check on fresh questions for the same skills. A gap is closed when students get the skill right on questions they have not seen. In Gaplight's first measured cohort, 2 in 3 mistakes were answered right on the next try once the gap was visible: one cohort, one exam, measured without a control group, so a first signal rather than a promise.

Without going back through every paper

The template below works by hand for one quiz. Gaplight's item analysis does the same on the school's own questions and updates with every answer: the questions the class loses, the wrong answer most students chose, and the students who need the teacher on that skill.