Writing multiple choice (MC) items is extraordinarily more difficult than just writing good constructed response (CR) questions. CR questions give the test taker the freedom to take any path they want and to demonstrate their understandings and misunderstandings without the scaffolding and limitations of the set of answer options that are the distinguishing characteristic of MC items.
When we use MC items, it is because for all the initial difficulty in writing and refining them, they are faster and cheaper to score than CR items. Machines have been able to do it quickly and for virtually no cost for decades. That’s why we used to use #2 pencils—that’s the kind of pencil that the machines were make to read most easily. That long term cost saving is truly the only reason to use them.
But what does it take to write an MC item that gathers the information that a CR item does? How do we avoid making the answer too obvious? How do we avoid the little hints that so often appear in MC items? Unfortunately, quite few researchers have really looked at the contents of items clearly enough to offer good guidance, but we do know that each distractor (i.e., incorrect answer option) should represent the result of a misunderstanding or misapplication of whatever knowledge, skills and/or abilities that that item is trying to assess.
So, let us consider a very simple question: How many states make up the United States of America?
We know that key (i.e., the correct answer) is 50. But can we come up with three or four good distractors?
It seems plausible that some significant share of test takers who get this question wrong will come up with 13, confusing the number of original colonies that rebelled against the British and formed the original collection of state. 13 is clearly a good distractor.
What are the other mistakes that someone might make? Can you think of any? I don’t think that any other key American government numbers are likely mistakes.
3 (i.e., branches of government) is too obviously wrong and too low.
538 (i.e. the number of members in the electoral college) is too obscure and too advanced. Anyone who even knows that that is a significant number clearly will know the number of states.
435 (i.e., the number of members of the house of representatives) is not quite so advanced as 538, but suffers similar problems.
27 (i.e., the number of constitutional amendments) again suffers similar problems.
So, we still need two or three or more distractors. What about sheer guessing? I expect that most people who would get this question wrong would simply not know an answer and would simply guess. What are likely guesses?
I think that 100 is a likely guess, and perhaps attractive to someone who doesn’t know the real answer. I’m concerned that it’s a little large, but a nice round number doesn’t seem crazy.
But we still need at least one more distractor, at least. Something smaller than 100—which feels a little large. Maybe not a round number, so it feels more precise. Mexico has 31 states, so that might be particularly attractive to Mexican migrants. Of course, that might raise a fairness issue. Maybe they’d be more likely to pick it because they recognize the number, or maybe less likely because they know it is only Mexico’s answer? I’m going to ignore that for now.
How many states make up the United States of America?
A. 13
B. 31
C. 50
D 100
Now, I don’t love that. I’m always nervous that test takers are more likely to pick middle values (i.e., under the goldilocks principle of juuuuuuust right). Obviously, though, we ought to use every answer position equally when placing the key. I suspect that item developers too often try to offer distractors are smaller and that are greater than the key, and I don’t want to fall into a too common pattern.We could replace 100 with 25, though that is not so round a number, and therefore perhaps not as attractive for guessing.
How many states make up the United States of America?
A. 13
B. 31
C. 25
D 50
Such a simple question, and with one clearly correct answer. And yet, it’s not obvious which set of distractors is better. We would love it if we could offer good distractors to attract as many guessers and other mistakes as possible, so it just identifies the test takers who really do know the correct answer.
Writing high quality multiple choice items is hard.