[Each day in October, I analyze one of the 31 item writing rules from Haladyna, Downing and Rodriquez (2002), the super-dominant list of item authoring guidelines.]
Writing the choices: Avoid giving clues to the right answer, such as
a. Specific determiners including always, never, completely, and absolutely.
b. Clang associations, choices identical to or resembling words in the stem.
c. Grammatical inconsistencies that cue the test-taker to the correct choice.
d. Conspicuous correct choice.
e. Pairs or triplets of options that clue the test-taker to the correct choice.
f. Blatantly absurd, ridiculous options.
This rule is huge. As is, it has six parts. In their 2004 book, this rule has five parts, and in their 2016 book, this rule has four parts from this list of six and two part from elsewhere. So, that prompts a question of how I should go about responding to this rule. Some previous rules seem like they should just be sub-parts of this one. But this is a fisking project, so I will address everything.
Item developers should not fall into patterns that give away the correct answers. That’s the real principle for their first sub-part, not just particular words. Always, never, completely, and absolutely should only be used in patterns that do not flag that option as correct or incorrect—just like all of the above and none of the above and countless other potential tells. This is not merely about their use in any particular item, but rather pattens in their use across item banks. It certainly should not lead to a prohibition that has nothing to do with how well these terms address content.
I have never understood Haladyna et al.’s advice on “clang association.” They seem to be saying that items should not repeat key words from the stem either in the correct answer option or in incorrect answer options. That does not make sense to me. Why not? They offer that this can simply be too big a clue to the correct answer option—which seems just to be part D of this rule—or it can be a sign of a “trick” item. But I already addressed their dumb Rule 7 about trick items. I do not believe in trick items. Moreover, if some word in a title or quote actually is often misunderstood or mislead, then that sounds like it is a good basis for a distractor. It should not be avoided.
Isn’t just grammatical inconsistency a repetition of Rule 23? See my response to that rule, from earlier in the month. This sub-rule is folded into option homogeneity in their 2013 book. Length is also included in their 2013 book, but as a separate subpart.
Conspicuous correct choice? I’m not really sure what that means. That sounds more like an issue with a lack of plausible distractors, which might explain why this sub-part is missing from their books.
Pairs and triplets? Just another example of not understanding homogeneity, Rule 23. I already addressed that.
Blatantly ridiculous options? Yeah, that’s again about plausible distractors. That is its own issue, and perhaps the most important single principle of multiple choice item construction, right up there with clarity. It is not just about cuing, nor is it appropriate to bury in some sub-part of a rule on cluing. So, this one gets its due attention in tomorrow’s post.
Where does that leave us? I and my colleagues worry about false-positive results. We worry about those alternative paths, including encouraging guessing with various tells. But this is not a good list of tells to worry about. It is not even Haladyna et al.’s complete list of tells, so what is this rule doing?
What is this rule doing? They report that 96% of the sources for their 2002 article support it, but is that any surprise? They have six parts that a source could support to be included! Why aren’t these listed as six different rules so we can see how many sources mention each sub-part, how many supported it and how many ignore it? Are we to believe that each of the 96% mentioned all six parts? Certainly not! So, what is going on here?
No, this is a not a great rule. It misses the point, confusing symptom and outcomes for the actual principle at stake. As so many other reason, this rule makes clear that this list is not about deep principles.
[Haladyna et al.’s exercise started with a pair of 1989 articles, and continued in a 2004 book and a 2013 book. But the 2002 list is the easiest and cheapest to read (see the linked article, which is freely downloadable) and it is the only version that includes a well formatted one-page version of the rules. Therefore, it is the central version that I am taking apart, rule by rule, pointing out how horrendously bad this list is and how little it helps actual item development. If we are going to have good standardized tests, the items need to be better, and this list’s place as the dominant item writing advice only makes that far less likely to happen.
Haladyna Lists and Explanations
Haladyna, T. M. (2004). Developing and validating multiple-choice test items. Routledge.
Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. Routledge.
Haladyna, T., Downing, S. and Rodriguez, M. (2002). A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education. 15(3), 309-334
Haladyna, T.M. and Downing, S.M. (1989). Taxonomy of Multiple Choice Item-Writing Rules. Applied Measurement in Education, 2 (1), 37-50
Haladyna, T. M., & Downing, S. M. (1989). Validity of a taxonomy of multiple-choice item-writing rules. Applied measurement in education, 2(1), 51-78.
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied measurement in education, 15(3), 309-333.
]