[Each day in October, I analyze one of the 31 item writing rules from Haladyna, Downing and Rodriquez (2002), the super-dominant list of item authoring guidelines.]
Writing the choices: None-of-the-above should be used carefully.
First, I cannot tell you how much I hate this rule, nor how much it betrays the deep disrespect and distain that so many have for the work of content development in large scale assessment. No one would suggest that point-biserial or IRT should be used carefully, because everyone assumes that psychometrics is always done carefully. What does this rule or guidelines mea? Don’t be sloppy? Item developers should never be sloppy, whether they are using “None of the above” or not. They are professionals, and no professionals should be sloppy in their work.
Second, their 2002 sources and the empirical research is just split on this. There is no consensus.
Third, the 2002 article has an explanation that might be the most nuanced portion of the whole piece.
Given recent results and these arguments, NOTA [none of the above] should remain an option in the item-writer’s toolbox, as long as its use is appropriately considered. However, given the complexity of its effects, NOTA should generally be avoided by novice item writers.
Frankly, this kind of analysis should be applied to virtually their entire list, but it is nice to see it as least once. Of course “generally be avoided” is not actually actionable advice. It means that they can use it, but…I guess they should be careful, just like everyone else. Yeah, item development is hard.
Their none of the above analysis cites it making items more difficult, but their analysis of Rule 26 (all of the above) expresses concern that it makes items less difficult. Were they simply reporting on the literature, these differing results would be just different results for different phrases. But as they are offering their views in their actual recommendations, guidelines or rules, it is not even clear why a phrases impact on item difficulty automatically makes it objectionable. In fact, a plurality (48%) of their 2002 sources are against use of none of the above and only and slightly fewer (44%) are fine with it. There is no consensus.
Last, their 2004 book says, “When none of the above is used, it should be the right answer an appropriate number of times.” No, I do not have any idea what that is supposed to mean. My frequent co-author suggest that they mean something like “should only be the key approximately 25% of the time (for 4-option items) or approximately 33% of the time (for 3-option items)." But they’ve never shown that kind of thinking about how to read, understand or analyze items, so I don’t think she’s right. Of course, it her explanation has the benefit of giving some meaning to his rule—which otherwise lacks any
[Haladyna et al.’s exercise started with a pair of 1989 articles, and continued in a 2004 book and a 2013 book. But the 2002 list is the easiest and cheapest to read (see the linked article, which is freely downloadable) and it is the only version that includes a well formatted one-page version of the rules. Therefore, it is the central version that I am taking apart, rule by rule, pointing out how horrendously bad this list is and how little it helps actual item development. If we are going to have good standardized tests, the items need to be better, and this list’s place as the dominant item writing advice only makes that far less likely to happen.
Haladyna Lists and Explanations
Haladyna, T. M. (2004). Developing and validating multiple-choice test items. Routledge.
Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. Routledge.
Haladyna, T., Downing, S. and Rodriguez, M. (2002). A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education. 15(3), 309-334
Haladyna, T.M. and Downing, S.M. (1989). Taxonomy of Multiple Choice Item-Writing Rules. Applied Measurement in Education, 2 (1), 37-50
Haladyna, T. M., & Downing, S. M. (1989). Validity of a taxonomy of multiple-choice item-writing rules. Applied measurement in education, 2(1), 51-78.
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied measurement in education, 15(3), 309-333.
]