[Each day in October, I analyze one of the 31 item writing rules from Haladyna, Downing and Rodriquez (2002), the super-dominant list of item authoring guidelines.]
Formatting concerns: Format the item vertically instead of horizontally.
This one is just dumb. The New York Regents exams violate it all the time. Less than half of Haladyna et al.’s 2002 sources even mention this dumb rule, and nearly one quarter of them explicitly disagree.
In our own research, over half of respondents said that this is irrelevant, though the vast majority of the rest agreed that it is a good thing—though at the lowest level of value (i.e., Useful, as opposed to Important or Very Important).
There certainly is no consensus on this, and Haladyna et al. write, “We have no research evidence to argue that horizontal formatting might affect student performance. Nonetheless, we side with the authors who format their items vertically.” This is not a good basis for including a rule on a list that is supposed to be grounded in the consensus of the literature. It makes clear that this list is little more than a collection of their own opinions masquerading as research findings.
And yet, their 2013 book calls this an “important” (p. 95) item writing guideline. Nowhere do they cite any evidence for this, though they hypothesize that vertical formatting may be less confusing specifically for anxious test takers…without a milligram of support for this contention.
Yeah, “important.” Totally.
[Haladyna et al.’s exercise started with a pair of 1989 articles, and continued in a 2004 book and a 2013 book. But the 2002 list is the easiest and cheapest to read (see the linked article, which is freely downloadable) and it is the only version that includes a well formatted one-page version of the rules. Therefore, it is the central version that I am taking apart, rule by rule, pointing out how horrendously bad this list is and how little it helps actual item development. If we are going to have good standardized tests, the items need to be better, and this list’s place as the dominant item writing advice only makes that far less likely to happen.
Haladyna Lists and Explanations
Haladyna, T. M. (2004). Developing and validating multiple-choice test items. Routledge.
Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. Routledge.
Haladyna, T., Downing, S. and Rodriguez, M. (2002). A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education. 15(3), 309-334
Haladyna, T.M. and Downing, S.M. (1989). Taxonomy of Multiple Choice Item-Writing Rules. Applied Measurement in Education, 2 (1), 37-50
Haladyna, T. M., & Downing, S. M. (1989). Validity of a taxonomy of multiple-choice item-writing rules. Applied measurement in education, 2(1), 51-78.
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied measurement in education, 15(3), 309-333.
]