As a discipline, psychometrics is very focused on reliability. Well, it’s a quantitative discipline, and neither fairness nor validity is reducible to a number, so of course psychometrics puts an emphasis on reliability.
Unfortunately, we keep seeing the psychometric view dominate the multidisciplinary endeavor of educational measurement. And that means that reliability becomes the overwhelming criterion used to evaluate all matters.
Intersection or Union?
After I presented a paper this week about the importance of the facilitator role in getting the most useful feedback from teachers on how their students might understand and make sense of test items (and in making the human-in-the-loop part of automated workflows effective), a very smart audience member suggested using rubrics for their feedback so that their responses could be compared and checked for inter-rater reliability and then….
I interrupted her to explain that that would miss the point. We need panels of teachers who come from different perspectives and contexts so that we can get the range of issues that test takers from a range of backgrounds, experiences and contexts might face. We need that breadth of feedback because we need items to work across all contexts. Focusing on agreement limits the feedback we hear, rather than allowing us to create items that elicit evidence of the targeted cognition for the range of typical test takers.
As I said, she was smart. She immediately got my point and reflected it back to me really well. She nodded and said understandingly that we wanted the union of feedback across sources instead of merely the intersection. That was great, and it lays out a graphic that we can reuse in the future again and again.
Given the nature of the conference and the session, I have to imagine that she was a psychometrician—someone formally trained in the discipline of psychometrics. Her disciplinary training had her reaching for reliability automatically. As psychometricians lead automated item evaluation (AIE) efforts, this becomes a real problem. Whether feedback comes from multiple individuals, multiple AI agents or some mix of the two, we need to consider and weight the breadth of feedback—the union of all the feedback and not just the intersection. I am quite concerned that those with psychometric training will be so conditioned by this emphasis on reliability that they will undermine the basic purpose of item evaluation.
Always Blame Unidimensionality
In another session, I heard a very smart graduate student presenting on his efforts to break down some content to generate items using some new AI item generation. He explained the multidimensional structure of the content well, and it all made sense. But then he explained his procedure to evaluate the quality of the items that were produced.
He used point-biserial correlations. The standard tool for predicting model fit—to a unidimensional model. In practice, this is how we examine individual items’ contributions to Cronbach’s alpha—the most commonly reported reliability coefficient in technical documentation.
Reliability. Again.
But he was very clear about the structure of his content. And gave more evidence that there was at least one more dimension in the data.
Of course, he was just doing what he was taught to do. The most important consideration of item quality, from the perspective of psychometrics, is reliability.
But The Standards tell us that the third source of validity evidence is evidence based on internal structure, and he was clear that there were a handful of dimensions to his construct.
But psychometrics had him looking to reliability between/among items…when in fact they should not have been expected to have high point-biserial correlations.
Viva la revolución!
This tyranny of reliability is so often an obstacle to the processes of developing valid items (i.e., items that elicit evidence of the targeted cognition for the range of typical test takers) and the evaluation of item quality. It trips up even incredibly smart people who simply have been trained into a discipline that give it too much prominence.
I hope that we can establish a new regime that is more supportive of valid items and test uses that may be put to valid uses. I hope that contributors to educational measurement will be able to recognize when their preferred tools are appropriate and when they should defer to others’ tools from the start.