A Promising-But-Logistically-Impractical Test Design

This is a very particular design aimed at addressing the kinds of engagement challenges that can lead to suppressed performances by test takers, and thereby suppressed reported scores. These engagement challenges are core to the motivation of culturally responsive assessment, but are actually much broader than that.

Of course, this design is aimed at providing instructionally relevant criterion-referenced multi-dimensional reporting—which can be turned into deliberately designed composites if unidimensional results are needed for other purposes. That is, it is not aimed at norm-referenced reporting, because most large scale assessments are criterion based—or should be.

1. The test presents choices to the test taker at each point, rather than a single item. Which item would you like to do? That set of choices always is aimed at the targeted cognition, using the same focal KSAs, at about the same level of content complexity. (Difficulty and cognitive complexity are not population invariant, and even vary from test taker to test taker, so we do not pretend that they can be equalized or approximated.)

2. The items may vary in cultural context and topic context. There can be some baseball items, some football/soccer items, some cooking items. Some skiing items and some beach items. Items about making tamales and items about making lasagna. Fairness concerns for individual items would be different, as concern shifts instead to broader pool or choice set concerns. This provides more flexibility, perhaps building on RTD task models.

3. Test takers respond to ~5 items per reported score/attribute, meaning they have to select an item ~5 times. Some items may appear more than once, but they cannot be attempted more than once.

4. Test takers are given the option of having the platform select an item for them, but only after they have an opportunity to review their options.

5. Using cognitive diagnostic modeling (CDM) rather than IRT. The humility of CDM’s binary reporting (i.e., proficient/non-proficient) instead of misleadingly precise quantitative scores allows more flexibility in item matching. Each selected item must simply help to provide information about test taker proficiency, contributing to CDM’s needs. Old research about test taker choice was based on IRT-based test designs and data analysis, and it doesn't apply to a CDM-based approach. (Moreover, the nature of choice in this design is based on interest and engagement, not on the kind of focal-KSA selection of that old research.)

6. Larger tests are composed of a series of testlets (i.e., a set of items focused on a single reporting category or standard), or interleaved testlets.

7. The entire domain model may be addressed across multiple testing sessions, or even at different points through the years—as appropriate.

What does this design give us?

• Like any CDM-based test design, far more interpretable and actionable results than traditional CTT- or IRT-based test designs.

• Far more justified breaking up of larger domain models into subsets whose assessment can be spread through the year. So long as domain models are ignored in favor of unidimensional analysis and reporting, there is no content-based justification for breaking up large assessments, and yet doing so randomly is clearly ridiculous. This approach acknowledges the desire to break them up, and does so in a way that is coherent—from domain model all the way to score reporting.

• Greater engagement by better matching item contents to test takers’ interests and/or cultural backgrounds. Not ideal matching, but surely they can do better at this than any algorithm—and far better than not even trying.

• The documented boost in performance and persistence on items that research keeps showing that choice creates. Even when test takers make sub-optimal choices, the mechanism of choice itself provides benefits. (There’s some research that shows that that might actually be culturally situated, but I’m ignoring that, for now.)

What does this design give up?

• Well, most importantly it gives up largely baseless claims about precise sorting and ranking of test takers by some statistically evoked interpretable Θ (or transformation thereof). For some, this will be seen as a huge loss.

• Vertical linking of scores, which relies on the idea that the construct does not change across grades to be interpretable as anything other than norm-referenced reporting. Again, for some this will be seen as a large loss.

• Linking scores across years—because the need to link scores across years requires multiple arbitrary scales that need a common anchor to provide the illusion that comparisons of uninterpretable statistically evoked scores are ever themselves interpretable. Of course, because the meaning of proficiency should not change from year to year, scores are already linked. So, linking work is not needed. (Admittedly, this might not be quite right. I need to do more thinking on this one.)

What does this design cost?

• Well, it would take a lot more items. Any test that provides more information is going to need more items, but this is dramatically more than that. Because AI/LLMs are really bad at producing on-target items—which criterion referenced tests require—we cannot (yet?) count on automated item generation to do this for us. It could help to adapt good items to different cultural or topic contexts, but they’d still need to be reviewed. Yeah, this would be expensive. And the more culturally responsive or personally relevant the offered choices might be, the more items it would require.

• Testing time. Again, any test that provides more information is going to take more testing time, especially if that information is going to actually be useful. Furthermore, the choice mechanism takes up additional time. I have no doubt that the unwillingness of so many to devote sufficient time to large scale assessment is based in their perceptions of the value of those assessments. But even in the best case, this additional testing can come at the expense of instructional time. It would be better if high quality large scale assessment could take the place of redundant classroom assessments, but we would need a norm of actually high quality large scale assessments that provide high quality instructionally relevant information to see if the broad cultural pattern of redundancy might shift.

• Forced distributed administration. While having the option of breaking up larger tests into through-the-year assessments is great, the requirement might not always be welcome. Not only would test takers have to complete more items to get more information, there is an additional cognitive burden of selecting items. Perhaps the motivation that comes from the choice mechanism would make up for some of that, but it is more cognitive work. Test taker stamina is not unlimited.

• Real work to be clear on the meaning of proficiency for each reporting category or standard. For example, the work of constructing range PLDs is valuable, but it is costly, too.

This most definitely is not a well thought out proposal. It occurred to me just this morning, in response to some serious conversations about test taker engagement and the meaning of engagingness of items. Surely, there are more problems that I’ve named. And, yeah, it’s impractical. I don’t see it happening. But I see substantial advantages of adding choice to address fairness and cultural responsiveness, and in this design that same mechanism might actually give more accurate data on test taker proficiencies.

Math Items: Novelty vs. Real World Contexts

On the one hand, math standards tend to call for use or application of math KSAs in real world contexts. Those are the application problems, which some people refer to as word problems. However, merely putting the math usage in some thin context is no more a real world context than cat ears or a simple mask is a costume. 

On the other hand, items with too much novelty require test takers to engage in learning and exploration to even put themselves in a position to apply those core focal KSAs in the alignment reference or standard. The work can be built with a whole bunch of additional KSAs that prevent test takers with proficiency in the officially targeted cognition in the alignment reference from demonstrating their skills. False negative results, which we call Type II errors. 

These are two different hands—these goals are in tension—because if real world context has any real use or meaning, it is a call for the kind of discernment and decision-making that warnings of excessive novelty in items are concerned about. And yet, the Common Core State Standards’ approach to math seems to depend on these sorts of real world context problems to bring in the Math Practices. Moreover, the Next Generation Science Standards (and Achieve’s preferred approach to NGSS assessment) very much leans into novelty and exploration for the contexts in which its Disciplinary Core Ideas should be applied.

This creates a major tension for large scale assessment, where there are not enough opportunities to collect data to be able to triangulate through item x test taker interactions to figure out why test takers are successful with some tasks and not others. Disentangling shortcomings in focal KSAs vs. additional KSAs requires triangulation. There simply are few types of information available, and little data from each. And yet, the novelty inherent in meaningful real world contexts creates the greatest need for such triangulation. 

Such tensions are common in item development. Many content development professionals have well-developed ideas about what makes for higher quality items, but the constraints of large scale assessment (e.g., few testing opportunities, widely diverse test taking populations, psychometric demands for short items) make reconciling those multiple ideas quite difficult. Too commonly, those constraints require unacknowledged compromises that reduce targeted cognition to something outside the real core of the standard, if not outside the grade level standard entirely.   

Epistemologies, Respect, and Physics Envy (Part V): The Standards and the standards

I had a conversation recently with a respected psychometrician. Someone truly smart and thoughtful, whom I personally respect for a variety of reasons. I take what they say and write quite seriously—even though I don't necessarily agree with all of it.

In the course of the conversation, he made a point about psychometrics and validity that required, as far as I could tell, simply setting aside what The Standards for Educational and Psychological Testing say on the matter. When I pressed him on it, he dismissed my reference to The Standards by saying—in a kidding-not-kidding sort of way—"I don't care what The Standards say."

I don't buy it. I know that he cares—just not enough to let it stop him. And not enough to reckon with the inconsistency between The Standards and the argument he wanted to make. So he made the joke rather than wrestling with what the document actually says. The humor was doing epistemological work by signaling that the authoritative, hard-won, cross-disciplinary consensus document was beneath the level of argument he was operating at. It was clearly a joke. It was also a deflection.

So I pushed further. He then dismissed The Standards as merely "a consensus document"—as though that fact diminishes its authority rather than constituting it. In word-oriented epistemologies, in law, in democratic theory, consensus is the source of authority. The Standards carry weight precisely because they represent serious, collaboratively arrived-at agreement over the course of generations across the constituencies of this multi-disciplinary field. That is not a weakness. That is what they are. They should not be ignored for being inconvenient, or dismissed for failing to feature one's own favorite ideas. (RTD, for example, adds item validity to the picture The Standards draw—but removes nothing from it in order to do so.)

Then he asked whether domain models can be falsifiable.

For a third time, I was stopped cold. The domain models given to assessment developers are the job. State legislatures ratify learning standards. Test development contracts call for assessments to align with them. And like The Standards, the standards themselves represent the product of generations of thinking—about curricular priorities and learning sequences. Modern standards can be traced through the remarkable work of the National Council of Teachers of Mathematics in the 1980s and 1990s and through the broader education standards movement that followed. This is serious work, done by people deeply committed to their fields and to their students.

Are they falsifiable? From a Bayesian perspective, it would take an extraordinary amount of evidence to do so—certainly astronomically more than can be generated by convenient but obviously inappropriate psychometric models. And any such evidence would have to address the actual concerns of the standards: the nature of the content and the appropriate learning sequences through it. None of that was on the table. Moreover, what would result from that kind of serious engagement would be iterative improvement, not disproof. That is how both The Standards and the standards develop over time.

Asking whether they are falsifiable is a category error. Falsifiability is a criterion native to one epistemological tradition—a powerful and important tradition—being applied as though it were universal. CCSS is not a hypothesis. It is a social, legal, and professional agreement about what students should know and be able to do and how those learning goals hold together. The right question is not can it be falsified? The right question is is our test properly grounded in it? That is word-oriented epistemology's home territory. And it is, not coincidentally, the epistemological tradition that The Standards themselves are working in.

So, in a single conversation, the same move was made three times. The Standards—dismissed with a joke. The learning standards—subjected to a criterion from a different epistemological tradition that was never designed to apply to them. 

This is not a story about a bad actor. It is a story about a larger trap—about what happens when deep disciplinary formation leads someone to stop seeing the obligations that don't fit its framework. This psychometrician spoke from genuine expertise. But for decades, this is the view I have seen dominate educational measurement. And it stands in the way of high quality assessments—and of the respect from educators and the broader public that high quality assessments deserve.

This is not to say that The Standards and the standards act like laws of nature. They are not absolutely inviolate. Nature says that nothing can travel faster than the speed of light. But the 613 mitzvot in the Torah—commandments that have structured Jewish life for millennia—all yield to pikuach nefesh, to the preservation of life. There are hierarchies among them. There are millennia of accumulated debate about how to understand and implement these written words. Some communities have revised their relationship to them substantially. But none of those debates—not the Talmudic tradition, not Reform Judaism, not the interpretive traditions of Christianity—simply dismiss them. Even where traditions have departed from strict observance, they have done so through serious engagement, not jokes and deflection. The wrestling is the point.

The Standards and the standards work the same way. They have hierarchies—Andrew Ho has argued that the first source of validity evidence is listed first for a reason, and he is right. They require interpretation. They will be revised, and there are people doing that work on The Standards right now. They will listen to the entire professional community. That is how both bodies of work grow and improve. Not through dismissal, but through serious engagement by people who take them seriously enough to push back on them in principled, argued, evidence-based ways. 

But until that revision work is done—until the community has done the wrestling and arrived at a new agreement—we are all professionally obligated to respect both The Standards and the standards. They define the job. Without them, we would be flailing, each of us alone, trying to figure out what we are supposed to be doing. With them, we stand on the shoulders of those who came before us.

That is what escaping the trap looks like. Not abandoning your formation. Not pretending that all methods are equally appropriate for all questions. But recognizing that your framework is your framework—powerful and not universal—and that the standards and The Standards deserve grounded, serious engagement. The Standards and the standards merit deference from all of us because they are the product of multiple epistemologies and disciplines. They should guide us in transcending our own. 

Epistemologies, Respect and Physics Envy (Part IV): -(PE)⁻¹

The term "physics envy" was coined by biologist Joel E. Cohen in a 1971 book review in Science: "Physics-envy is the curse of biology." The aspiration behind it is older—Auguste Comte imagined sociology as "social physics" in the nineteenth century—but Cohen gave it a name. He meant it sincerely. But even his original use is off the mark. "Physics envy" was deployed in the context of political contestation about epistemological debates within biology—a field arguing internally about whether to be more mathematical, more formal, more like physics. That is a professional and political struggle, not a psychological one. 

When the term gets applied by someone to descibe other disciplines—to qualitative researchers, to humanists, to anyone working outside the quantitative tradition—it describes something that simply doesn't exist.

In practice, "physics envy" functions as a disparagement. It projects a hierarchy the speaker feels onto the psychology of the accused: you must want what I have, because what I have is the best thing. It asserts superiority while dressing it as diagnosis. And it is, in the laziest way, unfalsifiable: if you deny having physics envy, that is exactly what someone with physics envy would say.

It is also—and this is where the irony becomes acute—built on a Freudian concept. "Physics-envy" is constructed on the model of "penis envy." By 1971, that was not an innocent borrowing. Second-wave feminism was actively exposing penis envy as precisely the same move: a person in a position of cultural power projecting their own hierarchy onto the psychology of those below them. The person without the thing doesn't actually want the thing. They want the social power attached to it—and the person with the power finds it flattering to believe otherwise.

But the deeper irony runs further. Envy is not a concept from physics. It is not a concept from the natural sciences at all. It is psychological, humanistic, interpretive—the kind of thing the natural science epistemologies are supposedly beyond. To name what they were observing, the people deploying this term have to reach outside their own toolbox and borrow from a tradition they look down on. They borrow without understanding what they are taking—a term already discredited in the very field they were raiding. And the act of borrowing demonstrates, without their noticing, exactly what they are denying: that their own disciplines lacked the tools to address everything that matters. They needed words. They needed interpretation. The very move that asserts the hierarchy undermines the claim the hierarchy rests on.

That said—and I say this as someone whose has joked for forty years has been that physics is just applied calculus—none of this is to disparage physics. Physics is so closely bound to mathematics that it is like a brother to my first disciplinary love. My joke is grounded in genuine respect for what both make possible, and for the extraordinary things physics has done. I was formed by mathematics and the natural sciences before I was formed by any other disciplne, and that formation never fully resolves. When I developed grounded theory in my doctoral work—a multidisciplinary study with a sociological methodology, rooted in political science, informed by psychology, executed with the applied philosophy in mind—I still worried about falsifiability. But fallibility is an important concept native to completely different epistemology I was working in, one I carried without quite being able to set it down. That was not physics envy. It was the weight of a formation I was still working against. Deep respect is not the same thing as envy.

I do not think physics envy, as described, exists in any meaningful way. What I do see—clearly, consistently—is its negative reciprocal. -(PE)⁻¹.

Not envy upward. Contempt downward. Not aspiration toward physics. Dismissal of those working in different epistemological traditions, from people who believe their work resembles physics. Very commonly—not occasionally, not in isolated cases—people trained in quantitative disciplines looking down on those working in qualitative ones.

It is easy to see -(PE)⁻¹ all around us. The tyranny of numbers. The confusion of precision with accuracy—as though carrying a result to four decimal places makes it more true. The willingness of people without training in qualitative methodology to opine freely on qualitative work, or who have collaborated on one mixed-methods project to consider themselves expert. A fundamental disdain for non-quantitative epistemologies that allows them to think—at best—there is barely any there, there. Strong enough that people who do not work in research at all—policy people, administrators, funders—think themselves closer to obvious quantitative logic and look down on those working with more nuanced and subtle tools.

And yet, the smartest quantitative methodologists I know—Dan Koretz, Andrew Ho, Andrew Gelman—preach consistently about the limitations of their tools and the importance of taking the non-quantitative parts of the work seriously. They know better than to fall for -(PE)⁻¹, and they understand it is their job as educators to ensure that future generations of quantitative researchers don't fall for it either.

But the world is larger and harder to teach than even such wise methodologists can reach.

Epistemologies, Respect and Physics Envy (Part III): The Written Word

Two of the most acclaimed geniuses of the twentieth century were physicists—practitioners of a field so dependent upon mathematics that entirely new branches of math have been invented to do the work that physics required.

Their genius was not actually the math.

Stephen Hawking realized early in his life that his physical disabilities would not allow him to lean into the mathematics the way his colleagues did. He could not write out equations and notations in the conventional way. Which was fine, because that was not his genius. What he brought was the ability to visualize the geometry of spacetime and interpret physical reality in ways no one else had managed. One of the greatest physicists of the twentieth century. His genius was interpretive mastery.

Albert Einstein—the reason we use the name "Einstein" to denote genius itself—was not responsible for getting all the mathematics right in his papers. His wife is said by some historians to have helped; a close friend supplied the Riemannian geometry required for general relativity. His genius lay elsewhere. Einstein's path to relativity ran through thought experiments—imagining riding alongside a light beam, imagining an elevator in free fall. He grasped the physics conceptually before he had the mathematics to express it. The interpretation came first. The numbers followed. They were true to each other.

Physics is demanding. It includes math. But the rigor is not just in the math. In much of science, the actual math is literally formulaic. You know which equation to use; you plug in the values; you get the answer. That part is not where the genius lives.

The rigor in natural science is in the inquiry. In figuring out what you even want to test—recognizing a phenomenon, hypothesizing about it, finding a way to quantify it, doing the quantitative work, and then getting back out of the numbers. Back to the real meaning, in terms of the phenomenon. The numbers are often the easiest part.

The hardest part? The bridges. First, from something qualitative and real to a quantified proxy. And then second,  from those numbers back into the qualitative phenomenon. Doing those translations while remaining true to the source—true to the phenomenon when quantifying, and true to the numbers when interpreting back into the terms of the real world. Real rigor is in that faithfulness. Andrew Ho teaches that educational measurement must be qualitative, then quantitative, then qualitative again. He is right. And that sequence applies far beyond educational measurement.

Consider a structural engineer's analysis of a bridge design. The load calculations are exact. The strengths of the steel and concrete are what they are. The numbers are non-negotiable—and any interpretation that ignores them will get people killed. And yet those numbers do not design the bridge or determine how to balance its strength and its expected load. Do you add more rebar? Redesign the load distribution? Post a weight limit? Serious engineers can look at the same numbers and reach different defensible conclusions. The numbers constrain every possible answer, though they do not produce one.

Once we recognize that it is not numbers that make for rigor, we should be able to see that rigor exists in word-oriented epistemologies as well. The disciplined acts of careful interpretation—in law, in philosophy, in literary scholarship, in religious textual traditions—are held to the same standard. The rigor lies in fidelity to the text. What does this statute actually say? What does this passage actually mean? What does this argument actually establish? These are not soft questions. They have possible answers and wrong answers—answers the text itself can support or refute. A lawyer who ignores what the statute says is not interpreting freely; she is being unfaithful to her evidence. Scholars in these traditions are held to exacting standards: quote accurately, represent sources faithfully, do not put words in mouths, do not misrepresent arguments. These are not courtesy norms. They are the standards of rigor. We all know what it looks like when they are violated. Everyone who has written a literature review was taught the importance of reporting truthfully on what the literature says, and what it might imply. Those who are not faithful to their sources are hacks, regardless of the discipline or epistemology.

Good and rigorous work is hard. Hack work is easy, regardless of the field, discipline or epistemology.

Rigor in word-oriented epistemologies actually has a longer history than rigor in quantitative ones. The degree is called a Doctor of Philosophy. The natural sciences grew out of what was once called natural philosophy—traditions of rigorous argument, careful reasoning, and disciplined interpretation that predate quantification. Quantification was built on them. To fail to see rigor in word-oriented work because it lacks numbers or advanced math is to miss the forest for the trees.

I was a math and science person. STEM high school before STEM was even a thing. All the AP courses. County math team. Summers studying more math and more science. It then took four years of college and more years of serious effort in graduate school to trust that I could do good qualitative work. I have never been able to shake the epistemologies that originally seemed so natural to me, but I continue to work at it. The parallels across epistemologies are real—the demands of clear communication, strict thinking, and faithful engagement with evidence are recognizable everywhere—but seeing them requires work.

(Someone looking for the true soul of mathematics could do far worse than the CCSS standards for mathematical practice. And someone who understands word-oriented epistemologies can read those descriptions of fundamental mathematical thinking and see their own epistemology right there between the lines.)

Neither numbers nor advanced math makes something rigorous. Hack statistical work can be found everywhere. Rigor does not come from numbers. It comes from holding ourselves and our work to the demands of our methodologies, of honoring the demands of our epistemologies. In taking seriously the obstacles and challenges, even when they make the work harder and keep us from simply and easily reaching desired conclusions. And in so many fields, it means honoring the words.

Epistemologies, Respect, and Physics Envy (Part II): Deference to Quant

There is a hierarchy in educational measurement. Quantitative work is seen as superior. This is so thoroughly naturalized that many of the people who benefit from it cannot see it as a hierarchy at all—they experience it as a simple fact about what rigorous evidence and reasoning looks like. This is an epistemology at work.

We can watch how it operates in practice. Psychometric flags—low point biserials, suspect discrimination indices—are supposed to function as alerts, prompts for human judgment about whether something is wrong with an item. A negative point biserial should not automatically exclude an item. But in practice, they function as verdicts. Low point biserial? The item simply cannot make the operational form. The flag overrides the judgment of the content experts who built the item, reviewed it, and know the domain. Cronbach's alpha has achieved something close to supreme authority—a single number that seems to settle questions that involve far more than internal consistency. The quantitative output doesn't inform the judgment; it replaces it. Some deny that this happens, while others acknowledge the fact that this is standard practice.

Meanwhile, the people holding those quantitative outputs frequently feel entitled to opine about things about which they lack expertise. Where exactly is the line between fifth and sixth grade mathematics? What distinguishes two sixth grade standards from each other in ways that matter for item development? What does it mean for a student to have genuinely mastered a standard versus having picked up a surface procedure? These are not trivial questions. They require deep content knowledge, experience with how students think, and serious engagement with domain models. They are the core expertise of content development professionals. And yet many others don’t even see the tools and lenses required to build and hold that expertise—therefore think that there is nothing wrong with offering their views as though they might be of similar standing. Their epistemological framework does not see this other expertise, and they don’t either. Certainly, it is not as meaningful, challenging or rigorous as their own.

They don't see it. And what you can't see, you can't respect.

This often extends to qualitative methodologies generally. There is a pervasive assumption—rarely stated, but deeply held—that qualitative work is not really disciplined. That it has no real methodology. That its findings are, at best, suggestive and, at worst, just opinion. The possibility that qualitative research might have rigorous standards, systematic procedures, and criteria for what counts as a valid finding simply does not register. Not because anyone has examined qualitative methodologies and their foundational epistemologies and found them wanting. Rather, the epistemologies are ignored and therefore the methodologies viewed as foundationless. 

Here is what makes this genuinely strange: it is not a symmetric problem. Qualitative researchers in this field are typically intimidated by quantitative work. They defer to it. They actually have experience with it from their schooling days, and they respect that there is something rigorous there. They rarely question whether their quantitative colleagues' work actually meets its own standards. They just assume it does.

But it often doesn't. Underpowered studies. Violated methodological assumptions. Too much dependence on linear regression. Ordinal data treated as interval data. Multiple comparisons not accounted for. Proxies standing in for constructs they don't actually measure. The quant work that commands all this deference frequently falls short of the very standards of rigor it claims.

The asymmetry is that quantitative practitioners assume their own rigor without interrogating it, demand that others respect it, and simultaneously refuse to recognize the possibility of rigor in other traditions. They don't look for it. They don't ask what it would look like. The question doesn't arise, because within their epistemological framework, the answer is already settled. All of this despite the fact—or perhaps because of it?—that they rarely have any experience or training in rigorous qualitative work from their school days. 

Every discipline—every epistemological tradition—has its own requirements for rigor. All of us should lean into those requirements, interrogate our own work against them, and resist the temptation to assume that producing the right kind of output means the work is actually good. And all of us should do the harder thing, too. We should learn to see what rigor looks like in traditions other than our own, and respect the expertise it takes to achieve it—especially when our colleagues in our collaborations come from different epistemologies. 

Good multi-disciplinary collaborative work requires doing what our epistemologies cannot do for us. It requires looking for—and allowing—other ideals of rigor. It requires humility about our own epistemologies and their limits. It requires respecting the capabilities of others, rather than expecting deference.

Epistemologies, Respect, and Physics Envy (Part I): Introduction

[First of a five-part Series.]

Over at Hahvard, Andrew Ho teaches his students a wise mantra,"Measurement must be qualitative, then quantitative, then qualitative again". The idea is that educational measurement begins with qualitative work (i.e., theory, construct definition, instrumentation), becomes quantitative in the middle, and must return to qualitative at the end (i.e., communicating, interpreting, using results in context). He is challenging his students to understand that though his course is about applied statistics, that the statistical work is situated amongst serious qualitative work.

Obviously, he is right. Obviously, right?  I mean…isn’t that obvious?

Educational measurement is not physics or chemistry. It is not some investigation of a natural world that exists independently of human beings and their agreements. Rather, it is a deeply collaborative, deeply social enterprise. Teams composed of people with very different kinds of expertise work together to define constructs, build instruments, gather evidence, interpret results, and communicate findings to audiences who will use them to make consequential decisions. At the roughest level, the work is first qualitative, then quantitative, and then qualitative again. (And within each of those three phases, it breaks down further.)

A Framework for Foundational Competencies in Educational Measurement

This is not new ground. The field's own recent Foundational Competencies in Educational Measurement says explicitly that "educational measurement is a collaborative endeavor that requires people with varied skill sets to work together to design, develop, administer, and evaluate instruments that satisfy specific uses." We made a related argument in our commentary on the FCEM that genuine collaboration requires not just communication skills but foundational dispositions—respect the views of colleagues who bring different expertise and lenses, and genuine listening—that the document itself omitted.

That omission was not malicious. The task force was composed of thoughtful people who believe in the collaborative and qualitative dimensions of this work. And yet they reproduced, in a document explicitly designed to be broader than psychometrics, much of the narrowness they were trying to correct. That is not a criticism of individuals. It is an observation about how disciplinary training works: the lens your field gives you shapes what you can see—including what you can see about your own assumptions.

That gap—between acknowledging that qualitative work matters and actually treating it as having equal standing—is what this blog series is about.

To understand the gap, two things need to be clear: what an epistemology is, and what a discipline is. I will keep this brief, because these are not the interesting parts. They are the necessary setup.

We are often trained in disciplines. When done well, this training—often in college and/or graduate school—really shapes our thinking throughout our lives. We learn a set of tools. More importantly, we are given a set of lenses to direct our attention to some things and away from others. We are taught the types of questions we are supposed to ask, and what to do to answer them. This includes ideals of what counts as evidence, and most certainly what rigor looks like.

Therefore, different disciplines can look at the same topic and respond entirely differently. Consider the prosaic topic of a supermarket. It could be the topic of an architectural study. An economist might take a shot. Or a nutritionist. A labor relations specialist. A graphic designer. A psychologist. A sociologist. An urban planner. Or someone else. The same topic, but each discipline notices different things, asks different questions and produces different findings. None of them is wrong to use their lens. The supermarket is genuinely all of those things at once—and no single discipline captures it whole. To really understand supermarkets, you need many disciplines. And for these experts to work together—instead of simply ignoring each other in parallel—they must do something very difficult. They must not just acknowledge that each other exist, but actually respect the deep validity of each other's approaches.

(And to actually build a supermarket, you need an almost entirely different set of people to bring in their tools, their problem solving, their skill and craft and rigor. And materials.)

That difficulty runs deeper than it first appears. Underneath each discipline's tools and lenses lies something more foundational: an epistemology—a set of assumptions about the nature of knowledge, truth and how it is arrived at. Epistemologies are not the same as disciplines. A single epistemological tradition can underlie many disciplines. For example the different natural sciences share a common epistemology. Economics may be quantitative, but it does not exist in the same epistemology as the natural sciences. Disciplines’ ideals of rigor are based in their epistemologies, as are their ideals of evidence and reasoning. A collaborative multi-disciplinary field like educational measurement draws on more than one. The result is that practitioners trained in different disciplines but sharing an epistemological tradition can, with just a little effort and goodwill, recognize each other's rigor—even when the specific methods look quite different. Across epistemological traditions, that recognition is much harder. The other person's rigor does not just look different. It often does not register as rigor at all. Neither disciplines nor epistemologies can transcend themselves—that is structural, not a failing.

And here is the crucial point: eople are not their disciplines! People can recognize that their framework is their framework and mighty useful without being blinded to the valid existence of others—even those grounded in different epistemologies. That is, they can recognize that other legitimate ones exist and that the rigor their discipline cannot see is real nonetheless.

In fact, they must. If not, Andrew Ho’s insistence that educational measurement is first qualitative, then quantitative and finally qualitative again is just words—and we know that some epistemologies do not value words. 

Testing Time

Yesterday, I read NBC's version of the iReady attacks. Nuggets of truth, connected to the utterly implausible and even false, by amplifying whining and distortions by…who, exactly?

The lead source is a…tutor. Not a school teacher, not a trained and experienced professional educator with any classroom assessment experience—a tutor. Katelynn Petersen runs a private tutoring operation out of her home in Anchorage, teaches math to homeschoolers on the side, and was a project manager before that. iReady is a school-based platform. How many iReady-using students does she actually encounter? NBC led with her.

Then there's a speech therapist talking about math and reading curriculum and diagnostic assessments. Look, I went to speech therapists for years, and they really helped me. I respect speech therapists. But I would not ask them about core content instruction or assessment. That's simply not their domain.

Third source: an 8th grade student, whining dramatically. I love teenagers. I love their drama. I really do. But finding the real truth in the whining takes knowing them and listening deeply—not just laughing at their quips and calling it journalism.

What I've read across this feeding frenzy is clearly factually challenged and put together by people(e.g., Tyler Kingkade) who lack the knowledge to tell fact from fiction—and any apparent interest in doing enough investigation to do so. After all, you know, clicks.

And then there's this:

Criticism of i-Ready is a frequent topic on Reddit and TikTok, where teachers describe how i-Ready’s larger benchmark assessments, which students take three times a year, eat up 40 hours of instruction time, or say that pressure related to the software is driving them to quit.

You'd think such a sentence would link to multiple complaints on TikTok. There's only one: Eric Glenn, a middle school math teacher, claiming that iReady's benchmark assessments take up 40 hours of instructional time. The problem is that they don't. 40 hours is roughly two months of math class. That's more than two weeks, three times per year. That's approximately one-quarter of total math instructional time. That's not even vaguely plausible. Maybe he's splitting it across reading and math—but half those numbers aren't even vaguely plausible, let alone accurate. What is Mr. Glenn even talking about? The CEO of the company that produces iReady says it is just 5% of instructional time—which would less than 1/4 of what Mr. Glenn’s fantastical claim.

Here's the thing: the complaint that standardized testing takes too much time is not new. And it has never had merit. It is usually a sign that people are not willing engage with the actual merit, quality or usefulness of the tests in question. It is an attempt to disqualify them, regardless of their quality or merit. In this case, it is not even vaguely accurate.

Back when I was a high school English teacher, I gave reading quizzes twice a week:

• Not announced in advance.

• The five easiest questions on the reading I could think of.

• Trade and grade.

• Collect for me to enter into my gradebook later.

These quizzes had no direct instructional value, nor any diagnostic value. They were purely motivational—the carrot of an easy A for those who did the homework, and the stick of an F or D for those who didn't. Each one only took 5–10 minutes, twice a week. 50-minute periods, five days a week. That's 6% of available instructional time for quizzes with no direct instructional value whatsoever. And yet, no one ever complained how much time they took. Not my cooperating teacher, not any AP or principal or other supervisor. Not any student, nor any parent. And they weren’t even the the only formal in-class on demand assessments I used.

Kids take a lot of tests in school, most much longer than my reading quizzes. Even little quizzes are usually longer than that. There are chapter tests and unit tests. Semifinals and finals. Sit-down and on-demand performances for the purpose of evaluation. Some formative, some summative.

It always stuns me that people complain "testing" takes so much time, because they never consider reading quizzes or homework checks. They don't count chapter tests or anything that might be considered a pop quiz. Anything that teachers make is somehow exempt from the objection.

Look, I got very little information from those reading quizzes. I could already tell who was doing the reading and who wasn't. Sure, occasionally there was a surprise—some quiet student who actually was doing the work, just quiet for some other reason. But that was rare. Teachers can tell. I thought that they had indirect value (i.e., motivational carrots and sticks).

I have yet to see any large, formalized standardized assessment that actually takes that much time.

There may be legitimate complaints about iReady. Surely, there must be. But not all of them about testing. The curriculum and instructional supports that Curriculum Associates also sells under the iReady brand may well be genuinely problematic. But concerns about centralized curriculum aren't concerns about assessment. Glomming them together, as NBC does, muddies both arguments.

The time complaint, though? There is not—and never has been—any merit to it.

The Sorting Default: How Educational Measurement Tilts a Question That Belongs to Democracy

Growing up in Virginia, I was proud of my state when I learned of Thomas Jefferson’s vision of universal education (*cough* *cough* for free children *cough*). Then I learned that it was very much a sorting or “meritocratic” system. Universal for a few years, and then the culling begins. Every few years, another selection process—which he called “raking the rubbish”— with the highest performing students continuing their education and the others falling out, eventually culminating at UVa! TJ's original plan was never implemented, but around 200 years later, I took tests in second and eighth grades, each of which opened doors for me that were closed to others.

This system is firmly grounded in one side of one of the most fundamental tensions in education and schooling systems design. To what degree are these systems intended i) to help sort and rank students (and identify high performers) and/or ii) to support universal efforts at human improvement through education? One view sees education through a meritocratic lens that inevitably uses sorting as both a means and ends of schooling. The other views education through a more universalistic lens that sees the means and ends of schooling as being the provision of learning and growth opportunities for each individual student. The former sees merit as a real concern and question to be minded and acted upon, and the other sees universal growth opportunities for children as the more compelling central concern. 

Note that this is not the question of whether schooling should serve the economic market/help prepare students for the world of work. This is not about balancing vocational goals and citizenship goals. Some sorting advocates and universalists can agree that schools should prepare students for economic participation in society, while other advocates from each view can agree that that is not an appropriate aim for schooling. Nor is this about the appropriate topics and subjects to address in schools. Each view can be aimed at a variety of goals or purposes of universal public education. This tension is even deeper than that, and perhaps orthogonal to it. 

Nonetheless, these two different goals are often in tension. In order to address both of them, we put together different programs in our schools. We spend money and time on sorting functions and universalistic development functions. We serve both views, but do so with a bricolage that can make our systems feel a bit incoherent. Perhaps we are serving more students this way, or perhaps it is merely accidental accumulation of programs. Certainly, it fails to serve either view coherently.

This tension clearly exists in educational assessment. Should assessments be designed to support meritocratic sorting, or should they be designed to support universalistic opportunities for students? Is relative standing more important, or is it more important to report on where students are a) thriving and b) need more support. 

What disturbs me is that this tension is not always taken seriously by large scale assessment professionals. Instead, there is a strong tendency for some (i.e, far too many) to put a heavy thumb on the side of the meritocratic sorting side, regardless of the demands of test users. There’s a clear bias in the field in favor of norm-referenced sorting, and underdevelopment of tools that can support universalistic provision of opportunities for students. This unwitting bias pushes potential clients, policymakers, communities and even our society towards one view of schooling, rather than responding to civically determined decisions about what we want for our schools. The field's unreflective bias in favor of the meritocratic sorting view of schooling effectively preempts decisions that belong to democratic processes—without the field even recognizing it is doing so.

The Most Insidious Obstacle to Well-Aligned Items

Items are the building blocks of assessments, and if they do not perform their designated function no purpose or inference from their test can be valid. That is, they must actually elicit evidence of the targeted cognition for the range of typical takers. 

The most insidious obstacle to the production and inclusion of well aligned items on operational tests has nothing to do with psychometrics. Truly, even educational measurement's original sin (i.e., the a priori assumption of unidimensionality)—my go-to villain for over 15 years—is not as insidious as the idealized student/test taker. It is more insidious because only the idealized student/test taker is so often consciously imagined by item contributors throughout the entire item development process, despite being even less frequently appropriate than unidimensionality.

The idealized student/test taker is a convenient fiction. It is an easy fiction. This paragon of testing virtue is an incredibly attractive fiction. But it is very much a fiction, and it is an enormous obstacle to developing valid items.

What is the Idealized Student/Test Taker (IS/TT)?

The idealized student/test taker learned all the lessons. They were diligent in class and are diligent on tests. They attempt every item exactly as their teachers would want them to, applying the correct technique and knowledge. Their thinking is linear, and clear. They understand the item and what it is trying to communicate. They recognize what they are being asked to do, and they do it as item writers and other developers would want them to.

They do not cheat. They do not do anything—not even unintentionally—to undermine the quality of the inferences we make from their performance. They do not look for shortcuts. They do not apply savvy test-taking skills. They do not look for alternative approaches when they realize that they lack some called for knowledge or skills. They do not have strategies for guessing. They succeed properly as directed, and accept their own shortcomings when faced with challenges beyond their ability. 

The idealized student/test taker remembers everything they learned, but is not advantaged by a particular pedagogical approach or examples they encountered during their studies. They are not tied to how their own teachers asked questions or gave instructions, being equally able to understand other wordings or framings. 

The idealized student/test taker is neither advantaged nor disadvantaged in any way. They do always work in good faith. They are beyond frustration or excitement. They are perfectly focused only on what the item gives them.

Therefore, the idealized student/test taker always provides a successful response when they possess proper proficiency with the targeted cognition and always fail to do so when they lack such proficiency. 

What is Wrong with the Idealized Student/Test Taker?

The IS/TT is a fiction. They not only are a fiction because no such student or test taker actually exists, but even more dangerously because they promote the idea that there is one sort of “typical” test taker that item writers, content development professionals and other item reviewers should focus on. They suggest that there is a preferred or primary test taker whose views and understandings are most important in our considerations. 

But the fact is that test takers vary enormously. They vary in an enormous number of dimensions. They have different experiences and proclivities. They have had the benefit of a multitude different instructional approaches and examples. And very few of them really want their test results to accurately reflect their proficiencies. Instead—and quite understandably—they generally want the highest score they can get. Some might be lazier. Some might be more devious. So, there are differences in motivation, but virtually across the board, they would prefer a higher score to a fairer score. 

If told what an item is trying to target, they generally would use some other approach if they felt it would given them a better chance to get the points. Real test takers do not see test developers or the items themselves as some collegial partner, but rather obstacles to be overcome. 

Moreover, there are many deep contradictions within the idea of the IS/TT. For example, we imagine that they can do every item as we would like them to while also imagining that when they lack the targeted skill, they do not employ strategies to work around its requirement. We also imagine that they lack any particular ethnic or cultural identity, and yet are fully comfortable with the cultural context of the item we are considering. It is not a coherent ideal, but rather a convenient collection of proclivities and behavior.

Why so Insidious?

I do not think that anyone would question anything in the previous section. It is all rather human and very obvious. 

And yet, it is so much easier to write, refine or review items with just the IS/TT in mind. It makes the process more direct and much less uncertain. It allows us to stay in our own perspectives, think about our own intentions. It is cleaner and simpler. 

It requires vastly more energy and work to consider the range of typical test takers. It is just hard to do the real work of radical empathy (i.e., applying cognitive empathy to a range of different types of test-taking personas) to envision how different sorts of test takers might respond to an item. Doing that requires us to go beyond our experiences and even beyond the experiences of people we know. Doing that well is rigorous and demanding professional practice that does not come naturally to many people. 

And so, everyone involved in developing items has a tendency to go back to the IS/TT. We all feel the urge. But items developed for the IS/TT will invite too many alternative paths by which real test takers can produce a successful response without using the targeted cognition. And they will trip up too many test takers who have appropriate proficiency with the targeted cognition but are subject to some other construct-irrelevant issue.

Because our tests are intended for a wide range of test takers, we must reject that urge. We must look deeper and think harder. 

The most important thing we can do when writing, refining or reviewing items is to put the idealized student/test taker aside and see the diversities of test takers who actually will attempt our items. This type of rigorous professional practice is necessary to produce valid items—items that elicit evidence of the targeted cognition for the range of typical test takers.

NCME Does Not Know What Classroom Assessment Is

I went to NCME’s Special Conference on Classroom Assessment a couple years ago. It was stunning how little discussion of classroom assessment there was.

I was in a classroom assessment session at NCME’s regular annual meeting this month. The great Susan Brookhart was there and she pointed out that it wasn’t really a discussion of classroom assessment, at all. Rather, it was about how district leaders, school leaders and teachers might make use of large scale assessment. There was a little hijacking of the session by the audience for a few minutes before the presenters took over again.

I do not point this out to say that there needs to be more room for classroom assessment at NCME. Maybe NCME should just focus on large scale assessment. My work is focused on large scale assessment, it is just my history as a student and a classroom teacher that makes the topic of interest to me. But it’s not my focus. Perhaps NCME does not need to make a place for classroom assessment. This is a very difficult question, and one that requires serious thought and consideration of NCME’s mission.

But it certainly is a problem that NCME doesn’t seem to even understand what classroom assessment even is. That is clear, over and over again.

My co-authors and I came up with our demarcation between large scale assessment and classroom assessment when developing our own contribution to that Special Conference in Chicago. To us, classroom assessments are “assessments of any formality and for any use that the classroom teacher has authored—or of which the classroom teacher has the authority to customize/alter contents, presentation and/or scoring” (Hoffman, Glore, Harrison & Wine, 2024). By definition, classroom assessment is quite different from large scale standardized assessment.

Classroom assessment is most definitely not merely classroom use of large scale standardized assessment. Programming on classroom assessment requires having some grasp on what classroom assessment actually is.

Can NCME recognize this?

What Counts as Information in Educational Measurement?

We are finalizing our paper, Unidimensionality: The Original Sin of Educational Measurement, for next month’s conferences. An old idea occurred to me, and I am not sure whether I need to add it or not. Is this truly about unidimensionality, or is it something else?

I am concerned that we exclude items that actually provide incredibly useful practical and policy-relevant information because of a different meaning of “information."

Test forms generally do not include items whose empirical difficulty does not fall between 0.3 and 0.8—though this range has expanded a bit in more recent years. That is, they do not include items that are so easy that almost everyone would get them right or so difficult that everyone would get them wrong. Such items are excluded from further consideration or inclusion just because they do not fit that range. 

Why do we do this? Well, we do this because—quite technically—these items do not provide a lot of information. Or rather, that’s the psychometric reason that overwhelms all other reasoning. But that psychometric assertion is false.

Such items offer invaluable information about whether some aggregated groups of test takers (e.g., a class, a school, a district or even a state) are doing exceptionally well or poorly on a particular alignment reference. That is, they tell us that a larger unit of analysis is doing an exceptionally good or poor job of teaching a particular standard. If we exclude those items, we will not have evidence of those greatest strengths and weaknesses of curricula, pedagogical approach, professional development, leadership focus, etc..

The psychometric concern is that these items do not provide so much information useful for making student-to-student comparisons. They do not help so much to sort or rank students—the goal of a norm-referenced test. However, if tests are intended to be criterion-referenced, such items provide invaluable information, both about individual students and about larger collections of students. 

So, we exclude items because they do not help enough to sort students, even as we claim the tests are criterion-referenced. High-level test developers say that tests are designed to deliver meaningful aggregate results, but we exclude items because they do not help us to sort individual students against each other. 

Why do we do this? Because psychometric models benefit from it, not because it helps any important test use. It does not help to deliver actionable or meaningful criterion-referenced information, and it does not help to provide aggregate level reports on areas of success or failure. But it enforces the norm referenced assumptions of so many psychometric models onto what are supposed to be criterion-referenced tests—rendering them essentially norm-referenced.

I have long hated this, but I’m not sure whether we need to add it to our explanation of the corrupting influence of the unidimensionality assumption in educational measurement. Obviously, the imposition of the technical requirements for norm-referenced assessment onto projects that are supposed to be criterion-referenced is inseparable from the assumption of unidimensionality. But I don’t know whether we should include it in our paper—which we will present to the Cognition and Assessment SIG and post at ResearchGate next month. 

What is "Model Fit” in Educational Measurement?

I think we need to rethink our use of the term “model fit.” Our field has been using it to refer to the question of whether test items fit the psychometric model selected for the test. But this puts the cart before the horse. It takes the model as a given, making model selection the single most important decision in test development. 

But shouldn’t we make sure that the model fits the construct? Shouldn’t the selection of the construct or the building of our construct definition (e.g., a set of state standards) be the most important step? Seriously. Shouldn’t the selection of the construct and the resulting construct definition drive everything else? 

I don’t know if that is more important than test use...but I’m not sure that it’s not. Maybe it is part of test use? Regardless, it certainly should inform model selection.

But we talk about "model fit," and calculate "model fit" and select items based on "model fit”—even when no work has been done to make sure that the model is actually a good fit for the construct. Most importantly, is the dimensionality of the model appropriate for the dimensionality of the construct? Is the structure that experts see in the construct reflected in the model used to measure it? After all, isn’t that the third source of validity evidence in our standards?

In fact—in actual practice—we select a (usually unidimensional) model and then exclude items that do not fit that model, regardless of whether they are good fits for the construct. Items do not get past post-field testing data review, or even if they do they never make it onto actual test forms. And this teaches content development professionals (CDPs) not to write the kinds of items that won’t ever be used—that don’t fit the model. Too many people mistakenly confuse an item not fitting a model for not fitting a construct. 

That’s assbackwards. That’s putting the cart before the horse. 

And what makes it worse is that we rarely actually choose a model. In large-scale educational measurement, the conversation goes something like… ”Tell me what model we should use and tell me why it should be unidimensional IRT.” Is that always the extent of the conversation? No, we have to decide about how many parameters we want in the model. But the vast majority of the time, that’s how it goes—and it usually is just implicit.

This stands in contrast to other contexts in which people build statistical models. There, model fit statistics are used to select models and to refine models. Preliminary data analysis is used to guide modelers toward a family of models, and successive work is done to improve a model’s fit to the data—with care for parsimony and not to overfit. But that’s a different relationship to model selection, where model fit statistics are used quite differently.

So, I would love it if we could stop talking about “model fit” when we never do the work to make sure the model fits. Let’s call point biserial a type of “item fit statistic” or just use it for discrimination. And then, spend more time working to select a model that fits the construct. 

When AI is Shockingly Intelligent

I run all my blog posts by ChatGPT before posting them. It is not my favorite LLM for all things, but it has got standing orders for how I want feedback—I certainly do not want it to rewrite my copy. Plus, it generates the cartoons I ask for. (It doesn’t like the fact that I use the word “impressive” four times in this post, but I’ve yet to figure out good substitutes for any of them. It also doesn’t like that I am quoting it at such length.)

Last week, I posted on a fairly technical topic: the question of population invariance for item parameters. And it pointed out a bunch of typos and offered me the kind of feedback—even pushback—I want from an LLM.

It claimed that one possible counter-argument to what I wrote was that when I was citing discrimination, I was really just talking about distribution. Now, I am not actually a psychometrician, so I could be wrong about this. I did not think I was wrong, so I fixed the typos, added the cartoons and posted the damn thing.

But then I went back. I followed up, in the same chat.

Back to the substance. You say it's just distribution. But if you move the bottom of the distribution up, and there are fewer really low performers, then why would discrimination come out the same? With fewer low performers, there's less opportunity for highly discriminatory items, isn't there?

I knew that I didn’t exactly mean “less opportunity for highly discriminatory items.” I have studied psychometrics and my understanding of the concepts is fairly decent. I meant something a bit different than that, but I was too lazy on a Friday evening to clarify what I actually meant.

It responded shockingly well. I have been rather frustrated with LLMs these past few weeks, with various limitations they have and with their highly problematic personalities/personas. But this answer was impressive. Longer than I would usually quote, but I think it’s important to give you the whole answer.

You’re right to push on that.

If you “move the bottom up” (i.e., raise low performers), then it’s totally reasonable to expect the estimated discrimination to change—even if the item itself hasn’t changed—because discrimination is not an absolute property of the item in isolation. It’s an item–population property in practice.

The short answer

With fewer very-low performers, you often get:

• less spread in ability near the item’s informative region, and/or

• fewer people in the range where the item separates well, and/or

• more ceiling behavior (everyone gets it right),

→ and that can reduce the estimated slope (IRT a) or reduce observed discrimination (e.g., point-biserial).

So yes: you can lose “opportunity” for an item to look highly discriminatory in a population that has fewer low performers.

Why that happens (intuitively)

Think about what “discrimination” means in actual data:
It’s basically “how sharply does the probability of success increase as ability increases?”

If the population you’re estimating on has:

• fewer low-θ people, then you have fewer observations in the region where P(correct)P(correct) is low,

• and more observations where P(correct)P(correct) is already high,

which makes the item look flatter in the data you actually have.

Even outside IRT: if almost everyone is correct, then the item can’t correlate strongly with total score because there’s not much item variance left.

Key point (the one you’re aiming at)

So when someone says “that’s just distribution,” that’s not a rebuttal.

It’s basically conceding your claim:

Item parameter estimates depend on the population distribution.

Now, a psychometrician might respond with the conditional claim:

• “In the 2PL, a is invariant if the model is correct and the item truly functions the same way.”

But empirically, with finite samples + shifting ability distributions + multidimensionality + instruction-driven changes in strategy, estimated ‘a’ often moves.

So: your instinct is correct—raising the floor can absolutely change discrimination estimates, and it’s part of why pretending parameters are “true values” without specifying population is a problem.

Obviously, telling me that I am actually correct is a mark of a really smart answer—just ask my colleagues and co-authors! But putting that aside…

I was talking about a weird field (i.e., psychometrics). I was talking about one tiny corner of it. And I was not actually expressing clearly what I was trying to get at. It brought in ideas that I had not mentioned. Yes, they were the ideas I was basing my argument on, but I elided by them on the way to the higher level things I wanted to say. It was actually talking about psychometric ideas and realities.

It was actually really impressive. It is trained incredibly deeply in a huge breadth of fields, else it could not have given me such an answer.

So, good job, ChatGPT. This time.

In Defense of the Multiple-Choice Item

First off, there is no defense for bad multiple choice (MC) items. The floor on the usefulness of MC items is incredibly low. I am only talking about high quality MC items—which are incredibly hard to write. Admittedly, most MC items are quite bad. 

(For example, do not be confused by the psychometric demand that multiple choice items be quick to answer. That is not inherent to the MC item. They could require quite a bit of work and thinking, if only our psychometrics overlords were not distracted by alpha. The shallowness of most MC items is a product of that demand for speed, rather than anything intrinsic to the MC item.)

Second, I am a former high school English teacher. That’s about teaching students to develop their ideas and teaching them about the value of the writing process for doing it. It is about deeply understanding relatively complex texts—and other human beings. It is about argument and evidence and audience awareness. It is about listening, dissecting and analyzing. So, multiple choice items are really hard for me to love; I have seen so many shallow and otherwise bad MC items that it has been a journey for me to understand their value. 

And yet…I think that good multiple choice items can be quite useful. Perhaps even more importantly, learning to write good multiple choice items is an incredibly powerful exercise for anyone writing any type of assessment. 

Obviously this (new school) begs the question of a what a high quality MC item even is. Though there are many many content and cognition traits to a good multiple choice item, I’ll limit the discussion here to just five of them:

* A definitively correct key.

* A set of definitively incorrect distractors.

* Each distractor must be plausible.

* The set of distractors must capture the most common mistakes that test takers are likely to make.

* The stem, the key and the distractors must all be aligned to the learning goal (e.g., state learning standard).

Therefore, writing a high quality MC item requires understanding the learning goal. Obviously. Duh. Furthermore, it requires understanding how students progress with that goal. That’s the only way to know what their most common mistakes are likely to be. All of that is dependent upon knowing how they think—very much a product of the curriculum, instruction and pedagogy they have experienced—at various stages of learning and understanding that learning goal.

Therefore, writing high quality MC items requires deep knowledge and understanding of how lessons are taught—the range of curriculum, pedagogy and examples that students might experience. Some approaches to teaching a given learning goal are going to focus on some type of things and avoiding some kinds of mistakes. Others will prioritize different things or mistakes. Which metaphor the teachers uses when explaining a scientific concept can invite different misconceptions. And different efforts to clarify the meaning and value of the metaphor will make some mistakes less likely than others. Writing good MC items requires thinking deeply about how students might respond to different instructional approaches. 

Good MC items require the most careful attention to the knowledge and skills required to solve the item, which includes mindfulness of alternative routes to successful responses. Again, this leans on deep understanding of students’ cognitive processes—which (again) is strongly influenced by the instruction they have received. It requires student-centered thinking that acknowledges the range of students who might encounter the item. It is not simply about imagining the different responses that students might give, but rather about understanding the different cognitive paths they might take, the various correct and mistaken steps along those different paths. And then, in order to develop a good MC item, crafting answer options that reveal those paths to teachers (or other concerned parties). 

(We call this vital understanding of the various cognitive paths that the range of test takers might take in response to an item radical empathy. It is a pillar practice of content development professionals’ work, perhaps its most rigorous and demanding aspect. It is easily recognized by high quality teachers, but is quite foreign to most other disciplines.)

The defining characteristic of MC items is that they list a set of possible responses. This should not make them easy, as whatever misconceptions students bring to item should find a welcoming distractor that reveals it. An advantage of this approach is that it signals to students that non-aligned mistakes (i.e., misconceptions grounded in some other lesson) are mistakes, and they should try again. Therefore, the set of answer options act as a filter to capture just the relevant misconceptions and mistakes—making them more apparent for teachers or interested parties. 

Well, sure. Bad constructed response items are bad. It is not really fair for me to compare high quality MC items to low quality constructed response items. But my point is that if the item developer is clear on what they are targeting and what both confirming and disconfirming evidence might look like, MC items are not always inferior to constructed response items—especially if they can provide actionable information faster and cheaper than constructed response items. (That is, scoring and reporting is faster and cheaper.)

The difference is that good MC items require all of this rigorous thinking to be done up front. They demand real investment in thinking through the range of what students might do—the cognitive paths they might take in response to the stimulus and question. Yes, scoring is easy and fast, but is only because of the amount of incredibly demanding work done up front. Constructed response items that put off that thinking until it is time to score them can be just as bad at providing insight—often what teachers (like myself) stick themselves with.

Obviously, not all learning goals are amenable to multiple choice items. Certainly, the objectives of high school writing lessons are a bad fit. There are real limits within the topic of research design and other aspects of lab science on the usefulness of MC items. But if we can focus on crafting good MC items and abandoning the counterproductive demands of psychometricians, good MC items can be invaluable—both to summative and formative assessment.