[First of a five-part Series.]
Andrew Ho teaches his students a mantra,"Measurement must be qualitative, then quantitative, then qualitative again" (Ho, 2024). The idea is that educational measurement begins with qualitative work (i.e., theory, construct definition, instrumentation), becomes quantitative in the middle, and must return to qualitative at the end (i.e., communicating, interpreting, using results in context). He is challenging his students to understand that though his course is about applied statistics, that the statistical work is situated amongst serious qualitative work.
Obviously, he is right. Obviously, right? I mean…isn’t that obvious?
Educational measurement is not physics or chemistry. It is not some investigation of a natural world that exists independently of human beings and their agreements. Rather, it is a deeply collaborative, deeply social enterprise. Teams composed of people with very different kinds of expertise work together to define constructs, build instruments, gather evidence, interpret results, and communicate findings to audiences who will use them to make consequential decisions. At the roughest level, the work is first qualitative, then quantitative, and then qualitative again. (And within each of those three phases, it breaks down further.)
This is not new ground. The field's own recent Foundational Competencies in Educational Measurement says explicitly that "educational measurement is a collaborative endeavor that requires people with varied skill sets to work together to design, develop, administer, and evaluate instruments that satisfy specific uses" (Ackerman et al., 2024, p. 9). We made a related argument in our commentary on the FCEM that genuine collaboration requires not just communication skills but foundational dispositions—respect for colleagues who bring different expertise and lenses, and genuine listening—that the document itself omitted (Wine & Hoffman, 2024).
That omission was not malicious. The task force was composed of thoughtful people who believe in the collaborative and qualitative dimensions of this work. And yet they reproduced, in a document explicitly designed to be broader than psychometrics, much of the narrowness they were trying to correct. That is not a criticism of individuals. It is an observation about how disciplinary training works: the lens your field gives you shapes what you can see—including what you can see about your own assumptions.
That gap—between acknowledging that qualitative work matters and actually treating it as having equal standing—is what this blog series is about.
To understand the gap, two things need to be clear: what an epistemology is, and what a discipline is. I will keep this brief, because these are not the interesting parts. They are the necessary setup.
We are often trained in disciplines. Taught a set of tools. More importantly, we are given a set of lenses to direct our attention to some things and away from others. We are taught the types of questions we are supposed to ask, and what to do to answer them. This includes ideals of what counts as evidence, and what rigor looks like.
Therefore, different disciplines can look at the same topic and respond entirely differently. Consider the prosaic topic of a supermarket. It could be the topic of an architectural study. An economist might take a shot. Or a nutritionist. A labor relations specialist. A graphic designer. A psychologist. A sociologist. An urban planner. Or someone else. The same topic, but each discipline notices different things, asks different questions and produces different findings. None of them is wrong to use their lens. The supermarket is genuinely all of those things at once—and no single discipline captures it whole. To really understand supermarkets, you need many disciplines. And for these experts to work together—instead of simply ignoring each other in parallel—they must do something very difficult. They must not just acknowledge that each other exist, but actually respect the deep validity of each other's approaches.
That difficulty runs deeper than it first appears. Underneath each discipline's tools and lenses lies something more foundational: an epistemology—a set of assumptions about the nature of knowledge, truth and how it is arrived at. Epistemologies are not the same as disciplines. A single epistemological tradition can underlie many disciplines. For example the different natural sciences share a common epistemology. Economics may be quantitative, but it does not exist in the same epistemology as the natural sciences. Disciplines’ ideals of rigor are based in their epistemologies, as are their ideals of evidence and reasoning. A collaborative multi-disciplinary field like educational measurement draws on more than one. The result is that practitioners trained in different disciplines but sharing an epistemological tradition can, with effort and goodwill, recognize each other's rigor—even when the specific methods look quite different. Across epistemological traditions, that recognition is much harder. The other person's rigor does not just look different. It often does not register as rigor at all.
And here is the crucial point. Neither disciplines nor epistemologies can transcend themselves—that is structural, not a failing. But people are not their disciplines. People can recognize that their framework is their framework and mighty useful without being blinded to the valid existence of others—even those grounded in different epistemologies. That is, they can recognize that other legitimate ones exist and that the rigor their discipline cannot see is real nonetheless.
In fact, they must. If not, Andrew Ho’s insistence that educational measurement is first qualitative, then quantitative and finally qualitative again is just words—and we know that some epistemologies do not value words.