Humans-in-the-Loop

As AIME-Con approaches, my thoughts turn to automated item generation (AIG) and plans to reduce the many human roles to a human-in-the-loop role.

Obviously, psychometric reviews are easily automatable. I mean, you wouldn’t call for automating other work you’re not actually expert in if you didn’t think all the field test data processing, all the psychometric forms reviews and all the final score determinations were already easily automatable, right? 

So, we can check off those three steps of psychometric work as automatable. 

What other points or types of review are automatable? You looked into all the issues of graphics review, right? You made sure that that was all automatable with LLM-based tools?

Blueprint conformity is pretty easy, right? Representation across the content, all the different types of items and stimulus types called for in the blueprint? That’s all in the metadata and you just need to match that up to blueprint requirements, right?

You do the accessibility reviews? Which ones did you check? You sure they are all automatable?

Fairness reviews? Sensitivity? Bias? Representation? You have a good balance of political/cultural sensitivity and authenticity, right? Just curious, which cultural perspectives do your tools take into account? 

ELA’s stimulus reviews? You’ve got your various fairness reviews? Content reviews? You’re satisfied that LLM-based technologies can identify grade-level testable points? 

How about client reviews? You know, before stimulus reviews, before item external reviews, before field testing, before operational testing. Are those going to be automated?

What about reviews of modified material for test takers with sensory and motor disabilities? Oh, I don’t mean the earlier accessibility reviews. I mean for the modified stuff. You accounting for that? 

Inter-item clueing? 

I’m curious about the teacher reviews. You know, the ones that enable the industry, state officials and politicians to say things like “Every question on our tests was reviewed by a team of teachers from our state.” You automating that? You’re automating the easier item starter/drafting stages, right?  The ones that allow state officials and politicians to (misleadingly) say, “All the items on our tests were written by teachers from our state.” You automating that? 

What about reviewing items for cognitive complexity and whether they are hitting the core vs. the margins of each alignment reference? Is your system reviewing for that? 

OK. Just making sure your plan for a human in the loop is accounting for all the human reviews and all the types of expertise that are so important to ensuring that we have items that elicit evidence of the targeted cognition for the range of typical test takers. 

Two definitions of item validity

There is good reason to think there is only one type of test validity: the degree to which evidence and theory support the intended uses of a test. We can ask whose intended uses count—the developer's, the state's, the district's, the teacher's—but the idea that test validity is test-use validity is strong and well established. We agree with it entirely.

But items exist before there are tests.

Items are developed, reviewed, and banked as items. They are bought, sold, and licensed as items. They carry labels: aligned to this standard, measuring that skill, at a certain level of proficiency or perhaps even at a level of cognitive complexity. Each label is a claim that the item produces a particular kind of evidence. Items, too, have intended uses.

None of this impugns the unitary view of test validity. Item validity is not a challenge to the established unitary view of validity because it is not an alternative view of test validity. It is the validity of a different object. It is a different level of analysis. Because items are so important in the processes of developing tests and the processes of taking tests, they merit rigorous scrutiny. Thus, mirroring The Standards’ idea of test validity, we can define item-level validity: the degree to which evidence and theory support the use of an item for its intended purpose.

This is a more straightforward idea than test validity. A test's intended uses can be contested, multiplied, and extended far beyond what its developers imagined. An item's intended use is already on its label: to elicit evidence of the cognition it targets.

Thus, we can operationalize item validity with a more directly useful definition: valid items elicit evidence of the targeted cognition for the range of typical test takers.

Each clause matters. Elicit: the item must actually prompt the cognition and produce observable evidence of that cognition. The targeted cognition: what the alignment reference specifies, not something adjacent to it. For the range of typical test takers: not just the idealized student the item writer pictured, but the full range of test takers who will actually encounter the item.

That is item validity.

The Meaning of the Standards for Mathematical Practice

When I first saw the Common Core State Standards' (CCSS) Standards for Mathematical Practice (SMPs), I was so incredibly excited. Simply brilliant. Now, I thought they were brilliant because they were kinda exactly what I wanted them to be. They were the most important parts of math, in my view. Back in the early 1990's, when I decided to teach the lessons of math without numbers or Bunsen burners, this was what I was looking to teach to students who got freaked out in math class.

Math teaches us how to think. Well, every content area should teach us how to think. Math is not special in that regard. As Ted Sizer said, schooling is about learning to use one's mind well. Math is one way to teach children how to use their minds well.

The thing is, it's not the more traditional math content standards that teach us how to think, how to strengthen our minds. They are just tools. And we can fill the toolbox with tools, but that does not teach us when to use them, or why. Tools are not thinking skills, and tools are not habits of mind. But in my view, math provides incredible opportunities to teach students these so valuable lessons. Math can be incredibly useful for providing lessons worth learning for a lifetime.

The hardest math problems are so often word problems. Real world content problems. Not the naked math problems whose solution paths are simply the algorithms that we were taught to apply. Not the problems that make obvious what tool to use. No, the challenge is figuring out how to get from the word problem to the naked math—and that's with grossly simplified problems without any of the actual messiness and complexity of the real world.

And that's where the SMPs come in. They are habits of mind that enable us to make sense of messy and complex situations in the real world, find the actual problem, and recognize the tools we might apply. And then to figure out which parts of the mess and complexity are the information that our tools need as input, or the gaps that we need to fill with outputs from our tools—often from our mathematical toolbox.

SMP 1: Make sense of problems and persevere in solving them.

SMP 2: Reason abstractly and quantitatively.

SMP 3: Construct viable arguments and critique the reasoning of others.

SMP 4: Model with mathematics.

SMP 5: Use appropriate tools strategically.

SMP 6: Attend to precision.

SMP 7: Look for and make use of structure.

SMP 8: Look for and express regularity in repeated reasoning.

Even on the surface, few of these appear to be truly focused on math or the procedural tools of the CCSS's content standards. Sure, from a mathematical perspective, the links are obvious. But links to other content are also pretty obvious from other content perspectives. The SMPs are a set of incredibly important thinking habits that allow us to better make sense of the world. And then, once we have done that, we might know what we are trying to accomplish and how our math toolbox might help us to do so.

Ironically, the only SMP that does not fit this view—that the SMPs are about bridging the gap from the complexity and messiness of the real world to our (math) toolbox—is SMP 5. Oddly, CCSS's official explanation of SMP 5 is not about the math toolbox of the Standards for Mathematical Content. The examples given are "pencil and paper, concrete models, a ruler, a protractor, a calculator, a spreadsheet, a computer algebra system, a statistical package, or dynamic geometry software." These are not the internal skills and procedures math students develop mastery of through their math education. No, instead they are the external tools that they might buy (or download). SMP 5 has always struck me as odd. The title suggests that it should be my favorite, perhaps the one that focuses all the others. But whoever wrote that explanation does not think like I think and does not understand the other math practices…perhaps at all.

The others, however, are clearly all about making sense of what we don't yet understand. As that same page says—in its closing section on connecting the practices to the content standards—these practices are what students draw on to "consider analogous problems, represent problems coherently, justify conclusions, apply the mathematics to practical situations."

The tragedy is that the norms of math instruction make it impossible to teach students to grow their command of these important habits. Rather than embracing the messy complexity of the world and our responsibility to help students make sense of it, math instruction and assessment remove as much complexity as possible. They want a clear sightline to each content standard and an uncluttered workspace in which to apply it. Disconnected tools, without real purpose. Which leaves students unable to transfer their use from the obvious, simplified problems of worksheets and math tests to the challenges and realities of their actual lives outside of math class.

This is no minor tragedy. It is the fundamental flaw of math education in America. And unfortunately, although large scale assessment has the ability to push instruction to pay more attention to the important parts of our state learning standards, math assessment has also bailed on the most important parts of the Common Core State Standards for Mathematics.

Why Item Alignment Matters

If items are the building blocks of assessment—and they are—then alignment to the domain model and alignment references are the chemical bonds that hold each block together. They are the difference between a bunch of stuff and crafting that stuff into something that can bear appropriate weight, fitting in with the other blocks for whatever purpose the test is put to.

Item alignment is incredibly important. Vitally incredibly important. So, putting aside fairness and bias (though, in fact, items with fairness issues are not aligned for everyone, so it’s an inhumane thing to do) and focusing for now just on the K-12 large scale assessment context, let’s quickly go through why alignment is important. 

Understanding alignment starts with understanding the role of learning standards. 

• Our democratic processes empower particular people to make decisions about our schools. This begins with state legislators and governors, who may have ratified some set of state learning standards—thus giving them the authority of laws. In other states, it is a state school board that ratifies them, or perhaps senior officials in a state’s department of education, giving them the authority of regulations. Either way, they are ratified democratically, and thus they have the moral and legal authority of laws and/or regulations. 

• Because most state constitutions say that public schools exist so that our democracy can function with an informed voting citizenry, state learning standards can be understood as the state’s view of what would make for an informed citizen, or for someone who has the tools they need to become or stay informed.

• Yeah, legal authority over education is in the hands of states. The federal constitution does not mention education, but state constitutions do. Legally, they are the authority, with power to dictate to districts, schools and their personnel. 

• Like all laws and regulations, they necessarily exist in a particular written form. They communicate expectations. Though there may be some room for reasonable interpretation, they exist in a literal form and not just as spirit, ethic or vibe. Like any complex law or set of regulations, each set of standards must be interpreted as a whole and on the more precise level of individual requirements—always firmly grounded in what the law/regulation actually says. 

• Standards tell school districts and teachers what is expected of them and their students. They express the democratically determined expectations of the state for its schools in a form that every educator, every administrator and every member of the public can see. Surely, it is representative democracy rather than direct democracy—but it is democracy nonetheless.

• Standards take power away from arbitrary and opaque whims of publishers and individual teachers. They are public, published and were subject to debate before ratification. The entire populace can see them, gain understanding of them, and even influence their form and contents. They can be altered through the same democratic processes that ratified them in the first place. 

• Unfortunately—like all laws and regulations—standards can be misunderstood. As in any regulated field, those bound by a rule tend to read it in whatever way demands the least change from what they already do or value. They can be misunderstood by non-experts or those who attempt to take portions of them out of context. They can be misunderstood by experts who don’t want to concede that they lack authority to shape schools to conform to their whims or will. And they can be misunderstood by the many levels of people who are supposed to do their work in compliance with them.This is no different than any other type of law or regulation. In other fields, we see lawsuits over government officials misapplying or misunderstanding laws or regulations. We even see that in education. Standards are referenced in education adequacy lawsuits, and perhaps there should be lawsuits over misaligned items and the most indefensible interpretations of standards. 

So, getting back to item alignment, items must align to standards. 

• Assessments communicate to districts and teachers how they should interpret the standards and what they should teach. They include operationalized examples of learning standards in action. Therefore, they impact curriculum and instruction. Improperly aligned items send the wrong signals to districts and teachers about what the standards are, actively misinforming the public. They replace the carefully and publicly crafted substance of the legally ratified law or regulation with the convenience or judgment of all-but invisible lower level officials and workers. And convenient misreadings, once made, don't stay put—they get built into curricula, professional development, and institutional habit, and each new cohort inherits them as though they were the standard itself. Therefore, poorly aligned items are quite literally anti-democratic. 

• Assessments gather information about (select types of) school performance. Most importantly, this information is reported to the public. The public does not see individual report cards, or even aggregate report cards. Schools generally report some sort of graduation rate statistic, and the public can see the performances of sports teams, school bands and drama clubs/classes. They might be able to attend a science fair or the like. But public reporting on large scale assessment results really is the only grade-by-grade reporting on the consensus most important mission of our schools: teaching core academic lessons. They are the only way that the voting public can evaluate the academic performance of their schools. Poorly aligned items misrepresent that performance because they claim to measure something that they do not. They mislabel the results. They actively deceive the public about a matter of public concern. They are, therefore, incredibly anti-democratic. 

Carefully aligned assessments (built of carefully aligned items) and the public learning standards to which they are supposed to be aligned are the democratic accountability mechanism of our public schools. They clarify expectations of our elected and senior appointed officials to districts and teachers and report on performance to the public. Interfering with either half of those mechanisms is unethical and undermines the moral legitimacy of our schools, of so much of our local and state taxes and of the (almost-but-not-quite) requirement that parents send their kids to public schools. 

Item alignment matters. A lot. 

How AI (and people) Get Reading Standards All Wrong

Here in the world of large scale assessment, it is abundantly clear that AI/LLM models are shockingly far from understanding the meaning of reading standards, as are far too many people. Perhaps at the most fundamental level, they simply do not unpack reading into the large collection of knowledge, skills, understandings and abilities (KSAs), and instead view standards as task lists that reading should enable. Therefore, any failure to answer a supposedly aligned question (or item) is a failure on that standard. That view simply ignores the fact that the larger endeavor of reading is made up of many, many smaller KSAs.

People do not make this mistake with mathematics. People seem to easily be able to understand that an arithmetic error can lead to an incorrect result, even when the student has shown that they understand the actual grade level skill in question. Performance may require both arithmetic and understanding of the more advanced concept, but they are different KSAs. Similarly, stumbling over vocabulary or understanding of the literal surface meaning of a text are failures of lower level skills, and certainly not middle or high school level skills. 

So, let’s dive into just one example of a reading standard and explore what it really means, and how we figure that out. 

The Standard

Taken from the Common Core State Standards, RI 8.2: Determine a central idea of a text and analyze its development over the course of the text, including its relationship to supporting ideas; provide an objective summary of the text.

ELA reading standards are based on a small set of anchor standards that develop and deepen through the grades. At first grade, this thread takes the form, Identify the main topic and retell key details of a text. At 11-12th grades, it takes the form, Determine two or more central ideas of a text and analyze their development over the course of the text, including how they interact and build on one another to provide a complex analysis; provide an objective summary of the text. So, let’s break down what it means in 8th grade—RI 8.2.

Give Up on “Reading Comprehension”

This thread is never about decoding words or making basic sense of sentences or what an individual sentence means. Those are different sorts of skills, lower level and lower complexity skills. Modern ELA reading standards are about making deep and more complex sense of what is already understood on the surface level—certainly this is the case by 8th grade. Simply misunderstanding what the text literally says is not a failure of the KSAs that make up this grade level standard.

The complex endeavor of reading is about putting together an array of skills that when utilized together proficiently give the reader an understanding of the layers, connections and architecture of a text. They provide a reader an understanding of how a text accomplishes its layers of nuanced goals. They are about idea, audience, technique, implication and various ways to evaluate the strength of a text—and of its contents. 

One could term this “reading comprehension,” but it would be quite different than what most people use the term to mean. What is really going on is the fact that reading skills—expectations for the KSAs learned by readers—continue to deepen far beyond the early- to mid-elementary school skills that basic and low expectations for “reading comprehension” suggest.

Just as the abstraction skills of algebra and the logical, ordered thinking skills of geometry are not necessarily consciously recognized as such when they are applied in adulthood, these more advanced reading skills ought to be applied unconsciously. It is easy for adults to recognize an arithmetic error or a failure to recognize a word, but most of the time these things are taken for granted. The unconscious and integrated use of these more advanced skills is incredibly important and is the real goal of education, regardless of the content domain.

Summarizing

Summarizing is an incredibly complex set of skills, and tracing through the grade progression of this standard (shown at the end of this post) makes clear how it is expected to deepen and gain detailed nuance over the years. But at its heart, summarizing is about i) recognizing what is important for this purpose and therefore worth including, and ii) presenting it in a sensible form that fits both the original summarized text and the purpose of the summary. 

Yes, summaries can have purposes. For example, consider the difference between “Summarize the events in this text” and “Summarize the important ideas in this text.” This standard is about informational texts, rather than literary texts, so one might ask for different types of summaries. Were the reader asked merely to “Summarize the text,” they would have to judge for themselves what aspect of the text should be the focus of the summary. That is an important summarizing skill—the ability to make that judgment. Of course, it is not expected at earlier grades, where texts are simpler. But it becomes more important in later grades as texts become more complex.

At early grades, summarizing is little more than retelling what has already been presented to the student. At later grades, it is about connections between ideas. What does it mean at this grade, grade 8?

Look to the Language of the Standard

The standard tells us the level of sophistication that is expected at this grade. Determine a central idea of a text and analyze its development over the course of the text, including its relationship to supporting ideas; provide an objective summary of the text.

* This is not just about one singular main idea, but rather about recognizing that there can be multiple central ideas in a text and distinguishing between them. 

* This is not just about recognizing that a text can have multiple ideas, but rather about recognizing that some ideas are more important or primary and other ideas are less important or play a supporting role.

* This is not just about recognizing central and supporting ideas, but rather about recognizing the nature of the relationship between a supporting idea and a central idea. 

Of course, any standards document or domain model is written for an expert audience. It is about organizing, labeling and classifying particulars, not about defining what everything means. In this case, the audience would understand that a supporting idea can widen or narrow the claim of a central idea. It can be an illustrative example of a central claim, or a purposive counter-example. It can detail a particular aspect or application, or offer a more generalized context for the central idea. Or work to support that central idea in some other way. The language of the standard is vital, but it assumes that the reader can understand it. 

It is difficult to convince LLMs that ELA content experts are not all white women.

The Requirement of Expertise 

Without expertise in the content domain, a reader is not going to be expert enough to understand what the language of the standard means. This is no slight to the non-expert—who is likely quite expert in some other field or topic. Rather, it is the basic fact that professionals in virtually every field need to communicate with each other in ways that they can understand. Expanding out every meaning and implication every time is simply not effective, and the idea raises real questions about how far that might be taken. 

Reading and understanding a content standard, learning standard, learning objective or any domain model requires that the team doing the larger project include people who have sufficient expertise to understand those documents and their elements. 

How Much of the Standard?

This standard, RI 8.2, appears to have two parts. The latter part is providing an objective summary, and the former part is about relationships of central and supporting ideas. 

One might read the standard as suggesting that the former part defines what the latter part means. That is, provide an objective summary in which the relationships of central ideas to their supporting ideas are more or less fully analyzed. That is a lot. It is clearly a defensible reading of the standard, but it asks too much to be done quickly. Rather, it is the kind of task that requires thought and planning and revisiting. In fact, the nature of modern reading standards is that they envision writing about reading, which makes the scope of thinking and deliberation this suggests more appropriate. (Again, experts know that this is what modern reading standards focus on, as they have read the entire domain model instead of just taking part of it out of context.)

But that is not the only reading. One might look at RI 8.2 and see that it is about understanding and relating central ideas to supporting ideas, which can culminate in a complete summary, but does not have to. That is, there is that cluster of KSAs about determining a central idea and analyzing the development of the idea through its relationship to its supporting ideas. And there is also the next layer of using that bundle of KSAs to build up a complete summary—which would also require recognizing the purpose of a summary and selecting appropriate central ideas for that summary. And providing a summary would also include figuring out how to present it coherently and understandably. 

So, the language of the standard, its order and its use of a semi-colon (;) suggests that one could analyze one central idea like this or do the larger summary including multiple central ideas. However, merely identifying a central idea is not enough. That is not even the 3rd grade skill. Moreover, the standard is clear that the KSAs include determining a central idea, rather than merely responding to someone else’s nomination of a central idea. That is, it implies that this cognitive effort of analysis is how one recognizes what is a central idea and what is a supporting idea, and being supplied with a central idea undermines the requirement to do that analysis. 

What We See From AI

Instead of those real grade 8 reading skills, we see something quite different from AI/LLM efforts to base items on this standard. We see less. We see so very much less. It is not merely the failure to recognize that item type modality (e.g., multiple choice) can undermine targeted cognition (e.g., consider the cognitive act of providing vs. recognizing), which is bad enough. Rather, this seems to be a refusal to take reading standards and ELA domain models seriously as descriptions of particular grade level skills. We have see this from multiple LLMs, using multiple versions, with every level of prompt engineering in our projects and for every ELA standard we have attempted. Again and again, we see a mistaken core idea that all that matters is completing a task, with no consideration of which KSAs the item actually taxes—which is what item alignment is about.

We see a radical lowering of cognitive demand and grade level. That is, many items simply ask the test taker to select the correct main idea of the text—even if phrased differently. We see selection of potential summaries (or analysis of central idea development) which simply requires eliminating the ones that include basic misunderstandings of the text—which are really just rather simplistic reading comprehension questions. 

Of course, it is a problem that AI/LLMs do not understand how to read or unpack modern reading standards. But there is a lot that AI/LLMs cannot do. The real problem is when people accept that the dumbed-down interpretations from AI/LLMs are all that reading standards even could mean, without asking anyone with the expertise to know better. We have already seen this kind of misunderstanding from quantitatively oriented practitioners for years. In this new AI era, their ignorance is getting built into the structures of generative AI, lowering the ceiling on test (use) validity and possibly shortchanging students of their opportunities to learn the advanced skills that will help them in their futures. 

Development of a Single Strand of a Reading Standard: REading Informational Texts 2

Kindergarten: With prompting and support, identify the main topic and retell key details of a text. 

Grade 1: Identify the main topic and retell key details of a text. 

Grade 2: Identify the main topic of a multi-paragraph text as well as the focus of specific paragraphs within the text. 

Grade 3: Determine the main idea of a text; recount the key details and explain how they support the main idea. 

Grade 4: Determine the main idea of a text and explain how it is supported by key details; summarize the text. 

Grade 5: Determine two or more main ideas of a text and explain how they are supported by key details; summarize the text. 

Grade 6: Determine a central idea of a text and how it is conveyed through particular details; provide a summary of the text distinct from personal opinions or judgments.

Grade 7: Determine two or more central ideas in a text and analyze their development over the course of the text; provide an objective summary of the text.

Grade 8: Determine a central idea of a text and analyze its development over the course of the text, including its relationship to supporting ideas; provide an objective summary of the text. 

Grades 9-10: Determine a central idea of a text and analyze its development over the course of the text, including how it emerges and is shaped and refined by specific details; provide an objective summary of the text.

Grades 11-12 Determine two or more central ideas of a text and analyze their development over the course of the text, including how they interact and build on one another to provide a complex analysis; provide an objective summary of the text. 

A Promising-But-Logistically-Impractical Test Design

This is a very particular design aimed at addressing the kinds of engagement challenges that can lead to suppressed performances by test takers, and thereby suppressed reported scores. These engagement challenges are core to the motivation of culturally responsive assessment, but are actually much broader than that.

Of course, this design is aimed at providing instructionally relevant criterion-referenced multi-dimensional reporting—which can be turned into deliberately designed composites if unidimensional results are needed for other purposes. That is, it is not aimed at norm-referenced reporting, because most large scale assessments are criterion based—or should be.

1. The test presents choices to the test taker at each point, rather than a single item. Which item would you like to do? That set of choices always is aimed at the targeted cognition, using the same focal KSAs, at about the same level of content complexity. (Difficulty and cognitive complexity are not population invariant, and even vary from test taker to test taker, so we do not pretend that they can be equalized or approximated.)

2. The items may vary in cultural context and topic context. There can be some baseball items, some football/soccer items, some cooking items. Some skiing items and some beach items. Items about making tamales and items about making lasagna. Fairness concerns for individual items would be different, as concern shifts instead to broader pool or choice set concerns. This provides more flexibility, perhaps building on RTD task models.

3. Test takers respond to ~5 items per reported score/attribute, meaning they have to select an item ~5 times. Some items may appear more than once, but they cannot be attempted more than once.

4. Test takers are given the option of having the platform select an item for them, but only after they have an opportunity to review their options.

5. Using cognitive diagnostic modeling (CDM) rather than IRT. The humility of CDM’s binary reporting (i.e., proficient/non-proficient) instead of misleadingly precise quantitative scores allows more flexibility in item matching. Each selected item must simply help to provide information about test taker proficiency, contributing to CDM’s needs. Old research about test taker choice was based on IRT-based test designs and data analysis, and it doesn't apply to a CDM-based approach. (Moreover, the nature of choice in this design is based on interest and engagement, not on the kind of focal-KSA selection of that old research.)

6. Larger tests are composed of a series of testlets (i.e., a set of items focused on a single reporting category or standard), or interleaved testlets.

7. The entire domain model may be addressed across multiple testing sessions, or even at different points through the years—as appropriate.

What does this design give us?

• Like any CDM-based test design, far more interpretable and actionable results than traditional CTT- or IRT-based test designs.

• Far more justified breaking up of larger domain models into subsets whose assessment can be spread through the year. So long as domain models are ignored in favor of unidimensional analysis and reporting, there is no content-based justification for breaking up large assessments, and yet doing so randomly is clearly ridiculous. This approach acknowledges the desire to break them up, and does so in a way that is coherent—from domain model all the way to score reporting.

• Greater engagement by better matching item contents to test takers’ interests and/or cultural backgrounds. Not ideal matching, but surely they can do better at this than any algorithm—and far better than not even trying.

• The documented boost in performance and persistence on items that research keeps showing that choice creates. Even when test takers make sub-optimal choices, the mechanism of choice itself provides benefits. (There’s some research that shows that that might actually be culturally situated, but I’m ignoring that, for now.)

What does this design give up?

• Well, most importantly it gives up largely baseless claims about precise sorting and ranking of test takers by some statistically evoked interpretable Θ (or transformation thereof). For some, this will be seen as a huge loss.

• Vertical linking of scores, which relies on the idea that the construct does not change across grades to be interpretable as anything other than norm-referenced reporting. Again, for some this will be seen as a large loss.

• Linking scores across years—because the need to link scores across years requires multiple arbitrary scales that need a common anchor to provide the illusion that comparisons of uninterpretable statistically evoked scores are ever themselves interpretable. Of course, because the meaning of proficiency should not change from year to year, scores are already linked. So, linking work is not needed. (Admittedly, this might not be quite right. I need to do more thinking on this one.)

What does this design cost?

• Well, it would take a lot more items. Any test that provides more information is going to need more items, but this is dramatically more than that. Because AI/LLMs are really bad at producing on-target items—which criterion referenced tests require—we cannot (yet?) count on automated item generation to do this for us. It could help to adapt good items to different cultural or topic contexts, but they’d still need to be reviewed. Yeah, this would be expensive. And the more culturally responsive or personally relevant the offered choices might be, the more items it would require.

• Testing time. Again, any test that provides more information is going to take more testing time, especially if that information is going to actually be useful. Furthermore, the choice mechanism takes up additional time. I have no doubt that the unwillingness of so many to devote sufficient time to large scale assessment is based in their perceptions of the value of those assessments. But even in the best case, this additional testing can come at the expense of instructional time. It would be better if high quality large scale assessment could take the place of redundant classroom assessments, but we would need a norm of actually high quality large scale assessments that provide high quality instructionally relevant information to see if the broad cultural pattern of redundancy might shift.

• Forced distributed administration. While having the option of breaking up larger tests into through-the-year assessments is great, the requirement might not always be welcome. Not only would test takers have to complete more items to get more information, there is an additional cognitive burden of selecting items. Perhaps the motivation that comes from the choice mechanism would make up for some of that, but it is more cognitive work. Test taker stamina is not unlimited.

• Real work to be clear on the meaning of proficiency for each reporting category or standard. For example, the work of constructing range PLDs is valuable, but it is costly, too.

This most definitely is not a well thought out proposal. It occurred to me just this morning, in response to some serious conversations about test taker engagement and the meaning of engagingness of items. Surely, there are more problems that I’ve named. And, yeah, it’s impractical. I don’t see it happening. But I see substantial advantages of adding choice to address fairness and cultural responsiveness, and in this design that same mechanism might actually give more accurate data on test taker proficiencies.

Math Items: Novelty vs. Real World Contexts

On the one hand, math standards tend to call for use or application of math KSAs in real world contexts. Those are the application problems, which some people refer to as word problems. However, merely putting the math usage in some thin context is no more a real world context than cat ears or a simple mask is a costume. 

On the other hand, items with too much novelty require test takers to engage in learning and exploration to even put themselves in a position to apply those core focal KSAs in the alignment reference or standard. The work can be built with a whole bunch of additional KSAs that prevent test takers with proficiency in the officially targeted cognition in the alignment reference from demonstrating their skills. False negative results, which we call Type II errors. 

These are two different hands—these goals are in tension—because if real world context has any real use or meaning, it is a call for the kind of discernment and decision-making that warnings of excessive novelty in items are concerned about. And yet, the Common Core State Standards’ approach to math seems to depend on these sorts of real world context problems to bring in the Math Practices. Moreover, the Next Generation Science Standards (and Achieve’s preferred approach to NGSS assessment) very much leans into novelty and exploration for the contexts in which its Disciplinary Core Ideas should be applied.

This creates a major tension for large scale assessment, where there are not enough opportunities to collect data to be able to triangulate through item x test taker interactions to figure out why test takers are successful with some tasks and not others. Disentangling shortcomings in focal KSAs vs. additional KSAs requires triangulation. There simply are few types of information available, and little data from each. And yet, the novelty inherent in meaningful real world contexts creates the greatest need for such triangulation. 

Such tensions are common in item development. Many content development professionals have well-developed ideas about what makes for higher quality items, but the constraints of large scale assessment (e.g., few testing opportunities, widely diverse test taking populations, psychometric demands for short items) make reconciling those multiple ideas quite difficult. Too commonly, those constraints require unacknowledged compromises that reduce targeted cognition to something outside the real core of the standard, if not outside the grade level standard entirely.   

Epistemologies, Respect, and Physics Envy (Part V): The Standards and the standards

I had a conversation recently with a respected psychometrician. Someone truly smart and thoughtful, whom I personally respect for a variety of reasons. I take what they say and write quite seriously—even though I don't necessarily agree with all of it.

In the course of the conversation, he made a point about psychometrics and validity that required, as far as I could tell, simply setting aside what The Standards for Educational and Psychological Testing say on the matter. When I pressed him on it, he dismissed my reference to The Standards by saying—in a kidding-not-kidding sort of way—"I don't care what The Standards say."

I don't buy it. I know that he cares—just not enough to let it stop him. And not enough to reckon with the inconsistency between The Standards and the argument he wanted to make. So he made the joke rather than wrestling with what the document actually says. The humor was doing epistemological work by signaling that the authoritative, hard-won, cross-disciplinary consensus document was beneath the level of argument he was operating at. It was clearly a joke. It was also a deflection.

So I pushed further. He then dismissed The Standards as merely "a consensus document"—as though that fact diminishes its authority rather than constituting it. In word-oriented epistemologies, in law, in democratic theory, consensus is the source of authority. The Standards carry weight precisely because they represent serious, collaboratively arrived-at agreement over the course of generations across the constituencies of this multi-disciplinary field. That is not a weakness. That is what they are. They should not be ignored for being inconvenient, or dismissed for failing to feature one's own favorite ideas. (RTD, for example, adds item validity to the picture The Standards draw—but removes nothing from it in order to do so.)

Then he asked whether domain models can be falsifiable.

For a third time, I was stopped cold. The domain models given to assessment developers are the job. State legislatures ratify learning standards. Test development contracts call for assessments to align with them. And like The Standards, the standards themselves represent the product of generations of thinking—about curricular priorities and learning sequences. Modern standards can be traced through the remarkable work of the National Council of Teachers of Mathematics in the 1980s and 1990s and through the broader education standards movement that followed. This is serious work, done by people deeply committed to their fields and to their students.

Are they falsifiable? From a Bayesian perspective, it would take an extraordinary amount of evidence to do so—certainly astronomically more than can be generated by convenient but obviously inappropriate psychometric models. And any such evidence would have to address the actual concerns of the standards: the nature of the content and the appropriate learning sequences through it. None of that was on the table. Moreover, what would result from that kind of serious engagement would be iterative improvement, not disproof. That is how both The Standards and the standards develop over time.

Asking whether they are falsifiable is a category error. Falsifiability is a criterion native to one epistemological tradition—a powerful and important tradition—being applied as though it were universal. CCSS is not a hypothesis. It is a social, legal, and professional agreement about what students should know and be able to do and how those learning goals hold together. The right question is not can it be falsified? The right question is is our test properly grounded in it? That is word-oriented epistemology's home territory. And it is, not coincidentally, the epistemological tradition that The Standards themselves are working in.

So, in a single conversation, the same move was made three times. The Standards—dismissed with a joke. The learning standards—subjected to a criterion from a different epistemological tradition that was never designed to apply to them. 

This is not a story about a bad actor. It is a story about a larger trap—about what happens when deep disciplinary formation leads someone to stop seeing the obligations that don't fit its framework. This psychometrician spoke from genuine expertise. But for decades, this is the view I have seen dominate educational measurement. And it stands in the way of high quality assessments—and of the respect from educators and the broader public that high quality assessments deserve.

This is not to say that The Standards and the standards act like laws of nature. They are not absolutely inviolate. Nature says that nothing can travel faster than the speed of light. But the 613 mitzvot in the Torah—commandments that have structured Jewish life for millennia—all yield to pikuach nefesh, to the preservation of life. There are hierarchies among them. There are millennia of accumulated debate about how to understand and implement these written words. Some communities have revised their relationship to them substantially. But none of those debates—not the Talmudic tradition, not Reform Judaism, not the interpretive traditions of Christianity—simply dismiss them. Even where traditions have departed from strict observance, they have done so through serious engagement, not jokes and deflection. The wrestling is the point.

The Standards and the standards work the same way. They have hierarchies—Andrew Ho has argued that the first source of validity evidence is listed first for a reason, and he is right. They require interpretation. They will be revised, and there are people doing that work on The Standards right now. They will listen to the entire professional community. That is how both bodies of work grow and improve. Not through dismissal, but through serious engagement by people who take them seriously enough to push back on them in principled, argued, evidence-based ways. 

But until that revision work is done—until the community has done the wrestling and arrived at a new agreement—we are all professionally obligated to respect both The Standards and the standards. They define the job. Without them, we would be flailing, each of us alone, trying to figure out what we are supposed to be doing. With them, we stand on the shoulders of those who came before us.

That is what escaping the trap looks like. Not abandoning your formation. Not pretending that all methods are equally appropriate for all questions. But recognizing that your framework is your framework—powerful and not universal—and that the standards and The Standards deserve grounded, serious engagement. The Standards and the standards merit deference from all of us because they are the product of multiple epistemologies and disciplines. They should guide us in transcending our own. 

Epistemologies, Respect and Physics Envy (Part IV): -(PE)⁻¹

The term "physics envy" was coined by biologist Joel E. Cohen in a 1971 book review in Science: "Physics-envy is the curse of biology." The aspiration behind it is older—Auguste Comte imagined sociology as "social physics" in the nineteenth century—but Cohen gave it a name. He meant it sincerely. But even his original use is off the mark. "Physics envy" was deployed in the context of political contestation about epistemological debates within biology—a field arguing internally about whether to be more mathematical, more formal, more like physics. That is a professional and political struggle, not a psychological one. 

When the term gets applied by someone to descibe other disciplines—to qualitative researchers, to humanists, to anyone working outside the quantitative tradition—it describes something that simply doesn't exist.

In practice, "physics envy" functions as a disparagement. It projects a hierarchy the speaker feels onto the psychology of the accused: you must want what I have, because what I have is the best thing. It asserts superiority while dressing it as diagnosis. And it is, in the laziest way, unfalsifiable: if you deny having physics envy, that is exactly what someone with physics envy would say.

It is also—and this is where the irony becomes acute—built on a Freudian concept. "Physics-envy" is constructed on the model of "penis envy." By 1971, that was not an innocent borrowing. Second-wave feminism was actively exposing penis envy as precisely the same move: a person in a position of cultural power projecting their own hierarchy onto the psychology of those below them. The person without the thing doesn't actually want the thing. They want the social power attached to it—and the person with the power finds it flattering to believe otherwise.

But the deeper irony runs further. Envy is not a concept from physics. It is not a concept from the natural sciences at all. It is psychological, humanistic, interpretive—the kind of thing the natural science epistemologies are supposedly beyond. To name what they were observing, the people deploying this term have to reach outside their own toolbox and borrow from a tradition they look down on. They borrow without understanding what they are taking—a term already discredited in the very field they were raiding. And the act of borrowing demonstrates, without their noticing, exactly what they are denying: that their own disciplines lacked the tools to address everything that matters. They needed words. They needed interpretation. The very move that asserts the hierarchy undermines the claim the hierarchy rests on.

That said—and I say this as someone whose has joked for forty years has been that physics is just applied calculus—none of this is to disparage physics. Physics is so closely bound to mathematics that it is like a brother to my first disciplinary love. My joke is grounded in genuine respect for what both make possible, and for the extraordinary things physics has done. I was formed by mathematics and the natural sciences before I was formed by any other disciplne, and that formation never fully resolves. When I developed grounded theory in my doctoral work—a multidisciplinary study with a sociological methodology, rooted in political science, informed by psychology, executed with the applied philosophy in mind—I still worried about falsifiability. But fallibility is an important concept native to completely different epistemology I was working in, one I carried without quite being able to set it down. That was not physics envy. It was the weight of a formation I was still working against. Deep respect is not the same thing as envy.

I do not think physics envy, as described, exists in any meaningful way. What I do see—clearly, consistently—is its negative reciprocal. -(PE)⁻¹.

Not envy upward. Contempt downward. Not aspiration toward physics. Dismissal of those working in different epistemological traditions, from people who believe their work resembles physics. Very commonly—not occasionally, not in isolated cases—people trained in quantitative disciplines looking down on those working in qualitative ones.

It is easy to see -(PE)⁻¹ all around us. The tyranny of numbers. The confusion of precision with accuracy—as though carrying a result to four decimal places makes it more true. The willingness of people without training in qualitative methodology to opine freely on qualitative work, or who have collaborated on one mixed-methods project to consider themselves expert. A fundamental disdain for non-quantitative epistemologies that allows them to think—at best—there is barely any there, there. Strong enough that people who do not work in research at all—policy people, administrators, funders—think themselves closer to obvious quantitative logic and look down on those working with more nuanced and subtle tools.

And yet, the smartest quantitative methodologists I know—Dan Koretz, Andrew Ho, Andrew Gelman—preach consistently about the limitations of their tools and the importance of taking the non-quantitative parts of the work seriously. They know better than to fall for -(PE)⁻¹, and they understand it is their job as educators to ensure that future generations of quantitative researchers don't fall for it either.

But the world is larger and harder to teach than even such wise methodologists can reach.

Epistemologies, Respect and Physics Envy (Part III): The Written Word

Two of the most acclaimed geniuses of the twentieth century were physicists—practitioners of a field so dependent upon mathematics that entirely new branches of math have been invented to do the work that physics required.

Their genius was not actually the math.

Stephen Hawking realized early in his life that his physical disabilities would not allow him to lean into the mathematics the way his colleagues did. He could not write out equations and notations in the conventional way. Which was fine, because that was not his genius. What he brought was the ability to visualize the geometry of spacetime and interpret physical reality in ways no one else had managed. One of the greatest physicists of the twentieth century. His genius was interpretive mastery.

Albert Einstein—the reason we use the name "Einstein" to denote genius itself—was not responsible for getting all the mathematics right in his papers. His wife is said by some historians to have helped; a close friend supplied the Riemannian geometry required for general relativity. His genius lay elsewhere. Einstein's path to relativity ran through thought experiments—imagining riding alongside a light beam, imagining an elevator in free fall. He grasped the physics conceptually before he had the mathematics to express it. The interpretation came first. The numbers followed. They were true to each other.

Physics is demanding. It includes math. But the rigor is not just in the math. In much of science, the actual math is literally formulaic. You know which equation to use; you plug in the values; you get the answer. That part is not where the genius lives.

The rigor in natural science is in the inquiry. In figuring out what you even want to test—recognizing a phenomenon, hypothesizing about it, finding a way to quantify it, doing the quantitative work, and then getting back out of the numbers. Back to the real meaning, in terms of the phenomenon. The numbers are often the easiest part.

The hardest part? The bridges. First, from something qualitative and real to a quantified proxy. And then second,  from those numbers back into the qualitative phenomenon. Doing those translations while remaining true to the source—true to the phenomenon when quantifying, and true to the numbers when interpreting back into the terms of the real world. Real rigor is in that faithfulness. Andrew Ho teaches that educational measurement must be qualitative, then quantitative, then qualitative again. He is right. And that sequence applies far beyond educational measurement.

Consider a structural engineer's analysis of a bridge design. The load calculations are exact. The strengths of the steel and concrete are what they are. The numbers are non-negotiable—and any interpretation that ignores them will get people killed. And yet those numbers do not design the bridge or determine how to balance its strength and its expected load. Do you add more rebar? Redesign the load distribution? Post a weight limit? Serious engineers can look at the same numbers and reach different defensible conclusions. The numbers constrain every possible answer, though they do not produce one.

Once we recognize that it is not numbers that make for rigor, we should be able to see that rigor exists in word-oriented epistemologies as well. The disciplined acts of careful interpretation—in law, in philosophy, in literary scholarship, in religious textual traditions—are held to the same standard. The rigor lies in fidelity to the text. What does this statute actually say? What does this passage actually mean? What does this argument actually establish? These are not soft questions. They have possible answers and wrong answers—answers the text itself can support or refute. A lawyer who ignores what the statute says is not interpreting freely; she is being unfaithful to her evidence. Scholars in these traditions are held to exacting standards: quote accurately, represent sources faithfully, do not put words in mouths, do not misrepresent arguments. These are not courtesy norms. They are the standards of rigor. We all know what it looks like when they are violated. Everyone who has written a literature review was taught the importance of reporting truthfully on what the literature says, and what it might imply. Those who are not faithful to their sources are hacks, regardless of the discipline or epistemology.

Good and rigorous work is hard. Hack work is easy, regardless of the field, discipline or epistemology.

Rigor in word-oriented epistemologies actually has a longer history than rigor in quantitative ones. The degree is called a Doctor of Philosophy. The natural sciences grew out of what was once called natural philosophy—traditions of rigorous argument, careful reasoning, and disciplined interpretation that predate quantification. Quantification was built on them. To fail to see rigor in word-oriented work because it lacks numbers or advanced math is to miss the forest for the trees.

I was a math and science person. STEM high school before STEM was even a thing. All the AP courses. County math team. Summers studying more math and more science. It then took four years of college and more years of serious effort in graduate school to trust that I could do good qualitative work. I have never been able to shake the epistemologies that originally seemed so natural to me, but I continue to work at it. The parallels across epistemologies are real—the demands of clear communication, strict thinking, and faithful engagement with evidence are recognizable everywhere—but seeing them requires work.

(Someone looking for the true soul of mathematics could do far worse than the CCSS standards for mathematical practice. And someone who understands word-oriented epistemologies can read those descriptions of fundamental mathematical thinking and see their own epistemology right there between the lines.)

Neither numbers nor advanced math makes something rigorous. Hack statistical work can be found everywhere. Rigor does not come from numbers. It comes from holding ourselves and our work to the demands of our methodologies, of honoring the demands of our epistemologies. In taking seriously the obstacles and challenges, even when they make the work harder and keep us from simply and easily reaching desired conclusions. And in so many fields, it means honoring the words.

Epistemologies, Respect, and Physics Envy (Part II): Deference to Quant

There is a hierarchy in educational measurement. Quantitative work is seen as superior. This is so thoroughly naturalized that many of the people who benefit from it cannot see it as a hierarchy at all—they experience it as a simple fact about what rigorous evidence and reasoning looks like. This is an epistemology at work.

We can watch how it operates in practice. Psychometric flags—low point biserials, suspect discrimination indices—are supposed to function as alerts, prompts for human judgment about whether something is wrong with an item. A negative point biserial should not automatically exclude an item. But in practice, they function as verdicts. Low point biserial? The item simply cannot make the operational form. The flag overrides the judgment of the content experts who built the item, reviewed it, and know the domain. Cronbach's alpha has achieved something close to supreme authority—a single number that seems to settle questions that involve far more than internal consistency. The quantitative output doesn't inform the judgment; it replaces it. Some deny that this happens, while others acknowledge the fact that this is standard practice.

Meanwhile, the people holding those quantitative outputs frequently feel entitled to opine about things about which they lack expertise. Where exactly is the line between fifth and sixth grade mathematics? What distinguishes two sixth grade standards from each other in ways that matter for item development? What does it mean for a student to have genuinely mastered a standard versus having picked up a surface procedure? These are not trivial questions. They require deep content knowledge, experience with how students think, and serious engagement with domain models. They are the core expertise of content development professionals. And yet many others don’t even see the tools and lenses required to build and hold that expertise—therefore think that there is nothing wrong with offering their views as though they might be of similar standing. Their epistemological framework does not see this other expertise, and they don’t either. Certainly, it is not as meaningful, challenging or rigorous as their own.

They don't see it. And what you can't see, you can't respect.

This often extends to qualitative methodologies generally. There is a pervasive assumption—rarely stated, but deeply held—that qualitative work is not really disciplined. That it has no real methodology. That its findings are, at best, suggestive and, at worst, just opinion. The possibility that qualitative research might have rigorous standards, systematic procedures, and criteria for what counts as a valid finding simply does not register. Not because anyone has examined qualitative methodologies and their foundational epistemologies and found them wanting. Rather, the epistemologies are ignored and therefore the methodologies viewed as foundationless. 

Here is what makes this genuinely strange: it is not a symmetric problem. Qualitative researchers in this field are typically intimidated by quantitative work. They defer to it. They actually have experience with it from their schooling days, and they respect that there is something rigorous there. They rarely question whether their quantitative colleagues' work actually meets its own standards. They just assume it does.

But it often doesn't. Underpowered studies. Violated methodological assumptions. Too much dependence on linear regression. Ordinal data treated as interval data. Multiple comparisons not accounted for. Proxies standing in for constructs they don't actually measure. The quant work that commands all this deference frequently falls short of the very standards of rigor it claims.

The asymmetry is that quantitative practitioners assume their own rigor without interrogating it, demand that others respect it, and simultaneously refuse to recognize the possibility of rigor in other traditions. They don't look for it. They don't ask what it would look like. The question doesn't arise, because within their epistemological framework, the answer is already settled. All of this despite the fact—or perhaps because of it?—that they rarely have any experience or training in rigorous qualitative work from their school days. 

Every discipline—every epistemological tradition—has its own requirements for rigor. All of us should lean into those requirements, interrogate our own work against them, and resist the temptation to assume that producing the right kind of output means the work is actually good. And all of us should do the harder thing, too. We should learn to see what rigor looks like in traditions other than our own, and respect the expertise it takes to achieve it—especially when our colleagues in our collaborations come from different epistemologies. 

Good multi-disciplinary collaborative work requires doing what our epistemologies cannot do for us. It requires looking for—and allowing—other ideals of rigor. It requires humility about our own epistemologies and their limits. It requires respecting the capabilities of others, rather than expecting deference.

Epistemologies, Respect, and Physics Envy (Part I): Introduction

[First of a five-part Series.]

Over at Hahvard, Andrew Ho teaches his students a wise mantra,"Measurement must be qualitative, then quantitative, then qualitative again". The idea is that educational measurement begins with qualitative work (i.e., theory, construct definition, instrumentation), becomes quantitative in the middle, and must return to qualitative at the end (i.e., communicating, interpreting, using results in context). He is challenging his students to understand that though his course is about applied statistics, that the statistical work is situated amongst serious qualitative work.

Obviously, he is right. Obviously, right?  I mean…isn’t that obvious?

Educational measurement is not physics or chemistry. It is not some investigation of a natural world that exists independently of human beings and their agreements. Rather, it is a deeply collaborative, deeply social enterprise. Teams composed of people with very different kinds of expertise work together to define constructs, build instruments, gather evidence, interpret results, and communicate findings to audiences who will use them to make consequential decisions. At the roughest level, the work is first qualitative, then quantitative, and then qualitative again. (And within each of those three phases, it breaks down further.)

A Framework for Foundational Competencies in Educational Measurement

This is not new ground. The field's own recent Foundational Competencies in Educational Measurement says explicitly that "educational measurement is a collaborative endeavor that requires people with varied skill sets to work together to design, develop, administer, and evaluate instruments that satisfy specific uses." We made a related argument in our commentary on the FCEM that genuine collaboration requires not just communication skills but foundational dispositions—respect the views of colleagues who bring different expertise and lenses, and genuine listening—that the document itself omitted.

That omission was not malicious. The task force was composed of thoughtful people who believe in the collaborative and qualitative dimensions of this work. And yet they reproduced, in a document explicitly designed to be broader than psychometrics, much of the narrowness they were trying to correct. That is not a criticism of individuals. It is an observation about how disciplinary training works: the lens your field gives you shapes what you can see—including what you can see about your own assumptions.

That gap—between acknowledging that qualitative work matters and actually treating it as having equal standing—is what this blog series is about.

To understand the gap, two things need to be clear: what an epistemology is, and what a discipline is. I will keep this brief, because these are not the interesting parts. They are the necessary setup.

We are often trained in disciplines. When done well, this training—often in college and/or graduate school—really shapes our thinking throughout our lives. We learn a set of tools. More importantly, we are given a set of lenses to direct our attention to some things and away from others. We are taught the types of questions we are supposed to ask, and what to do to answer them. This includes ideals of what counts as evidence, and most certainly what rigor looks like.

Therefore, different disciplines can look at the same topic and respond entirely differently. Consider the prosaic topic of a supermarket. It could be the topic of an architectural study. An economist might take a shot. Or a nutritionist. A labor relations specialist. A graphic designer. A psychologist. A sociologist. An urban planner. Or someone else. The same topic, but each discipline notices different things, asks different questions and produces different findings. None of them is wrong to use their lens. The supermarket is genuinely all of those things at once—and no single discipline captures it whole. To really understand supermarkets, you need many disciplines. And for these experts to work together—instead of simply ignoring each other in parallel—they must do something very difficult. They must not just acknowledge that each other exist, but actually respect the deep validity of each other's approaches.

(And to actually build a supermarket, you need an almost entirely different set of people to bring in their tools, their problem solving, their skill and craft and rigor. And materials.)

That difficulty runs deeper than it first appears. Underneath each discipline's tools and lenses lies something more foundational: an epistemology—a set of assumptions about the nature of knowledge, truth and how it is arrived at. Epistemologies are not the same as disciplines. A single epistemological tradition can underlie many disciplines. For example the different natural sciences share a common epistemology. Economics may be quantitative, but it does not exist in the same epistemology as the natural sciences. Disciplines’ ideals of rigor are based in their epistemologies, as are their ideals of evidence and reasoning. A collaborative multi-disciplinary field like educational measurement draws on more than one. The result is that practitioners trained in different disciplines but sharing an epistemological tradition can, with just a little effort and goodwill, recognize each other's rigor—even when the specific methods look quite different. Across epistemological traditions, that recognition is much harder. The other person's rigor does not just look different. It often does not register as rigor at all. Neither disciplines nor epistemologies can transcend themselves—that is structural, not a failing.

And here is the crucial point: eople are not their disciplines! People can recognize that their framework is their framework and mighty useful without being blinded to the valid existence of others—even those grounded in different epistemologies. That is, they can recognize that other legitimate ones exist and that the rigor their discipline cannot see is real nonetheless.

In fact, they must. If not, Andrew Ho’s insistence that educational measurement is first qualitative, then quantitative and finally qualitative again is just words—and we know that some epistemologies do not value words. 

Testing Time

Yesterday, I read NBC's version of the iReady attacks. Nuggets of truth, connected to the utterly implausible and even false, by amplifying whining and distortions by…who, exactly?

The lead source is a…tutor. Not a school teacher, not a trained and experienced professional educator with any classroom assessment experience—a tutor. Katelynn Petersen runs a private tutoring operation out of her home in Anchorage, teaches math to homeschoolers on the side, and was a project manager before that. iReady is a school-based platform. How many iReady-using students does she actually encounter? NBC led with her.

Then there's a speech therapist talking about math and reading curriculum and diagnostic assessments. Look, I went to speech therapists for years, and they really helped me. I respect speech therapists. But I would not ask them about core content instruction or assessment. That's simply not their domain.

Third source: an 8th grade student, whining dramatically. I love teenagers. I love their drama. I really do. But finding the real truth in the whining takes knowing them and listening deeply—not just laughing at their quips and calling it journalism.

What I've read across this feeding frenzy is clearly factually challenged and put together by people(e.g., Tyler Kingkade) who lack the knowledge to tell fact from fiction—and any apparent interest in doing enough investigation to do so. After all, you know, clicks.

And then there's this:

Criticism of i-Ready is a frequent topic on Reddit and TikTok, where teachers describe how i-Ready’s larger benchmark assessments, which students take three times a year, eat up 40 hours of instruction time, or say that pressure related to the software is driving them to quit.

You'd think such a sentence would link to multiple complaints on TikTok. There's only one: Eric Glenn, a middle school math teacher, claiming that iReady's benchmark assessments take up 40 hours of instructional time. The problem is that they don't. 40 hours is roughly two months of math class. That's more than two weeks, three times per year. That's approximately one-quarter of total math instructional time. That's not even vaguely plausible. Maybe he's splitting it across reading and math—but half those numbers aren't even vaguely plausible, let alone accurate. What is Mr. Glenn even talking about? The CEO of the company that produces iReady says it is just 5% of instructional time—which would less than 1/4 of what Mr. Glenn’s fantastical claim.

Here's the thing: the complaint that standardized testing takes too much time is not new. And it has never had merit. It is usually a sign that people are not willing engage with the actual merit, quality or usefulness of the tests in question. It is an attempt to disqualify them, regardless of their quality or merit. In this case, it is not even vaguely accurate.

Back when I was a high school English teacher, I gave reading quizzes twice a week:

• Not announced in advance.

• The five easiest questions on the reading I could think of.

• Trade and grade.

• Collect for me to enter into my gradebook later.

These quizzes had no direct instructional value, nor any diagnostic value. They were purely motivational—the carrot of an easy A for those who did the homework, and the stick of an F or D for those who didn't. Each one only took 5–10 minutes, twice a week. 50-minute periods, five days a week. That's 6% of available instructional time for quizzes with no direct instructional value whatsoever. And yet, no one ever complained how much time they took. Not my cooperating teacher, not any AP or principal or other supervisor. Not any student, nor any parent. And they weren’t even the the only formal in-class on demand assessments I used.

Kids take a lot of tests in school, most much longer than my reading quizzes. Even little quizzes are usually longer than that. There are chapter tests and unit tests. Semifinals and finals. Sit-down and on-demand performances for the purpose of evaluation. Some formative, some summative.

It always stuns me that people complain "testing" takes so much time, because they never consider reading quizzes or homework checks. They don't count chapter tests or anything that might be considered a pop quiz. Anything that teachers make is somehow exempt from the objection.

Look, I got very little information from those reading quizzes. I could already tell who was doing the reading and who wasn't. Sure, occasionally there was a surprise—some quiet student who actually was doing the work, just quiet for some other reason. But that was rare. Teachers can tell. I thought that they had indirect value (i.e., motivational carrots and sticks).

I have yet to see any large, formalized standardized assessment that actually takes that much time.

There may be legitimate complaints about iReady. Surely, there must be. But not all of them about testing. The curriculum and instructional supports that Curriculum Associates also sells under the iReady brand may well be genuinely problematic. But concerns about centralized curriculum aren't concerns about assessment. Glomming them together, as NBC does, muddies both arguments.

The time complaint, though? There is not—and never has been—any merit to it.

The Sorting Default: How Educational Measurement Tilts a Question That Belongs to Democracy

Growing up in Virginia, I was proud of my state when I learned of Thomas Jefferson’s vision of universal education (*cough* *cough* for free children *cough*). Then I learned that it was very much a sorting or “meritocratic” system. Universal for a few years, and then the culling begins. Every few years, another selection process—which he called “raking the rubbish”— with the highest performing students continuing their education and the others falling out, eventually culminating at UVa! TJ's original plan was never implemented, but around 200 years later, I took tests in second and eighth grades, each of which opened doors for me that were closed to others.

This system is firmly grounded in one side of one of the most fundamental tensions in education and schooling systems design. To what degree are these systems intended i) to help sort and rank students (and identify high performers) and/or ii) to support universal efforts at human improvement through education? One view sees education through a meritocratic lens that inevitably uses sorting as both a means and ends of schooling. The other views education through a more universalistic lens that sees the means and ends of schooling as being the provision of learning and growth opportunities for each individual student. The former sees merit as a real concern and question to be minded and acted upon, and the other sees universal growth opportunities for children as the more compelling central concern. 

Note that this is not the question of whether schooling should serve the economic market/help prepare students for the world of work. This is not about balancing vocational goals and citizenship goals. Some sorting advocates and universalists can agree that schools should prepare students for economic participation in society, while other advocates from each view can agree that that is not an appropriate aim for schooling. Nor is this about the appropriate topics and subjects to address in schools. Each view can be aimed at a variety of goals or purposes of universal public education. This tension is even deeper than that, and perhaps orthogonal to it. 

Nonetheless, these two different goals are often in tension. In order to address both of them, we put together different programs in our schools. We spend money and time on sorting functions and universalistic development functions. We serve both views, but do so with a bricolage that can make our systems feel a bit incoherent. Perhaps we are serving more students this way, or perhaps it is merely accidental accumulation of programs. Certainly, it fails to serve either view coherently.

This tension clearly exists in educational assessment. Should assessments be designed to support meritocratic sorting, or should they be designed to support universalistic opportunities for students? Is relative standing more important, or is it more important to report on where students are a) thriving and b) need more support. 

What disturbs me is that this tension is not always taken seriously by large scale assessment professionals. Instead, there is a strong tendency for some (i.e, far too many) to put a heavy thumb on the side of the meritocratic sorting side, regardless of the demands of test users. There’s a clear bias in the field in favor of norm-referenced sorting, and underdevelopment of tools that can support universalistic provision of opportunities for students. This unwitting bias pushes potential clients, policymakers, communities and even our society towards one view of schooling, rather than responding to civically determined decisions about what we want for our schools. The field's unreflective bias in favor of the meritocratic sorting view of schooling effectively preempts decisions that belong to democratic processes—without the field even recognizing it is doing so.

The Most Insidious Obstacle to Well-Aligned Items

Items are the building blocks of assessments, and if they do not perform their designated function no purpose or inference from their test can be valid. That is, they must actually elicit evidence of the targeted cognition for the range of typical takers. 

The most insidious obstacle to the production and inclusion of well aligned items on operational tests has nothing to do with psychometrics. Truly, even educational measurement's original sin (i.e., the a priori assumption of unidimensionality)—my go-to villain for over 15 years—is not as insidious as the idealized student/test taker. It is more insidious because only the idealized student/test taker is so often consciously imagined by item contributors throughout the entire item development process, despite being even less frequently appropriate than unidimensionality.

The idealized student/test taker is a convenient fiction. It is an easy fiction. This paragon of testing virtue is an incredibly attractive fiction. But it is very much a fiction, and it is an enormous obstacle to developing valid items.

What is the Idealized Student/Test Taker (IS/TT)?

The idealized student/test taker learned all the lessons. They were diligent in class and are diligent on tests. They attempt every item exactly as their teachers would want them to, applying the correct technique and knowledge. Their thinking is linear, and clear. They understand the item and what it is trying to communicate. They recognize what they are being asked to do, and they do it as item writers and other developers would want them to.

They do not cheat. They do not do anything—not even unintentionally—to undermine the quality of the inferences we make from their performance. They do not look for shortcuts. They do not apply savvy test-taking skills. They do not look for alternative approaches when they realize that they lack some called for knowledge or skills. They do not have strategies for guessing. They succeed properly as directed, and accept their own shortcomings when faced with challenges beyond their ability. 

The idealized student/test taker remembers everything they learned, but is not advantaged by a particular pedagogical approach or examples they encountered during their studies. They are not tied to how their own teachers asked questions or gave instructions, being equally able to understand other wordings or framings. 

The idealized student/test taker is neither advantaged nor disadvantaged in any way. They do always work in good faith. They are beyond frustration or excitement. They are perfectly focused only on what the item gives them.

Therefore, the idealized student/test taker always provides a successful response when they possess proper proficiency with the targeted cognition and always fail to do so when they lack such proficiency. 

What is Wrong with the Idealized Student/Test Taker?

The IS/TT is a fiction. They not only are a fiction because no such student or test taker actually exists, but even more dangerously because they promote the idea that there is one sort of “typical” test taker that item writers, content development professionals and other item reviewers should focus on. They suggest that there is a preferred or primary test taker whose views and understandings are most important in our considerations. 

But the fact is that test takers vary enormously. They vary in an enormous number of dimensions. They have different experiences and proclivities. They have had the benefit of a multitude different instructional approaches and examples. And very few of them really want their test results to accurately reflect their proficiencies. Instead—and quite understandably—they generally want the highest score they can get. Some might be lazier. Some might be more devious. So, there are differences in motivation, but virtually across the board, they would prefer a higher score to a fairer score. 

If told what an item is trying to target, they generally would use some other approach if they felt it would given them a better chance to get the points. Real test takers do not see test developers or the items themselves as some collegial partner, but rather obstacles to be overcome. 

Moreover, there are many deep contradictions within the idea of the IS/TT. For example, we imagine that they can do every item as we would like them to while also imagining that when they lack the targeted skill, they do not employ strategies to work around its requirement. We also imagine that they lack any particular ethnic or cultural identity, and yet are fully comfortable with the cultural context of the item we are considering. It is not a coherent ideal, but rather a convenient collection of proclivities and behavior.

Why so Insidious?

I do not think that anyone would question anything in the previous section. It is all rather human and very obvious. 

And yet, it is so much easier to write, refine or review items with just the IS/TT in mind. It makes the process more direct and much less uncertain. It allows us to stay in our own perspectives, think about our own intentions. It is cleaner and simpler. 

It requires vastly more energy and work to consider the range of typical test takers. It is just hard to do the real work of radical empathy (i.e., applying cognitive empathy to a range of different types of test-taking personas) to envision how different sorts of test takers might respond to an item. Doing that requires us to go beyond our experiences and even beyond the experiences of people we know. Doing that well is rigorous and demanding professional practice that does not come naturally to many people. 

And so, everyone involved in developing items has a tendency to go back to the IS/TT. We all feel the urge. But items developed for the IS/TT will invite too many alternative paths by which real test takers can produce a successful response without using the targeted cognition. And they will trip up too many test takers who have appropriate proficiency with the targeted cognition but are subject to some other construct-irrelevant issue.

Because our tests are intended for a wide range of test takers, we must reject that urge. We must look deeper and think harder. 

The most important thing we can do when writing, refining or reviewing items is to put the idealized student/test taker aside and see the diversities of test takers who actually will attempt our items. This type of rigorous professional practice is necessary to produce valid items—items that elicit evidence of the targeted cognition for the range of typical test takers.