Here in the world of large scale assessment, it is abundantly clear that AI/LLM models are shockingly far from understanding the meaning of reading standards, as are far too many people. Perhaps at the most fundamental level, they simply do not unpack reading into the large collection of knowledge, skills, understandings and abilities (KSAs), and instead view standards as task lists that reading should enable. Therefore, any failure to answer a supposedly aligned question (or item) is a failure on that standard. That view simply ignores the fact that the larger endeavor of reading is made up of many, many smaller KSAs.
People do not make this mistake with mathematics. People seem to easily be able to understand that an arithmetic error can lead to an incorrect result, even when the student has shown that they understand the actual grade level skill in question. Performance may require both arithmetic and understanding of the more advanced concept, but they are different KSAs. Similarly, stumbling over vocabulary or understanding of the literal surface meaning of a text are failures of lower level skills, and certainly not middle or high school level skills.
So, let’s dive into just one example of a reading standard and explore what it really means, and how we figure that out.
The Standard
Taken from the Common Core State Standards, RI 8.2: Determine a central idea of a text and analyze its development over the course of the text, including its relationship to supporting ideas; provide an objective summary of the text.
ELA reading standards are based on a small set of anchor standards that develop and deepen through the grades. At first grade, this thread takes the form, Identify the main topic and retell key details of a text. At 11-12th grades, it takes the form, Determine two or more central ideas of a text and analyze their development over the course of the text, including how they interact and build on one another to provide a complex analysis; provide an objective summary of the text. So, let’s break down what it means in 8th grade—RI 8.2.
Give Up on “Reading Comprehension”
This thread is never about decoding words or making basic sense of sentences or what an individual sentence means. Those are different sorts of skills, lower level and lower complexity skills. Modern ELA reading standards are about making deep and more complex sense of what is already understood on the surface level—certainly this is the case by 8th grade. Simply misunderstanding what the text literally says is not a failure of the KSAs that make up this grade level standard.
The complex endeavor of reading is about putting together an array of skills that when utilized together proficiently give the reader an understanding of the layers, connections and architecture of a text. They provide a reader an understanding of how a text accomplishes its layers of nuanced goals. They are about idea, audience, technique, implication and various ways to evaluate the strength of a text—and of its contents.
One could term this “reading comprehension,” but it would be quite different than what most people use the term to mean. What is really going on is the fact that reading skills—expectations for the KSAs learned by readers—continue to deepen far beyond the early- to mid-elementary school skills that basic and low expectations for “reading comprehension” suggest.
Just as the abstraction skills of algebra and the logical, ordered thinking skills of geometry are not necessarily consciously recognized as such when they are applied in adulthood, these more advanced reading skills ought to be applied unconsciously. It is easy for adults to recognize an arithmetic error or a failure to recognize a word, but most of the time these things are taken for granted. The unconscious and integrated use of these more advanced skills is incredibly important and is the real goal of education, regardless of the content domain.
Summarizing
Summarizing is an incredibly complex set of skills, and tracing through the grade progression of this standard (shown at the end of this post) makes clear how it is expected to deepen and gain detailed nuance over the years. But at its heart, summarizing is about i) recognizing what is important for this purpose and therefore worth including, and ii) presenting it in a sensible form that fits both the original summarized text and the purpose of the summary.
Yes, summaries can have purposes. For example, consider the difference between “Summarize the events in this text” and “Summarize the important ideas in this text.” This standard is about informational texts, rather than literary texts, so one might ask for different types of summaries. Were the reader asked merely to “Summarize the text,” they would have to judge for themselves what aspect of the text should be the focus of the summary. That is an important summarizing skill—the ability to make that judgment. Of course, it is not expected at earlier grades, where texts are simpler. But it becomes more important in later grades as texts become more complex.
At early grades, summarizing is little more than retelling what has already been presented to the student. At later grades, it is about connections between ideas. What does it mean at this grade, grade 8?
Look to the Language of the Standard
The standard tells us the level of sophistication that is expected at this grade. Determine a central idea of a text and analyze its development over the course of the text, including its relationship to supporting ideas; provide an objective summary of the text.
* This is not just about one singular main idea, but rather about recognizing that there can be multiple central ideas in a text and distinguishing between them.
* This is not just about recognizing that a text can have multiple ideas, but rather about recognizing that some ideas are more important or primary and other ideas are less important or play a supporting role.
* This is not just about recognizing central and supporting ideas, but rather about recognizing the nature of the relationship between a supporting idea and a central idea.
Of course, any standards document or domain model is written for an expert audience. It is about organizing, labeling and classifying particulars, not about defining what everything means. In this case, the audience would understand that a supporting idea can widen or narrow the claim of a central idea. It can be an illustrative example of a central claim, or a purposive counter-example. It can detail a particular aspect or application, or offer a more generalized context for the central idea. Or work to support that central idea in some other way. The language of the standard is vital, but it assumes that the reader can understand it.
It is difficult to convince LLMs that ELA content experts are not all white women.
The Requirement of Expertise
Without expertise in the content domain, a reader is not going to be expert enough to understand what the language of the standard means. This is no slight to the non-expert—who is likely quite expert in some other field or topic. Rather, it is the basic fact that professionals in virtually every field need to communicate with each other in ways that they can understand. Expanding out every meaning and implication every time is simply not effective, and the idea raises real questions about how far that might be taken.
Reading and understanding a content standard, learning standard, learning objective or any domain model requires that the team doing the larger project include people who have sufficient expertise to understand those documents and their elements.
How Much of the Standard?
This standard, RI 8.2, appears to have two parts. The latter part is providing an objective summary, and the former part is about relationships of central and supporting ideas.
One might read the standard as suggesting that the former part defines what the latter part means. That is, provide an objective summary in which the relationships of central ideas to their supporting ideas are more or less fully analyzed. That is a lot. It is clearly a defensible reading of the standard, but it asks too much to be done quickly. Rather, it is the kind of task that requires thought and planning and revisiting. In fact, the nature of modern reading standards is that they envision writing about reading, which makes the scope of thinking and deliberation this suggests more appropriate. (Again, experts know that this is what modern reading standards focus on, as they have read the entire domain model instead of just taking part of it out of context.)
But that is not the only reading. One might look at RI 8.2 and see that it is about understanding and relating central ideas to supporting ideas, which can culminate in a complete summary, but does not have to. That is, there is that cluster of KSAs about determining a central idea and analyzing the development of the idea through its relationship to its supporting ideas. And there is also the next layer of using that bundle of KSAs to build up a complete summary—which would also require recognizing the purpose of a summary and selecting appropriate central ideas for that summary. And providing a summary would also include figuring out how to present it coherently and understandably.
So, the language of the standard, its order and its use of a semi-colon (;) suggests that one could analyze one central idea like this or do the larger summary including multiple central ideas. However, merely identifying a central idea is not enough. That is not even the 3rd grade skill. Moreover, the standard is clear that the KSAs include determining a central idea, rather than merely responding to someone else’s nomination of a central idea. That is, it implies that this cognitive effort of analysis is how one recognizes what is a central idea and what is a supporting idea, and being supplied with a central idea undermines the requirement to do that analysis.
What We See From AI
Instead of those real grade 8 reading skills, we see something quite different from AI/LLM efforts to base items on this standard. We see less. We see so very much less. It is not merely the failure to recognize that item type modality (e.g., multiple choice) can undermine targeted cognition (e.g., consider the cognitive act of providing vs. recognizing), which is bad enough. Rather, this seems to be a refusal to take reading standards and ELA domain models seriously as descriptions of particular grade level skills. We have see this from multiple LLMs, using multiple versions, with every level of prompt engineering in our projects and for every ELA standard we have attempted. Again and again, we see a mistaken core idea that all that matters is completing a task, with no consideration of which KSAs the item actually taxes—which is what item alignment is about.
We see a radical lowering of cognitive demand and grade level. That is, many items simply ask the test taker to select the correct main idea of the text—even if phrased differently. We see selection of potential summaries (or analysis of central idea development) which simply requires eliminating the ones that include basic misunderstandings of the text—which are really just rather simplistic reading comprehension questions.
Of course, it is a problem that AI/LLMs do not understand how to read or unpack modern reading standards. But there is a lot that AI/LLMs cannot do. The real problem is when people accept that the dumbed-down interpretations from AI/LLMs are all that reading standards even could mean, without asking anyone with the expertise to know better. We have already seen this kind of misunderstanding from quantitatively oriented practitioners for years. In this new AI era, their ignorance is getting built into the structures of generative AI, lowering the ceiling on test (use) validity and possibly shortchanging students of their opportunities to learn the advanced skills that will help them in their futures.
Development of a Single Strand of a Reading Standard: REading Informational Texts 2
Kindergarten: With prompting and support, identify the main topic and retell key details of a text.
Grade 1: Identify the main topic and retell key details of a text.
Grade 2: Identify the main topic of a multi-paragraph text as well as the focus of specific paragraphs within the text.
Grade 3: Determine the main idea of a text; recount the key details and explain how they support the main idea.
Grade 4: Determine the main idea of a text and explain how it is supported by key details; summarize the text.
Grade 5: Determine two or more main ideas of a text and explain how they are supported by key details; summarize the text.
Grade 6: Determine a central idea of a text and how it is conveyed through particular details; provide a summary of the text distinct from personal opinions or judgments.
Grade 7: Determine two or more central ideas in a text and analyze their development over the course of the text; provide an objective summary of the text.
Grade 8: Determine a central idea of a text and analyze its development over the course of the text, including its relationship to supporting ideas; provide an objective summary of the text.
Grades 9-10: Determine a central idea of a text and analyze its development over the course of the text, including how it emerges and is shaped and refined by specific details; provide an objective summary of the text.
Grades 11-12 Determine two or more central ideas of a text and analyze their development over the course of the text, including how they interact and build on one another to provide a complex analysis; provide an objective summary of the text.