This is a very particular design aimed at addressing the kinds of engagement challenges that can lead to suppressed performances by test takers, and thereby suppressed reported scores. These engagement challenges are core to the motivation of culturally responsive assessment, but are actually much broader than that.
Of course, this design is aimed at providing instructionally relevant criterion-referenced multi-dimensional reporting—which can be turned into deliberately designed composites if unidimensional results are needed for other purposes. That is, it is not aimed at norm-referenced reporting, because most large scale assessments are criterion based—or should be.
The test presents choices to the test taker at each point, rather than a single item. Which item would you like to do? That set of choices always is aimed at the targeted cognition, using the same focal KSAs, at about the same level of content complexity. (Difficulty and cognitive complexity are not population invariant, and even vary from test taker to test taker, so we do not pretend that they can be equalized or approximated.)
The items may vary in cultural context and topic context. There can be some baseball items, some football/soccer items, some cooking items. Some skiing items and some beach items. Items about making tamales and items about making lasagna. Fairness concerns for individual items would be different, as concern shifts instead to broader pool or choice set concerns. This provides more flexibility, perhaps building on RTD task models.
Test takers respond to ~5 items per reported score/attribute, meaning they have to select an item ~5 times. Some items may appear more than once, but they cannot be attempted more than once.
Test takers are given the option of having the platform select an item for them, but only after they have an opportunity to review their options.
Using cognitive diagnostic modeling (CDM) rather than IRT. The humility of CDM’s binary reporting (i.e., proficient/non-proficient) instead of misleadingly precise quantitative scores allows more flexibility in item matching. Each selected item must simply help to provide information about test taker proficiency, contributing to CDM’s needs. Old research about test taker choice was based on IRT-based test designs and data analysis, and it doesn't apply to a CDM-based approach. (Moreover, the nature of choice in this design is based on interest and engagement, not on the kind of focal-KSA selection of that old research.)
Larger tests are composed of a series of testlets (i.e., a set of items focused on a single reporting category or standard), or interleaved testlets.
The entire domain model may be addressed across multiple testing sessions, or even at different points through the years—as appropriate.
What does this design give us?
Like any CDM-based test design, far more interpretable and actionable results than traditional CTT- or IRT-based test designs.
Far more justified breaking up of larger domain models into subsets whose assessment can be spread through the year. So long as domain models are ignored in favor of unidimensional analysis and reporting, there is no content-based justification for breaking up large assessments, and yet doing so randomly is clearly ridiculous. This approach acknowledges the desire to break them up, and does so in a way that is coherent—from domain model all the way to score reporting.
Greater engagement by better matching item contents to test takers’ interests and/or cultural backgrounds. Not ideal matching, but surely they can do better at this than any algorithm—and far better than not even trying.
The documented boost in performance and persistence on items that research keeps showing that choice creates. Even when test takers make sub-optimal choices, the mechanism of choice itself provides benefits. (There’s some research that shows that that might actually be culturally situated, but I’m ignoring that, for now.)
What does this design give up?
Well, most importantly it gives up largely baseless claims about precise sorting and ranking of test takers by some statistically evoked interpretable Θ (or transformation thereof). For some, this will be seen as a huge loss.
Vertical linking of scores, which relies on the idea that the construct does not change across grades to be interpretable as anything other than norm-referenced reporting. Again, for some this will be seen as a large loss.
Linking scores across years—because the need to link scores across years requires multiple arbitrary scales that need a common anchor to provide the illusion that comparisons of uninterpretable statistically evoked scores are ever themselves interpretable. Of course, because the meaning of proficiency should not change from year to year, scores are already linked. So, linking work is not needed. (Admittedly, this might not be quite right. I need to do more thinking on this one.)
What does this design cost?
Well, it would take a lot more items. Any test that provides more information is going to need more items, but this is dramatically more than that. Because AI/LLMs are really bad at producing on-target items—which criterion referenced tests require—we cannot (yet?) count on automated item generation to do this for us. It could help to adapt good items to different cultural or topic contexts, but they’d still need to be reviewed. Yeah, this would be expensive. And the more culturally responsive or personally relevant the offered choices might be, the more items it would require.
Testing time. Again, any test that provides more information is going to take more testing time, especially if that information is going to actually be useful. Furthermore, the choice mechanism takes up additional time. I have no doubt that the unwillingness of so many to devote sufficient time to large scale assessment is based in their perceptions of the value of those assessments. But even in the best case, this additional testing can come at the expense of instructional time. It would be better if high quality large scale assessment could take the place of redundant classroom assessments, but we would need a norm of actually high quality large scale assessments that provide high quality instructionally relevant information to see if the broad cultural pattern of redundancy might shift.
Forced distributed administration. While having the option of breaking up larger tests into through-the-year assessments is great, the requirement might not always be welcome. Not only would test takers have to complete more items to get more information, there is an additional cognitive burden of selecting items. Perhaps the motivation that comes from the choice mechanism would make up for some of that, but it is more cognitive work. Test taker stamina is not unlimited.
Real work to be clear on the meaning of proficiency for each reporting category or standard. For example, the work of constructing range PLDs is valuable, but it is costly, too.
This most definitely is not a well thought out proposal. It occurred to me just this morning, in response to some serious conversations about test taker engagement and the meaning of engagingness of items. Surely, there are more problems that I’ve named. And, yeah, it’s impractical. I don’t see it happening. But I see substantial advantages of adding choice to address fairness and cultural responsiveness, and in this design that same mechanism might actually give more accurate data on test taker proficiencies.