There is an absolutely crazy idea out there—and it is well established in our culture and our practices—that alignment and fairness are distinct things.
The usually brilliant Center for Assessment (man, is there a smarter and more thoughtful collection of people anywhere in our field?) just offered this:
These are not separate questions. They are not separate categories. They cannot be answered sequentially. They cannot be properly understood without braiding them together.
Anyone who tries to answer the former question (i.e., alignment without fairness) must necessarily select some ideal of the student/test taker to consider before all other test takers. This idealized student/test taker (IS/TT) is determinant of alignment, as though there is some base normal student, and everything else is aberrational or otherwise marginal. The selection of someone’s IS/TT is usually unconscious and unexamined, and it is a principal mechanism for fairness issues to bias consideration of the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests.
You might select some imagined version of yourself at that age. I might select a particular sort of student that I remember from my classrooms, or maybe another sort of student. Or maybe my brother. You might select your own child, or some particular idealization of normal children as you imagine them.
Just as item parameters will vary across populations, there is no singular typical test taker. There is no preferred test taker. There is no imagined test taker that we can use to ask the question, “Are these items ‘aligned to the intended construct’?” without declaring—usually unwittingly—this is the one sort of test taker I care about.
No, if we actually care about alignment, we can only ask “For which sorts of students are these items and this test ‘aligned to the intended construct’?” We must acknowledge that fairness is always a part of alignment.
Otherwise, we are just proudly accepting that we should have a fundamental bias in our considerations of alignment—which is a principal mechanism by which fairness issues embed themselves in our assessments.
Now, we can take two perspectives on this question. We can ask:
“For which sorts of students are these items and this test ‘aligned to the intended construct’?”
“What sorts of ‘factors unrelated to the knowledge and skills it is intended to measure’ might impact what sorts of students’ likelihood of responding to items successfully?”
We can focus on the targeted cognition and then focus on the construct irrelevant barriers. These questions remain somewhat intertwined, of course. But we cannot meaningfully consider either without keeping the range of typical test takers in mind the whole time—unless we are willing to concede that our own IS/TT is the singular sort of student/test taker that our test is for.
This is part of why I like to say: Fairness is so important that it merits its own chapter in The Standards; Fairness is so important that it should be embedded in every chapter in The Standards. I simply do not know how we can emphasize the importance of fairness without doing both.
Psychometricians think that reliability is so very important that they try to cram it into Chapter 1 of The Standards, displacing what The Standards actually says about dimensionality (i.e., “evidence based on internal structure,” p. 16) to claim that it means something rather different. I understand why a discipline so focused on reliability would want to make sure that their North Star is part of “the most fundamental consideration in developing tests and evaluating tests.” But if there is one chapter’s main idea that I would cram into Chapter 1, it is Fairness. And I would not be content with displacing just one thing within Chapter 1; I would lace it through everything.
(Those two categories are from a discussion of evaluating validity in the context of tests for students with significant cognitive disabilities, but this is an incredibly diverse population across an incredible number of dimensions. Yes, they have a range of disabilities, but they also are hugely diverse demographically, culturally and experientially, as well. And I have absolutely no doubt that the authors of those categories are entirely aware of this.)