At AIME-Con this month, I heard a very high level measurement expert at a major testing organization say, "Fairness (it's basically the same thing as DIF)…" I believe that that was the dumbest thing I have ever heard anyone say at a conference. I mean, not a dumber question, not a dumber comment in the halls, not a dumber joke. Just a sheer, dangerously stupid thing that can do harm to countless millions of test takers—the absolute worst. So bad that even though this was mid-session and a symposium presenters said it as part of his prepared remarks, I blurted out "No, it isn't." But none of the other talking heads ever pointed out that he was wrong, or how dangerous such thinking is. (I have since heard from others who were in the room who thought it was shocking, but no one else acknowledged it at the time.)
"Fairness is a fundamental validity issue and requires attention throughout all stages of test development and use." (p. 49, Standards for Educational and Psychological Testing)
"Fairness is so important that it merits its own chapter in the Standards; fairness is so important it should be infused in each chapter in the Standards." (Me, many times)
For norm-referenced assessments (and test development or reporting techniques that lean on norm-referenced elements), fairness issues are a fundamental threat to every score and every score interpretation. If the test scores for some significant share of the test takers are not accurate, then any inference based on relative standardizing loses credibility. How do we know that this test taker is actually anywhere that close to that percentile, or that this basket has anywhere close to that share of the population, if we cannot trust that 10-20% (or more!) of the test takers are properly placed on the unidimensional scale?
So, fairness issues undermine all IRT-based reporting—which is the vast majority of large scale assessments.
Fairness issues also threaten criterion referenced assessments. If we cannot be confident that we have dealt with the fairness issues, how can we be sure that this person's score (or score profile) is not one of the impacted scores (or score profiles)? How can we have any confidence that we are not over-inclusive in some categorizations and under-inclusive in others?
Fairness really is that important.
DIF (differential item functioning), on the other hand, is not. DIF is just one very, very limited technique that barely captures any issues, and at best just flags items that merit further examination.
DIF has many issues. First, in practice it relies on the same a priori assumption of unidimensionality as psychometric model selection. That is, it presumes that the targeted construct is unidimensional, and therefore revealed deviations from that unidimensionality are problematic. Of course, none of the supposedly criterion-based targets of large scale assessment actually are unidimensional. That's a big problem that is difficult to address.
Second, DIF can only look for bias after the fact, and only across tracked groups. So, we cannot run DIF for single parent households vs. dual parent households, or Mexican vs. Puerto Rican, child of immigrants vs. 4th generation. That's another big problem that is difficult to address—though it theoretically could be addressed.
Third, we do not actually run DIF across all the potentially relevant groups we could: urban vs. suburban vs. exurban vs. rural. We have test taker addresses, so we could. With K-12 assessment, we could run districts against each other or against all the others, but we don't. My guess is that that is a product of an even deeper problem that is even more difficult to address.
But even best case, DIF is an incredibly limited tool. DIF necessarily assumes that all the other items are fine, but this item might be a problem. So, fourth, DIF cannot see uniform bias across a set of items. That's an unsolvable problem.
Fifth, there are untrackable differences between test takers. For example, which students recently lost a grandparent or pet? Which students have which common hobbies? That is an unsolvable problem.
Sixth, DIF presumes that items that appear to behave oddly for some students are exclusively responsible for their own oddness (i.e., local independence). But sensitivity issues can impact test takers for some time after they are raised, meaning that a problem with item #n might create impacts in item #n+1. DIF is never going to be able to address that because fairness does not respect the assumption of local independence.
Seventh, for DIF to find anything it requires large enough sample sizes (i.e., power). But I do not know of anyone who thinks that fairness issues only exist when subgroup sizes are big enough for a particular statistical method.
Eighth, no serious thinker even claims that DIF is about fairness. DIF is about difference, and fairness is about causes and interpretations. This seems to be the kind of slip of a very sloppy thinker who has let his own casual and sloppy thinking overwhelm his supposed declarative knowledge. And then, he said the quiet part out loud.
Of course, none of that addresses the many larger and longer term issues that are part of fairness. DIF cannot even capture all the immediate item-level issues of differential item functioning between relevant subgroups. It is merely a Band-Aid that we hope might flag some of the issues that were not caught by the far more robust procedures that should be in place earlier in the item development process.
Only a psychometrician would say that fairness is “basically the same thing as DIF.” I would hope that only a very bad psychometrician would say something that stupid, but this guy has been promoted a lot over the years, so I've got to think that other psychometricians think that he is pretty good.
Which leaves me even more worried than I was a month ago about how poorly the discipline of psychometrics understands fairness, or thereby validity.