However, we’ve come to realize that ECD is actually short when it comes to evidence. I do not mean that it lacks procedures there—after all, it lacks procedures everywhere. Rather, ECD has far too little to say about evidence itself.
How do we know what counts as good evidence? How do we recognize evidence? How do we avoid bad evidence?
ECD’s framework includes Evidence Statements (i.e., descriptions of what evidence of the claims might look like in action). And like Task Models and Domain Models, ECD does not explain what evidence statements should look like. That is left to the individual practitioners and/or project teams. But we’ve come to realize that there are some essential problems with this too vague view of Evidence Statements.
Simple Evidence Statements have a number of significant weaknesses. This is why RTD suggest more robust evidence statements, and creating them in the context of strong Item Logic.
First, simple evidence statements often mistake an absence of evidence for evidence of absence. Whether the purpose of an assessment is summative or formative, identifying what students do not know and cannot do is at least as important as identifying what they can do. In fact, with formative assessment, it is even more important. This is not about being negative, rather this is about being instructionally minded. It is important to be sure one does not confuse an absence of affirming evidence for the presence of disconfirming evidence.
Second, simple evidence statements are prone to Type II errors (false negatives). This is in part due to the absence of evidence problem, but it is also due to their inability to disentangle different causes for mistakes or errors.
Third, simple evidence statements are prone to Type I errors (false positives). That characteristic of student or test taker work could be present due to that particular knowledge, skill, and/or ability (KSA), but it might be because the test takers took an alternative path that did not depend on that KSA.
Simple evidence statements likely work best in the context of some sort of portfolio assessment, in which raters are able to review a broader set of each student's or test taker’s work and look for larger trends and patterns. Taken together, the errors in that noisy data can cancel out and the signal of information can become apparent. This is really just a sample size issue; the noisier the data, the larger a sample size is needed.
However, neither formative assessment nor large scale standardized assessment has access to such large samples of a student's or test taker's work for each assessment target. Therefore, robust evidence statements are needed.
Robust evidence statements must include information about the context in which the evidence appears. What sort of directions or instructions prompted the work? Did they specifically ask for this sort of evidence, or did they merely provide an opportunity to develop it? How much scaffolding was present? Did the task allow for alternative paths? ECD talks about the evidentiary argument, and the importance of the source of and context for evidence is well known in that field of law. Assessment should take those issues just as seriously.