On the one hand, math standards tend to call for use or application of math KSAs in real world contexts. Those are the application problems, which some people refer to as word problems. However, merely putting the math usage in some thin context is no more a real world context than cat ears or a simple mask is a costume.
On the other hand, items with too much novelty require test takers to engage in learning and exploration to even put themselves in a position to apply those core focal KSAs in the alignment reference or standard. The work can be built with a whole bunch of additional KSAs that prevent test takers with proficiency in the officially targeted cognition in the alignment reference from demonstrating their skills. False negative results, which we call Type II errors.
These are two different hands—these goals are in tension—because if real world context has any real use or meaning, it is a call for the kind of discernment and decision-making that warnings of excessive novelty in items are concerned about. And yet, the Common Core State Standards’ approach to math seems to depend on these sorts of real world context problems to bring in the Math Practices. Moreover, the Next Generation Science Standards (and Achieve’s preferred approach to NGSS assessment) very much leans into novelty and exploration for the contexts in which its Disciplinary Core Ideas should be applied.
This creates a major tension for large scale assessment, where there are not enough opportunities to collect data to be able to triangulate through item x test taker interactions to figure out why test takers are successful with some tasks and not others. Disentangling shortcomings in focal KSAs vs. additional KSAs requires triangulation. There simply are few types of information available, and little data from each. And yet, the novelty inherent in meaningful real world contexts creates the greatest need for such triangulation.
Such tensions are common in item development. Many content development professionals have well-developed ideas about what makes for higher quality items, but the constraints of large scale assessment (e.g., few testing opportunities, widely diverse test taking populations, psychometric demands for short items) make reconciling those multiple ideas quite difficult. Too commonly, those constraints require unacknowledged compromises that reduce targeted cognition to something outside the real core of the standard, if not outside the grade level standard entirely.