Item difficulty is notoriously hard to predict, and yet people keep trying. IDM (item difficulty modeling) has been a recurring effort in educational measurement.
It should be obvious that generic surface features of items are never going to be sufficient to predict item difficulty. It should be obvious that item difficulty is driven by KSA-specific details that vary from alignment reference to alignment reference. (An alignment reference is whatever an item is written to align to. It could be a state learning standard, an element of a job/role analysis, or one level in a range performance level descriptor (rPLD).)
Consider the difference between these three questions. They are all single-digit multiplication questions, right? All straight off the multiplication table.
4 x 5 = ?
7 x 8 = ?
8 x 9 = ?
Obviously 4 x 5 is the easiest and 8 x 9 is harder, right? Duh. Bigger numbers. Is that what your model is going to say? Then why is 7 x 8 the hardest of the three?
I could easily find two reading passages for which the shorter one with slightly simpler vocabulary and sentence structure actually is much more subtle and nuanced, actually is more difficult to answer questions about for its subtlety and nuance—a shorter passage that experts will say is better. An IDM simply built on surface features is not going to catch that.
Similarly, to figure out how to predict which shape rotation item is harder, one must look at aspects of these items that are specifically relevant to the task of rotating two- and three-dimensional shapes.
Generic surface features—including surface features of reading passages—are never going to be enough. Some of the smartest people whom I have respected the most have attempted to do this, and they were foolish every time they tried.
But that's not what the real problem has been. That is not why IDM efforts in K-12 assessment can never work.
Item difficulty—be it CTT's empirical p or IRT's b parameter—is not simply a function of items. It is not population invariant. It is not stable or constant. It is not predictable simply from items because differences in instruction and experience alter item difficulty.
There are traces of this idea back in Goldstein (1983). Dan Koretz (2008) and others have pointed out that inappropriate test prep can alter the difficulty of particular item logics without improving the more generic KSAs. Ross Markle has spoken about how changes in item difficulty driven by improved instruction are the desired result of assessment programs that highlight areas in need of improvement. Parameter non-invariance is not just inexplicable item drift or a security problem. It is not necessarily any problem at all. It is what is supposed to happen.
Imagine two districts use the same test and it reports students in both are really bad with some alignment reference (i.e., those items are very difficult). One of them takes this information and revisits how that basket of KSAs is taught, improving instruction district-wide. In the next year or two, those items become much less difficult in that district, right? That's the whole point! Now, imagine that the whole state takes the hint and copies that district. This is what we want to happen!!! Those items not only get less difficult, they get less difficult relative to items built to other alignment references. (Yeah, the construct is multi-dimensional, as it always is in K-12 assessment.)
This is not merely an issue of surface features vs subtle differences between items. This has nothing to do with item features at all. This is about the other half of item difficulty determinants: instructional experiences.
Any predicted item difficulty is dependent upon some assumed constant curricular, instructional and pedagogical approach. The particular item might be fixed, but the test takers are not. Not only do all IDM efforts fail to account for this, but they cannot account for this because we do not track curriculum, instruction and pedagogy all the way down to the explanation, examples and practice that students are exposed to. Moreover, as Markle points out, we want instruction to change in response to high difficulty items.
So, if you want to do the work of trying to figure out the specific aspects of a stimulus or task that make items assessing the same alignment reference more or less difficult, go for it. That's a separate project for each standard at each grade—and likely even more than that if you are a fan of rPLDs. This could be invaluable work, but it is too deep in consideration of actual domain content and test taker cognition to be of interest to the dominant views in educational measurement.
But we should stop wasting our time on trying to predict item difficulty simply by looking at items assessing any content anyone is actually trying to teach. All we are doing is i) ignoring intended test use and ii) assuming particular instructional approaches used in particular sorts of districts. That is, entirely ignoring validity and inviting in a whole new swarm of fairness issues that most will not understand.