Rigorous Test Development

Valid item elicit evidence of the targeted cognition for the range of typical test takers.

Tests are invaluable for teachers, school leaders, district leaders, policy-makers, the public, parents and even students -- so long as they (and the inferences mades from the data they produce) are valid.

"Validity is the most important consideration in test evaluation." (1985 Standards for Psychological and Educational Testing)
"Validity is, therefore, the most fundamental consideration in developing and evaluating tests." (1999 Standards for Psychological and Educational Testing)
"Validity is, therefore, the most fundamental consideration in developing and evaluating tests." (2014 Standards for Psychological and Educational Testing)

Standardized the role of standardized testing in our schools has been on the rise for decades, and even 2015 ESSA (Every Student Succeeds Act) has been limited in diminishing that role. However, the tests that we use are of questionable validity, to much of the public and to many educators. For all the criticism that psyshometricians and assessment industry professionals may make of teacher-made tests, there have long been problems with standardized tests which render the usefulness questionable. Historically, the assessment industry has focused on maximizing reliability, with far less attention to validity.

AleDev Research & Consulting is the home of the RTD (Rigorous Test Development) project – an attempt to build a professionalized content development practice that focuses on individual item quality, particularly by leaning into the importance of validity throughout the content development process. It assumes that content development professionals develop professional judgment that can be raised, honed and calibrated by providing frameworks and clarifying expectations in ways that account for the constraints and demands of typical practice within test development, today. RTD is a conscious and deliberate attempt to respond to the disparity in status, training and shared knowledgebases between psychometrically oriented professionals and content development professionals.

At AleDev, we build protocols, approaches, guides and techniques for developing standardized tests with an emphasis on validity, above all else, because greater validity mean greater educational value and better data to inform policy-makers’ decisions. Through the RTD project, we are trying to help create better tests so that they can be useful both to those with who must take in a wide view of the progress of hundreds, thousands or millions of students and are those who are focus on individual students and the content they are trying to master.