The Use of a Diagnostic Competence Model About Children’s Operation Sense for Criterion-Referenced Individual Feedback in a Large-Scale Formative Assessment

• Published in: Journal of psychoeducational assessment Vol. 38; no. 4; pp. 426 - 444
• Main Authors: Schulz, Andreas, Leuders, Timo, Rangel, Ulrike
• Format: Journal Article
• Language: English
• Published: Los Angeles, CA SAGE Publications 01.07.2020
• Subjects: Children; Cognitive Measurement; Cognitive Processes; Criterion Referenced Tests; Diagnostic Tests; Elementary School Students; Feedback (Response); Foreign Countries; Formative Evaluation; Grade 4; Item Response Theory; Mathematics Tests; Number Concepts; Test Construction; Test Items; Test Validity; Germany; criterion-referenced feedback; cognitive diagnostic model; large-scale assessment; mathematics; disciplines and subjects; operation sense; validation
• ISSN: 0734-2829; 1557-5144
• DOI: 10.1177/0734282918823590

We provide evidence of validity for a newly developed diagnostic competence model of operation sense, by both (a) describing the theoretically substantiated development of the competence model in close association with its use within a large-scale formative assessment and (b) providing empirical evidence for the theoretically described cognitive levels of competences. The competence model describes students’ operation sense on four distinct levels. On each level, the model elaborates on the characteristics of tasks that students on this level are able to answer correctly. Moreover, the model explains this by referring to two kinds of cognitive processes that are supposed to be necessary to respond to these kinds of tasks successfully. In a validation study, about 85% of the variance in the item difficulties was explained by the four, a priori allocated, levels of operation sense. We discuss the relevance of the validation of the diagnostic competence model for the provision of criterion-referenced feedback in a large-scale formative assessment, including suggestions for teachers’ subsequent support activities, and the contributions of the model to the state of research about operation sense.