An investigation of construct relevant and irrelevant features of mathematics problem-solving questions using comparative judgement and Kelly's Repertory Grid

• Published in: Research in mathematics education Vol. 19; no. 2; pp. 112 - 129
• Main Authors: Holmes, Stephen D., He, Qingping, Meadows, Michelle
• Format: Journal Article
• Language: English
• Published: Abingdon Routledge 04.05.2017; Taylor & Francis Ltd
• Subjects: Construct Validity; Evaluation Methods; Foreign Countries; Item Response Theory; Mathematical analysis; Mathematical Logic; mathematical problem-solving; Mathematics; Mathematics Education; Mathematics Skills; Mathematics Teachers; Mathematics Tests; Problem Solving; Scores; Secondary School Students; summative assessment; Teachers; Test Items; Test scores; validity; United Kingdom (England)
• ISSN: 1479-4802; 1754-0178
• DOI: 10.1080/14794802.2017.1334576

The relationship between the characteristics of 33 mathematical problem-solving questions answered by 16-year-old students in England and the quality of problem-solving elicited was investigated in two studies. The first study used comparative judgement (CJ) to estimate the quality of the problem-solving elicited by each question, involving 33 mathematics teachers judging pairs of journal-style responses to the questions and the application of the Bradley-Terry model. In the second study a variant of Kelly's Repertory Grid was used with five mathematics teachers to identify 23 dimensions along which the problem-solving questions varied. Significant relationships between ratings on some dimensions and the problem-solving quality estimated in the first study were found. This suggests that the Kelly's Repertory Grid approach could be an effective way to identify features of questions that are relevant to the construct being assessed and features that could be potential sources of construct-irrelevant variance in test scores.