When a machine detects student reasoning: a review of machine learning-based formative assessment of mechanistic reasoning

• Published in: Chemistry Education Research and Practice Vol. 24; no. 2; pp. 47 - 427
• Main Authors: Martin, Paul P, Graulich, Nicole
• Format: Journal Article
• Language: English
• Published: Ioannina Royal Society of Chemistry 05.04.2023
• Subjects: Algorithms; Artificial Intelligence; Automation; Chemistry; Critical Thinking; Design; Education; Educational Assessment; Educational Research; Evaluation; Formative Evaluation; Heterogeneity; Logical Thinking; Machine learning; New technology; Reasoning; Research projects; Science Education; Science Process Skills; Scoring Rubrics; Student Evaluation
• ISSN: 1109-4028; 1756-1108
• DOI: 10.1039/d2rp00287f

In chemistry, reasoning about the underlying mechanisms of observed phenomena lies at the core of scientific practices. The process of uncovering, analyzing, and interpreting mechanisms for explanations and predictions requires a specific kind of reasoning: mechanistic reasoning. Several frameworks have already been developed that capture the aspects of mechanistic reasoning to support its formative assessment. However, evaluating mechanistic reasoning in students' open responses is a time- and resource-intense, complex, and challenging task when performed by hand. Emerging technologies like machine learning (ML) can automate and advance the formative assessment of mechanistic reasoning. Due to its usefulness, ML has already been applied to assess mechanistic reasoning in several research projects. This review focuses on 20 studies dealing with ML in chemistry education research capturing mechanistic reasoning. We developed a six-category framework based on the evidence-centered design (ECD) approach to evaluate these studies in terms of pedagogical purpose , rubric design , construct assessment , validation approaches , prompt structure , and sample heterogeneity . Contemporary effective practices of ML-based formative assessment of mechanistic reasoning in chemistry education are emphasized to guide future projects by these practices and to overcome challenges. Ultimately, we conclude that ML has advanced replicating, automating, and scaling human scoring, while it has not yet transformed the quality of evidence drawn from formative assessments.