Effectiveness of automated writing evaluation systems in school settings: A systematic review of studies from 2000 to 2020

• Published in: Journal of computer assisted learning Vol. 38; no. 2; pp. 599 - 620
• Main Authors: Nunes, Andreia, Cordeiro, Carolina, Limpo, Teresa, Castro, São Luís
• Format: Journal Article
• Language: English
• Published: Chichester, UK John Wiley & Sons, Inc 01.04.2022; Wiley; Wiley Subscription Services, Inc
• Subjects: automated evaluation systems; Automation; Control Groups; Education; Elementary School Students; Embedding; Evaluation; Feedback; Instructional Effectiveness; Learning; Learning Processes; Program Evaluation; Secondary School Students; Student writing; Students; System effectiveness; Systematic review; Systems analysis; Teachers; Teaching Methods; writing; Writing Evaluation; Writing Instruction
• ISSN: 0266-4909; 1365-2729; 1365-2729
• DOI: 10.1111/jcal.12635

Background Automated Writing Evaluation (AWE) systems to aid writing learning and instruction in primary and secondary education are growing increasingly popular. However, their effectiveness is hardly known. We conducted a systematic review focusing on the effects of these systems providing writing feedback to students in school settings. Objectives Our goal was to identify and characterize AWE systems tested in the last 20 years for Grades 1–12 and examine their impact on text quality and other writing‐related outcomes. Methods The review followed PRISMA guidelines. We identified eight studies reporting the effects on writing of six AWE systems on 1659 students 11–17 years of age. Results and conclusions Our review supported the usefulness of AWE systems for writing learning and instruction. Except for one, all studies showed a positive effect of automated feedback in at least one writing‐related measure. The integration of AWE systems into more extensive instructional programs, the amount of writing practice provided to students, the type of the control groups, and the role of teachers are factors influencing their impact on students' writing outcomes. Relevance Our review generally supported the value of AWE systems in the teaching/learning process of writing. A closer look into the conditions in which AWE systems are put to practice suggested that they are particularly effective when embedded into comprehensive instructional programs providing ample writing opportunities. Findings from this review expand knowledge on AWE systems as valuable tools to enhance writing in school settings. Lay Description What is already known about this topic The use of automated writing evaluation (AWE) systems is increasingly popular in education. Little is known about their effectiveness in school settings. What this paper adds This paper reviews studies reporting the effectiveness of AWE systems to enhance 11‐ to 17‐year‐old students' writing. The main characteristics of the AWE systems tested in the last 20 years for school settings, and their impact on writing, are identified and discussed. Implications for practice and/or policy Embedding AWE systems into more extensive instructional programs benefits students' writing more than using the systems in isolation. Providing students with ample opportunity to practice writing seems related to better text quality. The combination of teacher and system feedback seems to positively impact students' writing outcomes. Teachers' and students' perceived impact of use may be a critical factor.