AI-Automated EFL Writing Assessment Using Standardized Rubrics

• Published in: 영어교육연구 Vol. 37; no. 1; pp. 21 - 38
• Main Authors: Damian Heywood, Joseph Carrier, Kyu-Hong Hwang
• Format: Journal Article
• Language: English; Korean
• Published: 팬코리아영어교육학회(구 영남영어교육학회) 31.03.2025; Pan-Korea English Teachers Association; 팬코리아영어교육학회
• Subjects: 영어와문학; human assessment reliability; EFL composition; AI-automated assessment; writing assessment; standardized rubrics; assessment correlation; Computer-Assisted Language Learning (CALL); Large Language Models (LLMs)
• ISSN: 1226-6566; 2671-9460

This study investigates the reliability of AI-automated writing assessment in university-level EFL composition classes. The research compares human assessment and AI-generated assessment of 32 paragraphs written by Korean university students, using a standardized rubric adapted from the University of Michigan Writing Center. The study utilized Claude 3.5 Sonnet, accessed via API calls, to conduct multiple assessment iterations under controlled conditions. The assessment criteria included Content and Ideas, Organization and Structure, Language Use and Vocabulary, and Grammar and Mechanics, each scored on a five-point scale. Statistical analysis revealed high consistency in AI assessment (r=0.979). While correlation between human raters was moderate (r = 0.507), the AI system demonstrated reliable assessment patterns that aligned significantly with human scoring trends. The findings suggest that AI-automated assessment systems can provide reliable evaluation of student writing, though with some notable differences from human assessment patterns. This study contributes to the emerging field of AI-assisted writing assessment and provides a foundation for developing more sophisticated automated feedback systems for EFL writing instruction. KCI Citation Count: 0