Comparing the quality of human and ChatGPT feedback of students’ writing

• Published in: Learning and instruction Vol. 91; p. 101894
• Main Authors: Steiss, Jacob, Tate, Tamara, Graham, Steve, Cruz, Jazmin, Hebert, Michael, Wang, Jiali, Moon, Youngsun, Tseng, Waverly, Warschauer, Mark, Olson, Carol Booth
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.06.2024
• Subjects: Automated writing evaluation; Formative feedback; Writing instruction; Writing instruction; Formative feedback; Automated writing evaluation
• ISSN: 0959-4752
• DOI: 10.1016/j.learninstruc.2024.101894

Offering students formative feedback on their writing is an effective way to facilitate writing development. Recent advances in AI (i.e., ChatGPT) may function as an automated writing evaluation tool, increasing the amount of feedback students receive and diminishing the burden on teachers to provide frequent feedback to large classes. We examined the ability of generative AI (ChatGPT) to provide formative feedback. We compared the quality of human and AI feedback by scoring the feedback each provided on secondary student essays. We scored the degree to which feedback (a) was criteria-based, (b) provided clear directions for improvement, (c) was accurate, (d) prioritized essential features, and (e) used a supportive tone. 200 pieces of human-generated formative feedback and 200 pieces of AI-generated formative feedback for the same essays. We examined whether ChatGPT and human feedback differed in quality for the whole sample, for compositions that differed in overall quality, and for native English speakers and English learners by comparing descriptive statistics and effect sizes. Human raters were better at providing high-quality feedback to students in all categories other than criteria-based. AI and humans showed differences in feedback quality based on essay quality. Feedback did not vary by language status for humans or AI. Well-trained evaluators provided higher quality feedback than ChatGPT. Considering the ease of generating feedback through ChatGPT and its overall quality, generative AI may still be useful in some contexts, particularly in formative early drafts or instances where a well-trained educator is unavailable. •Human feedback was better than ChatGPT feedback for ⅘ elements of formative feedback.•Differences between ChatGPT and humans were modest when considering the overall quality of feedback and time-savings•ChatGPT and human feedback varied in quality based on the score of the targeted essay.•ChatGPT has potential as an evaluative tool given tradeoffs between quality and time.