The reliability and validity of automated tools for examining variation in syntactic complexity across genres

• Published in: International journal of applied linguistics Vol. 28; no. 1; pp. 165 - 188
• Main Authors: Polio, Charlene, Yoon, Hyung‐Jo
• Format: Journal Article
• Language: English
• Published: Oxford Blackwell Publishing Ltd 01.03.2018
• Subjects: Automation; Coh‐Metrix; English as a second language learning; Genre; genre variation; L2 syntactic complexity analyzer; Narratives; Syntactic complexity; 体裁差异; 句法复杂度; 句法复杂度分析工具
• ISSN: 0802-6106; 1473-4192; 1473-4192
• DOI: 10.1111/ijal.12200

This study investigates two automated systems, the L2 syntactic complexity analyzer (SCA) and Coh‐Metrix, for their analysis of English syntactic complexity as a way to capture variation across two genres. The purpose of the paper is to partially replicate previous studies that have found complexity differences between narrative and argumentative essays while also evaluating two automated tools that can be used for analysing variation in other contexts. We first test the reliability of SCA and Coh‐Metrix by comparing their results with the hand‐coded results of 30 essays. We then examine the two systems' ability to detect genre effects on syntactic complexity using 162 essays (narrative and argumentative) written by 81 ESL students. Results indicate that the majority of measures from the two systems are reliable, but some measures were not transparent and consistent. Additionally, both SCA and Coh‐Metrix measures show higher syntactic complexity in the argumentative essays than the narratives, confirming both the findings of previous research and that both tools may be valid for studying variation in complexity. Findings are discussed in terms of the motivation for genre differences. 本研究通过二语句法复杂度分析工具SCA以及Coh‐Metrix考察不同英语体裁写作中的句法复杂度差异。本研究目的是验证研究以往研究关于记叙文和议论文在句法复杂度上的差异,同时评估SCA和Coh‐Metrix对其他语境下句法复杂度差异分析的有效性。我们采用SCA和Coh‐Metrix自动分析以及研究者人工分析30篇文章的句法复杂度测试SCA和Coh‐Mextix的信度,然后通过这两种自动分析工具考察81位英语学习者共162篇文章(记叙文和议论文)中体现的句法复杂度体裁差异。结果表明, SCA和 Coh‐Metrix的大多数测量指标信度较高,但某些测量指标清晰度和一致性较差。另外, SCA和 Coh‐Metrix测量指标都表明议论文句法复杂度高于记叙文,证实了以往研究关于记叙文和议论文句法复杂度的发现,并表明这两种自动分析工具都能够有效考察句法复杂度的变化。根据上述研究发现,本文进一步讨论了句法复杂度体裁差异产生的原因。