A Case Study of AI Agent–Assisted Correction of Intermediate-Level Chinese Writing

บทคัดย่อ

This study aimed to investigate the effectiveness and applicability of an Artificial Intelligence (AI) Agent in assessing intermediate-level Chinese writing produced by learners of Chinese as a second language (L2). The research addressed two primary objectives: first, to examine the degree of consistency between AI Agent scoring and manual scoring conducted by professional teachers, and second, to identify advantages and limitations of using an AI Agent for essay correction in international Chinese language education. The research population consisted of international students in Chinese language programs at Central China Normal University (CCNU). Purposive sampling was employed to collect 31 final examination essays from students taking the Intermediate Chinese Writing II course, comprising 16 essays from the first semester and 15 from the second. All participants passed the HSK Level 4 and represented diverse cultural and linguistic backgrounds. The primary instrument was an AI Agent named “Essay Grading Assistant,” developed on the ByteDance Coze platform using the DeepSeek-V3.2 large language model. This system integrated Optical Character Recognition (OCR), Chinese word segmentation, and document analysis plugins to process handwritten examination scripts. Additionally, two university instructors with over five years of experience independently evaluated the essays using the same analytical scoring rubric.  

Data analysis involved descriptive statistics to examine score distributions, Spearman’s rank correlation coefficient to determine the relationship between AI-generated scores and human ratings, and the Intraclass Correlation Coefficient (ICC) using a two-way random effects model with absolute agreement to evaluate inter-rater consistency. The findings revealed a very strong positive correlation between the AI Agent scores and the teachers’ scores, with correlation coefficients of 0.917 and 0.902, respectively (p < .001). The ICC value reached 0.964, indicating excellent agreement. The AI Agent significantly improved grading efficiency, requiring approximately 15 seconds per essay, while enhancing scoring objectivity and consistency in evaluating surface-level linguistic features. However, several limitations were identified, particularly in deep semantic interpretation, pragmatic understanding, and the recognition of unconventional learner expressions. The study concluded that AI Agents possess strong potential as supportive tools for international Chinese writing assessment; nevertheless, a human–AI collaborative scoring approach remains the most appropriate model for ensuring both efficiency and pedagogical quality in second-language writing evaluation.