A Comparison of EFL Raters' Essay-Rating Processes Across Two Types of Rating Scales

作者:Li Hang; He Lianzhen*
来源:Language Assessment Quarterly, 2015, 12(2): 178-212.
DOI:10.1080/15434303.2015.1011738

摘要

This study used think-aloud protocols to compare essay-rating processes across holistic and analytic rating scales in the context of China's College English Test Band 6 (CET-6). A group of 9 experienced CET-6 raters scored the same batch of 10 CET-6 essays produced in an operational CET-6 administration twice, using both the CET-6 holistic essay-rating scale and an analytic rating scale designed specifically for the study, while providing think-aloud protocols. The findings indicated that important differences in both rating strategies adopted and the specific aspects of essays attended to existed across the scales. Specifically, raters were found to use more self-monitoring-interpretation strategies, the strategy of considering local language features, as well as certain self-monitoring-judgment strategies when rating holistically. They were found to use more frequently self-monitoring-judgment strategy, error-classifying strategy, and quality-assessing strategy when applying the analytic scale. As for the differences in text focus across the scales, it was found that the use of the holistic scale led to raters paying more attention to the general quality of language use and non-scale-related language features. Meanwhile, the application of the analytic scale resulted in raters giving more attention to coherence and grammar.