Relationships among Different Effect-Size Indexes for Inter-Rater Agreement between Human and Automated Essay Scoring

서지정보

· 발행기관 : 학습자중심교과교육학회
· 수록지 정보 : 학습자중심교과교육연구 / 23권 / 18호 / 901 ~ 919페이지
· 저자명 : 윤지여

초록

Objectives The purpose of this study is to investigate the magnitudes of and relationships among different effect-size indexes for inter-rater agreement between human and machine scoring in writing assessments.
Methods The procedure of meta-analyses consists of data gathering, including literature search with criteria for inclusion and exclusion, and data analysis, including data cleaning and coding, after tests of heterogeneity for each index, hierarchical weighted models, and moderator and contrast analyses were conducted. Appropriate analyses were conducted using rma, robust, and robu functions in the metafor and robumeta packages in R software Version 3.3.2.
Results Based on the results, the overall random-effects means for correlations, kappa values, and adjacent proportions of agreement between automated and human scoring of essay writing were .75, .48, and .99, respectively.
The results from hierarchical weighed models and heterogeneity tests indicate that the rates of agreement for each index were inconsistent across studies. According to moderator and contrast analyses, correlations and kappa values using 6-point scales significantly differed from those using 3-, 4-, and 5-point scales, respectively, at alpha level .05. On the other hand, the adjacent proportions of agreement on either 3- or 4-point scales significantly differed from the adjacent proportions of agreement on the 5- and 6-point scales, respectively, at alpha level .01. Regarding the exact and adjacent proportions of agreement, the average of discrepancies was 0.34 units, and the variance of discrepancies between exact and adjacent proportions of agreement was 0.004.
Similarly, the mean of discrepancies between the correlation and kappa was 0.27, and the variance of discrepancies between the correlation and kappa was 0.003.
Conclusions According to this finding, machine scoring is similar to human scoring in terms of relative consistency and absolute consensus. Compared to the evaluation criteria suggested by prior studies, the degrees of inter-rater agreement seen in this study were above the thresholds for correlations, moderate agreement for kappa, and in the range of consensus rates for adjacent proportion agreement. The rates of agreement for each index were inconsistent across studies. This means that the all agreement indexes had relatively large between-studies differences compared to the between-effects differences within the studies. It is necessary to investigate if some moderators explain the between-studies differences. The number of score-scale points used for measuring inter-rater agreement between human and machine scoring influenced the agreement rates. The relationships across the four indexes (i.e.,  , , , and ) from the study appear to be reasonably strong and linear. Regardless of the number of points on the score scales, kappa values are 0.27 points lower than correlations. In addition, the mean exact proportions of agreements is 0.34 points lower than the mean adjacent proportions of agreements. Since each inter-rater agreement index has its own disadvantages, such as scale dependency, not showing the degree of identical matching and matching patterns, and so on, it is advised to report several inter-rater agreement indexes.

참고자료

· 없음