Meta-Analysis of Inter-Rater Agreement and Discrepancy Between Human and Automated English Essay Scoring
(주)코리아스칼라
- 최초 등록일
- 2023.09.25
- 최종 저작일
- 2023.09
- 20페이지/ 어도비 PDF
- 가격 5,500원
* 본 문서는 배포용으로 복사 및 편집이 불가합니다.
서지정보
ㆍ발행기관 : 한국영어교육학회
ㆍ수록지정보 : 영어교육 / 78권 / 3호
ㆍ저자명 : Jiyeo Yun
목차
1. INTRODUCTION
2. REVIEW OF LITERATURE
2.1. Automated Scoring Systems
2.2. Inter-Rater Agreement and discrepancy
2.3. Meta-Analysis Models
3. METHODOLOGY
3.1. Data Gathering
3.2. Data Cleansing
3.3. Data Analysis
4. RESULTS
4.1. Description of Research Sources
4.2. Heterogeneity Tests
4.3. Overall Means of the Correlation and d
4.4. Consistency Across Studies
4.5. Influence of the Numbers of Point on the Scale
5. DISSCUSSION AND CONCLUSION
영어 초록
Studies on automatic scoring systems in writing assessments have also evaluated the relationship between human and machine scores for the reliability of automated essay scoring systems. This study investigated the magnitudes of indices for inter-rater agreement and discrepancy, especially regarding human and machine scoring, in writing assessment. The mean of the overall population correlation between automated and human scoring in essay writing was .78. The overall common d effect size was 0.001. Results from this meta-analysis indicated a strong relationship with no discrepancies between automated and human scoring. Both the I2 and Q values suggested that the population correlation values studied seemed to be heterogeneous, in contrast to homogenous d effect sizes. Therefore, it is necessary to investigate the sources of the between-studies variations for r correlations. Practical implications for ways of reporting results of automatic-scoring systems research and limitations of the study are also discussed.
참고 자료
없음