ChatGPT와 코퍼스 기반의 대학수학능력시험 모의평가 영어영역 실제 정답률 예측 가능성 모색 (Predicting the difficulty of college scholastic ability test based on ChatGPT and corpus linguistic features)

서지정보

· 발행기관 : 한국멀티미디어언어교육학회
· 수록지 정보 : 멀티미디어 언어교육 / 26권 / 4호 / 29 ~ 50페이지
· 저자명 : 이명원, 이예나, 최희, 최인철

초록

This study attempts to validate ChatGPT, as well as corpus linguistic features, in predicting the difficulty level of the College Scholastic Ability Test(CSAT). The Observed Proportion Correct (Pc) of the 2021 to 2024 CSAT mock tests was compared with those provided by ChatGPT-3.5, ChatGPT-4.0 and 10 corpus-based linguistic features, including Type, Token, TTR, AWL, Lexical Density, Average Words per Sentence, Flesch Reading Ease, Flesch-Kincaid Grade Level, and Gunning-Fog Index. The research findings are as follows: First, ChatGPT-4.0 bettter predicted item difficulty compared to ChatGPT-3.5.
Second, the Predicted Proportion Correct (Pc) of ChatGPT-3.5, ChatGPT-4.0 and some corpus features showed a high correlation with observed Pc. The observed Pc was largely accounted for by the difficulty predicted by ChatGPT-3.5 and ChatGPT-4.0, where ChatGPT-4.0’s predicted Pc demonstrated higher explanatory power than ChatGPT-3.5. Third, a regression model for predicting the difficulty of CSAT was established with ChatGPT and such corpus-based variables as the Gunning-Fog Index and Lexical Diversity. Overall, the findings endorse the potential use of ChatGPT, along with salient corpus linguistic features, as a suitable tool for predicting item difficulties prior to actual test administration.

영어초록

This study attempts to validate ChatGPT, as well as corpus linguistic features, in predicting the difficulty level of the College Scholastic Ability Test(CSAT). The Observed Proportion Correct (Pc) of the 2021 to 2024 CSAT mock tests was compared with those provided by ChatGPT-3.5, ChatGPT-4.0 and 10 corpus-based linguistic features, including Type, Token, TTR, AWL, Lexical Density, Average Words per Sentence, Flesch Reading Ease, Flesch-Kincaid Grade Level, and Gunning-Fog Index. The research findings are as follows: First, ChatGPT-4.0 bettter predicted item difficulty compared to ChatGPT-3.5.
Second, the Predicted Proportion Correct (Pc) of ChatGPT-3.5, ChatGPT-4.0 and some corpus features showed a high correlation with observed Pc. The observed Pc was largely accounted for by the difficulty predicted by ChatGPT-3.5 and ChatGPT-4.0, where ChatGPT-4.0’s predicted Pc demonstrated higher explanatory power than ChatGPT-3.5. Third, a regression model for predicting the difficulty of CSAT was established with ChatGPT and such corpus-based variables as the Gunning-Fog Index and Lexical Diversity. Overall, the findings endorse the potential use of ChatGPT, along with salient corpus linguistic features, as a suitable tool for predicting item difficulties prior to actual test administration.

참고자료

· 없음