ChatGPT로 음운론 시험 문제 풀기: 특성, 오류, 추론 능력 분석 (ChatGPT takes on a phonology exam: Analyzing its characteristics, errors, and reasoning ability)

서지정보

· 발행기관 : 한국언어정보학회
· 수록지 정보 : 언어와 정보 / 27권 / 2호 / 73 ~ 115페이지
· 저자명 : 오은진, 박선우, 조혜선, 송상헌

초록

This study aimed to test ChatGPT’s ability to answer linguistics (phonology) exam questions by comparing its responses to those of two graduate students in linguistics. GPT-3.5 and GPT-4 versions were employed. ChatGPT performed similarly to students on true-false and multiple-choice questions but scored higher on conceptual questions. Although ChatGPT explained phonological concepts well, it performed poorly in problem-solving questions requiring inference using such concepts. In particular, ChatGPT correctly analyzed questions composed from previously distributed data but did not conduct any meaningful analyses based on the given data for questions modifying the phonetic environments of phonological processes or phonemic compositions in words. Overall, while the students wrote brief answers covering the key content, ChatGPT tended to provide answers with general content or described content that was not directly related to the problems. Therefore, ChatGPT’s answers could be easily distinguished from those of the students. In post-hoc experiments using “step-by-step” and “bullying,” GPT-4 outperformed GPT-3.5. Specifically, GPT-4 derived accurate phonemic analysis results from a small number of “bullying” questions for the problem modifying the phonetic environment. Thus, GPT-4’s reasoning ability in analyzing new data was superior to that of GPT-3.5.

영어초록

This study aimed to test ChatGPT’s ability to answer linguistics (phonology) exam questions by comparing its responses to those of two graduate students in linguistics. GPT-3.5 and GPT-4 versions were employed. ChatGPT performed similarly to students on true-false and multiple-choice questions but scored higher on conceptual questions. Although ChatGPT explained phonological concepts well, it performed poorly in problem-solving questions requiring inference using such concepts. In particular, ChatGPT correctly analyzed questions composed from previously distributed data but did not conduct any meaningful analyses based on the given data for questions modifying the phonetic environments of phonological processes or phonemic compositions in words. Overall, while the students wrote brief answers covering the key content, ChatGPT tended to provide answers with general content or described content that was not directly related to the problems. Therefore, ChatGPT’s answers could be easily distinguished from those of the students. In post-hoc experiments using “step-by-step” and “bullying,” GPT-4 outperformed GPT-3.5. Specifically, GPT-4 derived accurate phonemic analysis results from a small number of “bullying” questions for the problem modifying the phonetic environment. Thus, GPT-4’s reasoning ability in analyzing new data was superior to that of GPT-3.5.

참고자료

· 없음