수용성 판단 실험의 설계 및 통계 분석 처리의 몇 문제- 서열형·연속형 데이터를 중심으로 - (Some issues in designing and analyzing Korean ordinal and continuous acceptability judgment experiments)

서지정보

· 발행기관 : 한말연구학회
· 수록지 정보 : 한말연구 / 64권 / 42호 / 1 ~ 27페이지
· 저자명 : 조용준

초록

Acceptability judgment experiments, often conducted as surveys, offer the advantage of obtaining a large amount of data with minimal time, effort, or cost. However, conducting acceptability judgment experiments involves several considerations. First is the issue related to the sensitivity of result analysis, second is related to the identification of careless responses, and third pertains to the statistical analysis of result data. This study empirically investigates these issues, focusing on ordinal data and continuous data.
Sensitivity, represented by statistical power, was assessed across various effect sizes with sample sizes ranging from 5 to 100, each simulated with 1,000 resamplings. Similar to Sprouse & Almeida 2017, for the Likert scale(LS), it was found that a minimum of 9 participants is required for an extra-large effect size, 19 for a large effect size, and 51 for a medium effect size. For the magnitude estimation(ME), in the case of an extra-large effect size, at least 11 participants are needed, 25 for a large effect size, and 75 for a medium effect size. In both methods, for a small effect size, statistical power did not reach 80%.
Additionally, relying on crowd-sourced large-scale acceptability judgment experiments, 30 standard sentences (gold standard) for identifying careless responses were provided, consisting of 30 grammatical and 30 ungrammatical sentences. These could be useful for developing practice items and items for identifying careless responders. Moreover, 20 sentences are presented for designing practice items, for participants to grasp a possible range of of acceptability judgments. This is expected to be very helpful in designing acceptability judgment experiments.
Finally, we examined various possible statistical analyses for ordinal and continuous data. For ordinal data, there are methods such as standardization for statistical analysis or multivariate cumulative logit analysis using raw values. For continuous data, there are methods such as standardization or natural logarithm transformation for statistical analysis. As an example, for each of the four syntactic phenomena, 200 samples were randomly selected, and statistical analyses were compared. Overall, we suggest that standardizing ordinal data for statistical analysis is preferable.

영어초록

Acceptability judgment experiments, often conducted as surveys, offer the advantage of obtaining a large amount of data with minimal time, effort, or cost. However, conducting acceptability judgment experiments involves several considerations. First is the issue related to the sensitivity of result analysis, second is related to the identification of careless responses, and third pertains to the statistical analysis of result data. This study empirically investigates these issues, focusing on ordinal data and continuous data.
Sensitivity, represented by statistical power, was assessed across various effect sizes with sample sizes ranging from 5 to 100, each simulated with 1,000 resamplings. Similar to Sprouse & Almeida 2017, for the Likert scale(LS), it was found that a minimum of 9 participants is required for an extra-large effect size, 19 for a large effect size, and 51 for a medium effect size. For the magnitude estimation(ME), in the case of an extra-large effect size, at least 11 participants are needed, 25 for a large effect size, and 75 for a medium effect size. In both methods, for a small effect size, statistical power did not reach 80%.
Additionally, relying on crowd-sourced large-scale acceptability judgment experiments, 30 standard sentences (gold standard) for identifying careless responses were provided, consisting of 30 grammatical and 30 ungrammatical sentences. These could be useful for developing practice items and items for identifying careless responders. Moreover, 20 sentences are presented for designing practice items, for participants to grasp a possible range of of acceptability judgments. This is expected to be very helpful in designing acceptability judgment experiments.
Finally, we examined various possible statistical analyses for ordinal and continuous data. For ordinal data, there are methods such as standardization for statistical analysis or multivariate cumulative logit analysis using raw values. For continuous data, there are methods such as standardization or natural logarithm transformation for statistical analysis. As an example, for each of the four syntactic phenomena, 200 samples were randomly selected, and statistical analyses were compared. Overall, we suggest that standardizing ordinal data for statistical analysis is preferable.

참고자료

· 없음