Angoff Standard Setting and a Reformed Curriculum: Were the Standards Credible and Responsive?

作者：Mohammad Ahmed M; Al Ozaire Ebaa; Ayed Adel Khader; Ware James^*

来源：Kuwait Medical Journal, 2010, 42(2): 106-111.

• assessment
• multiple-choice questions and option numbers

Objectives: Determine the quality of eleven knowledge based multiple-choice examinations (A-J) and compare pass-fail rates before and after standard setting as a criterion of credibility; intra-class correlation coefficients (ICC) for inter-rater reliability and correlations between mean test scores and standards set for all individual items as a measure of validity. Determine the effect of standard setting an exam 121 days after it had been previously standard set and two options removed using item analysis.
Design: Cohort study
Setting: In-course examinations delivered to 85-111 medical and dental students in one medical school.
Intervention: Modified method of Angoff standard setting multiple-choice examinations
Main outcome measures: KR-20 coefficients of reliability, mean test scores, point biserial coefficients, standards set and item score correlation, intra-class correlations and pass-fail rates
Results:All the examinations were of high quality according to the criteria used. The pass-fail rates of exams A and B not standard set were 56.4% and 51.8% which then fell to a range 6% - 29% (C-J) after standard setting. The ICC was 0.69 + 0.12, indicating good inter-rater reliability while the MTS item Diff correlations were low. The theoretical exam confirmed an acceptable standard set in terms of a statistically insignificant change in pass-fail rates.
Conclusions: These results support the use of standard setting as being credible and responsive, but only a moderate level of validity was achieved.

• 出版日期2010-6