Assessing the Reliability of Situational Judgment Tests Used in High-Stakes Situations

作者：Catano Victor M^*; Brochu Anne; Lamerson Cheryl D

来源：International Journal of Selection and Assessment, 2012, 20(3): 333-346.

DOI：10.1111/j.1468-2389.2012.00604.x

Assessing reliability of situational judgment tests (SJTs) in high-stakes situations is problematic with reliability inappropriately measured by Cronbach's alpha when test items are heterogeneous. We computed the corrected, weighted mean alpha from 56 alpha coefficients, which produced a value of a?=?.46 and reviewed appropriate types of reliability to use with SJTs. In the current longitudinal study, SJT testretest reliability was r?=?.82, compared with internal consistency, a?=?.46, and stratified alpha, a?=?.45 at Time 1 and a?=?.52 and stratified a?=?.51 at Time 2. We used a student sample (Time 1: n?=?185; Time 2: n?=?132) with items from a credentialing exam with should do instructions. The SJT correlated significantly with cognitive ability, r?=?.30, and agreeableness, r?=?.24. In Study 2, we assessed testretest reliability with Human Resource professionals (Time 1: n?=?94; Time 2: n?=?32) who had been recently credentialed and who participated in a pilot test of new SJT items with most likely/least likely do response options. The SJT testretest reliability was r?=?.66 compared with internal consistency, a?=?.43 and stratified a?=?.47 at Time 1 and a?=?.61 and stratified a?=?.67 at Time 2. We discuss the theoretical implications of the Study 1 results as well as the practical implications for use of SJTs in credentialing examinations.

• 出版日期2012-9