Treffer: The Role of Expertise in Scoring Situational Judgment Tests: Is the Juice Worth the Squeeze?

Drawing from theory regarding the general and domain‐specific knowledge typically measured by situational judgment tests (SJTs), this study examined the effects of subject matter experts (SME) qualifications on SJT scoring keys and their associated predictive validity. Although one may expect that scoring keys generated using the judgments of more qualified SMEs would result in higher predictive validity, an exploration of any gains in incremental validity associated with the keys is warranted to determine whether gains are meaningful, and perhaps more importantly, worth the additional resources required to obtain highly qualified SMEs. We created three distinct SJT scoring keys for an SJT designed to measure cross‐cultural competence based on a sample of crowdsourced novices, a sample of incumbents, and a sample of highly trained and experienced SMEs and examined their predictive validity. Findings from a time‐lagged, year‐long concurrent validity study (N = 350) provide some support for the idea that using particularly qualified, highly experienced SMEs to develop SJT scoring keys may provide a meaningful increase in the predictive validity of the assessment over using a crowdsourced novice sample or a convenience sample of incumbents that are often used in practice. Summary: Situational judgment tests (SJT) scoring keys are usually built using incumbent or novice judgments.Whether highly experienced subject matter experts (SMEs) outperform the usual incumbent‐derived scoring keys for SJTs has not yet been explored.Cost–benefit of using highly experienced and qualified SMEs in selection is rarely quantified.Experienced SMEs outperform incumbents and crowdsourced novices in keying.Expert keys add incremental validity beyond incumbent‐derived keys.Cost‐benefit analysis suggests expert SME investment may pay off in high‐stakes contexts.Investing in expert SMEs may offset costs by reducing expensive errors in high stakes settings. [ABSTRACT FROM AUTHOR]

Copyright of International Journal of Selection & Assessment is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)