Examiner judgement of professional competence in exercise physiology

Abstract

Exercise Physiology courses have transitioned to competency based, forcing Universities to rethink curriculum design and assessment to ensure students are competent to practice. This thesis explored the reasoning behind clinical judgment, specifically, capturing the factors that contribute to assessor decision making about students’ competency. Aims: The aims were twofold. The first was to determine the sources of variance in the examination process. The second was to document the factors impacting on examiner judgment and ratings of student performance. Methodology: Examiner judgement was explored from both a quantitative and qualitative perspective. 23 examiners viewed three video encounters of student performance on an OSCE. Once rated, analysis of variance was performed to determine where the variance in ratings was attributed. A semi-structured interview was conducted to draw out the examiners reasoning behind their ratings. Results: At a global level, analysis of variance indicated that the individual examiner had a minimal impact on the variance, with the majority of variance explained by the student performance on the task. A closer examination of the three domains indicated slightly different outcomes. The assessment of procedural skill, mimicked the global assessment ratings. For communication competency, examiners had different expectations about what they expected to observe. For technical competency, the examiners and the University they came from contributed to the majority of variance. Using qualitative analysis, five themes were identified to help explain the variance in examiner judgement. These were variable frames of reference, the use of varying inferences, variable reaction to language, varying use of the patient experience as a reference and varying levels of confidence in performing the assessment. Discussion: The major outcome from this study was that despite the documented differences in the examiner focus and judgement pathways, the ratings of student performance was relatively consistent amongst the examiners, with the differences in ratings largely attributed to the student’s varied ability. This finding was unexpected, as the examiners had diverse backgrounds, roles and experience. The thought processes behind judgement were diverse and if the qualitative results had been used in isolation, may have led to the researchers drawing conclusions that the examined performances would have yielded widely different ratings. A strength of the current study was that by combining the use of quantitative and qualitative approaches, the researchers were able to gain a more comprehensive picture of the assessment rational and impact on the ratings. Conclusion: This research was the first to explore the role of the examiner in the assessment of entry level exercise physiology competence. The examiner role is pivotal in determining graduate level student competency and ultimately the calibre of entry level exercise physiologists. This study highlighted the differences in the examiners judgement pathways but confirmed that the differences only had a small impact on the student ratings and that it was in fact the students’ ability that accounted for the variance in rating.