Assessment
This unit focuses on competencies the Clinician Educator (CE) will need to develop and implement a valid system of assessment. These skills are essential in the development of new curricula and in the management of existing curricula.
The Foundations Unit is a prerequisite for this unit.
Curriculum is a co-requisite.
Competencies Addressed in this Unit
A CE must be able to
Apply principles of assessment in developing a system of assessment for a program
Select assessment instruments that are relevant and appropriate for the learning context
Describe the challenges of self-assessment
Key Documents
Objectives
By the end of this Unit, the CE will be able to:
1. Demonstrate comprehension of the principles of assessment, by
recognizing the multiple purposes of assessment in a system of education
analyzing how a system of assessment fits within the cycle of curricular design
describing the differences between summative and formative assessment
applying the concept of “utility” to individual assessment instruments and methods
identifying and describing the professional responsibilities of educators to ensure that the assessment system meets the needs of trainees and the public
aligning types of assessment with the corresponding levels of Miller’s pyramid
comparing and contrasting the terms “assessment” and “evaluation”
justifying the importance of both self-directed assessment and reflection in the development and maintenance of competence of a clinician
describing the enablers of and barriers to self-assessment
2. Conduct a survey of important assessment instruments, by
comparing and contrasting the following methods or instruments of assessment, with an emphasis on their structure, the types of objectives they can assess, the context for which they are best suited (e.g., clinical setting, classroom environment, etc.), and their advantages and disadvantages:
written exams (constructed response and selective response)
oral examinations
OSCEs and their derivatives (e.g., OSATs)
ITERs and other rating-scale–based expert assessment forms
instruments for direct observation, such as Daily Encounter Cards, CEX
Multi-Source Feedback
chart audits and practice assessments
portfolios
other
selecting and implementing appropriate assessment instruments within a curriculum
3. Demonstrate comprehension of the principles of psychometrics, by
describing modern validity theory and the concept of reliability and applying these to the evaluation of assessment strategies
applying the concept of utility and analyzing the implications of the assessment system in the context of the candidate’s jurisdiction
comparing and contrasting classical test theory and generalizability theory
defining principles of standard setting and selecting appropriate methods for a given assessment instrument and context
applying the principles of qualitative research in the context of judging the quality of a portfolio as a composite assessment method (using principles of credibility, consistency, triangulation, transferability, etc.)
4. Demonstrating competence in remediation, by
applying a remediation framework to an encounter the candidate has had with a learner in difficulty, while emphasizing a step-wise approach
Assessment
Please refer to the McMaster Task List for Assessment Unit for a complete listing of the summative and formative assessments required for this unit.
Primary Review Form (To be filled by Unit Supervisor)
Suggested Readings
Formative assessment
Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77(1):81–112.
van de Ridder JM, Stokking KM, McGaghie WC, ten Cate OT. What is feedback in clinical education? Med Educ. 2008;42(2):189–97.
Introductory psychometrics
Downing SM, Yudkowsky R. Assessment in the health professions education. New York: Routledge; 2009.
Govaerts MJB, van de Wiel MWJ, Schuwirth LWT, van der Vleuten CPM, Muijtjens AMM. Workplace-based assessment: raters’ performance theories and constructs. Adv Health Sci Educ Theory Pract. 2012 May 17.
Kane MT. An argument-based approach to validity. Psychol Bull. 1992;112(3):527–35.
Messick S. Validity. In: Linn Rl, editor. Educational Measurement. 3rd ed. American Council on Education Series on Higher Education. Phoenix (AZ): Oryx Press; 1989.
Schuwirth LW, van der Vleuten CP. General overview of the theories used in assessment: AMEE Guide No. 57. Med Teach. 2011;33(10):783–97.
Principles of assessment
Harden RM. Ten questions to ask when planning a course or curriculum. Med Educ. 1986;20(4):356–65.
Holmboe ES, Hawkins RE. Practical guide to the evaluation of clinical competence. Philadelphia (PA): Mosby; 2008.
Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9):S62–67.
Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 2011;33(3):206–14.
Schuwirth LW, van der Vleuten CP. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 2011;33(6):478–85.
Remediation
Hauer KE, Ciccone A, Henzel TR, Katsufrakis P, Miller SH, Norcross WA, et al. Remediation of the deficiencies of physicians across the continuum from medical school to practice: a thematic review of the literature. Acad Med. 2009;84(12):1822–32.
Zamost S, Griffen D, Ansari A. Exclusive: Doctors cheated on exams. CNN Health. Available at: www.cnn.com/2012/01/13/health/prescription-for-cheating/index.html
Self‐assessment
Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296(9):1094–1102.
Eva KW, Regehr G. Rater-based assessments as social judgments: rethinking the etiology of rater errors. Acad Med. 2005;80(10 Suppl):S46–54.
Sargeant J, Armson H, Chesluk B, Dornan T, Eva K, Holmboe E, et al. The processes and dimensions of informed self-assessment: a conceptual model. Acad Med. 2010;85(7):1212–20.
Standard setting
Turnbull J. What is … normative versus criterion-referenced assessment? Med Teach. 1989;11(2):145–50.
Survey of important assessment instruments
Bandiera G, Sherbino J, Frank JF. The CanMEDS assessment tools handbook: an introductory guide to assessment methods for the CanMEDS competencies. Ottawa (ON): The Royal College of Physicians and Surgeons of Canada; 2006.
Gagliardi AR, Brouwers MC, Finelli A, Campbell CM, Marlow BA, Silver IL. Physician self-audit: a scoping review. J Cont Ed Health Prof. 2011;31(4):258–64.
Kogan JR, Conforti L, Bernabeo E, Iobst W, Holmboe E. Opening the black box of clinical skills assessment via observation: a conceptual model. Med Educ. 2011;45(1):1048–60.
Kogan JR, Holmboe ES, Hauer KR. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009;302(12):1316–26.
Tochel C, Haig A, Hesketh A, Cadzow A, Beggs K, Colthart I, et al. The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No. 12. Med Teach. 2009; 31(4):299–318.
Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7. Med Teach. 2006;28(2):117–28.