12.   Standard setting in OSCE


Knowing is not enough; we must apply!

Goethe

Contents:

Homepage - OSCE and OSPE

Compiled Responses

 The Beginning

 01.   Introduction and Basics of OSPE/OSCE

 02.   Is OSPE glorified spotting.. 

  03.   How to set up OSCE and OSPE stations..

 04.   How to collect the answer-sheets in OSCE/OSPE from students.

 05.   OSCE reliability

 06.   Levels of competencies in evaluation

 07.   Practical experiences of OSPE/OSCE

 08.   OSCE in testing Communication and attitudes

  09   Should knowledge be accessed in OSCE/OSPE

 10.   Simulated patients

 11. Examiner training...a must to do...

12.   Standard setting in OSCE

 13.   Disadvantages of OSCE

 

Standard setting in OSCE

Anshu : How does one decide the pass cut off in OSCE and other performance based examinations? We usually see that when there is batch with average performance, even a mediocre student scores well. Whereas when there is a batch full of good students, even excellent students get average marks. What is the criterion to discriminate between good, average and bad? How do we set standards?

Dr Tejinder Singh: For formative assessment and skill training, one should go by absolute standards. The level of batch mates should make no difference. For example, in the coffee example, earlier given by Barathi, if he does not turn the kettle on, he does not pass irrespective of how many marks he may score otherwise. This criteria referenced testing is vital to ensure that all students learn the basic minimum acceptable standard.

Sajjad: A useful article for standard setting
PDFs\Angoff standard setting.pdf< xml="true" ns="urn:schemas-microsoft-com:office:office" prefix="o" namespace="">

A very useful presentation on 'Standard Setting in Psychiatry Undergraduate OSCE'.
 www.nottingham.ac.uk/elan/XerteBenDiMambro.pps

Barathi: I am sure this article on OSCE in psychiatry will interest you --

Brian H, Mark H, Nancy M, Glenn R. Creating, Monitoring, and Improving a Psychiatry OSCE A Guide for Faculty Academic Psychiatry, 2002;26(3):134-161

Ciraj: As we are discussing about standard setting in relation to OSPE/OSCE, please find attached an excellent article by Norcini that gives an introduction to setting standards on educational tests.

Norcini JJ, Blank LL, Duffy FD, Fortana GS.The mini-CEX: a method for assessing clinical skills. Ann Internal Med 2003; 138:476-481.



Quality of assessment methods

Barathi: Dear All I would like to share a few points I read now.
Baartmann & coworkers in the Netherlands & Germany have presented a frame work of ten quality criteria for competence assessment programmes (Edn Research Review, 2007; 2:114-129).They argue that the criteria used to evaluate competency assessment programs should
involve traditional psychometric notions of validity and reliability but that these are not sufficient and be supplemented by newer edumetric measures.

The frame work of quality criteria they present is based on a literature review and is a synthesis of work by many different authors.
The ten criteria are:
1. Authenticity. The extent to which assessment reflects the competencies needed in the future workplace.
2. Cognitive Complexity: the assessment should reflect higher cognitive skills, gaining insight to the thinking processes applied by students.
3. Fairness. The assessment should reflect the knowledge, skills and attitudes of the competence at stake.
4. Meaningfulness. The assessment should get students to deal with meaningful problems relevant to their level of experience.
5. Directness. The extent to which teachers can immediately interpret assessment results without translation from theory into practice.
6. Transparency. The process must be clear and understandable to all participants.
7. Educational consequences. These include the effect the assessment has on learning and instruction.
8. Reproducibility of decisions. High stakes decisions should be made on multiple assessments carried out by multiple assessors on multiple occasions.
9. Comparability. The conditions under which the assessment is carried out should, as much as possible, be the same for all learners.
10. Costs and efficiencies. This relates to the practicability of the assessment procedures.

 RM Harden. Do you know? Medical Teacher 2008;30:642-643.

 Dr Tejinder Singh :Good one. However, I still like the 5 criteria suggested by van der Vleuten and Schuwirth.
These include-

  • Validity
  • Reliability
  • Educational impact
  • Cost and Acceptability
 

 

 

Discussion Summary

Introduction

Origin and evolution of OSCE

What is OSCE OSPE..

Important terminologies OSCE and OSPE

Process - OSCE and OSPE

Competencies

Advantages OSCE and OSPE

Limitation OSCE and OSPE

The final words