Protocol for Designing a Summative Clinical Radiology Exam for the Arab Board of Health Specializations (ABHS)

1. Introduction

The purpose of this protocol is to outline the steps for designing, implementing, and evaluating a summative clinical radiology exam within the Arab Board of Health Specializations (ABHS). This exam is aimed at assessing the clinical competencies of radiology residents, ensuring alignment with ABHS standards and international best practices.

2. Objectives

The objectives of the summative clinical radiology exam will be determined by a comprehensive job analysis of a radiologist’s responsibilities and alignment with the ABHS curriculum objectives. The job analysis will be conducted through:

• Surveys and Interviews: Engaging practicing radiologists and academic experts to identify core competencies required for effective practice.

• Focus Groups: Discussions with stakeholders, including educators, radiology department heads, and healthcare administrators, to validate competency domains.

• Task Analysis: Breaking down essential radiology tasks into measurable components to ensure they are represented in the exam.

• Curriculum Mapping: Ensuring the competencies identified in the job analysis align with the ABHS curriculum objectives and training outcomes.

These steps ensure that the exam evaluates real-world skills and knowledge necessary for competent practice in radiology, with special emphasis on evaluating clinical decision-making and diagnostic skills, assessing the ability to interpret imaging modalities effectively, and measuring communication and professionalism in clinical scenarios. Additionally, the process ensures alignment with ABHS’s competency framework and curriculum.

3. Structure of the Exam

3.1 Components

1. Objective Structured Clinical Examination (OSCE):

1. • Stations designed to evaluate image interpretation, procedure planning, and clinical reasoning.

   • Focus on key competencies such as:

- Diagnostic imaging interpretation (e.g., X-rays, CT, MRI, ultrasound).

- Procedure-based skills (e.g., ultrasound-guided interventions).

- Communication with patients and colleagues.

2. Structured Viva Examination:

1. • Case discussions with examiners to assess higher-order reasoning and professional behavior.

3. Image-Based Multiple-Choice Questions (MCQs):

1. • Questions presenting clinical scenarios followed by related imaging studies.

   • Focused on testing theoretical knowledge and applied problem-solving skills.

3.2 Duration

• OSCE: 8-10 stations, each lasting 10-15 minutes.

• Structured Viva: 2 sessions, each lasting 20 minutes.

• MCQs: 80 questions completed in 120 minutes.

3.3 Scoring System

• OSCE: Checklist-based scoring for each station with weighting for critical errors.

• Viva: Standardized rubrics focusing on communication, reasoning, and professionalism.

• MCQs: Automated scoring with pre-defined answer keys.

4. Exam Development Process

4.1 Blueprinting

• Develop a detailed blueprint ensuring:

- Alignment with ABHS’s curriculum objectives.

- Coverage of key radiology domains: diagnostic imaging, procedural radiology, and clinical integration.

- Representation of clinical scenarios commonly encountered in radiology practice.

4.2 Item Development

• Create OSCE stations, MCQs, and viva scenarios collaboratively with subject matter experts.

• Incorporate real-life clinical cases and imaging studies.

• Review items for clarity, validity, and relevance.

4.3 Validation

• Pilot test the exam with a small group of candidates.

• Analyze results to evaluate:

- Item difficulty and discrimination indices (for MCQs).

- Inter-rater reliability (for OSCE and Viva).

- Content and face validity.

5. Implementation

5.1 Logistics

• Equip testing centers with:

- Workstations for image interpretation.

- Standardized materials for OSCE stations.

- Trained standardized patients (if required).

5.2 Examiner Training

• Conduct workshops to train examiners on:

- Using scoring rubrics.

- Managing time and ensuring consistency.

- Providing constructive feedback to candidates.

5.3 Candidate Preparation

• Provide candidates with:

- Guidelines on exam format and expectations.

- Sample cases and stations for practice.

6. Quality Assurance

• Establish a central committee to oversee:

- Standardization across all testing centers.

- Continuous monitoring during exam administration.

• Conduct post-exam analysis, focusing on:

- Candidate performance.

- Examiner feedback.

- Psychometric properties of the exam.

7. Reporting and Feedback

• Generate detailed score reports for candidates, highlighting:

- Strengths and areas for improvement.

- Performance benchmarks.

• Provide aggregated data to the ABHS for curriculum enhancement.

This protocol ensures the summative clinical radiology exam meets the highest standards of validity, reliability, and fairness. By focusing on comprehensive evaluation methods, it aligns with ABHS’s mission to produce competent and ethical radiologists equipped for independent practice.

Integrative OSCE

Integrative OSCE (Objective Structured Clinical Examination) combines traditional OSCE components with additional elements, such as case-based discussions, reflective practice, and digital or simulated environments, to provide a more comprehensive assessment of a candidate's clinical competencies.

Key Features of Integrative OSCE

1.          Holistic Assessment:

o   Integrates procedural skills, clinical reasoning, and communication in a unified framework.

o   Moves beyond task-based evaluations to assess higher-order thinking and decision-making.

2.          Case Continuity:

o   Candidates progress through multiple stations that may involve interconnected tasks.

o   For example, interpreting an X-ray at one station may lead to discussing management in a subsequent case-based discussion.

3.          Diverse Competency Testing:

o   Combines technical skills (e.g., performing an ultrasound) with non-technical skills (e.g., ethical reasoning or breaking bad news).

4.          Technology Integration:

o   Uses PACS for digital image interpretation.

o   Incorporates simulators for interventional or procedural tasks.

5.          Feedback and Reflection:

o   Includes structured feedback to help candidates understand their strengths and areas for improvement.

o   Reflection may be built into the exam, asking candidates to discuss their thought process.

Steps to Design an Integrative OSCE for ABHS

1.          Define Objectives:

o   Identify key competencies aligned with ABHS standards, such as technical skills, clinical reasoning, and professionalism.

2.          Blueprint the Exam:

o   Map out stations to cover various domains (e.g., procedural skills, communication, clinical judgment).

o   Ensure a balance between knowledge-based and skill-based tasks.

3.          Develop Interconnected Stations:

o   Example: A case begins with a clinical scenario (history-taking), progresses to an imaging interpretation task, and ends with a management discussion.

4.          Incorporate Realism:

o   Use standardized patients, digital platforms (e.g., PACS), and high-fidelity simulators to simulate real clinical environments.

5.          Train Examiners:

o   Ensure consistency and fairness by providing training on scoring rubrics and station protocols.

6.          Evaluate and Reflect:

o   Include candidate self-assessment or a post-exam debrief to promote deeper learning.

Benefits of Integrative OSCE

1.          Comprehensive Evaluation:

o   Provides a multi-dimensional view of a candidate’s abilities.

o   Assesses both technical and non-technical skills effectively.

2.          Enhanced Validity and Reliability:

o   Reduces the risk of bias by integrating multiple evaluation methods.

3.          Improved Learning Outcomes:

o   Encourages candidates to think critically and connect clinical tasks with reasoning and communication.

4.          Alignment with International Standards:

o   Meets the evolving demands of competency-based medical education.

Potential Challenges

1.          Logistical Complexity:

o   Designing interconnected stations and coordinating resources can be challenging.

2.          Examiner Training:

o   Requires well-trained assessors to ensure standardization.

3.          Time and Cost:

o   More resource-intensive compared to traditional OSCEs.

Integrative OSCE Example in Radiology

Station 1: Image Interpretation

• Task: Interpret a chest X-ray and identify abnormalities.

• Competencies: Diagnostic accuracy, clinical reasoning.

Station 2: Case-Based Discussion

• Task: Discuss a clinical management plan for a patient with findings from Station 1.

• Competencies: Clinical reasoning, decision-making, communication.

Station 3: Procedural Task

• Task: Perform an ultrasound-guided biopsy on a phantom.

• Competencies: Technical skill, safety protocols, procedural knowledge.

Applications in ABHS

ABHS could adopt an integrative OSCE framework to:

1.          Enhance clinical assessments for final-year residency programs.

2.          Align its evaluation methods with international best practices.

3.          Provide a more realistic and comprehensive assessment experience for candidates.

This approach not only improves the validity of the assessment process but also ensures that ABHS-trained professionals are well-prepared for real-world medical challenges.