Predicted Readiness

Overview

The predicted readiness score is a model-based estimate of a trainee’s likelihood of earning a practice ready rating on his or her next SIMPL evaluation for that procedure. The model was developed using over 60,000 faculty-to-trainee evaluations for 193 procedures.

A predicted readiness score is based on the following factors:

• The specific procedure,

• The PGY-level of the trainee and the month of the academic year, and

• The cumulative number of practice ready evaluations by a trainee for a given procedure.

There is an important difference between a trainee’s predicted readiness and true readiness. True readiness is not directly knowable; however, based on the analysis of thousands of trainees, evidence suggests that past performance is a powerful predictor of future performance. The predicted readiness score increases as trainees perform to a practice-ready over time.

The approach we used to develop the predicted readiness score allowed us to include factors like differences among trainees, variation among raters, and the overall complexity of a case in an efficient and robust way. We included variation from each of these factors to account for the inherent uncertainty in predicting future performance while allowing a trainee’s prior performance to serves as the primary driver of changes in the score.

The predicted readiness score is explicitly predictive and can be interpreted in relation to a trainee’s future performance in the operating room. Moreover, as soon as a trainee has completed a single SIMPL evaluation for one of the 193 procedures, he or she can generate a predicted readiness score.

Individual trainees can increase their readiness scores and reduce the overall uncertainty in their scores as they earn more prior practice ready ratings and as they progress in their training. For example, for a procedure where the typical trainee has a 50% chance of earning a practice ready rating, an individual trainee who has earned 10 prior practice-ready ratings for that procedure, by contrast, has a predicted readiness score to 73%. The predicted probability score, therefore, provides a common scale that can be used to compare trainees’ progress toward different standards like a mastery standard of 95% or higher. Scores can also be used to quantify what the typical trainee’s likelihood is within a given PGY and month, which supports both absolute and relative benchmarks of performance.