Findings

This set of slides summarizing findings from a validation study that motivated some of the revisions to the survey for the ACC study. This validation study used secondary data from 4 prospective studies of implementation interventions that fielded the ORCA at one or more points in time. We want to call out a couple of things from these findings:

1. We predicted that baseline readiness (strength of evidence & organizational context) would independently predict implementation effectiveness, but failed to find any association. This does not mean that baseline readiness isn’t important or actionable (e.g., these were all implementation studies that collected ORCA data and they could have focused attention on those factors that were deficits at baseline), but it is certainly important to know.
2. It is important to consider who is responding to the survey, and if you are surveying multiple individuals per site (recommended) consider how you are using the data (e.g., analyzing individual surveys vs. creating an aggregate score). We generally found strong inter-rater agreement on readiness, but had weak reliability in the validation study because of a relatively small number of observations per site due to having small numbers of participants directly involved in the implementation projects.