Mason, C.A. (2009). Data Issues and Challenges in Documenting Effectiveness. In D. Green, J. Holstrum, C.A. Mason, D. Ross, and J. Eichwald (Eds.) Proceedings of the National Workshop on Accountability and EHDI Systems: Achieving Successful Outcomes, Strategies for Overcoming Obstacles; 2008 July 9; Breckenridge, CO. Atlanta, GA: Centers for Disease Control and Prevention.

https://www.cdc.gov/mmwr/volumes/66/wr/pdfs/mm6633a4.pdf

As EHDI programs progress to the point where they begin collecting long-term developmental outcome data, the issues they face in defining and determining which data to collect begin to increase dramatically. In regards to data, these challenges fall into two areas: (1) determining the domains or constructs to be assessed, and (2) ensuring collection of good quality data—meaning data that are reliable, valid, and interpretable.

A broad range of possible child and family outcomes exists. Child-focused outcomes include educationally relevant areas such as receptive or expressive language, cognitive development, and academic performance. Other child-focused domains that might be of interest to families and providers are socio-emotional development and attachment. Family-oriented outcomes include parenting behavior, parental knowledge regarding hearing loss and related service and intervention issues, and family advocacy.

Identifying and using appropriate assessment tools is an even greater challenge. Although there are numerous measurement instruments, many have poorly established psychometric properties, including poor reliability, limited established validity, lack of norms, limited independent evaluation, and limited experience with children with hearing loss. A further issue is whether these measures should be narrowly focused (e.g., diagnostic) instruments or broadly based developmental screening tools. The goal of developmental screening tools for children is to identify those who are at risk for language, cognitive, social, or emotional developmental delays, or a combination thereof. Although valuable for identifying children who are at risk, screening instruments provide only limited information on the full range of development. More detailed monitoring in at least some specific areas is warranted. By using a more focused measurement instrument to assess a construct such as expressive language, growth in language development can be monitored. However, such assessments are more time consuming and costly than a more limited developmental screen.

In addition, EHDI programs might need to collect developmental outcome data through partnerships with programs that are committed to using less than ideal measures.

The breadth and variety of instruments available for assessing developmental outcomes have implications for collecting useable data. Even when standardized scores are available, norming samples often are not equivalent, and their use and validity with children with hearing loss often limited or nonexistent. Consequently, additional data items will be necessary for the results to be useful for informing policy, research, or practice. Beyond simply collecting the child’s date of birth; date of administration of the instrument; raw score; and, if available, a standardized score, age-equivalent, detailed information regarding the measurement and assessment process also would be needed. This would include information such as the specific instrument and its version (e.g., version number and long or short forms), language (including American Sign Language), place of administration (e.g., home or clinic), respondent (e.g., mother, father, or the assessor via direct observation of the child), raw score, and answers to individual items on the instrument if possible. To maximize the utility of data, it will be important to be able to link this information with data regarding the timing and nature of intervention services received.

In 2006, the EHDI Data Committee issued a document describing the minimum data items that EHDI programs should collect for tracking and surveillance. At the time, few programs were in a position to collect developmental outcome data, and so the determination was made to focus on screening and diagnostic data. The time has come to revisit this issue to obtain a consensus to guide programs in collecting long-term developmental follow-up data.