Session 4: Adapting Assessment Tools

Aims

To educate fellows about how to select and adapt evidence-based assessment tools for community settings

Objectives

By the end of this session, participants will:
  1. Be able to define the major characteristics of a well-validated measure (at least 2 forms of validity and 2 forms of reliability)
  2. Identify 3 questions that assess the appropriateness of a measure for a particular community setting
  3. Design a study to develop a new measure, or adapt and/or validate a measure for use in a new setting

Key Terms

  • Validity: How well does the test measure what it is intended to measure?
  • Reliability: Reproducibility of a measurement

Agenda

  • 09:00-09:15: Why measurement matters
  • 09:15-09:45: How to choose evidence-based measurement tools for your project
  • 09:45-10:00: How to develop or adapt evidence-based assessment tools
  • 10:00-10:20: Example – Development of the Prodromal Questionnaire
  • 10:20-11:00: Fellows discuss assignments
  • 11:00 -11:10: Break
  • 11:10-12:00: RSG

Readings

Teaching Materials

Supplemental Resources

  • None

Assignments

  • Before class: Choose one or two measures needed for your community project.  Are these measures evidence- based?  Why or why not? Are these measures appropriate for the population in your project?  Why or why not?
  • During class: Discuss answers above. Do you need to choose new measures or adapt these measures in any way for your project?  Develop plan for choosing or adapting measures.

Summaries of Objectives

  1. Be able to define the major characteristics of a well-validated measure (at least 2 forms of validity and 2 forms of reliability):
    1. The two major forms of test validity are internal validity, which includes face validity and construct/content validity, and external validity.  External validity includes convergent validity (concurrent validity and predictive validity), as well as discriminant validity.
    2. Common forms of test reliability include inter-rater reliability, test-retest reliability and internal consistency.
  2. Identify 3 questions that assess the appropriateness of a measure for a particular community setting:
    1. Is your population similar to the normative sample used in the development of the measure?  Assess for: Age? Ethnicity? Gender? Educational attainment? Socioeconomic status?
    2. If the measure was translated into a second language, was it validated in the new language?
    3. Was the setting of the validation of the measure similar to your setting (e.g. school, community clinic, etc.)?
    4. Is the assessment modality appropriate (e.g. self-report, interview, focus group, computerized assessment)?
    5. Does this measure provide outcome data that is relevant to your population? What outcomes are important to them?
    6. Is it feasible to use this measure in your study or program?
  3. Design a study to develop a new measure, or adapt and/or validate a measure for use in a new setting.
    1. Common ways of adapting a measure are to shorten it, translate it, change the mode of administration, or use it with a new population or in a new setting.

Assessment Data and Learner Feedback

  • Learners discuss the objectives with the instructors and with each other in terms of choosing or adapting a measurement tool for their particular project.  This is completed in class, with feedback from the group.

Teaching Tips/FAQ

  • At UCSF, the example of developing and adapting a measurement tool was the Prodromal Questionnaire, a self-report measure for assessing a psychosis risk syndrome.  In other settings, it may be appropriate to describe the development of a measure with which the instructors are familiar or which is of particular relevance to the learners.
  • Learners should be encouraged to bring the results of these discussions back to their mentors in designing their projects.
  • Having the group provide feedback is a good way to utilize expertise within the group of both community and academic fellows, as well as to encourage more participatory discussion.

Comments


Comments