Jennifer Kim

Assessing Digital Literacy in Teachers

Context-specificity to reduce bias in self-assessments: Comparing teachers’ scenario-based self-assessment and objective assessment of technological knowledge

By T. Kastorff, M. Sailer, J. Vejvoda, F. Schultz-Pernice, V. Hartmann, A. Hertl, S. Berger, K. Stegmann

The Main Issue

Objective assessments and subjective self-assessments of digital literacy have inconsistent results. Neither objective tests nor subjective self-assessments paint an accurate picture of the true technological literacy level of educators. The study seeks to discover whether scenario-based self-assessments can bridge the gap between teacher self-assessments and objective assessments.

Which Frameworks are used to measure Technical Knowledge (TK) in Teachers?

TPACK

The TPACK framework for educational research by Koehler and Mishra (2009) stands for Technology, Pedagogy, Content and Knowledge. TK is defined as “understanding information technology broadly enough to apply it productively at work and in everyday life” (Koehler & Mishra, 2009). TK or technology knowledge is a synonym for digital literacy.

KMK

According to the KMK (Kultusministerkonferenz), a Standing Conference of the Ministers of Education and Cultural Affairs in Germany, basic digital skills can be narrowed down to five core concepts (KMK, 2019a):

Operating and applying technology
Searching for and processing information with technology
Communicating and collaborating with technology
Producing and presenting information with technology
Analyzing and reflecting information with technology

The Study

The study involved 75 participants, including 53 in-service teachers and 22 student teachers from German universities and schools. All participants were given:

Scenario-based self-assessment questionnaires
An Objective assessment of their technological knowledge (TK).

The Method

For the objective test, participants were given multiple choice tests to answer basic knowledge about their digital knowledge and technical skills.

For the scenario-based self-assessment, participants were given a scenario and they had to answer a questionnaire in which they rated themselves on a 5-point Likert scale ranging from 'never' or 'strongly disagree' (1) to 'very often' or 'strongly agree' (5).

Sample of Scenario-based Self-Assessment

The Results

The scenario-based self-assessment subsection on "operating and applying technology" predicted similar results to the objective assessment of TK. In other words, objective tests were accurate in rating the teachers' operating and applying knowledge.

However, other subcomponents like "producing and presenting information," "searching for and processing information," "communicating and collaborating," and "analyzing and reflecting on information," did not produce similar results as the objective TK assessment results.

Conclusion

The scenario-based self-assessments are a better predictor of technological knowledge in teachers than basic self-assessments or objective tests. They provide a clearer context for evaluating personal behavior. These assessments bridge the gap between objective assessments and self-assessments by being more closely aligned with objective assessments and less with general self-assessments that are biased towards social desirability.

Digital Literacy Assessment Tools

Objective Assessment Tools: Scenario-based Self-assessment tool:

Northstar Digital Literacy Assessment: Scenario-based assessment example 1

Technology Self-assessment Tool Scenario-based assessment example 2

References:

Kastorff, T., Sailer, M., Vejvoda, J., Schultz-Pernice, F., Hartmann, V., Hertl, A., Berger, S., & Stegmann, K. (2023). Context-specificity to reduce bias in self-assessments: Comparing teachers’ scenario-based self-assessment and objective assessment of technological knowledge. Journal of Research on Technology in Education, ahead-of-print(ahead-of-print), 1–14. https://doi.org/10.1080/15391523.2022.2062498

Koehler, M. J., & Mishra, P. (2009). What is technological pedagogical content knowledge? Contemporary Issues in Technology and Teacher Education, 9(1), 60-70. https://www.researchgate.net/publication/241616400_What_Is_Technological_Pedagogical_Content_Knowledge#fullTextFileContent

Kultusministerkonferenz (KMK) [conference of federal ministers]. (2019a). Bildung in der digitalin Welt. Stragegie der Kultusministerkonferenz [Education in a digital world. Strategy of the conference of federal ministers of Germany]. https://www.kmk.org/themen/bildung-in-der-digitalen-welt/strategie-bildung-in-der-digitalen-welt.html

Page updated

Report abuse