Module 3 Scoring
Module 3 Scoring
Embedded Files
What are different ways to score student concept maps?
What are different ways to score student concept maps?
In this module, we introduce five basic types of scoring methods:
Traditional
Categorical
Holistic Rubric
Structural (Holistic)
Comparison to Expert Map
We have also developed a slide deck that you can use to review the methods or introduce colleagues or students to scoring EM concept maps.
Each of these methods can be applied to hand-drawn or digital concept maps.
To follow along with the examples and exercises, you may want to download these files: exercises and examples
What scoring method should you choose?
What scoring method should you choose?
Scoring concept maps is a big topic and which method(s) you choose really depends on what you hope to learn from the assignment. Quantitative and qualitative methods help us learn different things about how students conceptualize a topic, and not all scoring approaches work with every assignment type. For example, scoring with the traditional or holistic rubric may not make sense for a “Fill-in-a-map” assignment because you do not expect the structure/complexity of the student maps to vary much. Comparison to a reference map probably makes the most sense for that type of assignment.
The Case Studies from Ohio State and Rowan include more examples of how to score different types of assignments and makes some recommendations for how to choose appropriate methods.
Do you even need to use one of these scoring methods? Maybe not.
The scoring methods featured in this module are research-based and provide reliable ways to quantitatively or qualitatively score cmaps; they are great for formal assessment at the course or program level and even research. But, if your primary goal is to get a quick sense of your students understanding of a topic and provide a grade or formative feedback, you may not need to use one of these methods. A simple rubric or student reflection and self-assessment could be sufficient for generating a grade. You may even be able to use an AI tool to help apply the rubric and give formative feedback, as described in the AI case study in Module 4. The structural (holistic) method could also be a good way to give feedback on the complexity of a concept map.
We encourage you to check out each of the methods, so you know what options are available.
Reminder: Before you start scoring concept maps...
Reminder: Before you start scoring concept maps...
If you have a set of digital cmaps generated (or converted) in CmapToolsTM, we recommend a few actions to make scoring easier, faster, and more consistent:
check for and fix broken links (under Tools menu);
use the Autolayout tool in CmapToolsTM for a clear, hierarchical layout (under Format menu); and
check for Spelling errors (under Tools menu).
Note: If students are submitting their assignment as a .cmap file, then you can remind them to take these steps before submission.
Scoring with the Traditional Method
Scoring with the Traditional Method
Traditional scoring involves counting components of a concept map, and is considered fairly straightforward. After some practice, traditional scoring can be done quickly and achieve high agreement across multiple scorers. It is the most commonly used scoring approach in the literature, and is often paired with a qualitative method that examines content.
One of the simplest qualitative complements to this method is generating a word cloud of all of the concepts for a quick visual understanding of prominent themes and major gaps.
In the video, we use the Exercise 3 concept map for traditional and categorical scoring if you'd like to follow along and practice.
Scoring with the Categorical Method
Scoring with the Categorical Method
The categorical method focuses on the cmap content and offers insight into knowledge gaps or different perceptions of a topic.
This method is often more time consuming than the traditional or holistic structure methods and requires time up front to calibrate category assignment, especially if there are multiple scorers.
Scoring with the Holistic Rubric
Scoring with the Holistic Rubric
The holistic rubric method considers the breadth, depth, and correctness of students' understanding of a topic.
Scoring with Structural (holistic), and Comparison/Match to Expert
Scoring with Structural (holistic), and Comparison/Match to Expert
Traditional, categorical, and holistic rubric are the most frequently used scoring approaches in the literature, but these slides illustrate how to apply two additional methods.
One of the challenges with using concept maps for assessment or research is the time required to reliably score the cmaps, especially for large classes or programs. To assist with scoring, we have developed an automated tool that can apply both categorical and traditional scoring to a batch of cmaps. For information on the tool's development and how to use it, please check out this page, which includes a link to download the software and videos walking you through examples.
Instructor Resources for Scoring Cmaps
Instructor Resources for Scoring Cmaps
Tutorial for automated traditional scoring (software created by J. Pelkey & M.K. Watson)
To acknowledge this work when you share the toolkit or its resources, please cite:
Barrella, E., Bodnar, C.A., Cano Morales, M.L., Carnasciali, M.I., Cruz, J., Dillon, H.E., Kecskemety, K., Jackson, A.M., Miskioglu, E., Rodríguez Mejía, E. (2023). EM Concept Map Toolkit. Licensed under CC BY-NC-SA
Page updated
Report abuse