Valid and Reliable Assessments

Assessments, so to speak, seems to be the confluence where instruction, learning, feedback, and reporting all come together. With that in mind, no discussion on assessments--in particular the way assessments are used in a standards based learning environment--can be undertaken without a firm understanding of some foundational concepts. While what follows here is by no means the entire discussion on this matter, it should serve to present enough information to get people started.

In brief, you do not need to be an assessment or measurement expert in order to develop and utilize the right kinds of assessments and the most reliable types of assessment practices in standards based grading.

Key Concepts*

• Assessing is measuring. Assessing is not for motivating, or for holding students accountable. It is for measuring learning and achievement. Period.

• Measurement error, which is inherent in every assessment, is the fact that no matter what, every assessment has a degree of inaccuracy in what it is reporting to us. This could be the result of a student being distracted for a moment, or forgetting something that they really do know (like when we forget a person's name that we use frequently enough to know it), or when they guess correctly. It could also be attributed to the way the assessment items (questions or tasks) are constructed, or a mistake in the way the teacher interprets the responses. Any single assessment has a degree of error. The only way to confidently conclude a level of achievement is to have multiple assessments (a minimum of 3 is the recommendation) for increased validity.

• At any time in an instructional environment, there are both assessments and systems of assessing happening. In SBG, both of these elements are a bit more articulated to both the teachers and the students. Marzano calls these types of classroom assessment (obtrusive, unobtrusive, student generated) and uses of classroom assessment (to provide formative scores, summative scores, instructional feedback).

• Formative vs. Summative: Often when teachers begin to shift towards the ideas of SBG, they adopt the practices of not including "formative" assessment scores in the calculation of a final grade. Instead, "formative" scores are for feedback, while the ultimate grade is determined by the results of a few summative assessments. However, when measurement error is considered, it is easy to see how an idea of achievement based off of a single summative assessment relative to a learning objective can be quite misleading. We should, however, ask ourselves, why aren't we including formative assessments when we are seeking to determine a student's achievement or learning? While not including formative scores in a traditional, points-based gradebook makes sense (since early learning counts against final learning), it does not make sense when were only considering the last 3 scores.

• Obtrusive Assessments: Instruction stops while students are being assessed. Think of the traditional final exam, or a whole class quiz, or a performance.

• Unobtrusive Assessments: Unobtrusive assessments do not interrupt the flow of instruction. This is usually done through observations by the evaluator (instructor). Examples could be demonstrations of speaking standards in a "fishbowl" activity, or observations by the HE teacher during an in-class (or out of class, for that matter) ultimate frisbee game, where skills relating to a specific standard are being demonstrated, or students utilizing statistical computations to solve a larger problem.

• Student Generated Assessments: This is perhaps the most powerful and telling type of assessment because students must have a very concrete grasp of what it is they are learning--or have learned--in order to decide how they will demonstrate this learning. Essentially, this is when a learner might come the evaluator with his or her suggestion on how they could demonstrate their mastery or learning of a learning objective.

• Formative Scores: In standards based learning and grading, when we are assessing to specific learning goals, formative scores are scores that we enter into the gradebook whenever we have assessed (whether obtrusively, unobtrusively, or through a student generated assessment) a student's demonstrated achievement towards a specific learning goal. The more formative scores we record for a student relative to the specific learning goal, the more valid is our idea of their true level of achievement. This is how we track and monitor, and even show growth. We can use formative scores to inform instructional decisions, show growth, generate summative scores, identify trends, and identify learning gaps and needs. Does this mean we need to record every single observation made in the classroom as a formative score? No, that would not be reasonable. What it does mean, is that whenever we feel we have a real sense of a student's demonstrated ability or achievement, we should record it to ensure we are evaluating his or her performance accurately, and making the right kind of instructional decisions based off of valid information.

• Summative Scores: In standards based learning and grading, summative scores are not something we enter, or are they something that come from a single assessment. Summative scores for a learning objective or standard are derived from the final formative scores (the last 3). For CJ and SMS, we will have several layers of summative scores: Summative scores for a learning standard (the average of the last 3 formative scores for a standard), summative scores for a domain (the average of the summative scores for the learning standards in the domain).

• Instructional Feedback: This is where we are not so much focused on documenting or measuring the learning, but getting immediate feedback on where students are. This is when we use exit slips, or grade homework as a class, allowing for corrections, or have students indicate their understanding through thumbs up or down. Yes, instructional feedback and assessing for a formative score can easily overlap. Instructional feedback is what we use to help us check and adjust our instruction and to provide further learning guidance to our students, while formative scores are used to measure the learning and the impacts of instruction within our students. Subtle, yet important, differences.

*These concepts are largely from Marzano, 2010, Formative Assessment and Standards-Based Grading

Developing Valid Assessments

Keeping in mind the key concepts to assessments and assessing, as well as the idea of levels of learning and how to use scales to help identify levels of learning, we can take a very easy look at developing assessments.

Assessments in standards based learning and grading should seek to identify the level of learning or achievement of the student, not so much the number of correct or incorrect responses.

An obtrusive assessment may look like this: several questions or tasks that relate to the emerging level of a standard (identification, vocab), and then several questions or tasks that relate to a developing level of a standard (recognition), and then several questions or tasks that relate to a proficient level of a standard (analysis, synthesis), and perhaps a single question or task related to an advanced level of learning (evaluation, creation). This is how the GRE is organized, and the SAT and the ACT. It is not about how many you answer correctly, but more about the level of questions you were able to handle.

Think about it this way: A traditional math quiz with 10 questions, and a student gets 7 of the 10 correct...a C grade. Were all of the questions requiring the same processes and knowledge? Were some questions more difficult than others?

A valid assessment would have have various levels of questions relative to the various levels of learning in regards to the learning objective. Students would receive a formative score based on the level of understanding/learning/achievement they demonstrated. While many scenarios could arise here (a student who does not answer developing questions correctly, but nails the advanced questions...or only answers a portion of the questions correctly at a certain level), the idea that students should receive scores based on the demonstrated level of achievement is what is key here. Yes, there are solutions to the other scenarios, but it is important that one become comfortable with the concept first.

Examples