Learn>Learning Catalog>Collecting Observations>Overview of Data Collection

Overview of Data Collection in the Classroom

Data collection in the classroom starts with lessons designed to give you the opportunity to observe the criteria of both the engagement and academic rubrics while implementing the trial lesson feature or instructional practice.

The lesson feature or instructional practice that is the treatment for the trial should be included in the learning design for each day of the trial. With that instructional practice being common to any day of the trial period, observations of a subset of criteria can be made on any day of the trial. As an example, if a lesson designed for day 3 of the trial only provides you the opportunity to collect observations on criteria 3 and 4 of the engagement rubric and criteria 1 and 2 of the academic rubric, design learning so that the remaining criteria can be observed on one of the next days of learning. 

Over the course of the trial period, try to collect three to four observations for each of the criteria in both the engagement and academic rubrics. 

Some common questions and concerns about data collection are located below.

One of the criteria we chose isn’t observable.

Option 1: If the team has four criteria chosen, just drop the criteria that is not observable for the trials. Collect data on the remaining three criteria for the remainder of the trials.

Option 2 (use this option only during baseline data reflection): If there is enough time to collect additional baseline data, revise the criteria into an observable statement. Collect baseline data on that criteria before the Trial 1 data collection window begins.

How do we know we are rating our indicator students similarly? (inter-rater reliability)

The data being collected does not have the precision of a measure, like a ruler or a thermometer. You are collecting indications that will be assumed to be connected to changes in learning design. Each teacher knows their students and how students respond. You are using your professional observation skills to maintain intra-rater reliability, meaning the degree of observation consistency by a single teacher. 

When you analyze the data you will be looking at the movement of observations from combined rubric levels 1 and 2 to combined rubric levels 3 and 4.

If it is helpful, the team can have another discussion to get a sense of what each engagement and academic criterion looks like in each of the team member's classrooms. All team members can come to some common understandings of the observable features of each criterion collected as data.

I didn’t have the opportunity to collect criteria observations.

It is important to design lessons to provide yourself the explicit opportunity to collect observations on the agreed-upon criteria. That may mean planning to collect data during particular lessons or making slight modifications to lessons so they include the opportunity to collect observations on criteria.

You may find collecting all engagement and/or academic criteria during one lesson difficult. The team can agree that we can split observations across different lessons. For example, I might be able to collect “Initiates and completes instructional tasks” on a Monday, but won’t collect “Participates in discussions” until the Wednesday lesson.

Some of my students are not at school every day or have dropped my class. 

Option 1: Reducing the number of indicator students is an option if there is not enough time to choose new students to observe. Strive to collect as much data across as many indicator students as possible.

Option 2: If there is enough time to collect baseline data on new students, identify other students to observe.

I couldn’t collect three observations in each of the criteria. 

Our goal is to collect three or more observations across all the criteria during each trial. That may not be feasible for each criterion in each trial. Try your best to collect as much as possible, balancing both engagement and academic observations.