Benjamin Karl

English Language Arts

Formative Assessment

Context of Data Assessment and Collection

My 10th Grade Honors English students were already engaging in Socratic seminar on a weekly basis, and overall achieving excellent results with respect to depth of thought, engagement, and understanding of the content under scrutiny. I wanted to bring this practice to my senior classes who were then studying Hamlet. I’d observed that my seniors (non-Honors track) seemed less willing to look smart in teacher-led discussions, yet expressed themselves well in small group or think-pair-share scenarios. Beyond merely scaling-up, I knew that in order to successfully integrate Socratic discussion here, I’d need to introduce another practice from my 10 Honors classes—that of preparing leveled questions as a pre-discussion technique.

There are various approaches to dividing questions into different levels; for our purposes, we used the following conventions and examples:

Level 1: Factual — Can be answered directly in the text

Example: Which character gives Hamlet his mission?

Level 2: Inferential — The answers is not obvious in the text; you must use clues and some personal connections to determine it. It may even be an arguable point. Example: Is Hamlet really crazy?

Level 3: Universal — The answer is supported by material in the text, but is not specifically about the text

Example: What power does belief play in personal destiny?

I created a lesson plan that included time dedicated to teaching this concept. In the course of the lesson, I displayed a slide with the above conventions and examples. Students worked in small groups to come up with additional examples, one level at a time. They then wrote those examples on the board and each was scrutinized by the class as a whole, looking for the following characteristics: (1) does it meet the fundamental criteria of the given level and (2) is it open-ended?

MATERIALS

Standards-aligned Lesson Plan

Daily Slides

Raw Data & Observations

Analysis of the Data

Students seemed to struggle the most with distinguishing between level 2 and level 3 questions, and with the concept of open-ended questions, the latter of which is evident in the linked data table. For the first seminar, with 48 students who were each tasked with producing 5 open-ended questions, only 15 of the expected yield of 240 questions were accurately produced (disqualification was based on either not being open-ended or not being leveled correctly). All students were given significant feedback and that number of accurately produced questions nearly doubled to 29 for the second seminar, but dropped again to 15 for the third seminar in the trial.

In follow-up conferences with students, I attempted to demonstrate the weakness of closed-ended questions through role-playing; I would have them ask their question and I would simply respond in the affirmative or negative, and then wait quietly, attempting to frustrate the student’s ability to hold actual discourse. The most common response was that students assumed their questions included a silent follow-up of “why or why not?” Because this type of questioning (core question with follow-up for support) is nearly ubiquitous among high school assessments, and therefore represents a singularly-ingrained experience for all of my seniors, I decided to reduce the stakes and only provide feedback, rather than a score, in regards to whether or not a question was open-ended.

Students then were only graded on whether their questions were leveled appropriately (this dimension is not independently evident in the linked data). Considering the lack of pressure on the open-ended quality, it is not surprising that that dimension returned quickly to baseline while the leveling quality rose steadily.

Analysis-based Conclusions

The data bear out the importance of not just appropriate leveling, but also the ability to formulate open-ended questions. About 40% of students made significant overall progress, 40% maintained average-to-high achievement, and 20% displayed variable performance. While a little less than 1/3rd of trials resulted in max discussion scores with no well-formed open-ended questions, nearly 2/3rds of the remaining max scores were correlated with at least one well-formed and open-ended level 3 question. Half of those were correlated with both well-formed, open-ended level 2 and level 3 questions.

The above correlations suggest that those students who are able to produce open-ended questions did not yet have fixed ideas about the content and were thus able to form and express novel thoughts as emergent from discussion. This is especially well supported by the significant correlation between max scores and the presence of level 3 universal questions which require the student to extrapolate their understanding to a separate context. Those producing closed-ended questions, then, may have essentially privately concluded their discussions before even coming to class, or may not have had time to fully engage with the content and establish anything more than a two-dimensional, single-context view of the material.

The implications of the data are that, despite systematic exposure to closed-ended-plus-follow-up questions on assessments, it is critical that students master the ability to formulate fully open-ended questions. It should be made a continuous practice in English and other subjects throughout each student’s education; the mere fact that my seniors, in the second-half of their final year of school, struggled so much with this suggests that prior exposure and practice was either too far removed to be effective, or not sufficiently ubiquitous to become habitual. Considering that failure to apply high-stakes to this task may have contributed to lower-than-expected results, it may be necessary to more heavily weight the scoring of question formulation in order to ensure that the practice is maintained and students are better able to reflect on what factors specifically led to their successes or failures in such discussions.

Google Sites

Report abuse