Embedded Files
Goals
Goals
The goals of the evaluation is to assess the effectiveness of the Reading Buddies literacy intervention program for 2nd grade students. Our key objective is to evaluate students' growth in:
Reading confidence
Fluency development
Comprehension skills
Cognitive engagement
By evaluating their growth in these areas, the goal is to determine the overall usefulness of the program to students' literacy skills and socio-emotional development, and identify where and how to continue to develop strategies to support these students.
Evaluation Plan
Evaluation Plan
When we started developing our evaluation plan, we looked to the Kirkpatrick Method for inspiration, and evaluated how to adapt it to an elementary school setting. Using its levels of evaluation, we used it to identify key questions and the artifacts and instruments needed to answer it.
Reaction
Reaction
How did the students respond to and engage with the program?
Journal reflections, mid-program feedback surveys
Learning
Learning
To what extent have the students improved in their fluency, confidence, and stamina?
Pre- & post-program assessments, facilitator observation of reading fluency
Behavior
Behavior
Are the students using reading strategies on their own, without teacher prompting?
Facilitator observation of fluency strategies & participation, journal self-assessments
Results
Results
What measurable improvements in students' reading and classroom engagement have been observed since the start of the program?
Comparing pre- & post-program assessment scores, identifying overall trends, teacher feedback
(Brown, 2015)
This ultimately let us to two major questions that would form the bedrock of our evaluation plan.
QUESTION
JUSTIFICATION
METHODS
How are students experiencing the peer reading sessions, and where can the structure (pairing, facilitation, time, book choice) be improved?
Understanding student experience will help improve engagement and learning outcomes. It’s especially useful for program designers and facilitators to ensure the environment is supportive and materials are developmentally appropriate.
Student journals (qualitative): Reflective responses after each session will provide direct insights into engagement, interest, and challenges.
Facilitator observation logs (qualitative + structured rubric): Track pair dynamics, participation levels, and session flow.
Mid-program feedback survey (quantitative + qualitative): Targeted questions on student satisfaction with pairings, book enjoyment, and time length
How much are students improving in reading fluency, confidence, and reading stamina throughout the program?
Stakeholders such as school administrators, funders, and instructional designers looking to scale or adapt the program.
Pre- and post-assessments (quantitative): Oral reading fluency checks (e.g., WPM and expression rubrics) at the beginning and end.
Self-assessment surveys (quantitative + qualitative): Students rate their own confidence, enjoyment, and ease of reading.
Facilitator logs (qualitative): Observations of student progress and behavioral indicators of stamina and enthusiasm.
Conducting the Evaluation
Conducting the Evaluation
Given the various constraints of this project, most notably our short time span and the numerous challenges that comes from testing designs with young children, we decided to try to run a simulation of our course with AI. Although we acknowledged it was unlikely to give us the complex data that evaluating this course over several weeks with real students, we thought it would be an interesting tactic to try for our initial testing.
Using the user personas we'd developed as a result of our learner analysis of our research, we assigned ChatGPT the role of two students: Alex and Kiana. We then proceeded to use the lesson plan and monitoring key we had developed as the script for our interactions, and played the role of the facilitator.
You can read the chat log here. (ChatGPT-4, personal communication, May 1, 2025)
Evaluation Report
Evaluation Report
Purpose
Purpose
This evaluation report is primarily intended for the Department of Education (DOE), but may be of interest to relevant stakeholders such as elementary principals, reading specialists, program coordinators, teachers, and parents.
The purpose of this evaluation was to examine the impact of a focused literacy support program on early readers’ confidence, fluency, and cognitive engagement. In particular:
To what extent did students demonstrate increased confidence and self-evaluation in reading?
How much fluency (e.g., words per minute) did students gain over the course of the program?
Were students able to summarize and identify key elements in texts more effectively?
Did students demonstrate greater reading stamina, vocabulary growth, and schema development?
How did students’ reactions and behaviors toward reading evolve throughout the sessions
Methods
Methods
Framework:
The evaluation used a mixed-method approach combining observational, performance-based, and self-reflective data. The goal was to capture both academic and socio-emotional development in response to targeted instruction.
Participants:
Two second-grade students identified for early reading recovery support participated in the program throughout a multi-week intervention cycle.
Instruments and data collection:
Observation Rubrics: Peer and facilitator scoring on participation, engagement, and group interaction.
Words-Per-Minute (WPM) Assessments: Pre- and post-intervention fluency tracking.
Reading Logs and Journals: Used to document reading habits, reflections, and comprehension efforts.
Worksheet Completion Rates: Tracked student consistency and engagement with reading tasks.
Pictorial Self-Evaluation Check-Ins: Students rated their enjoyment and confidence levels regularly using visual scales.
Analysis
Analysis
Thematic Analysis: Observational and journal data were sorted into patterns using affinity mapping techniques. (fig 1, Observations Affinity Map)
Comparative Analysis: Changes in WPM scores and completion rates were compared pre- and post-program. (fig 2, Reading Fluency (WPM))
Descriptive Statistics: Journal and worksheet completion rates were translated into percentage-based indicators of participation and consistency. (fig 3, Journal Completion Rates)
Observations Affinity Map
Reading Fluency (WPM)
Journal Completion Rates
Findings
Findings
Students demonstrated measurable growth in reading fluency, confidence, and engagement.
Journal reflections served as a reliable proxy for cognitive engagement and emotional processing.
Stronger outcomes emerged when academic tasks were paired with emotional reinforcement and scaffolding.
Peer pairings enhanced confidence but had limited impact on skill acquisition compared to adult guidance.
Challenges
Challenges
Limited transfer from social to academic confidence
Dependence on external motivation
Inconsistent engagement with journals and workbooks
Reading stamina and vocabulary hesitation
Conclusions & Recommendations
Conclusions & Recommendations
Improvements:
Improvements:
Reading fluency
Confidence
Social engagement
Effectively supporting emotional safety and reading skill development.
Challenges:
Challenges:
Fostering independent motivation
Reading stamina
Skill transfer
Responding more consistently to external reinforcement and interest driven tasks
While buddy systems helped socially, teacher modeling had a stronger impact on skill development. Students showed emerging metacognitive skills like decoding and self-correction, but still needed structured prompts to stay focused.
Recommendations:
Recommendations:
Prioritize teacher-facilitated scaffolding
Monitor reading stamina and build incrementally
Continue evaluation across iterations
References
References
Brown, A. H., & Green, T. D. (2015). The essentials of instructional design : Connecting fundamental principles with process and practice, third edition. Routledge.
Wylie, N. (n.d.). What is the Kirkpatrick Model? Cathy Moore. https://blog.cathy-moore.com/what-is-the-kirkpatrick-model/
Page updated
Report abuse