Embedded Files
CRITERIA 3
Building on your learning in Module 2, construct a program theory: theory of action and theory of change. Select a format that best communicates the goals and orientation of the program being evaluated.
PROGRAM THEORY
Chen states that "program theory is a systematic configuration of prescriptive and descriptive assumptions underlying a program" (2005, p. 18) The Making Sense of Evaluation handbook defines it as “a comprehensive description of how and why a desired change is expected to happen in a particular context. It tries to understand the causal links between what you do and what change occurs – why you believe it will happen” (Making Sense of Evaluation, p.4).
Program Context
The Starfish Pack program believes that by decreasing the stress caused by food insecurity, and providing consistent nutrition over the weekend, students will begin each school week happier, healthier, and nutritionally equipped to for greater cognitive functioning, focus, and academic receptiveness.
THEORY OF CHANGE
IF the Starfish pack program makes weekend meal backpacks available for food-insecure families, THEN needy students will participate in the Starfish Pack program.
IF needy students receive and consume the meals provided by the Starfish packs, THEN they will be well-fed over the weekend, and return to school after the weekend with full stomachs.
IF students come to school with full stomachs after the weekend, THEN they will be more receptive to instruction and demonstrate greater emotional, behavioural and cognitive functioning.
IF students demonstrate greater emotional and cognitive functioning, THEN social and academic performance will improve.
THEORY OF ACTION (Logic Model)
A logic model is "a graphical representation of the relationship between a program's day-to-day activities and its outcomes" (Chen, 2005, p. 34). It expresses the "who, what, where, when, and how" of the program.
CRITERIA 4
Based on your learning in Modules 1 and 2, review your program theory and evaluation questions to identify an evaluation approach. Provide a description of how that approach will operate in your specific program context. Also provide a rationale for why your approach is a good fit with your program context
Evaluation Approach
According to Chen, in order to be most useful, a program evaluation "needs to equip stakeholders with knowledge of those program elements that are working well and those that are not" (p.6). Green (2017) notes that evaluations for the purpose of learning and use require consideration of both process and outcomes in order to provide priority information for stakeholders.
Therefore, in order to assess whether the Starfish Program is being effectively implemented in the Vernon School District, and whether it is meeting its intended goals, I will be using a blended process and outcome-effectiveness evaluation model. It will make use of both formative (concerned with designing and using evaluation to improve a program) and summative (concerned with designing and using evaluation to judge a program's merit) assessment (Chen, 2005, p. 47).
Process evaluations are implemented during the operation of an existing program, and show how well the program is working, the extent to which it is being implemented as designed, and whether the program is accessible and acceptable to its target population (CDC, Types of Evaluation, 2020). It allows you to track information related to Who, What, When and Where program questions, for the purpose of enhancement.
In the context of the Starfish Program, we might ask:
is the program indeed reaching food-insecure students?
is there a strong communication link between the facilitators, schools, teachers and students/families?
has the facilitating organization succeeded in securing and maintaining the resources required to run the program throughout the school year?
Are there any existing barriers negatively affecting program implementation and efficiency?
However, along with analyzing whether the Starfish Program has been implemented as intended, we need to know whether or not the program itself has been effective in achieving its goals. The program demands a significant amount of resources, including: volunteer hours purchasing, packing, delivering and retrieving the packs; obtaining over $65,000 in donations from community partners and private donors; and the coordinated efforts of the Kalamalka Rotary Club, Okanagan Boys and Girls Club, School District 22, individual elementary schools, and teachers.
Outcome-effectiveness evaluations "measure program effects in the target population by assessing the progress in the outcomes or outcome objectives that the program is to achieve" (CDC, Types of Evaluation, 2020). The Starfish Pack program is a mature program, and “stakeholders inside and outside the program want to know whether the program is achieving its goals" (Chen, 2005, p.51).
For example:
does providing food-insecure students with weekend meals positively affect their school performance?
is there a notable difference in the participating students' demeanor, behaviour, and academic performance?
Are families satisfied with the weekend meals provided?
Rationale: why this approach is a good fit
The Starfish Pack program draws together a wide range of stakeholders, including charitable organizations, needy families, volunteer groups, schools and teachers, and community organizations and businesses. There are numerous challenges inherent in facilitating a program with such a complex chain of collaborators, increasing the possibility that the program is not being implemented as it was designed.
A monitoring approach is a perfect way to provide suggestions for improvement for stakeholders. Engaging stakeholders (students, families, schools, facilitators) in the evaluation provides empowerment, promotes inclusion, facilitates meaningful participation, and has the potential to significantly increase the likelihood that the evaluation will meet utility, feasibility, propriety, and accuracy evaluation standards (CDC, 1999; Donaldson, 2003).
Therefore, a process evaluation will get back to the basics of the Starfish Pack program's original intentions. In the case of the Starfish Pack program a process evaluation might provide information that:
helps the program expand to serve more students;
streamlines logistical matters;
increases effectiveness of communication between collaborators.
The outcome-effectiveness evaluation is a good way to introduce a sense of accountability to the program. While lofty in its intentions, is the program actually effective? On a practical level, is it yielding results? This form of evaluation allows for the collection of both qualitative and quantitative data to quantify and justify the program. The input of participants and their families, the school system, and the volunteer network can be compiled to "provide information that assists the stakeholders in figuring out what must be done next to securely attain- or even surpass-program goals" (Chen, 2005, p.4).
Key Evaluation Questions (UPDATED)
Chen, 2005, p.48
Students and Families
Are the students and families (program beneficiaries) satisfied with the program? Does the program address the needs of families?
Do students/families feel that the program is setting them up for greater success for the upcoming school week?
School District #22 (schools, principals, teachers)
Is the program reaching its intended audience? Is there a clear and effective collaborative process (teachers, principals, student support) in choosing students?
Do students exhibit improved social, emotional, behavioural and cognitive functioning after being enrolled in the program?
What are the challenges in facilitating the program from a school perspective?
Kalamalka Rotary Club & Volunteers, Save-on-Foods (Vernon Starfish Pack Program facilitators)
Is the target audience being reached?
How is the program functioning from an organisational perspective?
Is there a sufficient volunteer base to keep the program running smoothly? How sustainable is the program?
Is there a realistic possibility of decreasing program costs (groceries, etc.)?
References:
Centers for Disease Control and Prevention. (2019, March 8). Home. Centers for Disease Control and Prevention. Retrieved May 18, 2022, from https://www.cdc.gov/workplacehealthpromotion/index.html
Chen, H. (2005). A conceptual framework of program theory for practitioners. In Practical program evaluation (pp. 15-43). SAGE Publications, Inc.
Deeds, C. (2020, July 7). Food for thought: How food insecurity affects a child's education. American Youth Policy Forum. Retrieved May 26, 2022, from https://www.aypf.org/blog/food-for-thought-how-food-insecurity-affects-a-childs-education/
Green, J. (2017, April 12). Approaches to evaluation. YouTube. Retrieved May 26, 2022, from https://www.youtube.com/watch?v=zpwB_nSv6HM
Herman, J. L., Morris, L. L. & Fitz-Gibbon, C. T. (1987). Establishing the parameters of an evaluation. In Evaluator's handbook (pp. 9-25). SAGE Publications, Inc., https://dx.doi.org/10.4135/9781412984188
Making Sense of Evaluation: A Handbook for Everyone: https://dpmc.govt.nz/sites/default/files/2018-03/Evaluation%20Handbook%20Dec%202017.pdf
PACE MySPH (2012, December 11). Types of Evaluation. Retrieved May 27, 2022, from http://www.youtube.com/watch?v=IgwvRiGm424
Page updated
Google Sites
Report abuse