Program Evaluation - Stages 5 & 6

An effective program evaluation should implement a variety of data collection methods, in order to increase the credibility of the evaluation, and ensure that key evaluation questions are answered (Unicef, 2014). An evaluation should be begin by examining the data that are already available, and then decide what other data are required. Collecting and synthesising both qualitative and quantitative data will be essential to my program evaluation's thoroughness.

Quantitative data are usually numerical, and can typically be counted or measured. According to Herman (1987, p.20) quantitative data result from “measuring, summarizing, aggregating, and comparing measurements, and on deriving meaning from quantitative analyses.” In contrast, qualitative data rely on “observations, interviews, case studies, and other means of fieldwork” (Herman, 1987, p.23). Qualitative collection allows for descriptive input from the perspective of program participants.

The following are some common methods of collecting quantitative data:

• Surveys/Questionnaires - can be traditional paper copies, phone surveys, or online surveys using (eg) Google forms or services such as Survey Monkey. Surveys (typically Yes/No, or 1-5 scale) are an inexpensive method of collecting a large amount of detailed information from the population, although entering data can be a long process, and not all surveys may be completed or returned.

  • Surveys and questionnaires will be used extensively in this program evaluation - to efficiently gather data from parents, students, teachers and other program stakeholders.

• Program Data - typically refers to data already in existence. For example, this could be previous research, studies on similar programs, or data already collected and stored by the program undergoing evaluation.

  • Program data in the form of a record of previous participants, financials, and correspondence will allow the evaluator to make comparisons and track progress. Of particular use will be data related to the development of the program in other districts. New data from literacy and numeracy assessments will be added.

The following are some common methods of collecting quantitative data:

• Observations - This method is particularly effective when observations are made by a trained evaluator. They can be costly and time-consuming, however. In my program evaluation, teachers will be making educated observations of their students’ behaviour and progress.

• Interviews - are an excellent way to gather qualitative data. One-to-one interviews provide an opportunity for descriptive feedback by posing open-ended questions. However, interviews take much more time (and are therefore costlier) than surveys, and depend on a person’s availability, as well as their comfort level with the interviewer.

  • I will use interviews more sparingly than surveys, and will be strategic in my choice of interviewee. For example, interviews may be more suitable for younger students in the program unable to complete surveys. As well, due to the amount of time they require (and the corresponding high cost), a purposive sampling will be employed, focusing on families with more than one child in the program, and organizational stakeholders (Rotary Club leaders, etc.). This will introduce a greater margin of error in my evaluation, but it will provided useful descriptive data to complement the quantitative survey and program data.

  • Focus Groups - Focus groups allow for small groups to be consulted at one time, providing opportunity for descriptive feedback similar to one-on-one interviews, but with potential cost-savings.

References:

Data Collection Methods. (2016, April 2). YouTube. Retrieved June 7, 2022, from https://www.youtube.com/watch?v=NJ-gW6adQTc

Herman, J. L. (1987). Evaluator’s handbook. Accessible via Queen’s Library.

UNICEFInnocenti. (2014, December 15). Data Collection & Analysis. YouTube. Retrieved June 7, 2022, from https://www.youtube.com/watch?v=HFGVJJMDo4I

Many theorists emphasize that the most important aspect of program evaluations is use (Kirkhart, 2000; Patton, 2013; Weiss, 1998). In other words, evaluations are worthless if they do not answer questions that will help improve the program. Further, if evaluations are not presented effectively, their benefit can easily become diminished. The final summation is, then, a crucial finishing touch to an effective evaluation.

I will ensure my evaluation has an impact on the program by collecting and analysing meaningful data to provide to stakeholders - in particular the Kalamalka Rotary Club, School District 22, and the Starfish Pack Program organization.

By emphasizing collaboration and providing an opportunity for voice, the evaluation will generate 'buy-in' from stakeholders, increasing the likelihood of recommendations actually being applied. According to Greene, "program staff who participate in defining and framing the evaluation begin to think more deeply about what they are trying to accomplish" (Green, 1988, as cited in Weiss, 1998, p. 25) This process use approach will also create new knowledge as the evaluation is underway, again increasing application of evaluation discoveries.

Because the Starfish Pack program is a non-profit organisation, this evaluation will include suggestions for improvement that will be constructive and manageable, with an emphasis on financial feasibility and functionality.

Throughout my evaluation, data (from surveys, questionnaires, and analysis of existing program data) will be recorded and represented using software - programs such as Google Sheets (or Microsoft Excel), Google Forms, and graphing software. As such, this information will be easily transferrable and transformable into attractive media presentations.

Data Presentation

Much of the data collected will be from surveys, and although this data can be cumbersome to input, it can be presented in visually-friendly formats such as line graphs, bar graphs, pie charts, and tables. However, it is important not to fall prey to ‘data overload’, whereby huge swathes of qualitative data are presented. This can be disarming to the stakeholders, and mere data without analysis is not very effective.

Charts and graphs will allow stakeholders to see patterns in:

• financial statistics - annual costs to run the program; donations;

• Program enrollment - year-by-year to demonstrate program growth;

• Survey results - from students, parents, schools, facilitators;

The following graphics demonstrate how data inputted into Google Sheets can easily be transformed into attractive graphs and charts:

Google Forms is a simple and effective way of reaching a large population, and is particularly useful when collecting data from surveys or questionnaires. A Google Form survey can be created to include a wide variety of responses, including multiple choice, numerical scales (cp. Likert scale), short and long open-ended responses and more.

Information garnered can be easily collated for analysis. As with Google Sheets, data can be graphed and charted. See examples below:

Along with purely quantitative data, which can be both overwhelming and dry, an effective report will provide qualitative data from observations and interviews, which adds a 'human touch' to the evaluation. Statements from organizers, parents, and students attesting to the impact of the program in their lives can be as powerful as a string of numbers.

According to Unicef’s Qualitative and Quantitative Data presentation, effective presentations should be kept simple, avoiding overly complicated graphs, and should focus on the key evaluation questions. Visualizations should be clear, and strengthen the message. It is important note any potential biases in the data. I will need to explain that the purposive interviews conducted for my evaluation are not necessarily representative of the entire population.

Some helpful guidelines for effective reporting include:

1. Simplicity: A common myth is that complex analysis and big words impress people (StART). On the contrary, practical and understandable analyses are more appreciated.

2. Clarity - following correct form and order will ensure a clear presentation

3. Brevity - Length is not necessarily evidence of effectiveness. A report must be complete and thorough, but brief enough that "the patience of the reader is not lost and there is no confusion of ideas." (Jain, 2016)

4. Accuracy - Be sure to reference any potential weaknesses in data or data collection, and state any biases up front. Honesty is the best policy.

5. Presentation - attractive presentation - whether a written summary or a visual presentation, is important.

References:

Data Collection Methods. (2016, April 2). YouTube. Retrieved June 7, 2022, from https://www.youtube.com/watch?v=NJ-gW6adQTc

Jain, A. (2016, January 25). Top 11 Characteristics of a Good Report. Your Article Library. Retrieved June 2, 2022, from https://www.yourarticlelibrary.com/company/drafting/top-11-characteristics-of-a-good-report/75197.

Kirkhart, K. E. (2000). Reconceptualizing evaluation use: An integrated theory of influence. In V. Caracelli and H. Preskill (Eds.), The expanding scope of evaluation use. New Directions for Evaluation, 88 (pp.5-23). San Francisco; Jossey-Bass.

Patton, M. (2013, June 7). Utilization-focused evaluation for equity-focused and gender-responsive evaluations. YouTube. Retrieved June 7, 2022, from https://www.youtube.com/watch?v=jQP1FGhxloY

Students as Researchers Toolkit (StART). edugains.ca. (n.d.). Retrieved June 7, 2022, from http://www.edugains.ca/newsite/studentVoice/student_researchers_toolkit.html

UNICEFInnocenti. (2014, December 15). Data Collection & Analysis. YouTube. Retrieved June 7, 2022, from https://www.youtube.com/watch?v=HFGVJJMDo4I

Weiss, C. H. (1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19(1), 21-33.