N. Evaluate programs and services using measurable criteria
INTRODUCTION
Program evaluation is the equally important, less formal sister, to research which is covered in competency L. Evaluation tends to be program specific, whereas research can incorporate many different organizations. In order to ensure that a school library is meeting the needs of the entire school community, school librarians must make sure that evaluating the library’s services, programs, and resources is an ongoing consistent component of their job. The process of evaluation helps school libraries determine “results, costs, outcomes, impacts, successes, and other factors related to a library’s collections, programs, services, or resource use” (Matthews, 2018, p. 255). Evaluation also gives school libraries the ability to provide concrete evidence of the library’s value to stakeholders. School “librarians are being asked why activities and services are being done the way they are and whether the services being offered are the most appropriate given the characteristics and needs of a specific community” (Matthews, 2018, p. 255) [emphasis mine]. Matthews (2018), notes libraries should attempt to measure the extent, efficiency, effectiveness, quality of service, and impact of library resources, services, and programs on library users (p. 261). This document will discuss the components of program evaluation, and how a school library program should integrate them.
EVALUATION
The first thing a school librarian should determine before conducting an evaluation is the goals, objectives, and mission of the school library. According to Matthews (2018), once these are clearly defined, the library teacher may then identify and obtain the resources, programs, and services needed to reach those goals, and to ensure that the resources are being used in a way that fits the needs of the program, and is cost effective (p. 256). This can be done by keeping the following questions in mind when preparing an evaluation:
“What is our mission?
Who is our customer?
What does the customer value?
What are our results?
What is our plan?” (Matthews, 2018, p. 256).
Furthermore, Matthews (2018) identifies what evaluation does for a [school] library program, it “gathers, assesses, provides, justifies, informs, determines, compares, assists, monitors, uses, serves, and encourages” (p. 256-257). There are several different methods of evaluation a school librarian can utilize when evaluating, including: Outcome-Based Planning and Evaluation (OBPE), Performance-Based Evaluation, and Impact Evaluation.
OBPE
Program evaluation, specifically one that measures student outcomes like OBPE is integral to the success of a school library program. Children’s Access to and use of Technology Evaluation (CATE) Outcome-Based Planning and Evaluation (OBPE) Model (discussed in comp. L), examines student outcomes as a result of using library services, programs, and resources. An outcome is described as any measurable transformation, or “change in attitude, behavior, skill, knowledge, or status that occurs for users after a purposeful action on the part of the library and library staff” (Dresang, Gross, & Holt, 2006, p. 3). When a school library uses the OBPE method of evaluation to examine how effective library programs, services, and resources are, they are more likely to modify, update, or replace “legacy programs” rather than simply providing the same resource and services they have grown accustomed to. “This legacy of success bias, however, is not only invalid as a legitimate evaluation of the service, it is also unpersuasive—and thus likely to jeopardize youth services resources when assessed against more legitimate data and actual analysis” Bernier, n.d., p. 1). Therefore, it is imperative that school library teachers use evaluation as a way of ensuring that the library is truly able to meet the needs of the entire school community, rather than relying on past practices. According to Dresang, Gross, & Holt (2006), evaluation is meant to inform library staff “about the level of success of the program or service being evaluated. After understanding the conclusions drawn from the data, the library staff should be willing to make changes that will improve success from the user’s point of view” (p. 55). The willingness to change is very important, as it is only with change that the program can grow and evolve to meet the developing needs of the school community.
PERFORMANCE-BASED EVALUATION
According to Keller (2018), before conducting performance-based evaluation, the school librarian must first identify the school’s overall mission, vision, and philosophy, determine what aspects of the school library program are to be examined, and why, as well as which standards will be used—building, district, state, or national (p. 17). The evaluation can be focused on either “learners, school library, or teacher librarian” (Keller, 2018, p. 18). After deciding what area of the library is to be evaluated, the library teacher must then decide what method of evaluation will be used. For instance:
“Observations
Checklists
Surveys
Interviews
Conferences
Journals
Reflection prompts” (Keller, 2018, p. 17).
Once it is determined which area(s) of the program will be evaluated, and which methods will be used, the evaluator will then perform the evaluations, analyze the data, and formulate a report that includes recommendations based on the results of the evaluation. The school librarian should not attempt this process alone, it is best to organize a team of evaluators that includes students, faculty, parents, and stakeholders. When determining what areas to assess during the evaluation, school librarians can look to the American Association of School Librarians (AASL) for insight. AASL suggests school libraries should base their evaluation using “these assessment categories:
Vision/mission
Teaching/collaboration, reading, multiple literacies, inquiry, assessment
Planning
Staffing
Learning Space
Budget
Policies
Collection
Information Access
Outreach
Professional development
Leadership
Relationships” (Keller, 2018, p. 19).
While the library teacher may pick and choose which areas to evaluate, every facet of the library program should be evaluated regularly in order to ensure the library is functioning effectively, and able to meet the needs of the entire school community. Connecting the evaluation process to the AASL National School Library Standards for Learners, School Librarians, and School Libraries, as well as the library’s state, district, and building standards, should help identify what successful outcomes should look like. The AASL has developed a checklist to help library teachers further in this process “elements incorporated in the checklist represent the school library alignments from the AASL Standards Integrated Framework that are evident in an effective school library” (Keller, 2018, p. 19). The checklist and framework “do overlap in some ways, but not exactly. Therefore, both tools should be used as is or adapted for evaluating the school library” (Keller, 2018, p. 19). One important distinction to mention is that the standards have purposely been updated to concentrate on competencies, rather than outcomes “competencies as defined in the national standards, are observable behaviors that can be assessed or measured” (Keller, 2018, p. 20). Focusing on competencies gives the library a benchmark for the outcomes the library should strive to have learners meet.
IMPACT-BASED EVALUATION
Impact-based evaluation is meant to assess “any effect of the service (or of an event of initiative) that contributes to change to an individual, group, or community” (Markless & Streatfield, 2017, p. 107). Impact-based evaluation further looks to define what an “effective” school library looks like, and “whether and how the students and teaching staff in the chosen schools were affected by their interactions with the library and its staff” (Markless & Streatfield, 2017, p. 107). Different from outcome and performance-based evaluation, impact-based evaluation asks whether the library’s services, programs, or resources were beneficial in attaining observable outcomes? In other words “is there evidence that the program helped to achieve or was part of what caused the outcomes of interest?” (Markless & Streatfield, 2017, p. 108). Impact evaluation has been evolving into “emergent evaluation” or “developmental evaluation” (Markless & Streatfield, 2017, p. 112) which is “not straightforward to predict where change will appear and therefore, where to focus the evaluation” (Markless & Streatfield, 2017, p. 112). In other words, the particular program, learning goals, and outcomes will predict how to conduct the evaluation, which should be a fluid model.
Before beginning the evaluation, the school librarian should determine what the impact objectives are “stating clear objectives is a vital step towards evaluating impact—if you are not sure what you are trying to achieve how can you tell when you are succeeding?” (Markless & Streatfield, 2017, p. 110). Impact evaluation also looks to delve deeper into the evaluation process, the goal is to assess the effectiveness of Information Literacy (IL) programs which requires long-term evaluation efforts such as interviews, focus groups, or other “longitudinal studies” (Markless & Streatfield, 2017, p. 111) in order to discover how much of an impact the library had on the student’s learning. “Finding out whether students have enhanced their understanding and are applying what they have learnt involves more than looking at lesson plans and learning objectives” (Markless & Streatfield, 2017, p. 111). Furthermore, impact evaluation calls for “more inclusive evaluation” (Markless & Streatfield, 2017, p. 112), meaning that the goal is to include a “wider range of stakeholders, including marginalized groups, and involving them in designing the evaluations and in interpreting the findings” (Markless & Streatfield, 2017, p. 112). By incorporating a more diverse and inclusive range of stakeholders in all aspects of the evaluation, the library can make sure it is focusing on the needs of the library users, rather than what the library teacher feels the evaluation should focus on.
Markless & Streatfield (2017), recommend posing these questions to aid in figuring out where to target the evaluation:
“Who are the evaluations for?
Who should decide what to evaluate and how?
Who owns the results and decides what to do with them?” (p. 112).
Using this method of evaluation and answering these questions, can help the school librarian approach the overall evaluation with a more critical eye, focusing on emergent impacts. This method also looks to include a “Theory of Change (ToC)” (Markless & Streatfield, 2017, p. 113), which is a “description and illustration of how and why a desired change is expected to happen” (Markless & Streatfield, 2017, p. 113).
Setting up a ToC begins with determining what goals the program hopes to achieve long-term, then establishing “all the conditions (outcomes) that must be in place (and how these interdepend on each other) for the goals to occur—or the specific differences you want to make and how you expect to get there” (Markless & Streatfield, 2017, p. 113). These goals can be outlined in an “Outcomes Framework'' (Markless & Streatfield, 2017, p. 113), which gives the library a starting point to work from. When evaluating IL in a school library using the ToC, the school librarian must decide what successful learning looks like for students, and whether students are using the skills they learned in the library in their assignments. This can be determined through an “understanding of the learning processes involved; organization of ongoing feedback, peer support and monitoring; and provision of resources in mixed formats that can be accessed by everyone” (Markless & Streatfield, 2017, p. 113). While utilizing these evaluation methods may take a considerable amount of effort, it is believed that this will prove “easier for library leaders and IL interventions to be seen as credible” (Markless & Streatfield, 2017, p. 114).
EVIDENCE
Info 285 Research Methods in Youth Services - Final Project
For this paper, we were required to conduct an analysis of a library program and any documentation or analysis the library had completed on that program, identifying both the library and the librarian in charge of the program. We needed to describe the program, then using the OBPE method, create a comprehensive assessment of the programs, its outcomes, and any changes the library may have made as a result of their findings (assuming they had evaluated their own program). Finally, we needed to include our own evaluations on the outcomes of the program and any recommendations to improve the program. This assignment shows my ability to critically evaluate a library program, analyze the results, and offer recommendations. It also shows my knowledge of conducting evaluations, and the importance of that task.
Info 266 Collection Management Presentation 4
For this assignment, we were required to evaluate a school library’s core collection. We then needed to choose an area of general emphasis to concentrate on within the collection, then narrow that section down further by dividing it up into four areas of special emphasis based on curriculum needs. Lastly, we needed to make recommendations for improving and updating both the overall collection, as well as the general and special emphasis portions we chose to focus on.
I chose to evaluate the high school library collection where I currently work. For the general emphasis, I chose History/Social Studies as that subject is a requirement for all students. For the special emphasis portions, I chose materials that support four separate History/Social Studies courses: Civics, History of Sports and Society, Holocaust & Genocide, and AP Government. This assignment shows that I am capable of evaluating a library’s collection, deciding how to focus the budget, and making recommendations to obtain the resources and materials needed to help learners achieve success. It must be noted, however, if this were a true evaluation of a library program, I would have included students, faculty, parents, and stakeholders in this process.
Info 285 Research Methods in Youth Services - Discussion #3
In this assignment, we were told to evaluate the weekly readings, finding one overarching ethical issue or concern made apparent through the content of the readings. We then needed to note how we would overcome that concern by sharing how we would conduct an evaluation of a youth service library program. The concern I detected was “barriers in obtaining authentic youth voices during evaluation” (McGillis, 2020, p. 1). This assignment shows I understand the process of conducting evaluations in the school library, as well as the importance of having both library users and non users involved. It also shows I am dedicated to actively listening to youth when it comes to evaluating the library, as they are the ones who would be directly affected by any changes to the programs, services, and resources provided by the library.
Info 285 Research Methods in Youth Services - Brief 2
This assignment required us to find, evaluate, and synthesize three different scholarly articles on youth research conducted in the 1980’s. Completing this assignment shows I am able to use scholarly sources in order to inform my evaluation methods and practices in the library. Looking at prior research is beneficial to a library program because it shows the history of evaluation and how it has impacted youth services. It also helps reinforce the importance of conducting evaluations in order to improve the services, programs, and resources the library provides, as well as to prove value to stakeholders.
CONCLUSION
In conclusion, evaluation of a school library’s program, resources, and services is crucial to the success of a school library “many stakeholder groups find evaluation to be invaluable” (Matthews, 2018, p. 256). A school library’s effectiveness is contingent on its ability to provide beneficial resources, materials, programs, and services to the entire school community. Without consistent evaluation of those resources, materials, programs, and services, the library has no way of ascertaining whether it is in fact meeting the needs of its community “ongoing assessment of how well libraries provide existing services, as a basis for improving those services and for developing new and innovative services that compete with other providers, will be essential for the long-term survival of libraries” (Matthews, 2018, p. 263). Keeping this in mind, I am committed to implementing regular and consistent evaluation of any library program I am a part of.
References
Bernier, A. (Presenter). (n.d.). Youth research with experiments [Reading].
Dresang, E. T., Gross, M., & Holt, L. E. (2006). Dynamic youth services through outcome-based planning and evaluation. American Library Assoc.
Keller, C. (2018). AASL standards for supporting library evaluation. Teacher Librarian, 45(5), 17-21. https://search-ebscohost-com.libaccess.sjlibrary.org/login.aspx?direct=true&db=lls&AN=130291695&site=ehost-live&scope=site.
Markless, S., & Streatfield, D. (2017). How can you tell if it's working? Recent developments in impact evaluation and their implications for information literacy practice. Journal of Information Literacy, 11(1), 106-119. https://doi.org/10.11645/11.1.2201
Matthews, J. R. (2018). Evaluation: An introduction to a crucial skill. In K. Haycock & M.-J. Romaniuk (Eds.), The portable MLIS: Insights from the experts (2nd ed., pp. 255-264). Libraries Unlimited, an imprint of ABC-CLIO.
McGillis, J. (2020, March 15). Info 285 discussion 3 [Unpublished working paper]. Jennifer McGillis. https://docs.google.com/document/d/16F_XUR2kLuseH7ljdpdPDYu8JS4UfsEQoYT8lAVrRE4/edit?usp=sharing