Evaluate programs and services using measurable criteria
Introduction
Evaluation is an important component of the LIS profession. Using measurable criteria, librarians evaluate services, programs, and resources to ensure they are meeting the needs of their communities. Over the past decade, the interest in library assessment and evaluation has expanded, particularly in providing evidence and context for organizational, operational, and strategic planning and decision-making (Bakkalbasi, 2017). Librarians are held accountable to ensure services are being offered effectively, efficiently, and in a cost-effective manner (Matthews, 2018). Matthews (2018) defines evaluation as “the process of determining the results, costs, outcomes, impacts, successes, and other factors related to a library’s collections, programs, services, or resources used” (p. 255). Evaluation is useful for planning and delivering library services as the foundation for improvement (Matthews, 2018). By using specific measurement tools and criteria, librarians can evaluate and assess specific library components including the physical space, the website, instruction, reference services, and the collection. Librarians need to be knowledgeable of evaluation and assessment and how it drives change in the library.
Evaluation
The library must constantly evolve to meet the needs of their communities to remain relevant and competitive. To do this, they need to consistently evaluate and assess library operations to prevent failure (Matthews, 2018). Evaluation and assessment are often discussed together, such as Bakkalbasi’s (2017) chapter Assessment and Evaluation, Promotion, and Marketing of Academic Library Services referenced below. There is a strong importance with assessing and evaluating library outcomes and services. Increasing importance has been given to advocating for funding and resources, particularly at academic libraries. One of the best ways to advocate for the library is to tie services, programs, and resources to the effectiveness of the library on student success and retention (Bakkalbasi, 2017). To promote library effectiveness with institutional stakeholders, librarians need to participate in evaluation and assessment processes continuously (Bakkalbasi, 2017). A key component of evaluation and assessment is tying the process to ensuring services and resources are in alignment with library and institutional goals, missions, visions, values, and strategic planning (Bakkalbasi, 2017; Matthews, 2018).
Evaluation and assessment cycles are continuous and include developing a question or objective of the evaluation process, identifying evaluation measurements and criteria, analyzing and interpreting the data in terms of the goal or objective, integrating these results into operational improvements and strategic directions for the library, and disseminating the results to library staff, institutional stakeholders, and the greater LIS profession (Bakkalbasi, 2017). Disseminating the evaluation and assessment data is one of the most important parts of the cycle, not only does it allow the greater LIS professional community to work from these findings, but it places the library and institution in a good light to be proactively working for change (Matthews, 2018). Evaluations have many internal and external benefits including providing new input for developing new services, informing governing boards of community needs, determining how and if service objectives have been reached, comparing costs of services to each other to ensure cost efficiency, assisting library management to determine the level of staff training, and encourages the library to think about outcomes and impacts in relation to library users. In order to provide this data, it is important for librarians to know about evaluation criteria and what is recommended by library associations and organizations to achieve success in this area.
Measurable Criteria
Depending on the service, program, or resource being evaluated there is assessment criteria to consider and tools to use to gather important metrics. Specific assessment and evaluation criteria to ensure library success includes ensuring it is extensive, efficient, effective, a high quality of service, impactful, useful, and results in a change of knowledge and understanding skills or behavior (Matthews, 2018). There are many services librarians can evaluate and assess to see if they are meeting the criteria I just listed including reference, instruction, library spaces, and other library services. Assessing and evaluating reference is typically done by following the Reference User and Services Association (RUSA) guidelines which is used to assist libraries in identifying and recognizing observable behavioral attributes that could be correlated with positive patron interactions in five key areas including visibility and approachability, interest, listening and inquiring, searching, and follow-up (ALA, 2008). When assessing the instruction process, there are many tools to use depending on the type of library. For academic libraries, many rubric and assessment tools have been used such as the Association of College and Research Libraries (ACRL) Information Literacy Competency Standards, the Association of American Colleges and Universities (AACU) Information Literacy Valid Assessment of Learning in Undergraduate Education (VALUE) Rubric, and ACRL’s more recent Framework for Information Literacy for Higher Education (Becker, 2022). These methods all aim to get the best data that determines the effectiveness of the instruction on student learning and their ability level with information literacy skills (Becker, 2022).
One large area where libraries evaluate and assess effectiveness is in the library spaces, both in person and online. User Experience (UX) is a newer but critical assessment pathway to assess how library users are interacting with library touchpoints. The goal of UX is to “craft systematic library experiences designed to deliver totality” (McDonald, 2022, p.192). UX uses various tools such as usability testing, touchpoint analysis, journey mapping, and conducting a contextual inquiry to gather information on the usability, accessibility, and effectiveness of library spaces (Schmidt, 2023). Libraries also use typical quantitative and qualitative methods and tools such as surveys, focus groups, and more for understanding the user perspective (Matthews, 2018). Some organizations have created tools that allow libraries to more easily use these research methods to evaluate user perspectives on their services, programs, and resources. These tools include the International Organization for Standardization (ISO) performance indicators, American Research (ARL) Libraries Statistics and Assessment Program’s LibQUAL+ and ClimateQUAL+ to automate a survey process of gathering information (Bakkalbasi, 2017). Libraries even use automated tools to gather statistical data about the use of online resources such as with Project COUNTER (Counting Online Usage of Networked Electronic Resources) and MINES for Libraries which is a survey tool that collects data on the purpose and use of electronic resources (Bakkalbasi, 2017).
Regardless of the criteria and tools used, it is important to consistently evaluate and assess library services, spaces, programs, and resources to ensure community needs are met and that the library is working to meet, follow, and improve strategic initiatives of the library and their parent institution. Improvement and funding can only come from data-driven information and support demonstrating the value of libraries and their success in uplifting the community.
Evidence
Info 210 Discussion #3 - Evaluating Virtual Reference Services
My first piece of evidence is a discussion post for Info 210 Reference and Information Services. The purpose of this discussion was to identify a library that offered virtual reference services and evaluate the reference transaction according to how it met the criteria laid out by the RUSA guidelines. I chose to assess the virtual chat reference offered by College of Southern Nevada (CSN) Libraries to see how effective an online transaction would be at meeting my reference needs. In this evaluation process, I reviewed how the reference transaction fared in the five main areas of visibility and approachability, interest, listening and inquiring, searching, and follow-up. This evidence shows my knowledge and understanding of evaluating a library service using assessment criteria and tools, in this instance the RUSA guidelines.
My second piece of evidence is a white paper titled Assessment Programs in Community College Libraries for Info 230 Issues in Academic Libraries. This paper focuses on why assessment planning is important, particularly in the context of how libraries can advocate for themselves and gain additional funding when they are able to provide data-driven information to institutional stakeholders about the tie between library services and student success. This paper provides an introduction on the importance of assessment, a background on the history and use of assessment in libraries, and a proposal on how community college libraries can improve their services and implement data-driven change through assessment. This paper also includes a recommendation for the types of assessment tools to use when evaluating services and programs including ALA’s LibQUAL+ survey tool. This evidence shows my knowledge of the evaluation and assessment process and the importance of continuously assessing service, programs, and resources to ensure library success and the importance of showcasing library value to institutional stakeholders through assessment tools.
Info 254 - VALUE Rubric Assignment
My third piece of evidence is an assignment created for Info 254 Information Literacy and Learning. In this assignment, we were to assess four student research papers and determine the student’s level of information literacy knowledge. We used AACU’s Value Rubric to assess different areas of the paper to determine a student’s information literacy knowledge. Areas of assessment and evaluation included on a scale of 0-3 how well the student determined the extent of information needed, accessed the needed information, evaluated the information and its sources critically, used the information effectively to accomplish a specific purpose, and how they accessed and used the information ethically and legally. This evidence shows my knowledge of evaluating instruction and information literacy sessions and gave me a chance to evaluate real student’s work and how they processed information literacy instruction using an assessment rubric.
My fourth piece of evidence is a usability test I conducted for Info 287 User Experience. In this assignment, we were instructed to conduct a usability test on a library’s website. I chose to test the CSN Libraries website. For this assignment, we had to develop a script for how we would conduct the test and how we would instruct a user to move about the website, we had to create three scenarios in which we wanted the users to navigate the site and find the required information, and then we had to conduct the test, and report our findings. The goal of a usability test is to see if the website is intuitive enough for users to find the correct and needed information, and to physically observe users while they are doing it to see how they navigate the website and what pathways work for them. Typically usability testing is done soon after the first stage of website completion, and once usability testing is completed, the designer can edit the site to reflect the needed changes identified through evaluation and assessment. This piece of evidence shows my knowledge of evaluation and assessment tools such as user experience tools to understand the usability, effectiveness, and accessibility of a library’s website.
Conclusion
As a future academic librarian, I will continue to use my knowledge and understanding of evaluation and assessment to continue to assess and improve service, programs, spaces, and resources. In my current position in my academic library, I am on many library committees or teams that work on assessing and evaluating services, programs, and instruction, and plan to stay on them to better serve our library community. I plan to stay on top of new information regarding evaluation and assessment through professional development with webinars and conferences developed by ACRL and ALA.
References
American Library Association. (2008). RUSA guidelines for behavioral performance of reference and information service providers. ALA. http://www.ala.org/rusa/resources/guidelines/guidelinesbehavioral
Bakkalbasi, N. (2017). Assessment and evaluation, promotion, and marketing of academic library services. In T. Gilman (Ed.), American Librarianship Today (pp. 211-221). Rowman & Littlefield.
Becker, B. (2022). Teaching users: Information and technology literacy instruction. In S. Hirsh (Ed.), Information Services Today (3rd ed., pp. 216-228). Rowman & Littlefield.
Matthews, J.R. (2018). Evaluation: An introduction to a crucial skill. In K. Haycock & M.J. Romaniuk (Eds.), The Portable MLIS. (2nd ed., pp. 255-264). Libraries Unlimited.
McDonald, C. (2022). User experience. In S. Hirsh (Ed.), Information Services Today (3rd ed., pp. 192-202). Rowman & Littlefield.
Schmidt, A. (2023). Week 1 [lecture notes]. SJSU Canvas.