I am able to evaluate programs and services on specified criteria
Introduction
The purpose of library services and programs is to serve the information needs of users. But how do librarians know if that goal is being achieved? How do they know that patrons are receiving optimum services? These questions highlight the important role of assessment and evaluation to librarianship. In order for librarians to know that services are effective and users are satisfied, services and programs need to be assessed. Evaluation reveals the strengths and weaknesses of services. This information can then be used to guide service improvements. Evaluation helps the librarian fulfill her fundamental obligation to patrons to provide the best information services possible.
Evaluation plays another important role in libraries. As state and local budgets continue to shrink, taxpayers and their elected representatives are increasingly concerned with accountability. As Evans, Ward, and Rugaas (2000) explain, “People paying taxes to local government seek value for money from local taxes” (p. 242). Public organizations must demonstrate the value of their services or face the possibility of reduced budgets or, in extreme cases, closure. In this climate, quantitative data that demonstrates the value of services is vital to library survival. This data is gathered through evaluation.
Evaluation, then, is an essential component of library operations. It guides the librarian’s efforts to improve library services and can also be used to demonstrate the value of library services to government officials, funding agencies, and the public. Through my coursework in the SLIS program, I have used specific criteria to evaluate a library collection, reference interviews, and an information retrieval system. This experience demonstrates my ability to evaluate library programs and services using specified criteria.
Commentary
The library fulfills its mission to patrons primarily through its collection: the books, periodicals, and other materials maintained by the library in order to meet the information needs of users. Librarians evaluate collections in order to ensure that the collection is in fact supporting this mission. During my internship at San Jose State’s King Library I had the opportunity to evaluate the library’s Asian religions collection. A large part of the evaluation involved developing quantitative norms and standards. The library’s collection development policy described a “goal level” for the collection – a collection level adequate to meet the information needs of undergraduate students taking courses in the humanities. This goal level, however, was not defined in quantitative terms. The absence of a precise standard of evaluation made it difficult to assess the adequacy of the collection.
In order to resolve this problem, I had to look outside the King Library’s own collection development policy to develop quantitative norms. Specifically, I used standards associated with the WLN (Western Library Network) Conspectus Method. (For a summary of the WLN Conspectus Method, see Overview of the conspectus process on the University of Michigan-Flint Library’s web site.) The WLN Conspectus Method made it possible to specify the collection’s goal level as two quantitative standards: number of monographs in the collection and percentage of titles in major subject bibliographies. I was able to assess the strengths and weaknesses of the collection by comparing collection data with goal level standards. Evaluating the Asian religions collection gave me the opportunity to develop and apply specific, quantitative criteria in the evaluation of a library service.
In addition to a library’s collection, information retrieval systems (IRSs) – especially the catalog and article databases – serve an essential role in helping the library fulfill its mission to users. IRSs are what make it possible to identify and find items in the collection. Given their importance to libraries, librarians evaluate IRSs to determine their efficiency and effectiveness at retrieving information for users.
In LIBR 202 I used specific criteria to evaluate an information retrieval system. The IRS was a database of bibliographic records created as a group project for the course. The IRS was evaluated using two criteria: recall and precision. Recall refers to the number of relevant items in a retrieved set as a percentage of the total number of relevant items in the database. Precision refers to the number of relevant items in a retrieved set as a percentage of the total number of items in the set.
Measures of recall and precision alone are not adequate to evaluate an IRS. An IRS, for example, may achieve (on average) 50% recall. Is this good performance, or is it merely adequate or even unsatisfactory? Evaluating an IRS based on recall and precision requires an additional step: establishing normative standards for each value. In other words, it requires establishing values for “good recall/precision” as opposed to “satisfactory recall/precision” or “poor recall/precision.” Unfortunately, there is no consensus among librarians regarding such normative values. Ideally, an IRS would achieve 100% recall and precision (i.e., retrieve all relevant records and nothing but relevant records). However, this is an unrealistic goal. Normative standards for IRS evaluation need to be based on more modest expectations.
In order to evaluate the database I needed to establish normative standards for recall and precision. I defined those standards based on average recall and precision ratios for other IRS as reported in Tonta (1992). Because these ratios were based on searches using controlled vocabularies, I adjusted my standards for evaluating natural language searches. Having defined normative standards for recall and precision in both natural language and controlled vocabulary fields, I was then able to evaluate IRS performance based on an analysis of retrieved sets. Interpreting my results included an evaluation of the controlled vocabularies used to index article records as well as the quality of the indexing itself (i.e., how consistently and accurately were subject terms applied to records by the indexers).
Reference services are another key way librarians fulfill their mission to users. Reference services give librarians an opportunity to serve patrons directly. In order to ensure that reference services do in fact meet the information needs of users, librarians have developed criteria for the evaluation of reference services. In LIBR 210 I had the opportunity to apply criteria in the evaluation of two reference interactions: a face-to-face reference interview and a virtual reference interaction. I analyzed and evaluated both interactions based on two criteria: completeness and quality. Completeness was evaluated with reference to the five steps of a reference interview as described by Bopp (2001): “open the interview,” “negotiate the question,” “search for information,” “communicate the information to the user,” and “close the interview” (p. 48). A complete interview would include all five steps; an incomplete interview would lack one or more steps.
Quality was assessed in two ways. The first was an informal analysis based on Bopp’s (2001) description of the five steps. As Bopp explains, each step should include certain elements. For example, does the librarian open the interview with a smile, a friendly greeting, and body language that communicates attentiveness? If the patron asks a non-factual question, does the librarian use open-ended questions when negotiating the question? I was able to informally evaluate the quality of each interview based on the presence or absence of these different elements.
In order to assess the quality of each interview, I also employed the formal evaluation criteria developed by the Reference and User Services Association (RUSA). These Guidelines for Behavioral Performance of Reference and Information Service Providers are organized within five categories (approachability, interest, listening/inquiring, searching, and follow-up) and specify a list of features that characterize effective reference transactions, e.g., “Is poised and ready to engage approaching patrons” (1.2), “Maintains or re-establishes eye contact with patrons throughout the transaction” (2.4), and “Constructs a competent and complete search strategy” (4.2) (Reference and User Services Association, 2009). The presence or absence of features in a reference interaction is a measure of its quality. I used these criteria to evaluate both face-to-face and digital reference interviews. I also addressed the advantages and disadvantages of digital reference in comparison to face-to-face reference interviews.
I have focused on a standards-based approach to evaluation, but library services may also be evaluated through comparison to benchmarks. In this approach, performance data from a high-performing institution – or data averaged across a number of organizations – sets a standard for the performance of comparable institutions. For example, a librarian evaluating her collection might compare the size and coverage of her collection to the collections of comparable libraries. Meeting or exceeding the benchmark would be an indication of quality. Falling short of the benchmark suggests the need to improve the collection.
Librarians may also take a more user-centered approach to evaluation. In this case, the performance of a program or service is evaluated based on users’ perception of services. This approach relies on user surveys that measure levels of user satisfaction with library services, programs, and/or more general characteristics of the library (such as the courtesy of library staff). Survey questions may also gather data concerning user expectations of services. The LibQUAL survey is perhaps the most prominent example of this approach to evaluation.
Since the purpose of library services is to serve users, some argue that the user’s perception is all that matters when evaluating a service. From this perspective, patron satisfaction is directly correlated with effective, quality services. While few (if anyone) would dispute the importance of user satisfaction as a measure of performance, there are problems with this approach. As Evans et al. (2000) point out, “A person may be satisfied with information provided by an information professional and not be aware the information is incomplete, or perhaps even incorrect” (p. 253). Examples like this suggest the need for a multi-faceted approach to service evaluation, i.e., one that combines a user-centered approach with more objective standards or criteria.
Evidence
I am submitting two assignments as evidence of my mastery of this competency. The first is the collection evaluation I created for my internship at San Jose State’s King Library. (I also submitted this assignment as evidence for Competency F.) In the assignment I describe specific criteria used to evaluate library collections and explain how I used those criteria to evaluate the Asian religions collection at the King Library. The assignment demonstrates my ability to apply specific criteria in the evaluation of a library collection.
The second assignment is an analysis and evaluation of a face-to-face reference interview. In the paper I describe a specific reference interaction and then evaluate the librarian’s performance based on Bopp’s (2001) description of the steps of a reference interview and the RUSA Guidelines for Behavioral Performance of Reference and Information Service Providers. The paper demonstrates an understanding of criteria used to evaluate reference transactions and an ability to apply those criteria to a specific reference interview.
Conclusion
For the librarian committed to providing excellent information services and programs, providing the service is not enough. The librarian needs to know if the service is effective. She needs to know if the service or program is satisfying the needs of users. More specifically, she needs to understand its strengths and weaknesses in order to implement changes and enhance service performance. This information is gathered through assessment and evaluation. Evaluation, then, is essential to meeting the information needs of users. Through my SLIS coursework I have gained an appreciation for its importance as well the skills needed to evaluate library services and programs using specified criteria.
References
Reference and User Services Association. (2009). Guidelines for Behavioral Performance of Reference and Information Service Providers. Retrieved November 2, 2009, from http://www.pla.org/ala/mgrps/divs/rusa/resources/guidelines/guidelinesbehavioral.cfm
Bopp R. E. (2001). The reference interview. In R. E. Bopp & L. C. Smith (Eds.), Reference and information services: An introduction (3rd ed., pp. 47-68). Englewood, CO: Libraries Unlimited.
Evans, G. E., Ward, P. L., & Rugaas, B. (2000). Management basics for information professionals. New York: Neal-Schuman.
Tonta, Y. (1992). Analysis of search failures in document retrieval systems: A review. The Public-Access Computer Systems Review, 3(1), 4-53. Retrieved November 7, 2009, from http://epress.lib.uh.edu/pr/v3/n1/tonta.3n1