Competency Description and Scope:

User research is critical for libraries. The community's needs cannot be served if the library is unaware of them. Simply put, if the library does not know what its patrons need, it cannot provide for them. By using evaluation as a kind of reflection, LIS professionals can better respond to people's needs in the community. Results will give visual proof to illustrate the value of libraries to the user community, as discussed by Susan Alman in her chapter on communication, marketing, and outreach techniques (Hirsh, 2015, pp. 374–386). Budgets are another important factor to consider. The budget must be updated every fiscal year to pay the costs of programs and services, as well as collections, facility upkeep, technology, and workers. Advocacy efforts will be strengthened by proof of research methodologies that explain user needs (Hirsh, 2015, p. 374–386). For this competency, the goal of understanding the fundamentals of evaluation may feel overwhelming; however, a well-done evaluation serves internal library needs by assisting the library in achieving new goals or achievements, establishing a positive external presence in the public eye, allocating scarce resources to where they are most effective, providing a better understanding of their patrons, and, most importantly, serving community needs.

Evaluation Steps

Evaluation is an organized procedure with distinct phases. A "typical" assessment strategy is shown below using a logic model. In competency L, where I describe outcome-based planning and evaluation (OBPE), I briefly cover program assessment. However, in this competency, I want to focus on establishing a logic model that is relevant to diverse groups and where the evaluation process is adaptable to different experiences. In other words, because no two groups will approach evaluation measures the same, there should be a straight-forward procedure. I believe OBPE applies to committees and other groups whose members are usually on the same page or division. For example, in my IMPACT projects, I am not always paired with people from my same division and it's always approached differently. However, I noticed that the basic logical steps are the same. Thus, I will approach this competency as I would with various people and their diverse experiences (i.e., those not in the LIS program). This logic model is based on the W.K. Kellogg Foundation's "Logical Model Development Guide" (2006). It displayed proactive and outcome-oriented results with realistic practice, in my opinion.

Creating a logic model
Step 1: Plan the evaluation with questions and indicators such as the focus area, the audience, questions, information use, and technical assistance.
Step 2: Design instruments that best fit the data that needs to be collected.
Step 3: Collect data and consider sample size, area, and how the instrument will be distributed.
Step 4: Enter and clean the gathered data after the response or collection period ends.
Step 5: Analyze the data with software tools.
Step 6: Interpret the results with the team.
Step 7: Create the report with any notable questions or address any problems.
Step 8: Reflect and communicate with the team to present to the audience.

A logic model is essentially a comprehensive, graphical representation of a program. It defines the program's desired results as well as the route the program will follow to accomplish those outcomes. Additionally, it is a dynamic document; it should evolve in tandem with what the team finds out. Developing a logic model is an element of good program design, not evaluation. I include this stage since developing a good design and strategy for the team can save time and avoid stressful situations. Although logic models are to support program planning, creating a logic model will increase the quality of the evaluation significantly (Rossi and Lipsey, 2019):

• Creating a language that is understood by all members of the team.

• Specifying the program's underlying assumptions.

• Increasing communication while also fostering clarity and transparency.

• Contributing to the ongoing enhancement of quality.

• Ensuring that objectives, actions, and results are all aligned with one another.

In the following sections, I will discuss in further detail about the mindset of evaluation, the many types of evaluation, the instruments that are commonly used for evaluation, and finally, the process of producing evaluation data, as well as evaluating and interpreting that data as a team.

Getting into the Evaluation Mindset

Evaluating anything requires a certain frame of mind in addition to being a logical procedure. It is far more important to understand how to think critically about evaluations than it is to master the specific methods involved in carrying out a formal evaluation. An evaluation mindset is similar to navigation in that it uses the environment, guides, maps, and data to contextualize the whole trip and determine whether one is on the correct track. A mindset for evaluation is to be curious about how things work, to think critically, to reflect often, and to question basic ideas and assumptions. It takes humility as well as confidence in the task and openness to try again. Students in LIS will benefit from developing their critical thinking skills, as it will make them better leaders, builders, and achievers of certain goals.

Types of Evaluation

There is no one correct or clear method for constructing an evaluation. At certain phases of a program, specific sorts of assessments are advantageous. Each of these categories covers unique issues and may use distinct techniques. The three types of evaluation I will discuss are front-end, formative, and summative (or impact) evaluation. The evaluation will be significantly better when one aligns the outcome with the method:

Front-End Evaluation: When contemplating the development of a program or attempting to get a better understanding of your community or its members.
Questions it answers:
What does the audience need?
What concerns do they have?
What are they excited about?
What program is best for this audience and these outcomes?
What is the audience’s incoming knowledge and understanding?

Formative Evaluation: At the beginning of your program or when you are in the process of putting it into action.
Questions it answers:
Will this work?
Is it working?
How might we improve it?
Is it meeting our audiences’ needs?

Summative (or Impact) Evaluation: After your program has been put into action and is functioning effectively enough to evaluate its effects.
Questions it answers:
What impact did our program have with stakeholders?
Did our efforts achieve what we hoped to achieve?
Did we have our desired outcomes?

Instruments used for Evaluation

Data collection instruments might take the form of a survey, an interview form, a feedback board, or any other tool that facilitates data collection. Research methodologies such as action research, bibliometrics, case studies, content analysis, correlational research, ethnography, historical research, and surveys are a few examples of some additional methods (ALA). When it comes to data collection, one of the most common forms is the survey. A term I often read about in literature are the terms "validity" and "reliability". Both terms are important when it comes to research and assessment. A "valid" instrument has been shown to be able to measure what it is meant to measure, and a "reliable" instrument will keep its level of accuracy over time and in different settings. For example, a test that looks at whether or not a person likes math is not a good way to tell if that person would pursue a STEM field. Instead, it is better to utilize closed-ended questions to gather and evaluate information more effectively. These may be multiple-choice questions with pre-determined responses or rating scales. Incorporating meaningful response categories for gender, color, ethnicity, wealth, and other personal inquiries will further communicate how much the researcher values them. Open-ended questions require more work to respond to and interpret, so they should be used rarely. When one does not know what replies will be provided or when extensive solutions are required, open-ended inquiries are preferable. Evaluation tools that seem like a natural extension of the program will elicit a more positive reaction (Kolderup, F.T., 2008).

After that, after all of the data has been inputted and cleaned up, the team will be prepared to do an analysis on it. In this case, "analyze" refers to the process of condensing and representing the data, as well as maybe conducting statistical tests. The majority of the time, the data being evaluated will need descriptive statistics, such as percentages, to illustrate groupings. If demographic data are available, it is typically easier to begin there. Before doing more analysis on qualitative data, it will likely be necessary to "code" it too. For example, creating a set of categories to summarize the replies, labeling the responses with the categories, and then tabulating the number of responses in each category constitutes this procedure. As part of the coding process, it will be important to develop a "code book" with a description of the categories and samples of replies that suit each category (Osborne & Nakamura, 2000).

Interpretation and Report of the Data
During the analysis phase, I typically collect and summarize the data. While these summaries are essential, the context is still required. The next stage is to determine the significance of the data. Every effective assessment must contextualize the data and explain to the stakeholders what questions the data answers and how to utilize the data. Frequently, the why is the most crucial aspect of interpretation. The data may not immediately explain why a given event occurred the way it did. There are a number of other concerns that should be addressed during interpretation, such as whether or not the findings corroborate or contradict those of prior research, how the findings compare to staff expectations, and whether or not the findings suggest any unprecedented challenges. It is important that the findings address the future of the information institution.

An evaluation report has at least three basic sections: one describing the project, one describing the evaluation, and one describing the outcomes. Other than the main body of the report, it may also include a table of contents, a list of attachments, a logic model, and suggestions for future action, among other things. Funders, partners, objectives, target populations, projected timelines, and planned activities are all fair game when outlining the initiatives (Hirsh, 2022). Sample, methods, the manner in which the data were collected, the time at which the data were collected, the number of respondents, the response rate (if appropriate), and any potential biases may all be included in the description of the evaluation design. (Kolderup, F.T., 2008) Finally, the explanation of the findings may include a summary of the outcomes, conclusions, and suggestions, as well as information about when the data was acquired and any potential biases. Even if they're just a line or two long, these additions to the report guarantee that the results are given in context and help the reader better understand them.

Concluding Thoughts

Everyone goes through nearly the same logical steps when evaluating, but the evaluations are not all done the same and the strategies vary depending on the team. It is important that data be stated and easy to measure so that it is possible to analyze and improve the desired focus. Evaluation also enables the library to establish if its goals are being met and to discover how it may become more effective. I discovered that data can be utilized to tell a narrative and is founded on quantifiable criteria that highlight crucial results. I also discovered that the data is based on criteria that can be measured. This competency will be completed by presenting and discussing various programs and services in terms of how they indicate mastery.

Importance to the Profession:

Libraries must justify their activities and services by describing what they are doing and why, and by showing how they have effectively enriched the lives of their patrons (Hernon & Schwatrz, 2015). The goal of collecting data frequently (Rossi et al., 2019, p. 291-306) is to motivate libraries in enhancing a program, strengthening library advocacy, comprehending a program's impacts, and guiding decisions about the program's financing or structure. Different procedures and methods may be necessary for each assessment, making each evaluation unique. The evaluation process is influenced by the objective of the evaluation, the conceptual and organizational structure of the program being assessed, and the available resources (p. 31–91). By basing their planning and decision-making processes on factual information, libraries are better able to demonstrate their worth to the community and increase the quantity and quality of the services they provide (Hernon et al., 2014).

Personally, I have found so much value when it comes to working with groups not in the same field. I was offered perspectives I would have not noted if I was working alone. My first evidence piece had been uploaded to competency E, but I wanted to further explain our process because of how much the team process taught me. Our meetings were often more than an hour and most of the time dedicated to how we wanted to evaluate our project. I had been used to instantly collecting data and never understood why it made me feel overwhelmed after. When we had a logical approach to our process and then explored the reasoning, it did sincerely save a lot of time. For my second evidence, I used a well-known guideline to evaluate important factors when it came to library service. Lastly, I was bit hesitant to share this piece because it is still a work in progress, but the process was challenging and a great lesson when I realized my team members were not in my department nor had any idea of what it meant to evaluate a service. I know it is still in the midst of development, I want to highlight how far my team had come when trying to balance out the experience field. Their experience is very important and I think as long as there is a logical model to follow, anyone can pick up how to approach the evaluation process.

Preparation to Understand Competency N

When I first started creating evaluation processes for both my MLIS courses and in my professional work, I always felt overwhelmed. Much of this probably came from my inexperience with logical models, an evaluation mindset, and an overwhelming amount of collected data. I had a better idea of where I was at in the evaluation process when I learned about research methods. I mentioned in Competency L the assortment of both qualitative and quantitative methods for measuring user satisfaction, economic impact, and social impact in library programs and services. On the qualitative side, open-ended comments or suggestions such as interviews, surveys, questionnaires, and focus groups These research methods provided a professional and tangible performance objective that I did not know where to apply until I had a fundamental understanding of an evaluation strategy plan and the content within. I had to better my planning process in order to better gauge the perceived value of particular stages of planning. As a result of SJSU, I am now able to assess the quality of information presented to me. I now understand how to establish new programs and how to build up assessment criteria for existing ones based on an institution's purpose and vision statements. When conducted correctly, evaluations provide valuable insights for a wide range of stakeholders, including internal groups like workers and administrators, as well as external groups or grants.