Albertsons Library Access Services Evaluation

Evaluators: Mary Ly-Nguyen, Halah Mohammed, Amy Pallotta

Background

Organization

Albertsons Library is a public entity located in the Pacific Northwest and operates on the main campus of Boise State University (BSU). Founded in 1932, Albertsons Library comprises nine units and serves a community of more than 27,000 individuals. It is the established hub for learning and information sharing and represents BSU’s vision of innovation, global impact, and equitable education.

Program and Stakeholders

Access Services is one of the primary units of Albertsons Library and serves as the first point of contact for all library users. It is dedicated to delivering an expansive and diverse database of online and printed media and resources. It also offers services like computer labs, printing, scanning, and collaborative study spaces.

The staff consists of permanent employees, temporary employees, student employees, and computer lab student employees.

Three types of groups were involved in this evaluation: upstream stakeholders, direct impactees, and indirect impactees. The upstream stakeholders included executive staff at the library. The direct impactees included a combination of permanent employees, temporary employees and student employees. All other library patrons represented the indirect impactee group.

Evaluation Request

In fall 2022, Dr. Connie Chyung connected a team of students to the head of Library Access Services, Shelly Dotty, as part of a semester-long project for the OPWL 530 evaluation course. Shelly requested an evaluation for her unit and is the primary client for the project. The evaluation team consisted of three BSU graduate students from the Organizational Performance and Workplace Learning (OPWL) program – Halah Mohammed, Mary Ly-Nguyen, and Amy Pallotta. Dr. Chyung, the course instructor, served as the formal advisor for the team. She provided professional feedback and input throughout the evaluation to ensure that the team was conducting rigorous research, exercising professional ethics and meeting the OPWL program learning goals.

Evaluation Methodology

Evaluation Purpose and Type

The team selected Chyung’s (2019) 10-step evaluation procedure as a guide to systematically complete the evaluation. The procedure includes three phases: identification, planning, and implementation. Using an evidence-based approach helped ensure that the findings were valid and reliable. 

Based on multiple conversations conducted through virtual meetings and emails with the client, the evaluation team established two primary purposes for conducting the evaluation: (1) assess the services provided by the Access Services unit and (2) how well those services are delivered given the current staffing model. 

The team decided that a formative evaluation could best address the goals of the project through its identification of strengths and weaknesses of the services. The results from the assessment could then provide insights as to how the unit can improve its services and enhance the overall experience for its users.

Dimensions and Data Collection Method

It was determined that the Kellogg's program logic model (PLM) would be the most appropriate tool for identifying the critical functions of the unit, narrowing variables to measure, uncovering assumptions, and establishing how to address those assumptions. Using the model (illustrated in Appendix A), the team identified two key dimensions and their importance weighting to focus their research on. Those key dimensions were:

1. Staffing system efficiency/tools (Critical)

2. Policies, procedures, processes, standards (High)

The team used multiple sources of data from different classifications of Access Services employees: the manager, permanent employees, temporary employees, and student employees. The team also used three types of data collection methods: 

• Extant data: Documentation on expectations, standard operating procedures, schedules, calendars, and task distribution lists.

• Survey: Anonymous differentiated web-based survey

• Interview: Semi-structured Zoom interviews with current employees 

Due to an insufficient number of participants for one-on-one interviews, the team decided not to include the interview data in the evaluation; therefore, they focused on reporting extant data and surveys.

Evaluation Results

The team analyzed collected data using three types of four-level rubrics. Then, they triangulated the results from the individual rubrics into a three-level rubric with the following labels: Doing Well, Needs Some Attention, and Needs Critical Attention. The overall results are summarized in Table 1.

Results from Staffing System Efficiency/Tools Dimension

An analysis of the Staffing System Efficiency/Tools dimension revealed that approximately 42% of employees rate the current staffing level and staffing system efficiency as “About right” and “Somewhat over-staffed”. Approximately 15% of employees rate the current staffing level and staffing system efficiency as “Extremely short-staffed”. However, the response distributions were different between the two employee groups:

• 57% of student employees rate the current staffing level and staffing system efficiency as “About right” and 43% of student employees rate it as “Somewhat over-staffed”.

• None of the full-time employees rate the current staffing level and staffing system efficiency as “about right”. 60% of them rate the current staffing level and staffing system efficiency as “Extremely short-staffed” and 40% of them rate it as “Somewhat over-staffed”.

The data reflected different perspectives and potentially distinct experiences from the two groups. As a result, the team has determined that staffing system efficiency and tools “need some attention”.

Results from Policies, Procedures, Processes, and Standards Dimension

For the Policies, Procedures, Processes, and Standards dimension, the team used data from the administered survey and reviewed extant data and documentation.  The scoring of employees’ survey responses to questions about policies, procedures, processes, and standards revealed that: 

• The overall average score for both employee groups is 3.6. This means that on average both employee groups perceive this dimension to be in ”good standing”. Both employee groups show similar average scores.

The extant data, and documentation on policies, procedures, processes, and standards were of excellent and good standards, meaning the documentation was mostly clear, concise, and organized. The overall rating of the documentation is “excellent”; however, there are areas that can be improved upon (the team’s suggestions are listed in the recommendation section of the full report). Altogether, through rubric triangulation, the team determined that the policies, procedures, processes, and standards dimension is “Doing Well”. 

Conclusions

The overall results of the evaluation indicated that the efficiency of staffing systems and tools needs some attention while policies, procedures, processes, and standards are doing well. This information suggested that some areas were doing well and some areas needed improvement. These conclusions were reached by (1) collecting survey data, (2) reviewing extant data, (3) scoring data using a rubric, and (4) triangulation of the data.

To explore the results in greater detail, the team suggested that Access Services conduct a task analysis and a needs assessment. 

• A task analysis can help staff understand what they specifically need to do to perform with maximum efficiency. To conduct one, Access Services can form a team to prioritize tasks to analyze and observe employee performance. 

• A needs assessment can help Access Services identify strengths and challenges. This information helps them understand where they are currently and to develop a plan to get them where they need to be. 

Additionally, the team recommended the unit review, organize, and prioritize important documents to improve the existing knowledge repository. 

Limitations

The limitations experienced during the project were primarily related to time and  availability of data:

• Surveys were administered and collected over a three-week period. A delay in identifying survey errors and time spent updating and sending out a new survey affected the survey participation rate.

• The team was not able to generate sufficient interest and participation in interviews within the project’s timeframe. To overcome project limitations, the team connected with Dr. Chyung for professional input and based all decisions on team discussions and shared agreement.

To overcome project limitations, the team connected with Dr. Chyung for professional input and based all decisions on team discussions and shared agreement.