A Needs Assessment in Higher Education

Background

Organization

Online U (a pseudonym), provides accredited, higher education opportunities to people around the world by reducing the barriers of entry and completion in receiving a college education. The program serves over 20,000 students in over 150 countries and aims to improve lives through access to spiritually based, online affordable higher education.

One way the organization achieves its mission is by managing a robust network of service volunteers who serve as academic and spiritual support for the students enrolled at Online U. Service volunteers primarily focus on student success by overseeing weekly study groups and providing mentoring, encouragement, and coaching. Service volunteers are led and trained by Area Managers (paid positions) and Area Coordinating Mentors (ACMs, unpaid volunteers). To best organize its training operations in the field, Online U divides the globe into 21 areas which are overseen by 14 Area Managers and 1-2 ACM’s per area. While service volunteers traditionally serve 2 years terms, ACMs are long lasting volunteers who have years of dedication and commitment to serving the organization in a volunteer capacity. The organization operates in a hierarchical manner in which communication and training is delivered from the top down. This project focuses on the ACM level within the organizational chart shown in Figure 1.

Figure 1: Online U Organizational Chart

Challenge

Online U has experienced rapid enrollment growth over the last ten years. As student enrollment has increased, so has the need for volunteers. In 2021, Online U saw a stark increase in both enrollment and volunteers, causing operational challenges and opportunities for the organization to restructure. Various levels of service volunteers were created to streamline communication and training as described in Appendix A, page 2. The Latin American Area Director, and primary stakeholder, discussed challenges with recent leadership changes amongst the Area Managers and inconsistencies regarding job expectations for the ACMs. Concerns were expressed that ACMs are not regularly attending the required weekly Area Coordination Council (ACC) meetings. It was discovered that weekly meetings contain imperative training and knowledge to help ACMs perform their job duties and better support the service volunteers and students in which they oversee.

Performance Gap

A significant gap exists between desired performance and the actual/current performance as shown in Figure 2.

Figure 2: Performance Gap

Impact

As volunteers and key leaders within the organization, ACMs are required to be present at all ACC meetings. It is vital that the ACMs know how to perform their duties because they play a large role in the success of the students by overseeing the training of the service volunteers. The ACMs often learn critical job functions and standard training protocol provided at ACC meetings. If current performance continues, there is a potential negative impact on the organization as a whole. Direct impactees are the service volunteers whom the ACM’s oversee. If performance continues, some ACM’s may not be prepared to adequately train their service volunteers. Indirect impactees are the students who are enrolled at Online U. Unprepared ACMs and service volunteers may adversely impact the students ability to receive adequate academic and spiritual support. This consequence can impact over 20,000 and may contribute to low retention. Due to these potential risks, the need was deemed critical in closing the performance gap.

Methods

To conduct the needs analysis, the team followed a systematic and systemic approach to collect and analyze data and determine the root causes of the performance issue. We used a variety of frameworks and data collection methods to identify potential solutions and prioritize recommended interventions.

Table 1: Frameworks and Purpose for each model

Frameworks

The following frameworks and reasonings are described as follows:

  1. Van Tiem et al’s (2013) HPT Model - This model served as the overall systematic Human Performance Improvement Model used to help conduct the needs assessment. It was also used systemically to gain a deeper understanding of the gap we were addressing, craft interview questions to analyze the organization and environment during performance analysis, and identify possible solutions to help close the performance gap.

  2. Gilbert’s (2007) Behavior Engineering Model - When analyzing the root causes of the performance gap, we used Gilberts BEM to develop our survey questionnaire and codebook to aid in our analysis process. Survey questions were designed based on the factors of performance (Individual, Environmental) and components of behavior (Data, Resources, Incentives, Knowledge, Capacity, and Motives)

  3. Hale’s Hierarchy Interventions Matrix - This model was used to identify and select feasible interventions. By using this model to make data driven decisions, we were able to identify interventions based on 6 subtypes: information focused, consequence focused, design focused, capacity/capabilities focused, action focused and/or congruence focused.

  4. Watkins Multicriteria Analysis - This model was used to evaluate interventions based on variables such as cost, ease of implementation, likelihood to implement, scalability and pushback. This model allowed us to prioritize interventions, address limitations, and make recommendations.

Data Collection Methods

To fully understand the scope of the problem, we used a variety of data collection tools: extant data/documents, interviews, and structured surveys.

Extant Data - We used existing extant data and reviewed job descriptions located in the ACM/Area Manager Training Handbooks to gain an understanding of their tasks, expectations, and responsibilities.

Interviews - We conducted an informal interview with the Client/Area Director to collect information on the organization's structure and its communication channels. We collected information to clarify the performance gap and conducted a performance gap analysis. We also did an organizational and environmental analysis at the time of the client interview.

Survey - We developed an anonymous online survey based on Gilbert's (BEM) and delivered it to all ACMs. Survey questions were designed based on the factors of performance (Individual, Environmental) and components of behavior (Data, Resources, Incentives, Knowledge, Capacity, and Motives) and were used as the basis for all data gathering. We distributed the survey to nine ACMs. Eight out of nine ACM’s completed the survey giving us an 89% completion rate.

Cause Analysis

During analysis we discovered that extant data did not have updated responsibilities listed in the ACM Training Handbook. We also discovered that Area Managers were not collectively requiring their service volunteers to attend the meetings. The survey revealed that service volunteers are not aware they should be attending the meetings and/or watching the follow-up recording.

Survey Results

Table 2 shows survey results of the performance problem using the codebook as described in Appendix A, page 24 and Gilbert’s (2007) Behavior Engineering Model (BEM).


Root Causes

We identified three primary root causes:

1. Confusion about expectation to attend

2. Unclear expectation to review Zoom recordings

3. Intangible consequences/rewards

Summary of Findings

Table 2: Summary of Findings (click to expand)

Intervention Selection Feasibility Analysis

The team took a systematic and systemic approach to produce a list of interventions that were likely to address the stated performance problem. Using Watkins Multicriteria Analysis, we were able to determine feasible solutions while taking into consideration the environmental and individual factors that might inhibit the likelihood of implementation.

Multicriteria Analysis

Table 3: Intervention Prioritization using Watkins Multicriteria Analysis

Recommended Solutions

We determined four key recommendations to address the gap.

  1. Clarify expectation for Area Managers to require ACMs to attend

  2. Update Area Manager and ACM Handbook

  3. Reframe purpose and benefits of ACC

  4. Clarify expectation of attendance


We determined these recommendations to be the most feasible to implement for the organization, as they all have a relatively low cost, high ease of implementation, and high scalability. Furthermore, all recommendations except clarifying expectations of attendance were rated high in likelihood of implementation and low pushback. While clarifying expectations of attendance showed slightly less feasibility, likelihood of implementation and low pushback, the medium ratings in these categories and high ratings in the other categories were still high enough to categorize it as having strong potential for improving the organization. While somewhat less feasible, it may still be useful for the organization to consider implementing area manager follow up for missed meetings and including ACC attendance in area manager annual reviews.


References

Gilbert, T. (2007). The behavior engineering model. In Human competence: Engineering worthy performance (tribute edition) (pp. 73-107). Pfeiffer.

Hale, J. (2007). The performance consultant's fieldbook: Tools and techniques for improving organizations and people (2nd ed.). Pfeiffer.

Van Tiem, D., Moseley, J., & Dessinger, J. (2012). Fundamentals of Performance Improvement: Optimizing results through people, processes and organizations (3rd ed.). John Wiley & Sons.

Watkins, R., Meiers, M. W., & Visser, Y. (2012). A guide to assessing needs: Essential tools for collecting information, making decisions and achieving development results. The World Bank.

Appendices

Appendix A. Needs Assessment Final Report