Evaluation

PSWC Evaluation

Osemome Ndebbio, Patricia Tjan, Sam Wilson, and Kevin Kepple (2023)

This evaluation was completed by students of OPWL 530 in the Organizational Performance and Workplace Learning program at Boise State University in the Fall of 2023. The team used a goal-based / goal-free approach to derive improvements needed for increasing membership numbers and improving the quality of services.

Background

Organization

The Professional Society West Coast chapter (PSWC; a pseudonym) is a nonprofit dedicated to professionals specializing in performance improvement. The organization is committed to boosting the efficiency of both individuals and organizations to foster heightened functionality, productivity, and success in the workplace.

A membership program was established with the chapter’s founding in 1962, to increase membership every year and improve membership engagement through marketing outreach of events such as informative webinars. Although marketing engagement has improved, membership numbers have consistently decreased each year (see Figure 1).

Figure 1
PSWC Membership from 2022-2023.

Bar chart comparing PSWC membership from 2022-2023

Figure 2
PSWC Membership Evaluation: Program Logic Model (PLM).

Program & Stakeholders

The membership program is an ongoing initiative led by a volunteer board of directors to gain and retain organization members. The program engages its members through webinar events, contests, and social media posts. The OPWL team created a Program Logic Model to clarify the PSWC Membership Program focal points to understand the organization's strategy (see Figure 2 for a summary).

Cultivating a robust network to exchange vital insights and information is a key outcome of a Professional Society membership. This includes sharing innovative practices, navigating industry trends, and offering a supportive environment to manage arising challenges and opportunities. Aside from serving as a nexus for learning and development with webinars, conferences, and mentorship, the association acts as a catalyst for career development through active participation in a globally recognized professional community.

There are three types of stakeholders for the Membership Program.

Upstream Stakeholders
The key upstream stakeholders for the new Membership Program and its design, development, and delivery:

S.M., President
J.C., President-Elect and Marketing Chair
T.S., Finance, and Membership Chair

2. Direct Impactees
The current and future members of PSWC, directly impacted by the Membership Program:

46 total current members at the time of this proposal
(24 Regular members, 21 Student members, 1 Retired member)
PSWC volunteer workers if there is a change in workflow or required hours

3. Indirect Impactees
The success or failure of the Membership Program would impact the program’s immediate recipients and other groups of people:

PSWC parent organization
Community of Practitioners

External resources
The primary indirect impactees of the program are the parent P.S. organization and sibling charters. The secondary indirect impactees of the program include an increasing community of virtual/real-world practitioners. Any outside resources PSWC currently subscribes to (website tools, marketing tools, etc.) could be impacted if found not to be fulfilling their complete potential.

Evaluation Request

The overall approach to this evaluation was guided by Chyung's 10-step evaluation procedure (2019). The process shaped the overall design based on stakeholder needs and the stakeholder use of the evaluation findings. Based on discussions with T.S. (Finance and Membership Chair), the evaluation team concluded that the evaluation's primary intent was to determine causes for the low member retention and find ways to improve retention and modestly grow membership.

The stakeholders wanted the evaluation team to assist the membership department in assessing the program, determining factors that drive membership retention and loss, and recommending changes to minimize attrition and improve retention. Therefore, a formative goal-free and goal-based evaluation was conducted to determine the driving factors for membership retention and attrition with performance improvement recommendations.

Methods

The evaluation methodology focused on collecting predominantly qualitative data to understand the nuances of member experiences and program effectiveness. The data collection spanned several key areas:

Membership Retention Data: We conducted extant data reviews to analyze current trends in membership retention.
Stakeholder Interviews: We carried out semi-structured interviews with five upstream stakeholders to gain insights into the program's operational and strategic aspects.
Membership Satisfaction Surveys: To gauge member satisfaction levels, we administered surveys to 18 members using Qualtrics. These surveys were sent out by the PSWC board to both current and past members.
Webinar Data and Marketing Material: We reviewed extant data on existing webinars, including program membership surveys and metadata, as well as marketing materials, social media communication, and mass emails.
Webinar Observations: We observed and assessed webinars, utilizing a webinar observation tool to rank various aspects on a 3-point scale, and then calculated an average score to evaluate overall webinar quality.

For each of these data collection methods, specific instruments were developed:

Interviews: Semi-structured interview questionnaires were designed to gather detailed information relevant to the two dimensions of the evaluation. (Appendix B.2)
Surveys: A survey was created on Qualtrics, with informed consent forms provided to ensure ethical data collection practices. (Appendix B.3)
Extant Data Tools: Excel was used to systematically organize and analyze data such as webinar attendance rates and social media engagement metrics. (Appendix B.4)
Webinar Observation Tool: This tool allowed us to quantitatively assess webinar quality across various metrics, providing a composite score for each webinar observed. (Appendix B.5)

Results

Figure 3
PSWC Dimensions: Retention Rate; Member Engagement and Participation 2023.

Based on the findings from the analysis, the success of the current program was rated "Fair" (See Figure 3). Results of the evaluation also show that the PSWC membership program is gaining engagement, but has yet to increase retention rates, specifically with students. By maintaining and expanding current resources, events, and future projects, retention should improve over time.

Some identified factors contributing to membership loss include, but are not limited to:

Lack of networking: Webinar analysis and survey results indicate network-building as a primary factor for joining the organization. There is little opportunity to network during webinar events and few options exist other than cold contacting to connect with professionals.
Lack of ROI: Upstream stakeholder interviews and member surveys express a desire for short-term projects rather than long-term volunteer commitments.
Lack of additional resources: The current PSWC page is outdated. Past membership surveys demonstrate a need for templates, a regular blog, and/or newsletters.

After the primary analysis, our team sought to compile the findings into a more holistic view of the organization using the Rummler & Brache 9 Box model (2013).

Integration of Interview Data with the 9 Box Model: The first step involved organizing the interview data into distinct themes aligned with the categories of the 9 Box model. This process contextualized the qualitative data within the broader framework of the organization’s operational structure.

SWOT Analysis: Building on this thematic organization, we then incorporated all the collected data—including the interview data and findings from previous steps of data analysis—into a SWOT (Strengths, Weaknesses, Opportunities, Threats) analysis (see Figure 4). This analysis distilled our data set into a clear, structured assessment of PSWC's membership program.

The SWOT analysis allowed us to systematically identify and highlight areas where the chapter excels and the aspects that require further development or improvement. It provided an integrated perspective, combining various data points and insights into a cohesive evaluation.

Final Determination: After synthesis through the 9 Box model and SWOT analysis, a final determination was made regarding PSWC's performance. Key strengths of the chapter include areas of effective member engagement and program delivery, as well as critical opportunities for improvement, including strategies for membership retention and enhanced member experiences.

The comprehensive evaluation of PSWC's membership program highlights several key strengths, notable weaknesses, and significant opportunities for growth, alongside potential threats that require strategic attention.

Figure 4
PSWC SWOT Analysis Results: Strengths, Weaknesses, Opportunities, and Threats.

Infographic on PSWC SWOT Analysis displaying the Strengths, Weaknesses, Opportunities, and Threats

Conclusions

Recommendations

Figure 5
6 Recommendations to Improve PSWC Membership.

Meta-Evaluations

Based on these meta evaluations, the team made the following improvements:

Condensed Dimensions - The team's initial three dimensions were reduced to two based on Dr. Chyung’s feedback, that the second and third dimensions were similar - as they both evaluated membership engagement.

Survey Questions - Before the membership survey was sent to active PSWC members, the team received combined feedback from Dr. Chyung and our class peers to remove repetitive questions and to clarify which questions fell under which dimensions.

Rubric - During the process of creating rubrics and criteria, the team also received feedback from Dr. Chyung and the client to improve the rubric so the evaluation accurately reflected the dimensions assessed.

Limitations

The following section acknowledges and outlines the constraints of this evaluation which impacted the breadth and depth of our findings.

Non-Participation of Members in One-on-One Interviews: Lack of participation from current members limited the depth and diversity of member perspectives.
Limited Relevance of Existing Membership Survey Data: The survey data accessible to us was in misalignment with analysis parameters.
Limited Scope of Webinar Observations: The evaluation's primary focus on webinars neglected other important aspects of the membership experience like networking events.
Lack of Longitudinal Data: The restriction of data collection to the past year potentially limited our understanding of trends over time.
Non-inclusion of Non-Member Perspectives: The evaluation did not include perspectives from former members, who could offer insights into the chapter's value proposition.
Undefined Baselines for Social Media Engagement: Our failure to establish clear baselines for assessing social media engagement in line with the chapter's goals meant the analysis of the impact and effectiveness of social media activities is imprecise.
Exclusion of Email and Website Analysis: Our assessment did not consider the chapter's email communications or the content and functionality of the website.

References

Campaignmonitor (n.d.). What are good open rates, CTRs, & CTORs for email campaigns? Campaignmonitor.com. https://www.campaignmonitor.com/resources/knowledge-base/what-are-good-email-metrics/#:~:text=A%20good%20click%2Dthrough%20rate%20should%20be%20between%202%2D5,suggest%20checking%20out%20our%20report

Chyung, S. Y. (2019). 10-step evaluation for training and performance improvement. Thousand Oaks, CA: Sage.

Rummler, G. A., & Brache, A. P. (2013). Improving Performance: How to Manage the White Space on the Organization Chart. Jossey-Bass.

W. K. Kellogg Foundation. (2004, January 2). Guiding program direction with logic models. Retrieved from https://wkkf.issuelab.org/resource/guiding-program-direction-with-logic-models.html

Appendices

Kepple, K., Ndebbio, O., Tjan, P., & Wilson, S. (2023). PSWC Membership Program Evaluation. https://drive.google.com/file/d/1HJkzb20XO71O2TlaFVN_lYzehZQQ9FCl/view?usp=sharing

Page updated

Report abuse