PrintCom Evaluation
Background
Organization
PrintCom (pseudonym) is a $14 million-dollar commercial printing company located in the Mid-West. For years the company serviced its clients using traditional commercial offset printing presses and succeeded well. However, around 2017 the company was approached by a large existing customer with a high-profile project that would require a significant upgrade in their printing equipment to complete. PrintCom saw this as an opportunity and took advantage of it. In 2017, the senior executive team decided to take a calculated risk and purchase $3 million worth of new print on-demand (POD) equipment. This significant upgrade in printing technology greatly enhanced the company’s printing capabilities and has opened up new opportunities for growth and expansion.
The first two years after the transition from the old equipment to the new Print-on-Demand (POD) equipment was challenging, but the company was able to grow sales by $1,000,000 in the second year. In 2020, the company is at a point where they would like to evaluate the sales production data from the last two years and determine how best to leverage the new equipment in the future.
The company executive authorized an evaluation team consisting of three masters degree students enrolled in the evaluation course offered by the Organizational Performance and Workplace Learning (OPWL) department at Boise State University to conduct a formal evaluation of PrintCom's efforts. We, the evaluation team, conducted this evaluation as our course project (Hansen et al. 2020) and it is presented here in a summarized case study.
Evaluation Program
This evaluation project investigated how effective the new POD equipment has been for the company. A key purpose of this formative evaluation was to collect, analyze, and disseminate data to help the project partners determine the effectiveness of the company’s marketing and promotional efforts. In addition, the evaluation assisted in answering how well the PrintCom customer service employees understand the benefits provided by the on-demand printing equipment.
After several conversations with the client, the evaluation team used the Kellogg Foundation’s Program Logic Model (2004) or "PLM" as a basis for conducting the Evaluation Project for PrintCom. The PLM included the following elements:
● Resources: Equipment, tools, materials, data, personnel, etc that will be used
● Activities: Process that the program will execute with the program resources
● Outputs: Direct products that are immediately generated by the program activities
● Outcomes: Specific changes the program participants make as a result of program activities
● Impact: Intended or unintended changes that happen to the organization as a result of the program activities.
Program Stakeholders
There are three types of stakeholders for the program, Upstream Stakeholders, Direct Impactees, and Indirect Impactees.
Upstream Stakeholders
There were several stakeholders who played an important role in making the decision to purchase the new POD equipment and to move forward with the implementation of the new equipment. The key upstream stakeholders are the following:
● Brad S. - Business Owner
● Jeremy S. - General Manager
● Marlin S. - Chief Financial Officer (CFO)
● Brenda J. - Customer Service Manager
Direct Impactees
The print production and customer service employees of PrintCom are the direct impactees of having the new POD equipment. The company has approximately 80 employees and the changes made directly impact the job description, hours, processes and procedures of those employees.
Indirect Impactees
The customers of PrintCom are the indirect impactees of the changes the company has made. Additionally, the customers of PrintCom’s customers are also indirect impactees since they will be the recipients of the additional benefits coming from the new printing capabilities of PrintCom.
Evaluation Request
The purpose of this formative evaluation was to determine...
How well PrintCom’s current sales and marketing of the on-demand print services appeal to new customers,
How they can improve their sales and marketing strategies for the remainder of 2020,
How well their customer service employees understand the benefits provided by the on-demand printing equipment
Whether or not additional resources are needed to increase or improve the customer service team's understanding of the equipment is required.
The key findings of this evaluation will help PrintCom (in particular, the General Manager, Business Owner, CFO, Customer Service Manager, and the Sales and Marketing Director) make changes to the organization’s processes to generate more sales through their marketing efforts, additional training, targeting new clients, and getting existing clients to transition to the on-demand print model.
Program Evaluation Methods
As the overall approach to this evaluation, the evaluation team followed Chyung’s (2019) 10-step evaluation procedure. The 10-step procedure assisted the evaluation team in designing an evaluation based on the stakeholders’ needed use of the evaluation findings. The evaluation team also used multiple levels of the four-level training evaluation framework (Kirkpatrick, 1996) for evaluating the two dimensions:
● Level 2 Learning - How well do the PrintCom customer service employees understand the benefits provided by the on-demand printing equipment?
● Level 4 Results - How well does PrintCom's marketing efforts of their on-demand print services appeal to new customers?
These were the two primary dimensions this evaluation focused on were:
● Marketing and Sales Support: How well does PrintCom's marketing efforts of their on-demand print services appeal to new customers? (Most Important)
● Knowledge: How well do the PrintCom customer service employees understand the benefits provided by the on-demand printing equipment? (Important)
The evaluation team used multiple sources of data, including the General Manager, customer service representatives (CSRs), and sales data. Additionally, the evaluation team used multiple types of data collection methods:
● Knowledge Assessment - Web-based knowledge assessment was administered to CSRs.
● Interview - Semi-structured telephone interviews were conducted with the General Manager and CSRs.
● Extant data review - Existing sales data was reviewed.
The multiple types of data collection methods were selected to complement strengths and weaknesses of each method, and the data collected from multiple types/sources was triangulated to draw credible conclusions. The three evaluation team members also compared their analysis and interpretation of extant data to avoid drawing biased conclusions.
Summary of Evaluation Results
Dimension 1: Marketing and Sales Support
The team evaluated the answer to the question “How well does PrintCom's marketing efforts of their on-demand print services appeal to new customers?”
The dimension 1 interview questions produced a rating of "Mediocre." The results of the sales data analysis resulted in a overall rating of "Poor" for the sales. The ratings from both data collection methods were combined using a dimensional rubric which resulted in an overall rating of “Poor” for dimension 1: Marketing and Sales support.
Dimension 2: Knowledge
The second dimensional question the team evaluated is “How well do the PrintCom customer service employees understand the benefits provided by on-demand printing to their customers?” To evaluate this dimension, a knowledge assessment/survey of the CSRs, the interviews with the CSRs, and the interview with the General Manager were used.
The CSR interviews produced a rating of "Good" and the General Manager interview questions produced a rating of "mediocre."
The results of the CSR knowledge assessment and survey items resulted in an overall rating of "mediocre."
The ratings from the data collection methods were combined using a dimensional rubric, which resulted in an overall “Mediocre” rating for dimension 2: Knowledge.
Conclusions
The evaluation of the two dimensions revealed room for improvement in terms of PrintCom’s marketing and sales support, and the customer service employees’ understanding of the benefits provided by on-demand printing to their customers.
The evaluation team conducted a SWOT (strengths, weaknesses, opportunities and threats) analysis in addition to the above findings. The data analysis and SWOT analysis resulted in multiple recommendations for PrintCom to improve their POD services. Recommendations included:
focusing the marketing and advertising strategies to target ideal clients;
creating a tailored marketing plan;
tracking and measuring marketing;
developing POD sales goals and expectations for the customer services team;
providing feedback to the customer service team on how well they are doing at promoting POD services;
creating a POD training program for CSRs to further educate them on the benefits of POD services;
additional sales training;
and creation of POD sales tools and resources.
References
Chyung, S. Y. (2019). 10-step evaluation for training and performance improvement. Sage Publications.
Hansen, T., Ratliff, J., & Wisdom, J. (2020 Jan). Evaluation of PrintCom’s On-Demand Printing Services [MS Word document]. OPWL, 530 course site. https//blackboard.boisestate.edu
Kirkpatrick, D. (1996). Evaluating training programs: The four levels. Berrett-Koehler.
W. K. Kellogg Foundation (2004, January). Logic model development guide. Retrieved from: https://www.wkkf.org/resource-directory/resources/2004/01/logic-model-development-guide