Evaluation of a Project Management Office

Background

In early 2019, Company X (a pseudonym) hired an outside consultant to evaluate opportunities to improve productivity. The evaluation report recommended that the organization could benefit from a formal project management office to oversee all projects. The Project Management Office (PMO) was launched in July of 2019 to support performance improvement and better resource management through accountability.

Organization

Company X is a professional association providing industry-specific training, publications, and conferences. The organization was founded over one hundred years ago and has its main offices in Philadelphia, Pennsylvania.

Program and Stakeholders

Senior management engaged an outside consultant to complete a needs assessment of the organizations work process and efficiency. The consultant concluded the organization would benefit from a formal project management office to define and maintain standards and reporting for project and program management within the organization. This would introduce economies of repetition in the execution of projects and executive reporting.

There are three types of stakeholders associated with this project:

Upstream Stakeholders

Two stakeholders were instrumental in the PMO’s creation:

  • David Griffiths (pseudonym), Chief Strategy Officer

  • Jonathan Distasio (pseudonym), Senior Manager, Project Management Office

Direct Impactees:

  • Current project team members – (n=40) All employees and contractors who are allocated to one or more of the four projects in the PMO.

  • Business analysts – (n=2) Associated with all projects being managed in the PMO.

  • Vendors and consultants affiliated with projects – (n=6 companies) As pending their involvement (either as subcontractors or as support roles) must adhere to the PMO guidelines.

  • Business/product owners and executive sponsors – (n=8) Must not only adhere to but additionally coach/encourage the behavior to improve implementation.

Indirect Impactees:

  • Direct reports of those serving on project teams

  • All employees not connected to a project team but impacted by revenue goals

  • External customers

  • Board members

Performance Evaluation

Six months after launching the Project Management Office, the client, Senior Manager of the PMO, requested a formative evaluation to assess:

  • Effectiveness of the PMO

  • Adequacy of support for staff and the organization in implementing project management practice

This evaluation was conducted by a team of graduate students for a class. The evaluation began February 2, 2020 and concluded April 12, 2020.

Evaluation Methods

Evaluation Purpose and Type

The evaluation team’s primary goal for conducting the evaluation was to assess whether the PMO provided sufficient resources and support for business units to manage projects that result in revenue growth. The stakeholders asked the evaluation team to speak to both proponents and critics of the PMO to determine areas for improvement.

The evaluation team conducted a formative, goal-based evaluation of the PMO. The evaluation measured the PMO against its intended goals:

  • define and maintain standards and reporting for project/program management within the organization in order that economies of repetition in the execution of projects and executive reporting are introduced,

  • performance is improved, and resources are better managed through accountability.


The team took a systemic approach to the PMO’s evolution within the organization and provided recommendations where appropriate to improve the performance. Based on this, and because the PMO has only been established since July of 2019, the evaluation team opted to focus two dimensions.

Table 1

Dimensions and Dimensional Questions

(Hazen, et al., 2020, Table 4, p. 16)

Dimensions and Data Collection Methods

In consultation with the client, the evaluation team used Kellogg's program logic model to identify dimensions to investigate and outline the PMO’s impact.

  • Resources: Personnel, tools, annual budget, etc. used by the PMO.

  • Activities: Process and workflows the PMO will execute.

  • Outputs: Products that will be produced to support desired outcomes

  • Outcomes: Specific changes in the stakeholders

  • Impact: Expected changes in the organization and its society

Following this, the evaluation team began to develop specific dimensions. The evaluation team proposed two dimensions to analyze and assigned importance weighting to each of the dimensions.

  • Results: How well has the implementation of the PMO program impacted the business’ ability to achieve KPI’s and/or financial goals? Impact, Most Important

  • Performance Support: How well does the PMO Center of Excellence, (which houses all templates, job aids, dashboards, forms, and visible project reporting) support business units in integrating expected PM performance? Activities, Very Important

The evaluation team used Rummler and Brache's (Rummler, et. al., 2012) Nine Boxes Model to evaluate performance support at the performer, process, and organizational levels.

Table 2: Dimensions and Dimensional Questions

(Hazen, et al., 2020) Table 5, p. 17)

Data Collection Tools


The evaluation team used the following sources of data:

  • Survey - Two anonymous surveys were administered; one to senior management and another to project team members.

  • Interview- Senior management was interviewed to assess the degree of process change and shifts in outcomes for teams associated with the introduction of the PMO.

  • Extant data review- Data from the organization included PMO dashboards, project status reports, financial reports, and project team use of other PMO templates.


The evaluation team observed the following limitations during this evaluation process:


  • In addition to changing strategies, the working environment of Company X was severely impacted by the pandemic. Not only did the company defer to remote working, but the stay at home orders in several states impacted the Company’s priorities and revenue. As a result, most resources were not available for surveys and interviews. Our project liaison, Deb, obtained as much information as ethically and physically possible during this difficult time for the Company.

Results

The Project Management Office (PMO) has had a somewhat positive impact on the organization’s ability to achieve KPIs (see table 3). However, it is unclear if the desired impact on financial goals was achieved. Two of the initial four projects were underway during our evaluation. The other two projects delivered products to the marketplace in March just as the Covid-19 crisis inhibited purchases by customers. The COVID-19 crisis impacted the evaluation team’s ability to fully measure the impact the PMO had on Company X’s results.

Further, the evaluation identified that the PMO has had a somewhat positive impact on the business units integrating expected project management into work stream performance. However, there were some aspects of this dimension which did not score favorably with Company X employees. While the job aids and reports found in the PMO Center of Excellence were helpful overall for project efficiency and productivity, the PMO templates and content to include in reports changed a number of times leading to confusion when trying to locate the current version of a template and required report revision. Employees also had mixed feelings about the PMOs impact on their efficiency.

Table 3: Dimensions

(Hazen, et.al., 2020, Table 9, p. 11

Recommendations

The evaluation team conducted an ethical data collection effort and synthesized the data to come to the following recommendations. These recommendations are considered for normal working conditions following the resolution of Pennsylvania’s shelter in place order, effective March 17, 2020.

Recommendation #1: Restructure the Center of Excellence for easier user experience.

  • Support documents and templates available should be the current version only. An archive can exist for compliance/legal purposes, but should not be readily accessible for the project teams.

  • Documents should be tagged with key words and searchable.

  • Define singular owner(s) for templates to ensure timely transition of updated documents.

Recommendation #2: Examine reporting to remove duplicate information, contributions, and establish cadence for proper execution.

  • Compare/contrast all reports for duplicate reporting. Streamline where possible.

  • Meet with focus groups to determine which reports are most effective for the project teams.

  • Determine which departments need to contribute to which reports and determine the proper cadence. Allocate a singular point of contact/owner for accountability purposes.

Recommendation #3: All project team members attend change management workshops to better incorporate the PMO behavior and culture into the workforce.

  • Hire an outside consultant to supply change management workshops at leadership and individual contributor levels to assist with overall cultural transition.

  • Develop a short term sustainment plan to reinforce positive culture changes within the workforce.


References

Chyung, S. Y. (2019). 10-step evaluation for training and performance improvement. SAGE.

Hazen, D., Imler, N., Minna, M., Skoro, E. (2020) Evaluation of a Project Management Office [MS Word Document]. OPWL 530 course site. https://Blackboard.boisestate.edu

Rummler, G. A., & Brache, A. P. (2012). Improving performance: How to manage the white space on the organization chart (3rd ed.). Jossey-Bass.