Organization

The Right Company (pseudonym) is a financial planning association providing industry-specific training, publications, and conferences. The company was founded over 100 years ago and is headquartered in Philadelphia, Pennsylvania. In early 2019, as part of a strategic planning initiative, the company hired an outside consultant to complete a needs assessment to improve productivity. The resulting report recommended that the company would benefit from a formal project management office to oversee all projects. The Project Management Office (PMO) was launched in July of 2019 to support performance improvement and better resource management through accountability.

The PMO was initially tasked with applying standard project management processes and reporting for four cross-functional project teams. The PMO published templates and project dashboards, collaborated with product owners to create work breakdowns and deliverable schedules, and established protocols for weekly project team meetings and bi-weekly reports to leadership. 

The PMO senior manager, Jonathan Distasio (pseudonym), hosted series of educational sessions with employees across the organization to explain the PMO standards. While a project manager was not assigned to every project or program, the templates, processes, and protocols developed during the first project cycle managed by the PMO were expected to be used by all project teams moving forward in The Center of Excellence. 

The Center of Excellence was developed as a resource center for the PMO to support Right Company’s goals of maturing the PMO in the short-term (10 months) and in the long-term (two-plus years) based on the Program Logic Model (see Appendix A). The Center of Excellence houses all PMO templates, job aids, dashboards, forms, and visible project reporting.

Program and Stakeholders

There were three types of stakeholders for the PMO illustrated in Figure 1.

Figure 1. Program Stakeholders

Evaluation Request

Six months after launching the Project Management Office, the client, Senior Manager of the PMO, Jonathan Distasio (pseudonym), requested a formative evaluation be completed to assess: 

• Effectiveness of the PMO.
• Adequacy of support for employees and the company in implementing the PMO.

The evaluation team furthered referred to as “the team” consisted of four master’s degree students. The team conducted this evaluation as part of that Evaluation course project (Hazen et al., 2020) offered by the Organizational Performance and Workplace Learning (OPWL) department at Boise State University.

Evaluation Methods

Evaluation Purpose and Type

Based on discussions with Mr. Griffiths and Mr. Distasio, the team spoke to both proponents and critics of the PMO to determine potential areas for improvement. The team conducted a formative, goal-based evaluation of the PMO. This evaluation measured the PMO against its intended goal to define and maintain standards and reporting for projects and project managers so that economies of repetition in the execution of projects and executive reporting are introduced, performance is improved, and resources are better managed through accountability. The team used a systemic approach to review the PMO’s evolution and effectiveness within the organization. This review highlighted performance of the PMO and how it directly affected the company’s productivity and key performance indicators (KPI’s). 

Dimensions and Data Collection Methods

Considering the client’s request, and because the PMO was established in July of 2019, the evaluation team opted to focus on two dimensions to measure the effectiveness of the PMO and support for employees. Each dimension was given an importance weighting, as detailed below: 

1. Results: How well has the implementation of the PMO program impacted the business’ ability to achieve Key Performance Indicators (KPIs) and/or financial goals? Impact, Most Important 

2. Performance Support: How well does the PMO Center of Excellence, (which houses all templates, job aids, dashboards, forms, and visible project reporting) support business units in integrating expected PM performance? Activities, Very Important

The team followed Chyung’s (2019) 10-step evaluation procedure. By adhering to the 10-step procedure, the team was able to design an evaluation based on the stakeholders’ needs and intended use of the evaluation findings. 

The team selected two dimensions to evaluate, one from the impact and one from the activities section of the Program Logic Model (see Appendix A). While impact is related to the organization’s ability to reach desired KPI and financial results, the activities section of the program logic model focuses on performance support. The team integrated into the analysis Rummler and Brache’s Nine Box Model (2012) which identified support at the performer, process, and organizational level the evaluation team used the following sources of data: 

• Survey – Two anonymous surveys were administered: one to senior management and another to project team members. 
• Interview- Senior management was interviewed to assess the degree of process change and shifts in outcomes for teams associated with the introduction of the PMO. 
• Extant data review- Data from the organization included PMO dashboards, project status reports, financial reports, and project team use of other PMO templates.

The team utilized critical multiplism by using varied types of data, then triangulating data collected using diverse sources and methods (interviews, surveys, and extant data). This enabled the team to draw credible conclusions. The team also compared their analysis and interpretation of extant data to avoid drawing biased conclusions. 

Results

The Project Management Office’s results and performance support received a combined rating of Somewhat Positive (see Table 1). Each dimension had its own opportunities for improvement (See Table 2). 

Table 1. Overall Dimension Rating

Different data sources revealed important information. Figure 2 illustrates the different results between interviews with managers and team members.

Figure 2. Interview Results

Extant Data: The data review checklist indicated the following for each dimension.

• Results: The PMO Status Reports and Executive Summaries both had a positive impact on the PMO’s impact on the company’s KPIs and financial goals. Both offered clarity and insight into the project timeline and deliverables and allowed the PMO and project members to make appropriate changes if needed. 
• Performance Support: Templates had an overall negative impact, not because of the documents, but due to multiple versions being housed on the Center of Excellence, and the difficulty to locate correct versions. Job aids had a slightly positive impact as they helped create a uniform process.  

Surveys: Upon synthesis of the evaluation, the team recognized the survey questions did not accurately depict what the evaluation team was trying to determine. As a result, the survey results were as follows:

• Survey results at a manager level were consistent. Most were somewhat positive to mostly positive, with consistent answers denoting that the PMO and PMO tools supported project timelines and meeting deliverables.  
• Survey results for project team members were less consistent. While the results were overall somewhat positive, more participants answered questions negatively than their managers.  

A complete review of the survey questions resulted in the determination that the survey questions asked were ambiguous and could be interpreted differently depending on job role or project role. 

Table 2. Evaluation Results

Conclusions

The Project Management Office (PMO) has had a somewhat positive impact on the organization’s ability to achieve KPIs. However, it is unclear if the desired impact on KPIs and financial goals were achieved. Two of the initial four PMO projects are still underway. The other two projects delivered products to the marketplace in March 2020, just as the COVID-19 crisis inhibited purchases by customers. The COVID-19 crisis impacted the evaluation team’s ability to fully measure the impact the PMO had on Right Company’s results.

Further, the team identified that the PMO has had a somewhat positive impact on the company’s ability to integrate expected project management into work performance. 

However, there were some aspects of this dimension that did not score favorably with employees. While the job aids and reports found in the PMO Center of Excellence helped overall project efficiency and productivity, the PMO templates and content to include in employee reporting changed several times. This led to confusion when trying to locate the current version of a template and required report revision. Employees also had mixed feelings about the PMO’s impact on their efficiency. This formative evaluation uncovered challenges facing the organization as it seeks to mature the project management culture. Table 3 outlines the team’s recommendations.

Table 3. Recommendations

References

Chyung, S. Y. (2019). 10-step evaluation for training and performance improvement. Thousand Oaks, CA: SAGE.

Hazen D., Imler, N., Minna, M., and Skoro, L. (2020) Evaluation of a Project Management Office. [Unpublished student report]. Department of Organizational Performance and Workplace Learning, Boise State University.

Rummler, G. A. & Brache, A. P. (2012). Improving performance: How to manage the white space on the organization chart (3rd ed.). San Francisco, CA: Jossey-Bass.

Appendices

Appendix A. Program Logic Model