Diagnostic Framework and Health Check Tool for Engineering and Technology Projects

Purpose: Development of a practitioner oriented diagnostic framework and health check tool to support the robust assessment of engineering and technology projects. Design/methodology/approach: The research is based on a literature review that draws together insights on project assessment and critical success factors to establish an integrated systems view of projects. This is extended to allow a comprehensive diagnostic framework to be developed along with a high-level health check tool that can be readily deployed on projects. The utility of the diagnostic framework and health check tool are explored through three illustrative case studies, with two from Canada and one from the United Kingdom. Findings and Originality/value: The performance of engineering and technology projects can be viewed through a systems perspective and being a function of six contributing subsystems that are: process, technology, resources, impact, knowledge and culture. The diagnostic framework that is developed through this research integrates these sub-systems to provide a robust assessment methodology for projects, which is linked to existing best practice for project reviews, performance management and maturity models. The case studies provide managerial insights that are related to the diagnostic framework but crucially also position the approach in the context of industrial applications for construction engineering and technology


Introduction
On the first day at work, new engineering project managers are presented with a plethora of different challenges. There is, of course, a need to rapidly gain an understanding of the organisational systems and engineering processes as well as management structures and form effective working relationships with both internal and external stakeholders for the projects. However, where the project manager is taking on an existing portfolio of engineering projects, there is the pressing need to assess these projects and ascertain their 'health'. This needs to be carried out in order to ascertain the current level of progress, which includes the level of project activities and engineering tasks carried out according to the schedule, the level of expenditure according to the project budget as well as the level of technical performance or quality level achieved according to the project or engineering specification. But crucially the project manager must also understand whether the project management methodology has been implemented adequately through ensuring project controls and governance arrangements are effective and also whether the performance of the project is consistent with the views of the project stakeholders, including internal senior management, engineering staff and where applicable external client representatives.
While there are a number of well recognised project management protocols and guidelines in place, such as the Project Management Body of Knowledge (PMI, 2000), these procedures are focused on the delivery of defined projects and they do not necessarily focus on helping a new project manager to quickly assess the status of projects in full execution mode. Furthermore, project maturity models (Crawford, 2007) focus on the level of adoption of a project methodology and corresponding procedures within an organisation as part of understanding the overall status of capabilities to support effective project management. Such tools do not readily allow a newly appointed project manager to rapidly and efficiently assess a portfolio of projects in order to gain a robust view on the health of the projects.
An inability by new engineering project managers to undertake an effective project assessment and health check in a timely fashion will have the potential to seriously impact on the performance of their projects. This is the case since, if there are underlying problems with the projects (such as deficiencies in process design), then a lack of awareness will likely cause such problems to become exacerbated and therefore project performance will be reduced even further. Moreover, even if there are not any underlying problems or issues at that point, if new project managers are not able to become rapidly effective across the portfolio of projects within their responsibility, this could also cause the performance of the projects to be impacted detrimentally. Therefore, gaining access to a reliable diagnostic tool that can help project managers to quickly and efficiently assess the status of a group of projects has the potential to be of significant use to project management practitioners.
Consequently, this paper explores project performance assessment through providing details of a new diagnostic framework and health check tool developed to help project managers assess the status and health of projects. This framework will incorporate an integrated systems view that positions projects in a broader context in order to capture the full complexities of the project environment and contributing factors that ultimately influence project performance.
The paper includes a supporting literature review on project assessment that includes consideration of the critical success factors for engineering projects. Through building on the literature review, a diagnostic framework has been developed that can be used to support project health checks. The framework and supporting tool will be applied to three case study investigations, which highlight the types of challenges that engineering project managers may encounter as well as the scope for the health check tool to identify underlying issues and thereby help formulate project recovery plans when required.

Literature review
Ensuring engineering and technology projects are on a sound footing will of course need to involve assessment of the performance of the project (in regard to schedule, expenditure and quality outputs) and this can be related to the maturity of the project methodology employed (Jin, Chai & Tan, 2014). A key determinant of project success therefore rests on the adoption and implementation of supporting process methodologies. However, project performance can be viewed as being a function of a wider set of factors beyond the use of fixed project methodologies, guidelines and procedures. Consequently, a more systemic perspective would allow the full range of critical success factors for projects to be captured. Indeed assessing the performance and health of engineering projects from the construction sector has been related to a series of key performance indicators (KPIs), including a cost performance indicator that provides a 'snap-shot' view of the project's cost performance on the day of the project health check (Mian, Humphreys & Sidwell, 2004). The authors also identify seven primary themes of critical success factors (CSFs) that can be considered in regard to assessing the health of a construction sector project and these are as follows: cost, time, quality, relationships, safety, environment and stakeholders. Achieving time, cost and quality (including safety) targets for an engineering project can be addressed through process implementation, whereas maintaining effective relationships and stakeholder management needs to draw on softer, more people-oriented inputs to the project. Process implementation is therefore an essential part of project performance assessment although there are other determinants that can be considered as being part of the overall project context. This represents a holistic view of projects, whereas traditional project management approaches rest on delivery of project outputs within the so called 'iron triangle' of constraints, i.e. according to fixed budget, schedule and quality or performance parameters (Jha & Iyer, 2007). These parameters are important but some studies have indicated wider criteria that although encompass these constraints also seek to establish a more holistic view. Developing an improved approach to the assessment of performance for engineering projects will benefit from taking an integrated systems view allowing such a holistic perspective to be incorporated.
Atkinson's square route model for understanding project success (Atkinson, 1999) includes the iron triangle alongside three other sets of project success criteria, which are the information system (including reliability, validity and information quality), organisational benefits (such as improved efficiency, effectiveness, increased profits and meeting strategic goals) and stakeholder community benefits (such as satisfied users, personal and professional development, environmental impact). Moreover, project success factors can be related to the cultural dimension of projects (Zwikael, 2012), where the particular configuration of factors (such as those relating to project planning) will be contingent on the project environment and circumstances. This position supports project management being viewed adaptively in terms of the need for project frameworks to be tailored to meet the project background including but not limited to cultural aspects. Such a cultural dimension would need to accommodate the normal patterns of working on engineering projects, including the contributions from the project stakeholders and partner organisations. These patterns of working include the level of information sharing as well as the trust that is developing from open and regular dialogue and reciprocity of both benefits and risks for supplier and clients. An inability to adequately understand this cultural landscape can represent a further barrier to project performance. Dvir and Shenhar (2007) have pursued a holistic perspective on project management that is based on five main dimensions, which are as follows: Project efficiency (in relation to meeting budget and schedule requirements), impact on the customer (in relation to achieving the stated requirements for the project, including customer benefits, satisfaction and arising loyalty), impact on the team (in relation to the impact the project has on the team members including levels of satisfaction, retention and resulting personal growth), business results (in relation to the outcome of the project in terms of any return on investment, market share or growth that is achieved) and preparation for the future (in relation to the how project deliverables result in new technologies, new markets and new capabilities). The authors advocate using the framework as a tool to identify the optimal project management approaches to be deployed for a given project, thereby moving away from a 'one sits fits all' approach and rigid adoption of project management methodologies for all projects circumstances. This approach reinforces an adaptive perspective of projects, where implementation of project guidelines is dependent of the internal and external context of the projects. Conversely, rigidly applying project guidelines and protocols including full adoption of all procedures would be hampered by such an adaptive and flexible approach. A balanced adoption of guidelines would therefore appear to be a sensible approach.
Capturing performance according to a broader set of criteria is also a feature of the balanced scorecard (Kaplan & Norton, 2003), which includes metrics according to four perspectives that are financial, customer, internal business processes, and learning and growth. Measuring performance through such measures avoids a narrow financial view that has been historically used to assess organisational and even project performance. The balanced scorecard has been employed widely and in different organisational settings (Marr & Neely, 2003;Bigliardi & Bottani, 2010) to ensure that data and information relating to project performance that ultimately leads to impact on the wider business can be efficiently captured and monitored.
Considering the broader outputs of projects can be viewed in the context of the overall impact that is generated. Project delivery may result in new organisational capabilities being developing that in turn underpins the financial sustainability of a company, or the project may result in a new platform technology resulting in a whole new series of engineering products that can be offered by the company. Considering the wider impact of a project can be considered as being a further dimension of project performance and development of an assessment framework would need to include such a factor.
In the case of research and technology projects, the scorecard approach has been adapted to measure the performance of an industrial supported university institute (Philbin, 2011). This study highlighted how scorecard metrics can be utilised by management and customer stakeholders through access to a convenient source of data and information across all the individual project activities undertaken within the institute, such as those relating to funding, research, teaching and education, knowledge transfer, and experimental facilities development.
This approach emphasises the benefits of moving beyond simply measuring project performance according to meeting milestones and staying within project budgets. These impact-driven measures are important but the additional capture and quantification of less tangible project outputs, e.g. the level of training and learning undertaken by staff and students at the university institute, allows a wider view of project performance to be established. This research reinforces the advantages of using a holistic set of success factors for projects that encompass areas such as the long-term impact on skills, competencies and the development of new organisational capabilities.
Consideration of a broader range of project success factors needs to encompass an appropriate knowledge dimension (Davenport, De Long & Beers, 1998) and development of an increased understanding of how an organisation's knowledge base has improved through delivery and implementation of a given project should be addressed (Maqsood & Finegan, 2009). This knowledge could include, for example, skills acquired by project engineers involved with the design of a new chemical processing facility, or planning expertise gained by project schedulers involved in a city wide roll-out of a new telecommunications infrastructure. In either case, projects can benefit from analysis of knowledge inputs and outputs, and adoption of a systems perspective can help underpin this objective along with supporting both hard (i.e. technical) and soft (i.e. social) requirements of projects. In further work, a transport project assessment methodology has been developed within a framework for sustainable development (Joumard & Nicolas, 2010), which highlights the economic, social and environmental criteria for assessing projects. It should be noted, however, that data and information that is collected and processed in support of projects should be according to optimal data quality levels in order to avoid excessive data costs (Haug, Zachariassen & Van Liempd, 2011). Availability and access to project data and information is clearly important to project success and effective software systems need to be deployed in order to allow data capture, storage, processing and communication. Project managers have a key role in this regard and can be considered as nodes within the organisation for information gathering and dissemination. The use of data and information within the project environment contributes to the overall knowledge position that needs to integrate knowledge within the project alongside knowledge sourced from the wider organisation. Knowledge can therefore be seen as a further important factor contributing to project performance and would need to be a central part of any project assessment methodology.
In other work a systems based project health check methodology has been reported (Jaafari, 2007). The approach was not designed to measure project performance against an existing plan and also not intended to be a project maturity model. Instead, the health of the project is assessed through evaluating project practices according to a set of pre-defined variables that are collectively used to characterise the health of the project under assessment. This framework has been extended (Almahmoud, Doloi & Panuwatwanich, 2012) to help project managers identify the root causes of difficulties at the early stages of a project and the authors propose that further research is carried out to investigate health checks across the full project lifecycle in order to identify trends and underlying factors. The literature therefore points to the merits of implementing project health checks and such adoption is likely to benefit from a periodic assessment allowing continuous improvements throughout the project lifecycle.
Project management needs to be supported by the required project resourcing, including the pivotal role of the project manager (Müller & Turner, 2007) as well as other staff involved both within the supplier or delivery organisation and also those working in any partner organisations (Kerzner, 2013). These resources need to be have access to reliable ICT (information and communications technology) systems (Froese, 2010), such as software systems to support cost management and scheduling activities. An inability to gain access to such systems can clearly have a negative effect on the performance of projects and indeed barriers to the use of ICT on inter-organisational construction projects have been identified as the fragmented and temporary nature of construction projects as well as different working practices and objectives of the contracting parties (Adriaanse, Voordijk & Dewulf, 2010).
In addition to the barriers to project information use and deployment that may exist, new project managers may incorrectly assume that co-workers and partners providing data have the same priorities as the project goals. This may lead to arriving at false conclusions based on the information provided (Kennedy, 2014). The adoption of recognised approaches to support effective risk management is also an important feature of engineering projects and failure modes and effects analysis (FMEA) can be a particularly useful tool in this regard (Philbin, 2010). It can be seen that project resources play a pivotal role in the success of projects but deployment of such resources can be predicated on available technology (namely ICT but also including more specialised technologies needed to deliver the engineering project).
Consequently, development of an improved framework to support project assessment and health checks needs to adequately incorporate both resources and technology.
Project health status can also be viewed in terms of the knowledge that is generated and how it can subsequently be captured and utilised further on other projects. Schindler and Eppler (2003) have identified two ways to harvest knowledge from projects. The first being processbased methods, which focus on capturing key learnings from a project via a procedural approach, such as from a post-project appraisal meeting. The second being documentationbased methods, which allow learning from experiences to be gathered and documented through, for example, writing short articles on the project that detail project insights, or project workers' completing their own learning histories. The authors also advocate not simply reviewing project learning at the end but instead instituting continuous project learning through regular reviews to capture learning and best practice across projects. Some practitioners, however, warn that post-project lessons learned documents are often an exercise in fulfilling an organisational requirement and the findings presented are often shelved to remain unread by future project teams (Kennedy, 2004).

Integrated systems view
Developing a diagnostic framework to support the assessment of projects will help project managers that have recently been appointed to assess the health of the projects they are charged to deliver. Such a framework will need to capture a number of key areas and adopting a holistic or systems perspective will be beneficial in this regard. Consequently, Figure 1 provides the conceptual model for an integrated systems view of projects that includes the wider set of underlying factors (as sub-systems) that ultimately contribute to project performance. This conceptual model for an integrated systems view of projects has been established through building on the findings from the literature review, which has clearly identified six main determinants (or sub-systems) of project performance, which are process, technology, resources, impact, knowledge and culture. These sub-systems can be broadly classified as being part of three integrated levels, which are the project infrastructure (resources and technology sub-systems), project organisation (process and knowledge sub-systems) and project environment (culture and impact sub-systems).
Development of a systemic project assessment methodology and supporting diagnostic framework therefore needs to incorporate the six sub-systems as part of the main review and assessment criteria. These sub-systems are directly linked to the findings from the literature review of project assessment. Table 1 provides supporting definitions for the conceptual model, including the integrated systems view as well as the contributing sub-systems.

Integrated systems view
Broad-based perspective of projects that allows a holistic (systemic) consideration of all relevant determinants of project performance and implicitly the areas needed for review and analysis to support project assessment.

Process sub-system
The adoption and implementation of structured procedures and process-driven guidelines throughout the project lifecycle allowing project inputs and outputs to be efficiently managed.
Technology sub-system The availability of technology, including ICT and project-specific technology, to support the project environment, enable planning and control functions to be exercised and facilitate project governance.

Resources sub-system
The resources, including project staff and project infrastructure, required to underpin project delivery. Resources including project management and engineering staff, governance representatives as well as wider project stakeholders. Impact sub-system The overall outcomes for the project incorporating initial project outputs and the wider set of benefits arising from project delivery, including environmental and sustainability outputs as well as secondary impacts, such as those relating to the development of new competencies and additional business areas.
Knowledge sub-system The data and information architecture required to enable effective decision-making across projects as well as efficient planning and control that is supported by technology (ICT) adoption and utilised by project resources.

Culture sub-system
The supporting environment that incorporates norms of working and patterns of behaviour, levels of trust and reciprocity for both benefits and risks for key project stakeholders. Table 1. Supporting definitions for conceptual model As described previously, an ability to conduct a project health check has the potential to significantly impact the performance of engineering projects through diagnosing underlying issues and allowing corrective action to be undertaken. Project health checks can be undertaken on a periodic basis, for example as part of project initiation and also in support of project stage gate reviews. Furthermore, should a project manager suddenly depart the organisation, a project health check can be carried out by the replacement project manager in order to rapidly gauge the status of the project and help inform next steps. To illustrate such applications of a project health check and use of the proposed diagnostic framework, Figure 2 provides a schematic view of the project lifecycle for a manufacturing project including project health checks.

Diagnostic framework
The integrated systems view can be developed further through the design of a rigorous diagnostic framework, which includes supporting criteria according to the six sub-systems (see Table 2). These diagnostic criteria have been developed by the authors through considering the current best practice and approaches for project assessment as well as critical success factors and related areas of project management.

Diagnostic sub-systems Supporting review criteria
Process 1. Effective project planning processes, including business planning, resourcing planning, financial planning, quality planning and risk planning. 2. Project initiation and start-up as well as project close-out. 3. Project delivery and technical work processes that are context specific.

4.
Project governance arrangements to ensure necessary level of project steering and coordination. 5. Other control processes, including change control administration. 6. Project reporting processes, including internal and external reporting. 7. Contracting and procurement processes to underpin project contracts, sub-contracting and purchasing activities. 8. Stakeholder relations and communications. 9. Periodic review of project learning and best practice throughout project.
Technology (namely ICT systems) 10. ICT systems to support project cost profiling and business modelling, e.g. through local spreadsheet tool or use of proprietary software. 11. Enterprise resource planning (ERP) system required for recording and monitoring project expenditures against budget forecasts. 12. Project schedule software required to support use of Gantt charts for scheduling of tasks and activities as well as identification and management of critical paths. 13. Earned value analysis (EVA) software required to support project performance measurement. 14. Risk management software system for identification and mitigation of project risks. This can also include engineering risk systems, such as those based on FMEA (failure mode and effects analysis). 15. Specialist project lifecycle software systems (where appropriate) to support engineering lifecycle, from concept to design, manufacture & installation, acceptance & testing and finally evaluation.

Diagnostic sub-systems Supporting review criteria
Resources 16. Project manager appointed with required background, competencies and skills to drive forward project delivery. 17. Programme or portfolio manager overseeing high-level progress in the case where the project is part of a wider programme or portfolio at the company. 18. Project board or appropriate governance board including senior management from the supplier organisation and where appropriate representatives from the client organisation. 19. Project delivery staff, including engineering and technology specialists required to support delivery of the project's technical outputs. 20. Project support staff, such as project administration and other staff as required. 21. Access to generalist staff at the company providing other functions, such as procurement, invoicing and payments, health & safety, marketing and sales support.
Impact 22. Options appraisal carried out to support funding of project and selection of project approach. 23. Industrial value proposition and underpinning value drivers are clear and consistent with the company's delivery and growth strategy. Value drivers can be in quantitative financial terms as well as qualitative value streams. 24. Wider organisational benefits arising from delivery of the project have been clearly identified and communicated. 25. Skills improvement and development of enhanced and new competencies for project team members. 26. Creation of future revenue opportunities, including new areas of business, product extensions and options for strategic development initiatives. 27. Project outputs related to wider stakeholders, e.g. environmental sustainability and social benefits arising from the long-term impact of the project.
Knowledge 28. Access to background data and information from third-party sources, such as proprietary databases, information repositories and electronic libraries. 29. Generation of new technical data and information required to meet project milestones and deliverables to client organisation. 30. Availability of technical knowledge within the project and specifically to all project staff that require access. 31. Communication of arising knowledge from the project to the client through project reporting processes. 32. Data and information required to support the project management process, including financial data, commercial information (e.g. contractual terms and conditions), organisational procedures and policies, such as those relating to human resources, recruitment, procurement, etc.
Culture 33. Supporting culture that promotes open and honest working across the project with high levels of integrity shown by those involved with the project. 34. Patterns of working, interaction and communication to support sharing of issues within the project to minimise negative impact on project expenditure, schedule or quality targets. 35. Trust developed between the supplier and client so that challenges can be addressed in an open and efficient manner. 36. Broad view and perspective of project manager to ensure project is connected to wider organisational structures and processes. 37. Culture that supports reciprocal benefits for supplier and client, i.e. promoting a 'win-win' culture.

Project health check tool
The diagnostic framework can be used to assess the health of a project or group of projects at any point during the project lifecycle. Furthermore, the framework will be of particular use at the beginning of a project and also to help a newly appointed project manager assess the health of his or her projects. The diagnostic framework can be deployed to support assessment of projects through use of the diagrammatic tool depicted in Figure 3. This assessment tool is based on use of the web diagram technique, which although is a standardised diagrammatic tool it is based on the novel use of the six contributing sub-systems that have been developed in this research study. The web diagram technique has been used by other authors for different purposes, such as enabling natural resource managers to self-assess their adaptive capacity (Brown, Nelson, Jacobs, Kokic, Tracey, Ahmed et al., 2010). Moreover, Dvir and Shenhar (2007) have developed a web diagram view as part of the so called diamond approach for categorising projects. Nevertheless, adoption of the six sub-systems identified in this research study through the web diagram technique represents a novel application of the technique.
Furthermore, the web diagram technique supports the requirements for developing a project diagnostic framework through providing a convenient and rapid assessment of the health status for engineering projects. The tool can be used to effectively rate the project according to the six sub-systems (through being at a low, medium or high capability level) that have capacity to contribute to project performance.

Case study investigations
The following case study investigations are provided for illustrative purposes in order to highlight how use of the diagnostic health check tool could have potentially identified the underlying project issues that needed to be addressed for each of the cases. The case studies are independent of each other and have been selected in order to highlight the scope and applicability (including international relevance) of the diagnostic health check tool. The cases are drawn from the authors' collective experience in project management, which combined represents over 30 years. The case studies have been developed through a process of reflective inquiry (Schon, 1983) that has integrated both qualitative information and quantitative data in order to allow sensemaking (Weick, 1995) of the projects to be undertaken. This sensemaking is particularly useful in elucidating practitioner focused insights into the management of complex engineering and technology projects, which can often be characterised as being unstructured or at the 'fuzzy front end' (Khurana & Rosenthal, 1997). The names of the people mentioned in the engineering and technology case studies have been anonymised.

Case study #1: Construction project in Canada
Eugene was hired to take over as project manager of a $200 million construction project.
When he was parachuted into the job, eight months had passed on the scheduled three year total duration. Field construction was planned to kick off in four months, with the engineering design and procurement phases being reported as nearing completion.
Eugene started his career in the military and moved to the private sector with considerable experience in operations including sponsoring major projects, but he lacked any exposure to managing projects within a corporation. He was a firm believer in the concepts of chain of command and being personally accountable for one's work. Eugene's new employer executed few projects and operated within a hierarchical structure with project managers having little authority over the people assigned to fulfil the required tasks. During the first few weeks at the new job, Eugene learned the status of his project through oral reporting at regular meetings chaired by the company's president. All indications were that everything was on track and the project was on a solid financial footing.
Eugene began to feel discomfort regarding the validity of the assurances that the project was under control. Although equipment and materials were reported to be on schedule, he also heard from some people at the company that there was a policy of following just-in-time deliveries to reduce cash flow demands. Previous projects had encountered major impacts from late and wrong material deliveries. Therefore, he requested details of the purchased materials including delivery dates, details of how these materials would be transported as well as specifics on the budget versus forecast costs. He encountered considerable resistance from the people responsible for each of the areas for which he wanted data. Although the functional supervisors of the people working on the project appeared supportive, Eugene was instructed by his boss to back down on his demands. The basis of the request was that if everyone at the company was working toward a successful project, questioning the integrity of his co-workers was seen to show a lack of trust. Since he was new to the organisation, Eugene accepted that perhaps matters really were on track. He also noted that if these people were accepting accountability for their areas of responsibility, any subsequent problems would fall onto their shoulders.
After a few months, Eugene was feeling the heat as the project was clearly not on track.
Although many parts were on schedule, some were clearly late as evidenced by not arriving at site when they were due and some critical components had not yet been ordered. Although Eugene had influence over the people working with him on the project, their direct supervisors controlled the workers' salaries and therefore commanded greater power. The purchasers were indeed instructed by their functional management to postpone all expenditures until the latest acceptable timeframe. Shipping methods to be used from some vendors were unresolved and additional expediting costs were required to get these parts to site. Further analysis of the financial reports as information was uncovered showed that expenditures were not referenced against budget items. The total spent to date was assumed to indicate the project was on schedule, instead of the reality that it was both late and over budget on the costs already incurred. As the project manager, Eugene was seen by senior management to be the point of contact for all issues on the project and he was seen to have the final accountability for the project's performance. No one else was held to account for the overall project performance.
Eugene's predicament is more common than people not familiar with project management would expect. A project manager will be more successful by insisting at the onset on transparent and honest reporting, the authority to enforce directions and strengthening trust on the team through solid data to back assertions. Deploying the diagnostic and health check tool would have helped Eugene to undertake a system based assessment of the project from the outset and Figure 4 provides an illustrative view of such an assessment using this diagnostic approach. The tool identifies that although the technology and resources subsystems had a high capability level, the impact and knowledge sub-systems had a medium capability level with culture and process sub-systems being only at a low capability level.
Implementation of corrective steps to improve these capability levels could have helped recover the project and thereby help Eugene in his predicament.

Case study #2: Facilities development project in Canada
Susan was transferred from the company's fabrication plant to be the project manager of a field construction project that was building a new facility at a lump sum price for an operator client. The project had eight months remaining on the two year construction project. The previous project manager, Troy, had been fired due to being seen as unorganised and unresponsive to his management. The original scope of work had a forecast cost at completion near or slightly below the selling price. The project was, however, currently forecast to be acceptably profitable because of the profit expected from work extras that represented an additional 40% in value above the original contracted price. Once on the job, Susan thoroughly reviewed the scope and estimates for the final cost and agreed that the project would perform to the satisfaction of management.
After four months, Susan was presented with subcontractor invoices that were above the set price on the purchase orders. When she contacted one subcontractor to inquire about the high price, she was informed that the vendor had never agreed to any firm price and had only provided a rough estimate at the start of their work. When the extra work was added to the project scope, the cost of services would necessarily be higher than their original estimate.
When she reviewed the paper trail, Susan discovered that there was no record of a set price having been communicated with these vendors and Troy had mistakenly (or intentionally misleadingly) entered the purchases in the system as lump sum firm values. Some of the other project team stated they had heard Troy say that he had a verbal agreement with these vendors on a price, but there was no hard copy to support this claim. The site superintendent admitted that he had instructed these subcontractors to do additional work "since they were on site anyway and the changes helped make the finished product better." After six months on the job, Susan was told that the client had rejected her project's most recent progress payment invoice with the claim that the amount already exceeded their budget for the work. Susan contacted her counterpart at the client's office and explained that the additional cost was for the extra work that they had approved. The client representative sounded shocked and stated that they had approved some minor changes, but the total should not have been more than 5% above the contract price and much of the work now being invoiced was not wanted and should not have been performed if the cost was that much. Again turning to the superintendent, Susan discovered that operators and maintenance people working for the client had come to the construction site with special requests for changes and extra work. These people lacked the authority to approve extras and the client had clearly communicated the mechanism for managing scope changes on the project. In large, these instructions had not been followed for the extra work performed. In some cases the client expressed interest in considering some of the work and requested estimates after which they would issue formal change orders if they wanted it executed. No change orders had been issued.
Much of the extra work had been performed while Susan was Project Manager. It was difficult for her to explain that it was done without formal direction from the client. After the project was completed, the negotiated settlement with the client resulted in a one million dollar loss for Susan's company, instead of the profit being forecast when she took over. After completion, she was let go from her company.
It is common to take people at their word and when Susan was updated on the project status, she had no real reason to not believe what she was being told. It is also common for people to be hesitant in addressing matters around money and Susan may have been reluctant to remind the client of the higher price she was going to expect them to pay. Her problems could have been reduced by assuring that the information she received was understood in the same way by all parties in the contracts. Deploying the diagnostic and health check tool would have helped Susan to undertake a system based assessment of the project from the outset and Figure 5 provides an illustrative view of such an assessment using this diagnostic approach.
The tool identifies that although the resources sub-system had a high capability level, the technology, impact and culture sub-systems had a medium capability level with the process and knowledge sub-systems being only at a low capability level. Implementation of corrective steps to improve these capability levels could have helped recover the project and thereby help Susan in her predicament.

Case study #3: Research and technology projects in the United Kingdom
James was a project manager working for an independent research organisation and he managed a number of research projects involving synthetic chemistry, materials formulations and processing technologies for high-performance materials applications. He was informed that a fellow project manager, Henry, had resigned and was leaving the organisation and that James would be taking over responsibility for two of projects managed by Henry. One project was valued at £1.2 million and involved the development of processing technologies and the other project was valued at £1.8 million and involved mechanical testing of a series of polymer based formulations. Before his departure Henry declined several attempts by James to organise a project handover meeting but Henry did explain to James that the projects were all on track and performing well.
After Henry had left the research organisation, James arranged a project review meeting with the scientific team working on the main two projects he had taken over from Henry. At these meetings he discovered that the actual status and performance of both projects was rather different to the picture painted by Henry. The scientific staff informed him that there had been no project review meetings previously carried out and neither a project plan or task schedule had been prepared. The scientific team had been working in the related area to the two projects and consequently they had been booking time to the projects but the work carried out only partly met the requirements of the project. At this point James clearly become very concerned about the two projects and he then made further investigations, including reviewing the project expenditure on the organisation's ERP (enterprise resource planning) software, which revealed that one project was underspent while the other was overspent. James also had meetings with the senior management of the department in which both he and Henry had worked and these discussions revealed that Henry had been informing senior management that the projects had been progressing satisfactorily and they were on track to meet a series of major technical milestones due later in the year.
Once the full status of the projects was ascertained by James he was confronted with the need for urgent action in order to recover the projects. He quickly focused on preparing outline project management plans through capturing the views of the scientific staff working on the projects as well as the senior management views on the overall client's requirements for the projects. James started holding periodic project review meetings, which focused on reviewing technical progress across all the main projects tasks as well as providing updates on project expenditure, health and safety issues, risk management and progress towards the key project milestones. He also tried to improve the project culture through being more open and honest with other team members and with internal senior management.
Recovery plans were instigated that provided the necessary detail to enable the scientists to focus on the required experiments and trials needed to meet the project milestones due later in the year. In the case of the underspent project, the significant level of new work undertaken on the project allowed the expenditure to recover and at the end of the project the expenditure was within 5% of the financial limit of liability. Whereas in the case of the overspent project, the expenditure was already high, although once a detailed recovery plan had been assembled and reviewed, it was possible to re-negotiate the limit of liability with the project sponsor. A 20% higher limit of liability was agreed (increasing from £1.8 million to £2.2 million) and although some additional experiments were also required to justify this increase, it was nevertheless possible for this project's expenditure at the end of the financial reporting year to also be within 5% of the revised limit of liability. Both projects successfully delivered the technical milestones that were due later in the year.
This case highlights how corrective action when undertaken in a timely fashion and when focused in areas that were deficient can result in sustained recovery of the projects. In the case once James had been appointed, if he deployed the diagnostic and health check tool he would have been able to undertake a system based assessment of the projects from the outset and Figure 6 provides an illustrative view of such an assessment using the diagnostic approach. The tool identifies that although the impact sub-system had a high capability level, the resources, technology and knowledge sub-systems had a medium capability level with the process and culture sub-systems being only at a low capability level. In the case corrective actions were undertaken in the process and culture sub-systems and this resulted in a recovery for both projects. Figure 6. Application of health check tool to projects from case study #3

Conclusions and future work
The management of engineering and technology projects can be dependent on a wide range of critical success factors, including those related to process adoption, technology availability, resource configurations, broader impact, knowledge-based drivers and cultural considerations.
These factors can be viewed as being systemic in nature jointly impacting on the performance of projects and generating emergent effects, such as improved adoption of processes allowing enhanced use of technology within a project leading to improvements in knowledge availability for the project team members (resources). Consequently, this paper has formulated a new diagnostic framework and supporting health check tool based on an integrated systems view of engineering projects incorporating six contributing sub-systems. This new approach provides project managers with a convenient and robust approach to assess the health of projects. This will be of particular benefit to projects managers who have been newly appointed and are taking over responsibility for an existing project or group of projects. In this predicament the tool would help identify corrective measures that need to be undertaken to recover failing projects but it would also identify areas of a high capability level within the project that need to be maintained. Additionally, the diagnostic framework can be applied periodically and throughout the project lifecycle to measure performance across the six sub-systems that contribute to project performance.
The paper includes three illustrative case studies, two from Canada involving construction and facilities development projects and one from the United Kingdom involving research and technology projects. The projects in all three cases encountered difficulties, which for the two cases in Canada resulted in significant project overspend (in the first case study) and the loss of at least one job (the project manager in the second case study). Access to the diagnostic framework and health check tool would have provided a valuable lever and practical technique to help the respective project managers assess the project landscape for both of these projects and thereby identify and implement recovery plans to bring the projects back on track. In the third case study of research and technology projects in the United Kingdom, significant difficulties were encountered but the projects were successfully brought back on track. In this scenario, the health check tool provides an illustrative view of the predicament faced by the project manager in the case and it highlights how corrective steps were taken in focused areas to underpin the recovery of the two projects.
Future work is suggested on exploring the systemic nature, dependencies and connectivity between the sub-systems that are part of the integrated systems view of projects so that a greater understanding can be developed of how the diagnostic framework and health check tool can be deployed on complex engineering projects. Application of the framework and tool to different industrial sectors and applications is also recommended in order to investigate context specific factors and identify patterns in project trajectories, i.e. understanding whether projects in certain industries tend to exhibit representative profiles (high, medium or low capability levels) across the sub-systems in the framework. Industries that can be explored in this regard include the pharmaceutical, transport, telecommunications, defence and aerospace sectors.