The Cost of Drug Development

There is some debate about the real cost of drug development but what you can be sure of is that it is a very large number and increasing year on year. Back in 2003 the US Journal of Health Economics (DiMasi, 2003) published a figure of US$ 802m as the estimated cost of a pharmaceutical companies bring a new drug to market. This figure was based on the research and development of 68 drugs launched by 11 pharmaceutical companies. More up to date estimates suggest that these cost have risen to US$1.3 billion or even as much as US$1.7 billion (Collier, 2009). According to CenterWatch less that 10% of clinical trials are completed on time. The largest causes of delays are contract and budget negotiation, patient recruitment and enrolment and protocol design and refinement. (CenterWatch, June 2009). With a typically drug trial costing between $5m and $15mm each, these delays lead to a significant overspends. We argue that the root cause of these delays and overspent is insufficient commitment to the feasibility studies needed to optimise the study protocol and commercial relationships between the sponsor (Pharmaceutical Company), clinical research organisation (CRO) and investigator sites. We that propose a more collaborative approach to these feasibility studies based on the principles of value management could significantly reduce these overspends and identify significant reductions in cost. 

Reasons for delays and overspends

The typical clinical research procurement process has a major limitation because the CRO project managers and clinical research associate (CRA) who have all the experience implementing studies are often not closely involved the development of the initial protocol and procurement process.  This leads to the following issues:

1)    Project objectives and protocol are often defined by the Sponsor without input from either the in-house or out-sourced research teams hence these objectives can be arbitrary and unrealistic.

2)    The feasibility is sometimes done in isolation from the CRO which are going to complete the study or the CRO is asked to do feasibility as part of the selection process. As a result many assumptions about the best way to implement the protocol are made within either no or insufficient input from the CRO. The assumptions made at this point in the process may not reflect the most practical approach to key areas like recruitment of patients, selection of centres etc, which can have a profound impact on timelines and costs at a later stage.

3)    During the procurement process CROs rarely have longer than two week to tender for the work and the contracts are usually some form of fixed price. The aim of the CRO at this point is to win the contract and limit the liability. Very little time exists to do any proper feasibility study into the proposed protocol and the CRO is in sales mode so is unlikely to be honest about a poor quality approach. The more likely strategy is to ensure that the scope is tightly defined and CRO is protected from the weakness in the protocol by clearly defined assumptions which pass the risk back to the sponsor.

4)    During implementation phase virtual all project plans need to be amended, often in response to a more realistic assessment of site set up times, patient recruitment and practicalities of the protocol.  The CRO will expect the sponsor to cover the costs of these amendments. Many of these changes could have been readily foreseen in the feasibility phase if the people involved in completing the study were involved in the design of the protocol.

Furthermore the design may not allow for new innovations in techniques which could have saved significant time and increased patient recruitment and retention. Typical examples include:

• The protocol does not reflect the typical medical practice in the selected country.
• Site set up is delayed by contract negotiations with investigational sites.
• Competitive studies change site selection.
• Over optimistic recruitment due to insufficient risk analysis.

In summary may studies fail because they set unrealistic timelines, don’t do adequate feasibility, sometimes they end up starting projects well into the process. As a result many trial runs late and the cost overruns become significant. For a £10m study an overspend of 10% is not unusually. We seem to have forgotten the advice of Abraham Lincoln “Give me six hours to chop down a tree and I will spend the first four sharpening the axe.”

We believe that more collaborative but challenging approach to the design and procurement of clinical trials, with more focus on the feasibility phase, could results in significant savings and a reduction in the delays by addressing the root causes early in the process.  Value Management is a widely recognized framework to improve the collaboration and cooperation between the parties in projects.

Value Management; a Step by Step Approach to Collaboration

Many sectors facing the same challenge of getting the best value out of the supply chain, have moved towards a value management approach. The UK office of Government Commerce (Office of Government Commerce, 2007) has summarised a wide range of case studied in which value management has saved significant project cost. For this approach an integrated feasibility team (from both the sponsor and preferred supplier(s)) focus on value not cost by:

• Understanding and challenging the relative importance of the sponsor objectives. This helps define realistic criteria for the success of the project from the outset and prevents the adoption of arbitrary timescales. Value management ensures that everyone understands the relative importance of the project objectives, together with the project constraints and risk from an early stage.
• Identifying alternative courses of action including different techniques, approaches and methods which may be more cost effective. The early input from CRO project managers and CRA’s into the design of the protocol can significantly improve the selection of the most appropriate investigative sites, diagnostic techniques and data collection processes.
• Analysis and evaluation of different approaches against defined criteria to select the best option. This practical and collaborative input to feasibility reduces the chance of delays and additional costs introduced by amendment of the protocol once the study has started.

The change however in not just procedural it is also cultural with sponsors and suppliers adopting a more open approach to collaborative working. Some of the areas for change are:

• Joint leadership and commitment from the top, with a collaborative approach to project management. This eliminates the need to duplicate project managers in the Sponsor and the CRO with the formation of an integrated study team.
• Rewards for CRO against balanced scorecard which measure the overall success of the study financially, in terms of quality, for the staff and the customers, not just the just cost. Ideally CRO and Phama staff incentives would be linked to the successful completion of the project within the agreed objectives.
• An integrated knowledge management database using shared electronic data management systems. Much has been done in this area to use technology to improve the flow of information within the study teams but an integrated approach using an integrated system has potential to further improve the follow of information between the sponsor and the CRO. 
• A flexible work force with strong team working (secondments etc) between the partners in the project in this way the members of the Phama team get an appreciation of the challenges faced by the CRO and vica-versa.
• Multi disciplinary team working across all phases of the study to bring the key people involved in the design, feasibility and implementation of the study together at critical times.  
• Most controversially the use of open book accounting in which the true costs of the study are open and reward is based on a more holistic assessment of performance.

Bridging the Gap between Feasibility and Implementation

The processes for value management are well defined and include clear steps completed at each of the stages. A number of workshops are conducted at each stage of the project. The aim is to bring together the different teams in a collaborative view of the projects objectives and strategy. The main studies conducted are:

VM 1: stakeholder needs, objectives and priorities

The primary objective of value management at concept stage is clarifying why the need exists, to ensure that the project objectives have been thoroughly analysed and understood. The workshop would be conducted early in the development of the protocol with input from the CRO who will be bidding for the contract to complete the study. It gives the CRO the opportunity to understand the sponsor’s objectives and influence the approach for the protocol to avoid unnecessary complications and unrealistic constraints.

VM2:  project definitions and options

The purpose of VM2 workshop is to ensure that the project options are evaluated and the one which offers best value for money is selected.

Selecting the best value option requires:

• Review the validity of the objectives.
• Agree modifications.
• Evaluate the feasibility of options.
• Develop the preferred option (best value).
• Determine if the preferred option (best value) can be improved or enhanced.
• Agreed recommendations about the preferred option.
• Produce a programme for development.
• Develop an action plan.

This workshop would ideally be conducted during the feasibility phase just prior to the issue of the RFP so that the CRO organisations who are bidding can bring their experience to the design of the protocol.

VM3/4: value engineering to reduce cost

Value Management during the feasibility stage is termed ‘Value Engineering’. The objectives are to ascertain elements of the study design and specification that add no value to the project in terms of satisfying the brief and objectives or those which could be provided through another method thus increasing value. It could be completed between the submission of proposals for the study and the award of contract. Typically one preferred bidder would be asked to participate in the value engineering workshop.

Value Engineering provides a structured approach to ensure that the most cost-effective means of implementing the project are identified.

VM 5/6 Value Reviews

Value reviews repeat the exercise undertaken in VM 3/4 with during the implementation of the study. Value reviews shall be undertaken as part of the project’s change control mechanism to allow the opportunity to value manage / the project to determine if the functionality could be altered and be more cost effectively so the project may be delivered within the original budget.

Ideally a value review must be undertaken at least once during start up and implementation phases to identify more effective ways of working. More frequent reviews may be necessary due to the complexity of the project.

Conclusion for Clinical Research

Value management is a tried and trusted way to reduce the overall cost of projects and has a successful track record in many sectors. It brings together the complete project team to understand the challenges and identify smarter, quicker and more cost effective ways of completing the project, often by simple changes to the approach and requirements. The clinical research and development sector could learn much from these more collaborative and value focused approaches. For more information including clinical trials project management training visit www.parallelprojecttraining.com

1.   Bibliography

CenterWatch. (June 2009, June). U.S Sites Rate Medpace, Kendle and ICON at TOP CROs in 2009. The CenterWatch Monthly .

Collier, R. (2009, February 3). Drug development cost estimates hard to swallow. Canadian Medical Association Journal , 180(3): 279–280.

DiMasi, H. G. (2003). The price of innovation: new estimates of drug development costs. Journal of Health Economics , Vol 22, pp. 151-185.

Office of Government Commerce. (2007). Achieving Excellence in Construction. Retrieved from Office of Government Commerce: http://www.ogc.gov.uk/documents/CP0064AEGuide4.pdf

Using the integrated Parallel Learning System Parallel Project Training has launched a distance learning package for the APM project management qualification APMP. The APMP is ideal for project managers who want a practical approach to project management and is the ideal complement to PRINCE2. The package offers unrivalled range of distance learning materials including

1. A high quality printed study guide. This APMP study guide covers the entire scope of the APMP qualification and complement completely the APM Body of Knowledge (BoK).
2. On-line APMP e-learning in eight modules including
   1. How to Pass The APMP Exam
   2. Project Management in Context
   3. Organisation and Governance
   4. Techniques
   5. Business and Commercial
   6. Planning the Strategy
   7. Executing the Strategy
   8. People and the profession
3. Podcasts cover most of the APMP topics. These have been highly acclaimed. Once listener said

I just wanted to thank you for the Podcasts you have on iTunes.

I recently left the Forces after 22 years and attended an APMP course as part of my resettlement.

I downloaded your podcasts and listened to the each evening and when commuting.

Thanks to them I passed my APMP.

1. Online tutor support via our moderated forum. Post questions here and we will answer them within 24 hours.
   
2. An opportunity to take the APMP exam the APM open exam days or at one of our public courses in London or Reading. The exam starts at 13:00 on the last day of the course.
   

For more information and APMP Distance Learning from Parallel Project Training visit

We’re with you all the way

This is a summary of the presentation to the South West Branch of the APM on the 19^th May 2010. It summarises the presentation and the feedback from workshop sessions to identify the barriers to the implementation of earned value.

Introduction

Earned Value Management (EVM) is a fundamental process in both the APM’s and PMI bodies of knowledge and has its roots in the United States Department of Defence (DOD) Cost/Schedule Control Criteria System (C/SCSC), which has been specified for all DOD contracts for which the government carries the risk since 1967.

Earned Value Management is a simple practical concept with significant benefits for project managers including:

• An integrated view of the three key elements of project status, planned cost, project progress and actual expenditure.
• The ability to rapidly determine the status of a project based on performance indexes.
• The ability to rapidly highlight work packages that are overspending and / or behind plan.
• An estimate of cost at completion and completion date based on current performance.

And yet very few project managers’ use earned value management. Those that do, tend to be mostly in the defence, nuclear and some transport sectors. In this paper we argue that successful implementation of EVM is closely linked to a development of an organisations project management maturity.

Barriers to Wider Use of EVA

Based on research with PMI and APM seminars we can identify the barriers to adoption of EVM as being focus around the three elements of process, people and systems and the implementation of the fundamental principles of basic project management. In summary these barriers are:

Embedded processes to establish project scope, define realistic but challenging norms based plans, rules assessing progress and effective control of project change are all vital processes that must be in place to support the implementation of EVM. Without these fundamental processes then the earned value data will soon lose consistency and soon become discredited.

People within the organisation need the aptitude, training and discipline to implement effective EVM. They need the ability and training required to understand and use the data produced, they need to be familiar with the jargon used and trust it to make decisions. Some project managers see the increased visibility provided by EVM as a threat to there ability to manage the clients and their own senior managers expectations. Data needs to be provided within an understood context.

The systems which support EVM need to be fed with high quality baseline data with integrate cost and time information within the plan and accurate and timely cost information in a structure consistent with the project Cost Breakdown Structure (CBS). The latter of these often requires an interface to the organisations finance systems and possible modification to coding structures.

During the workshop we identified many potential barriers and aids to the implantations of earned value in organisations. These include:

Road to EVA Maturity

Critical to the successful implementation of EVM is the effective application of the core fundamentals of the project control cycle. In particular the following key elements need to be in place for each stage of the project control cycle

Plan the Work

The plans used for EVM need to implement key elements of good practice. They must integrate cost and time information through a resource and cost loaded Gantt chart. The work packages within this plan must be integrated with the CBS against which we are going to track costs. Finally it is useful if a rolling wave approach is used to development the detail within the plan a stage by stage.

Work the Plan

It is a basic tenet of project management that we work in accordance with the plan, but in many organisations this can be a real challenge, either because the plan lacks credibility, buy-in or commitment from the project team. For these reason many individuals or teams execute tasks in different sequences from the planned activities, complete work not included within the plan or ignore it all together. Detailed integrated baseline review of the project plan before baseline can ensure that the team has a detailed understanding of the project plan and iron out difficulties before implementation starts.

Measure Progress

Progress can be measured or estimated in several ways, from measurement of physically completed work to estimating the work left to complete. However in all but the most physical projects this estimate progress is too subjective. To effectively implement EVM we need defined rules for assessment of progress on work types and these need to be understood and adhered to across the organisation. For example for a design or software project we might make a rule that activities only earn value once they are complete and which point they earn all the value associated with that task.

Take corrective action and manage change

Uncontrolled or sluggish control of change can cause major problems for the implementation of EVM. Completing activities not shown on the plan, changes in scope or changes to the sequence of activities need to be captured and incorporated into the baseline through an efficient change control system. Without a timely and rigorous change control system the reports and performance measures produced by EVM systems will be inaccurate, misleading and could lead to incorrect management action.

For an organisation to benefit from the implementation of EVM the organisation needs the level of PM maturity which enables it to:

1. Prepare and agree robust plans integrating both time and cost
2. Execute work according to the plan
3. Accurately measure progress and record cost in a format consistent with the plan
4. Implement efficient and effective change control process.

These in fact are the corner stones of good project management. So the question remains as to what level of maturity is required to successfully implement EVM?

The Project Management Maturity Model

A Project Management Maturity meets these needs, drawing on established concepts from existing models such as the Capability Maturity Model (CMM) from Carnegie-Mellon Software Engineering Institute and the Business Excellence Model from the European Foundation for Quality Management (EFQM). Project Mangement Maturity describes four levels of increasing project management capability, termed Naïve, Novice, Normalised, and Natural. The aim is to provide a structured route to excellence in project management, with recognisable stages along the way which organisations can use to benchmark themselves against.

The various levels are defined as follows:

1. The Naïve project management organisation (Level 1) is unaware of the need for management of projects, and has no structured approach to projects. Management processes are repetitive and reactive, with little or no attempt to learn from the past or to prepare for future threats or uncertainties.

1. At Level 2, the Novice project management organisation has begun to experiment with project management, usually through a small number of nominated individuals, but has no formal or structured generic processes in place. Although aware of the potential benefits of managing projects, the Novice organisation has not effectively implemented project management processes and is not gaining the full benefits.

1. The level to which most organisations aspire when setting targets for management of projects is captured in Level 3, the Normalised project management organisation. At this level, management of projects is built into routine business processes and project management is implemented across all aspects of the business. Generic project management processes are formalised and widespread, and the benefits are understood at all levels of the organisation, although they may not be fully achieved in all cases.

1. Many organisations would probably be happy to remain at Level 3, but we can define a further level of maturity in project management processes, termed the Natural project management organisation (Level 4). Here the organisation has a fully project-based culture, with a proactive approach to project management in all aspects of the business. Project-based information is actively used to improve business processes and gain competitive advantage.

Earned Value Management and Project Maturity

We are proposing that earned value management requires a foundation of project maturity before it begins to add value.

For organisations at the “Naïve” level of maturity EVM will be over complex and
inappropriate, project management development efforts should concentrate on basic roles and responsibilities, definition of project objectives, scope and basic timeline planning and control.

The ad hoc processes within a “Novice” organisation are unlikely to respond well to the rigour and visibility introduced by the introduction of EVM, significant problems are likely with robustness of baseline plans, progress reporting, understanding within the organisations of EVM performance measures. However if the organisation has the commitment to develop to a normalised level of performance then EVM, especially required by a client, may be the vehicle to achieve this improvement, although it may take some time for the benefits to be realised.

EVM is ideally suited to a ‘Normalised’ organisation seeking to move towards the best practices exemplified by a ‘Natural’ organisation. The rigor, integration and visibility introduced by EVM will rapidly highlight any weakness in the organisations people, processes or systems and act as a driver to stimulate management action to further improve the organisations capability.

Organisations that have reached a ‘natrural’ level without using EVM for project control are likely to benefit little from its introduction. It is something that they may do if they have a client need or particular requirement. With a mature organisation then the changes required to implement EVM should be relatively painless giving increased levels of confidence in the project delivery capability.

Conclusions

EVM offers project management the promises of significant benefits, however very few project managers have adopted it because of significant barriers within their organisations. These barriers are linked to the project management maturity of an organisation. In order to be ready to adopt EVM organisations should be at least at a ‘Normalised’ level of project management and for these organisations EVM will give significant benefits and support their development. In addition, to enable really effective decision making EVM needs a maturity of understanding and support by Senior Managers. Project Managers will then, not only be helping themselves and their projects to understand whether they are ‘on target’, but also their Senior Managers. This should prevent the necessity of Project Managers to continually having to sell and explain the concept to their superiors.

Earned Value

Is it worth the effort?

Earned value promises significant benefits to project managers including

•    An integrated system to measure cost and schedule performance.

•    The ability to measure efficiency and progress towards the project goal.

•    Ability to quickly identify underperforming work package and instigate corrective actions.

•    Accurate forecasting of outturn cost and time.

Despite these valuable benefits earned value management (EVM) is not widely adopted by project managers. In this interactive seminar we will explore why earned value is not used as much as it should be. What are the barriers to wider adoption of EVM? Will technology help us overcome these barriers?

The seminar will start with a review of EVM principles, benefits and implementation processes. We will then several review case studies of organisations who have adopted EVM, the benefits they experience from the introduction of EVM, barriers they have experienced and the critical enablers of successful EVM implementation.

Click here to download slides

The need for capable and professional project managers is clear to many businesses. With ever increasing complexity, risk and financial implications, modern society demands that projects are undertaken with a clear and apparent expectation that the project not only finishes as planned and to budget, but that any potential hitherto unknown opportunities are converted into material financial benefit. In order to do this the hypothesis is that we need professionally competent project managers.

It is virtually impossible to be categorical about the benefits of training. This is because it is rather self evidently impossible to re-run a project and compare it with a control project.

In the final analysis we will never really know, but this argument could be used in any sphere of professional life. Would we want our heart operations carried out by unqualified people, should our train drivers be people without the requisite training? Of course not. If training does not improve competence then let’s stop training aircraft pilots and see what happens.

There are three main perspectives on a project and the advantages to be gained from having a qualified and competent hand at the rudder of the project in which they have an interest.

The Customer – where a significant investment has been made in the creation of some new facility, service or product, the customer is the one who has paid. What most customers expect is the delivery of what they asked for, when they wanted it and for the price they expected to pay. They expect at some point for the cost to be overtaken by the benefits derived.

A competent project manager will be equipped to push out these boundaries to extend wherever possible the benefit derived from the known scope either by under spending, over delivering or beating the schedule. They will also seek out opportunities to extend the scope, after discussion and with due consideration to take advantage of changing circumstances or new information.

The Supplier – in a lot of organisations the project manager is delivering on behalf of the supplier be they an internal service provider or on the other end of an external contract. Either way the customer organisation is a separate entity to the project managers’. The suppliers want to get rewarded in line with their expectations set early in the concept stages. They do not want to be extricating the project manager (and themselves) from difficult conversations at every turn. They expect the job to get done without fuss and difficulty.

A competent project manager will make sure that they are fully aware of the needs and constraints of their own organisation. They will go to great lengths to understand the nature of the requirement on the part of the customer and make sure they are set up to deliver those components. They will also make sure that there is as little uncertainty as possible both in terms of the required deliverable and also the manner in which it will be delivered. It is this risk reduction that marks out the truly competent project manager.

The project manager – wants to deliver the job on time and on budget and to get rewarded for doing so. Apart from immediate praise and financial benefit, most competent PM’s have an eye to the bigger picture. They will continue to be able to deploy their excellent project management skills only for as long as they are offered the opportunity. The best way to continue this virtuous circle is to deliver the current one as well as is humanly possible.

Competent project managers will continue to seek out new and challenging opportunities to maintain their own marketability and remuneration potential. They will be lifelong learners, eager to demonstrate their continued professional development and contribution to the general knowledge and capability of their own profession.

Why should an organisation seek to develop its project managers? What’s in it for them?

The benefits of project management training for organisations are:

•    They will have a pool of qualified individuals from which to draw the best placed individual for the job in hand. This pool will be comprised of the trained individuals. This will grant an organisation the option to choose who does the next job, knowing they are all just as capable as each other.

•    The amount an organisation spends on its project training will have a direct result on its project success. If an organisation spends 7% of its project budgets on training successful achievement of its objectives will rise from 30-70% (source IDC).

•    The trained project managers will be familiar with concepts and terms in a consistent and clear way. They will be able to repeat their experiences with reference to their successes of the past and develop a huge lessons learned capability. This standardized approach reduces re-work and cost.

•    The project maturity of an organization is partly a function of the competence of the people employed and it has been shown that there is a direct correlation with a nominal maturity level and the overspend (or otherwise) of that organizations projects.

•    In an environment where there is a rigid structure of financial incentives to manage projects well, the introduction of personal development can be a huge factor in the retention of good staff and their motivation. Training staff can be a relatively low cost mechanism for providing motivation when other forms are not available.

•    An organisation with a reputation for good or even excellent project management will bring with it a following of potential recruits and plaudits, raising morale, profile and confidence. This will result in easier recruitment, a higher calibre of staff and teams and a groundswell of satisfaction.

Why would an individual want to undertake project management training?

What is in it for the individual?

•    Individuals can improve their capability and learn new knowledge, so they can practice new skills in the workplace. This use of a wider range of tools and techniques will raise their own confidence giving them greater confidence in their ability.

•    They can recognize their role as a profession and relate directly to others in the same situation. This allows a direct synergy in building support groups and networks, giving a second order access to advice and help.

•    They will be able to pursue aspirations to develop a professional career, with the increased recognition of professional standards and certification. This brings with it necessary continual professional development, but this merely acts to cement lifelong development and focus.

•    Where their acquired knowledge is transferable they will have the opportunity to move organizations and adopt similar roles elsewhere. This is self evident.

David seems to have stirred up a bit of hornets’ nest with his piece about the negative value of the PMO, in project magazine. Having led a couple of PMOs and worked with many I can see the value in David’s arguments. A poorly run PMO can do little or nothing to support the delivery of successful projects in an organisation. They can be over obsessive about corporate reporting processes, following the process for the sake of the process and confuse accountability for the reporting of project progress. A well run PMO can provide support to project managers then they find makes a significant contribution to the successful delivery of the project, through providing the proper tools and skills to deliver the projects. This is especially true in organisation with lots of small and diverse projects which are not significant enough to invest in their own processes and system. However the PMO needs to remember it is there to improve project performance.

Advantages and disadvantages of a PMO

How the measure the value of a PMO?

How can we evaluate if the value of PMO exceeds its cost? Would the project managers pay for the PMO as part of the project costs? One way of evaluating the performance is to use them as an internal consultancy team, with targets to win support assignments from project managers. In this way they can generate income to invest in services systems, processes and training to offer the project managers. These project managers are only likely to buy the services that they perceive as adding real value.

Many organisations invest significant time, cost, and management effort in project management training. Project Management is about a team delivering clearly defined objectives to achieve a business benefit. How can we apply the same principles to Project Management Training Programmes? How can we measure the benefits of the project management training? How do we assess the acquisition of new knowledge and skills and the application of these in the work place? How do we measure the business benefits that flow from these new approaches to project management? Can we produce a realistic but prudent assessment of Return on Investment for a project management training programme? This paper looks at the current thinking in training evaluation and how it can be applied to Project Management development programmes. 

Overview of Training Evaluation

‘The first important contributions and still the most influential’ framework for training evaluation (Bramely, 1996) was proposed in 1959 by Donald Kirkpatrick’s “Techniques for Evaluating Training Programmes”(Kirkpatrick D.L., 1996). He identified four levels against which objectives should be set and evaluated. These are: 

Level 1: The reaction of the trainees to the programme 

Level 2: Measuring the amount of Learning of princples, facts, skills and attitudes 

Level 3: Changes in Behaviour
in the job 

Level 4: Results or changes in organisational effectiveness. 

More recently, Jack Philips (2003) proposed return on investment (ROI) as the fifth level in the Kirkpatrick framework. Measurement of ROI is important in Project Management Training Programmes because they must demonstrate a return from the significant investment in training. Other models exist such as CRIO (Warr, Bird and Rackham, 1974) and Peter Bramely’s evaluation against Effectiveness, Behaviour, Knowledge, Skills and Attitudes. Neither of these approaches have the clear link to the measurement of the impact on the business. CRIO focuses on the event itself while Bramely’s model lacks the structure and the link to return on investment. Hence, for this paper we will use Kirkpatrick’s four levels as extended by Jack Philips to include ROI and it’s application to project management training programmes. 

Measuring ROI for Training Programmes

To measure the ROI from the programme we need to follow the steps shown in figure 1. In summary the first step is to develop plans and baseline data, this includes plans for data collection at each of the lower four levels in the model (reaction, learning, application and business impact). Next, we must isolate the effects of the programme from other changes taking place in the business. Once these have been isolated, we can convert the business improvements into a financial return and together with a record of the full cost of training (both direct and indirect) we can calculate an ROI. Benefits that do not deliver a measurable financial return are intangible benefits. In the next section, we identify specifically how to measure ROI for a Programme. 

Application of ROI Model to BNFL ES Programme

Stage 1 Develop Evaluation Plans and Baseline Data

A number of data collection methods are available at different levels (see box 1). For example, tests are exclusively limited to the evaluation of learning (level 2) while focus groups capture data on application and business impact. The first step is planning which method of data collection is appropriate to the organisation and tabulate these decisions in a data collection plan as shown in Table 2. This defines the objectives, measures, data collection method, timing and responsibility for the data collection. 

Table 1 Data Collection Methods used at each Level
 

Table 2 Sample Data Collection Plan for Project Management Training Programmes

Level 1 reaction Level Evaluation

Evaluation of programme at level 1 uses a questionnaire completed immediately at the end of the courses. Typically the questionnaire covers the areas of briefing, course material (including content), delivery, outcomes, administration and further training requirements. To ensure 100% of the forms are returned they are collected at the end of every session. A target is usually set of achieving an average of at least Good (4 out of 6) across all these area. 

Level 2 Learning

Three mechanisms that are proposed to measure the effectiveness of learning. First, the programme includes a number of formal external exams and assessments and second the programme design was based on a knowledge based training needs assessment / competence assessment by line managers. Finally workplace assignment can assess the extent to which the knowledge can be applied in the workplace . 

The first approach utilises the progressively graded examinations and assessment administered by the Association of Project Management (APM) as shown in figure 1. These summative criterion references tests, evaluate that participants have acquired knowledge at the levels defined by the APM. 

The second approach will be to repeat, after say 12 months, the initial knowledge based Training Needs Analysis, based on tools. These norm-based tests will evaluate the extent to which the knowledge of the participants has improved. 

The final approach is to use work placed assignments. These can be designed so that participants have to demonstrate that they are able to apply the knowledge gained in a work place setting. They also support the application of the knowledge gained which is evaluated at the next level. 

Level 3 Application

The level of application cannot be evaluated until some time (3 to 6 months) after the training has been completed. However, three sets of measures are proposed. First, a review of action plans using telephone interviews 3 to 4 weeks after the training. During the interview participants should be asked about specific examples of how they have used the new knowledge from the training. The weakness of this approach is the difficulty validating the benefits of the individual actions. Second, focus groups could be used at each site to discuss the application knowledge gained during the training programme, these should be completed 3-4 months after the training. While the focus groups will provide significant feedback on the application of the learning, they may lack external credibility and may be difficult to get full coverage of a large population. The third measure is external audit and/or internal review to determine the extent to which the skills and processes have been implemented within day-to-day management of the projects. This approach, while expensive, has external credibility. 

Level 5 Business Impact

The ultimate value of the programme is its impact on business performance. The measures at this level need to reflect the objectives of the organisation. However they would typically include: 

1. Completion of project milestones to plan
   

1. Completing the full scope of work
   
2. Accelerating work through change control
   
3. Meeting Environmental, Health and Safety Requirements
   

The targets for these measures need to be agreed based on current performance and anticipated improvement. However not all the changes in these measures will be due to the training programme, so we must find ways to isolate the impact of the training from the other changes that are taking place in the organisation.
 

Isolating the Effects of Training

It is very unusual for training to be the only initiative or change taking place in a business. In fact the demand for learning is often triggered by an external change in the business environment or an internally driven change from within the business. It is unjustified to claim that all the improvement (or deterioration) in business performance is due to a training programme. Philips (2003) identifies several ways to isolate the effect of training, see Box 2. If baseline data exists, then trend analysis is the most convincing and reliable measure. It would be worthwhile supplementing this with focus groups and estimates from senior managers to validate the results of the trend analysis and improve buy-in. 

Converting to Monetary Data

To calculate ROI we need to both measure the return and the full cost of the training programme. The return can be often derived from the improvement in business measures however Philips has identified a number of guiding principles to ensure that the results are prudent. These are: 

1. When a higher level evaluation is conducted it must be supported by data collected at lower levels. (i.e. poor quality training (level 1) is unlikely to lead to significant business improvement)
2. When an evaluation is planned at a higher level, the previous level of evaluation does not need to be comprehensive. (i.e. if current business measures are being used to monitor the implementation then the need to project audit is reduced)
3. When collecting and using data only use credible sources, it is better to have a lower but justifiable ROI than a measure based on poor quality data.
4. When analysing data always choose the most conservative among alternatives
5. At least one method must be used to isolate the effects of training
6. If no improvement data is available for a population assume that no improvement has been occurred.
7. Estimates of improvements should be adjusted to use worst case estimates to account for potential errors
8. Extreme data items or unsupported claims should not be used for ROI calculations
9. Only the first year of benefits should be used for short term programmes
10. All costs should be loaded into the ROI calculation
11. Intangible benefits are those that are deliberately not to be converted into robust monetary benefits.
12. The results of ROI measurement should be communicated to all stakeholders.

 Following Phillip’s guiding principles ensures that the reported ROI is conservative and will be credible with the sponsors of the programme. 

Discussion and Conclusions

The initial very positive level 1 reaction feedback is a critical initial step towards the overall success of the programme. Equally, if not more important, is supporting and stimulating the application of the new knowledge within projects. Monitoring and evaluation of these higher levels will be a vital part of the feedback loop that ensures the overall success of the programme. Plans for monitoring these higher levels using trend line analysis of existing business measures, telephone interviews, focus groups and estimates from senior managers have been proposed. 

Bibliography

Bramley, P. ‘Evaluating Training’ Chartered Institute of Professional Development, 1996, ISBN 0-85292-636-7 

Philips J, ‘Return on Investment in Training and Performance Improvements Programs’, third edition 2003, Butterworth-Heinemann, 2003, ISBN 0-7506-7601-9 

Kirkpatric, D. L. ‘Techniques for Evaluating Training Programs’, Training and Development, American Society for Training Directors, Jaunary 1996. 

For more information on the benefits of project management training

What is meant by a complex project?

The world if full difference when we talk about complex projects. Is complexity the same as complicated? By complex do we just mean a large project, one which is geographical distributed or using complex technology? Or are these mega project just more complicated.

Complexity

Within business complexity is defined as

“Complexity describes the variances in approach and their consequences on the business”

I think complexity is closely related to ambiguity of project goal and method of delivery, because complexity is caused by the challenges of planning the project in the first place and likely level of anticipated change. This complexity makes it impossible to sit down and plan the project in a clear way from start to end; we need a slightly more flexible approach to the traditional “plan the work” “work the plan”. This can be represented by an adaptation of Rodney Turners classic goal method matrix

The higher the levels of ambiguity about the project goal and uncertainty about the delivery method make it more difficult to manage in a traditional way. This seems more important than the pure size of the project. Some large complicated projects have clear goals and known methods and these can be managed in a very traditional way. This definition differs from the normal multi organisation; multi-phase definition defined by the APM, but is nevertheless more relevant to help think about the complexity of the projects we manage.

Comments?

The social media revolution

The last ten years have see an explosion in social media with an estimated 307.4million unique visitors to social network sites in December 2009, an increase on 82% on the previous year (Nielsen Company, 2010). Many dozens of sites exist but the most recognisable and relevant to project managers are probably LinkedIn and Twitter, but these goliaths of the cyber world are not alone. Social networking sites abound, the growth of easy to use ‘blogging’ sites and the availability of open source software (such as WordPress) means that just about anyone armed with very little technical knowledge can express and exchange ideas in an interactive way with many thousands of others in a matter of days. The combined social networks and blogs are now the most popular form of online activity, according to the market research company Nielsen.

Social networks do not need to be external to an organisation like LinkedIn. An internal social network is a closed or private community that consists of a group or people within a company, association, society, education provider or other selected list. These facilities are often created for specific groups to support the real social networks within an organisation to help establish communities of practice to share knowledge, information and experience about a particular area or business interest.

The project management community have been part of this revolution with 2.8million project managers registered on LinkedIn alone. Resourceful project managers are always looking for tools to help them manage projects more effectively but are these social networking tools a help or an unwanted diversion with the potential to detract from delivery focus? In this article we will look at how social networks could be one part of the solution to the management demands of our increasingly complex projects.

The Growing Complexity of Global Project Management?

Projects have not escaped the globalisation trends. Modern technologies enable teams to work around the world with apparent ease. But in so doing they introduce for themselves a whole raft of new problems than did not emerge when teams were all co-located in a single geography or even office. This dispersal is a one of the key factors in the nature of the projects complexity.  The problems created as a result stem from several factors all of which have their roots in ambiguity and uncertainty. Research in the Australian Commonwealth Defence Department identified seven themes which contribute to this increased complexity (Remington, 11-13 October 2009) which can be summarised as:

These complexity factors put our traditional classical assumptions of project management under strain. We assume that we can reduce the complexity by de-composing the project into ever more tightly defined and simpler compartments. Paradoxically it has been shown that these reductionist approaches in fact lead to increased complexity within the project as each further level of breakdown generates just more interfaces that need to be managed and controlled.

A “clockwork universe” in which the future is predictable and repeatable with the outcomes proportional to the inputs does not exist for anything other than relatively simple projects. As the complexity increases these rules of predictability become even less reliable. We work in vain to increase formal control over workers activity to obtain more predictable outcomes whereas in reality this merely leads to more bureaucracy and less control in the real world of the project.

We also assume we can eliminate all major risks through identification, assessment planning and management but fail to recognise the dramatic effect of the ‘unknown unknowns’ associated with projects involving diverse and geographically distributed project teams.

In reality, apart from very simple co-located projects these factors were never true but we never really faced up to the fact. As teams become more diverse these paradigms are stretched to breaking point and now demand a re-think.

The Management of Complex Projects

The management of complexity is a topic of much debate (Whitty SJ, 2008) and a full discussion is beyond the scope of this article but although the guidance is still developing, current thinking would steer the project manger to consider a number of key areas;

Each of these would justify a volume of work on their own. The social networking theme of this month’s magazine drives us to consider the last topic here, that being the contribution that a developed social networking facility can make to reducing complexity and making the project mangers efforts more effective.

Can a social network help?

Every project relies on an informal communications network to support the more structured approaches enshrined in communication plans, stakeholder management plans, reporting procedures and so on. These informal meetings used to happen in the corridor, at the ‘water cooler’, coffee shop, elevator, in the car park. They are a key component of the social glue which keeps the project team moving in the right direction, highlighting risks, resolving issues and cementing progress before they get anywhere near the more formal systematic project management umbrella.   This glue holds a project together. It represents the linkages between individuals and coalitions within the wider project team.  Each social network has a ‘social capital’ which forms from the sum of the knowledge in the network, the strengths of the links between members, the alignment of the actors objectives, the collaboration between them and the effectiveness of the team leadership. (Hanifan, 1920). The skill of the best project mangers has always been to balance the use of the formal project approach with empathy for the social network to build a high performing team. With increasingly divers geographic teams we have lost much of this social glue.

Social Networking in the Virtual Team

Recognising the importance of the project social network is the first step in understanding how fundamental it is to the successful delivery of project. In a virtual team it is not possible to bring groups together on a regular basis (if at all), individuals cannot be forced into interacting with people they have never met. Could it be that the use of the social network tools will allow us to replace the lost interaction of the co-located team with a surrogate version hosted in one of these systems? Could they offer a replacement for that chance corridor meeting, the informal conversation over the skinny latte with extra caramel, the beer after work?

It’s possible that as with a traditional co-located team, debates and discussion could take place outside the constraints of formal communication, with on-going discussion of project issues and risks using the informal networks between team members. It might just be that we do not need to provide a procedure so individuals can interact with each other. Now there’s a thought.

Would I put my project on facebook?

A public forum like Facebook or LinkedIn may not be the most appropriate place to build a project network for obvious security and confidentiality reasons. The technology is so cheap and available though that this functionally can be easily replicated on a company intranet (available only within the organization) of extranet (which is available to partners). Typically based on propriety or open source platforms they have the potential to support and facilitate this valuable informal communication and knowledge sharing across a geographically dispersed project team. Many organistions already use internal discussion boards to provide technical support or project blog sites to keep a diversely distributed project team in touch with the latest project news. The extension of these approaches to develop a project social network is a relatively easy step.

If it’s such a good idea, why don’t we all do it?

We have been socially interacting for millions of years and arguably doing reasonably well as a result. As soon as a new mechanism, concept or idea comes along there is a natural human tendency to seek out opportunities from it but at the same time a reluctance on the part of the ‘establishment’ to change. Eventually we adapt but it can take a while. This cultural reticence stands in the way of an immediate and wholehearted adoption of things (like) social networking systems. This will over time wash out of the system as people reared and inculcated on these new and innovative techniques take their place amongst the traditionalists.

For now though there are a few things that need to be considered by an organization considering adopting such a system;

Conclusion

Modern communication technology has enabled projects to become increasingly fragmented and multinational adding exponentially as they go to the complexity of trying to manage them using the traditional methods. The co-located social networks that used to support the development of a performing team are increasingly more difficult to develop and maintain. Social networking systems seem to offer one way of maintaining this loose, unstructured but essential contact so as to underpin a more diverse project team. These tools are not without their dangers but rest assured that the downsides will not be solved through misguided attempts to regulate them. How effective will they be? It has taken thousands of years to become used to the written word, hundreds for telephones, tens for visual tools, perhaps the dawn of a new and more liberated way of interacting may be closer than any of us think. Ask a young adult just out of university, here’s a quote from one of them.

“I think social media, when used effectively, can be very useful for project management. Project teams are increasingly composed of young professionals. Rather than stifle their interest in social networking, why not utilize the creativity and initiative? Applications such as Twitter and Facebook can be effective tools for communicating in real time with project team members. Information sharing tools such as Digg, StumbleUpon, and Google Buzz are great ways to send relevant articles and research papers to the team as well as promote your project”

Watch this space!

Bibliography

Hanifan, L. J.-1. (1920). The Community Cente. Boston: Silver Burdett.

Nielsen Company. (2010, January 22). Led by Facebook, Twitter, Global Time Spent on Social Media Sites up 82% Year over Year. Retrieved March 2, 2010, from Nielsen Wire: http://blog.nielsen.com/nielsenwire/global/led-by-facebook-twitter-global-time-spent-on-social-media-sites-up-82-year-over-year/

Remington, K. a. (11-13 October 2009). A model of project complexity :distinguishing dimensions of complexity from severity. Proceedings of the 9th International Research Network of Project Management. Berlin.

Whitty SJ, M. H. (2008). And then came Complex Project Management (revised), . Int J Project Management , doi:10.1016/j.ijproman.2008.03.004.

Parallel Project Training

John Bolton and Paul Naybour both co-founders of Parallel Project Training the newest APM accredited training provider are enthusiasts in the adoption of any mechanism to make learning more accessible and effective and are committed to the development of a profession of project managers. Parallel Project Training uses modern technology such as podcasts, e-learning, communities of practice, moderated forums and webinars to complement traditional printed study guides and training workshops. Join them at a free webinar on the 19th April to discuss project complexity, social networking and the issues raised in more detail. Visit www.parallelprojecttraining.com to book your place.

During the past decade project management has undergone a transformation with through greater codification and formal certification. This certification and been based on the Association for Project Management (APM), Prince2TM and Project Management Institute (PMI) under taken by seemingly ever increasing number of organisations and individuals. With over half a million members of the PMI, 250,000 Prince2 Practitioners and 17,500 members of the APM, the world has seen a rapid growth in project management certification, much of it in the last ten years.

Traditionally this certification has been delivered as five days in a hotel with an exam on a Friday. Is this really the best way of developing project managers, or was it organised this way for the convenience of the training organisation and the HR department?

The weaknesses of the current 5-day project management training paradigm.

Project Managers are inherently activists, they learn by doing, so to remove them from a project for five days to develop new skills, always creates an artificial environment. Learning cannot be applied to the work place in bite size pieces. Putting the exam at the end of the course creates the pressure to focus on passing the test and not learning new skills. Some enlightened organisations recognised this limitation and committed to slightly longer modular programmes of two times 3 days or even 3 times 2 days. This at least gave time for individuals to apply what they were learning as they went. However in the tougher economic times organisations are beginning to challenge the paradigm of a week out of the office to pass a qualification. The challenge now is how to cover the wide range of project management topics in less time without reducing the quality of the learning.

Innovation has the solution

As with many such challenges technical innovation has the solution. Learning does not have to be done in the classroom. We can learn in many ways, through on the job exercise, reading books listening to the radio, playing on the computer. All the learner needs is access to the right material to help and guide the learning process. This needs to be available in a wide range of media to suit the learning style and approach of the individual. This is what is offered by the Parallel Learning System.

Parallel Learning System

The Parallel Learning System provides a wide range of learning activities including to a study guide printed as a book, e-learning, podcasts and workshops. Each can be used individually or as a combined approach to learning. Each element is widely available, the book and e-learning are easily purchased from Amazon and the podcasts can be downloaded free of charge from the web or iTunes. This whole approach means that the workshops can focus on skills development and preparing for the exam.

Click here for more information on innovative approaches to project management training.