David seems to have stirred up a bit of hornets’ nest with his piece about the negative value of the PMO, in project magazine. Having led a couple of PMOs and worked with many I can see the value in David’s arguments. A poorly run PMO can do little or nothing to support the delivery of successful projects in an organisation. They can be over obsessive about corporate reporting processes, following the process for the sake of the process and confuse accountability for the reporting of project progress. A well run PMO can provide support to project managers then they find makes a significant contribution to the successful delivery of the project, through providing the proper tools and skills to deliver the projects. This is especially true in organisation with lots of small and diverse projects which are not significant enough to invest in their own processes and system. However the PMO needs to remember it is there to improve project performance.

Advantages and disadvantages of a PMO

How the measure the value of a PMO?

How can we evaluate if the value of PMO exceeds its cost? Would the project managers pay for the PMO as part of the project costs? One way of evaluating the performance is to use them as an internal consultancy team, with targets to win support assignments from project managers. In this way they can generate income to invest in services systems, processes and training to offer the project managers. These project managers are only likely to buy the services that they perceive as adding real value.

“It’s a course in a book” is how this study guide describes itself. It is aimed at people who wish to pass the APMP but just cannot spare the time to attend the traditional workshops.

So, how does it fare? Getting a 5-day course into a book is a tall order, and at over 270 pages it could look a bit daunting, especially considering that you also need to have the APM’s Body of Knowledge alongside it. However, when you open the book and browse through a few pages you are immediately given a very good first impression. It looks accessible and interesting.

A few years’ ago I successfully studied for a similar professional examination using a guide rather like this one. I therefore have experience of how helpful these can be if they are put together in the right way; and I think that the authors have achieved just that.

There is a good balance of text, diagrams, multiple-choice quizzes, open questions and reflections on real-life practice. It is presented cleanly and clearly and has a nice work-book feel to it – I actually felt like I wanted to grab my pen and answer some of the questions.

With mock exam papers, glossary and exam tips it has all the items that you would expect from such a guide. But it’s the main sections of the book that bring the value, and here I think that it achieves its ambitions – “a course in a book”.

John Edmonds is Head of Training at pearcemayfield and has over 25 years of programme and project management experience in the educational and financial services sectors. He is one of the authors of the latest version of PRINCE2, and is the author of a blog called ‘The Opsimath’. John is chairman of the education, training and development charity ‘Roots’.

Do you want to carry out formal risk management?

This risk management practitioners course builds on the knowledge gained in our risk management foundation. This event leads to the APM’s Risk Management Level 2 Certificate, which is designed to determine an individual’s knowledge and understanding as well as capability in project risk management, sufficient to allow an individual to undertake formal project risk management.

APM Risk Certificate Level 2 Details:

Category Open APM Project Management Training Courses

Where: Manchester – Manchester

Date: 01.07.2010 -02.07.2010

Time: 09:00 -17:00

For More Details

Parallel Project Training are pleased to announce that they are launching two course based on the PMI body of knowledge (PMBoK)

PMP Certification (5-days) from £6999 for 14 people

The first course is a five day PMP certification course which meets the PMI requirements for 35 hours of directed learning. The PMBoK is covered in detail, including lots of exam hints and tips and a mock exam.

PMP Exam Prep course (3-days)

second is a PMP exam prep course ideal for those who are self studying for the PMP and covers the PMBoK overview focusing on exam prep and practice.

Both courses will initially be open to corporate clients for in-house training for up to 14 delegates. Full details are available on the Parallel Web site at PMP Certification Training from Parallel Project Training

Many organisations invest significant time, cost, and management effort in project management training. Project Management is about a team delivering clearly defined objectives to achieve a business benefit. How can we apply the same principles to Project Management Training Programmes? How can we measure the benefits of the project management training? How do we assess the acquisition of new knowledge and skills and the application of these in the work place? How do we measure the business benefits that flow from these new approaches to project management? Can we produce a realistic but prudent assessment of Return on Investment for a project management training programme? This paper looks at the current thinking in training evaluation and how it can be applied to Project Management development programmes. 

Overview of Training Evaluation

‘The first important contributions and still the most influential’ framework for training evaluation (Bramely, 1996) was proposed in 1959 by Donald Kirkpatrick’s “Techniques for Evaluating Training Programmes”(Kirkpatrick D.L., 1996). He identified four levels against which objectives should be set and evaluated. These are: 

Level 1: The reaction of the trainees to the programme 

Level 2: Measuring the amount of Learning of princples, facts, skills and attitudes 

Level 3: Changes in Behaviour
in the job 

Level 4: Results or changes in organisational effectiveness. 

More recently, Jack Philips (2003) proposed return on investment (ROI) as the fifth level in the Kirkpatrick framework. Measurement of ROI is important in Project Management Training Programmes because they must demonstrate a return from the significant investment in training. Other models exist such as CRIO (Warr, Bird and Rackham, 1974) and Peter Bramely’s evaluation against Effectiveness, Behaviour, Knowledge, Skills and Attitudes. Neither of these approaches have the clear link to the measurement of the impact on the business. CRIO focuses on the event itself while Bramely’s model lacks the structure and the link to return on investment. Hence, for this paper we will use Kirkpatrick’s four levels as extended by Jack Philips to include ROI and it’s application to project management training programmes. 

Measuring ROI for Training Programmes

To measure the ROI from the programme we need to follow the steps shown in figure 1. In summary the first step is to develop plans and baseline data, this includes plans for data collection at each of the lower four levels in the model (reaction, learning, application and business impact). Next, we must isolate the effects of the programme from other changes taking place in the business. Once these have been isolated, we can convert the business improvements into a financial return and together with a record of the full cost of training (both direct and indirect) we can calculate an ROI. Benefits that do not deliver a measurable financial return are intangible benefits. In the next section, we identify specifically how to measure ROI for a Programme. 

Application of ROI Model to BNFL ES Programme

Stage 1 Develop Evaluation Plans and Baseline Data

A number of data collection methods are available at different levels (see box 1). For example, tests are exclusively limited to the evaluation of learning (level 2) while focus groups capture data on application and business impact. The first step is planning which method of data collection is appropriate to the organisation and tabulate these decisions in a data collection plan as shown in Table 2. This defines the objectives, measures, data collection method, timing and responsibility for the data collection. 

Table 1 Data Collection Methods used at each Level
 

Table 2 Sample Data Collection Plan for Project Management Training Programmes

Level 1 reaction Level Evaluation

Evaluation of programme at level 1 uses a questionnaire completed immediately at the end of the courses. Typically the questionnaire covers the areas of briefing, course material (including content), delivery, outcomes, administration and further training requirements. To ensure 100% of the forms are returned they are collected at the end of every session. A target is usually set of achieving an average of at least Good (4 out of 6) across all these area. 

Level 2 Learning

Three mechanisms that are proposed to measure the effectiveness of learning. First, the programme includes a number of formal external exams and assessments and second the programme design was based on a knowledge based training needs assessment / competence assessment by line managers. Finally workplace assignment can assess the extent to which the knowledge can be applied in the workplace . 

The first approach utilises the progressively graded examinations and assessment administered by the Association of Project Management (APM) as shown in figure 1. These summative criterion references tests, evaluate that participants have acquired knowledge at the levels defined by the APM. 

The second approach will be to repeat, after say 12 months, the initial knowledge based Training Needs Analysis, based on tools. These norm-based tests will evaluate the extent to which the knowledge of the participants has improved. 

The final approach is to use work placed assignments. These can be designed so that participants have to demonstrate that they are able to apply the knowledge gained in a work place setting. They also support the application of the knowledge gained which is evaluated at the next level. 

Level 3 Application

The level of application cannot be evaluated until some time (3 to 6 months) after the training has been completed. However, three sets of measures are proposed. First, a review of action plans using telephone interviews 3 to 4 weeks after the training. During the interview participants should be asked about specific examples of how they have used the new knowledge from the training. The weakness of this approach is the difficulty validating the benefits of the individual actions. Second, focus groups could be used at each site to discuss the application knowledge gained during the training programme, these should be completed 3-4 months after the training. While the focus groups will provide significant feedback on the application of the learning, they may lack external credibility and may be difficult to get full coverage of a large population. The third measure is external audit and/or internal review to determine the extent to which the skills and processes have been implemented within day-to-day management of the projects. This approach, while expensive, has external credibility. 

Level 5 Business Impact

The ultimate value of the programme is its impact on business performance. The measures at this level need to reflect the objectives of the organisation. However they would typically include: 

1. Completion of project milestones to plan
   

1. Completing the full scope of work
   
2. Accelerating work through change control
   
3. Meeting Environmental, Health and Safety Requirements
   

The targets for these measures need to be agreed based on current performance and anticipated improvement. However not all the changes in these measures will be due to the training programme, so we must find ways to isolate the impact of the training from the other changes that are taking place in the organisation.
 

Isolating the Effects of Training

It is very unusual for training to be the only initiative or change taking place in a business. In fact the demand for learning is often triggered by an external change in the business environment or an internally driven change from within the business. It is unjustified to claim that all the improvement (or deterioration) in business performance is due to a training programme. Philips (2003) identifies several ways to isolate the effect of training, see Box 2. If baseline data exists, then trend analysis is the most convincing and reliable measure. It would be worthwhile supplementing this with focus groups and estimates from senior managers to validate the results of the trend analysis and improve buy-in. 

Converting to Monetary Data

To calculate ROI we need to both measure the return and the full cost of the training programme. The return can be often derived from the improvement in business measures however Philips has identified a number of guiding principles to ensure that the results are prudent. These are: 

1. When a higher level evaluation is conducted it must be supported by data collected at lower levels. (i.e. poor quality training (level 1) is unlikely to lead to significant business improvement)
2. When an evaluation is planned at a higher level, the previous level of evaluation does not need to be comprehensive. (i.e. if current business measures are being used to monitor the implementation then the need to project audit is reduced)
3. When collecting and using data only use credible sources, it is better to have a lower but justifiable ROI than a measure based on poor quality data.
4. When analysing data always choose the most conservative among alternatives
5. At least one method must be used to isolate the effects of training
6. If no improvement data is available for a population assume that no improvement has been occurred.
7. Estimates of improvements should be adjusted to use worst case estimates to account for potential errors
8. Extreme data items or unsupported claims should not be used for ROI calculations
9. Only the first year of benefits should be used for short term programmes
10. All costs should be loaded into the ROI calculation
11. Intangible benefits are those that are deliberately not to be converted into robust monetary benefits.
12. The results of ROI measurement should be communicated to all stakeholders.

 Following Phillip’s guiding principles ensures that the reported ROI is conservative and will be credible with the sponsors of the programme. 

Discussion and Conclusions

The initial very positive level 1 reaction feedback is a critical initial step towards the overall success of the programme. Equally, if not more important, is supporting and stimulating the application of the new knowledge within projects. Monitoring and evaluation of these higher levels will be a vital part of the feedback loop that ensures the overall success of the programme. Plans for monitoring these higher levels using trend line analysis of existing business measures, telephone interviews, focus groups and estimates from senior managers have been proposed. 

Bibliography

Bramley, P. ‘Evaluating Training’ Chartered Institute of Professional Development, 1996, ISBN 0-85292-636-7 

Philips J, ‘Return on Investment in Training and Performance Improvements Programs’, third edition 2003, Butterworth-Heinemann, 2003, ISBN 0-7506-7601-9 

Kirkpatric, D. L. ‘Techniques for Evaluating Training Programs’, Training and Development, American Society for Training Directors, Jaunary 1996. 

For more information on the benefits of project management training

What is meant by a complex project?

The world if full difference when we talk about complex projects. Is complexity the same as complicated? By complex do we just mean a large project, one which is geographical distributed or using complex technology? Or are these mega project just more complicated.

Complexity

Within business complexity is defined as

“Complexity describes the variances in approach and their consequences on the business”

I think complexity is closely related to ambiguity of project goal and method of delivery, because complexity is caused by the challenges of planning the project in the first place and likely level of anticipated change. This complexity makes it impossible to sit down and plan the project in a clear way from start to end; we need a slightly more flexible approach to the traditional “plan the work” “work the plan”. This can be represented by an adaptation of Rodney Turners classic goal method matrix

The higher the levels of ambiguity about the project goal and uncertainty about the delivery method make it more difficult to manage in a traditional way. This seems more important than the pure size of the project. Some large complicated projects have clear goals and known methods and these can be managed in a very traditional way. This definition differs from the normal multi organisation; multi-phase definition defined by the APM, but is nevertheless more relevant to help think about the complexity of the projects we manage.

Comments?

We are pleased to announce that we will be holding our first 5-day APMP in London on the 17th May. The pre-study material for this course includes a copy of our study guide, access to our e-learning, podcasts and pre-course forum support. Book now for 20% early bird discount at APMP Project Management Training London 

The winner of our APMP study guide competition was Kevin from Benfleet. He is delighted with his prize as he is a new project manager just starting his professional journey. Commiserations to all those who missed out but as compensation we are offering £10 discount on our APMP study guide when you purchase it from www.parallelprojecttraining.com. Just use the “comp2012″ as a coupon code at checkout.

Parallel Project training are launching the world’s first ever how to pass the APMP Webinar on the 3^rd May. Scheduled at 18:00 it is ideal for those who are self studying for the Open APMP exam on the 10^th May.

With the authors of our APMP study guide this webinar gives you all the trainer hints and tips for passing the APMP exam. This 2 hour session will include:

1. The structure of the APMP exam.

2. Managing time in the APMP exam.

3. What sort of questions will come up in the APMP?

4. How to structure answers to help the examiner award marks.

5. The APM project management mind set.

6. APMP Exam management tips.

The webinar will be highly interactive and can cover hot topics, such as earned value as requested by the participants. We recommend listening to our podcasts and getting a copy of our APMP study guide before the webinar. For details click on here

The good people at project management and recruitment specialists Arras people are to test out the Parallel Learning System by self studying for the APMP. They will be working as a learning set of four, using the Parallel APMP study guide, Podcasts and e-learning modules. They will be posting daily updates and reports on their blog how to manage a camel. It will be interesting to see how they all get on. To help everyone who is self studying we have set up an APMP self study forum on our website. Post any questions and we will help you. Remember we are with you all the way