Download the presentation on the Donald Kirkpatrick model. Library of articles from the HR League: Model for assessing the effectiveness of training D

Training assessment is carried out to understand how to increase its effectiveness, in what ways it can be improved.

In this regard, it is proposed to answer the following eight questions:

· To what extent does the training content meet the needs of the participants?

· Is the choice of teacher optimal?

· Does the teacher use the most effective methods to maintain participants' interest, impart knowledge, and develop skills and attitudes?

· Are the training conditions satisfactory?

· Are participants satisfied with the class schedule?

· Do audiovisual aids improve communication and maintain participant interest?

· Was the coordination of the program satisfactory?

· What else could be done to improve the program?

Learning assessment has four levels:

1. Reaction

2. Learning

3. Behavior

4. Results

First level assessment - reaction determines how program participants respond to it. When training is conducted internally, the response of participants is not always interpreted as customer satisfaction. The fact is that participation in such trainings is mandatory. People simply have no choice. The company's management determines the need for this training and obliges employees to take part in it. It would seem that in this case, we need to talk about the reaction of management. Kirkpatrick emphasizes that in this case too The reaction of the participants is a very important criterion for the success of the training, for at least two reasons. Firstly, people share their impressions of the training with their management, and this information goes higher. Consequently, it influences decisions about continuing training. Secondly, if participants do not respond positively, they will not be motivated to learn. According to Kirkpatrick, a positive reaction does not guarantee the successful development of new knowledge, skills and abilities. A negative reaction to training almost certainly means a decrease in the likelihood of learning.

Learning determined as changing attitudes, improving knowledge and improving the skills of participants as a result of their completion of the training program. Kirkpatrick argues that changes in participants' behavior as a result of training are only possible when learning occurs (attitudes change, knowledge improves, or skills improve). Behavior - n At this level, an assessment is made of the extent to which the participants' behavior has changed as a result of training. Kirkpartick points out that the lack of changes in participants' behavior does not mean that the training was ineffective. Situations are possible when the reaction to the training was positive, learning occurred, but the behavior of the participants did not change in the future, since the necessary conditions for this were not met. Therefore, the lack of change in the behavior of participants after the training cannot be a reason for making a decision to terminate the program. Kirkpartick recommends that in these cases, in addition to assessing the reaction and learning, check the presence of the following conditions:

· Participants' desire to change behavior.

· Participants have knowledge of what and how to do.

· Availability of an appropriate socio-psychological climate.

· Reward participants for changing behavior.

TO results refers to changes that occurred due to the participants undergoing training. As examples of results, Kirkpatrick cites increased productivity, improved quality, decreased accidents, increased sales, and decreased employee turnover. Kirkpatrick insists that results should not be measured in money. He believes the changes listed above could, in turn, lead to increased profits. Assessment at this level is the most difficult and expensive. Here are some practical tips that can help you evaluate your results:

· if possible, use a control group (those who did not receive training),

· carry out the assessment after some time so that the results become noticeable,

· whenever possible, conduct assessments before and after the program,

Conduct the assessment several times during the program,

· compare the value of the information that can be obtained through the assessment and the cost of obtaining this information.

Training assessment is carried out to understand how to increase its effectiveness, in what ways it can be improved.

In this regard, it is proposed to answer the following eight questions:

· To what extent does the training content meet the needs of the participants?

· Is the choice of teacher optimal?

· Does the teacher use the most effective methods to maintain participants' interest, impart knowledge, and develop skills and attitudes?

· Are the training conditions satisfactory?

· Are participants satisfied with the class schedule?

· Do audiovisual aids improve communication and maintain participant interest?

· Was the coordination of the program satisfactory?

· What else could be done to improve the program?

Learning assessment has four levels:

1. Reaction

2. Learning

3. Behavior

4. Results

· Participants' desire to change behavior.

· Participants have knowledge of what and how to do.

· Availability of an appropriate socio-psychological climate.

· Reward participants for changing behavior.

· if possible, use a control group (those who did not receive training),

· carry out the assessment after some time so that the results become noticeable,

· whenever possible, conduct assessments before and after the program,

thesis

1.6 D. Kirkpatrick’s model for assessing the effectiveness of training

As has been mentioned several times above, the problem of assessing the effectiveness of training still remains relevant for many companies involved in the development of their personnel.

It is interesting that there is still controversy surrounding the approach proposed by Kirkpatrick; back in the late 1950s, he formulated the now famous 4 levels of performance assessment, but his training evaluation model remains one of the basic ones, and his books are the most cited. Kirkpatrick views assessment as an integral part of the training delivery cycle, which includes 10 stages:

1. Determination of needs.

2. Setting goals.

3. Definition of subject content.

4. Selection of training participants.

5. Formation of an optimal schedule.

6. Selection of appropriate premises.

7. Selection of appropriate teachers.

8. Preparation of audiovisual media.

9. Program coordination.

10. Program evaluation.

Kirkpatrick believes that in most cases, evaluation is limited to the use of post-training questionnaires - studying the immediate reaction of trainees to the training. He calls these questionnaires “smile-sheets,” meaning that participants most often use the questionnaires to express gratitude.

Kirkpatrick's four levels define the sequence in which learning assessments are conducted. He writes: “Each level is important and affects the next level. As you move from level to level, the assessment process becomes more difficult and time-consuming, but also produces more valuable information."

· Level 1 - Reaction

· Level 2 - Learning

· Level 3 - Behavior

· Level 4 - Results

1. Reaction

Evaluation at this level determines how program participants respond to the program. Kirkpatrick himself calls this a customer satisfaction score. When training is conducted internally, the response of participants is not always interpreted as customer satisfaction. Often management determines the need to participate in an educational program. Kirkpatrick emphasizes that in this case, the reaction of the participants is a very important criterion for the success of the training, for at least two reasons.

· Firstly, people one way or another share their impressions of the training with their management, and this information goes higher. Consequently, it influences decisions about continuing training.

· Secondly, if participants do not respond positively, they will not be motivated to learn. A positive reaction does not guarantee the successful development of new knowledge, skills and abilities. A negative reaction to training almost certainly means a decrease in the likelihood of learning.

2. Learning

Learning is defined as the change in attitudes, improvement in knowledge and improvement in skills of participants as a result of their completion of a training program. Changing the behavior of participants as a result of training is possible only when learning occurs (attitudes change, knowledge improves, or skills improve).

3. Behavior

At this level, an assessment is made of the extent to which the participants' behavior has changed as a result of training. Kirkpartick points out that the lack of change in participants' behavior does not mean the training was ineffective. Situations are possible when the reaction to the training was positive, learning occurred, but the behavior of the participants did not change in the future, since the necessary conditions for this were not met. Therefore, the lack of change in the behavior of participants after the training cannot be a reason for making a decision to terminate the program. In these cases, in addition to assessing reaction and learning, it is recommended to check for the following conditions:

· Participants' desire to change behavior.

· Participants have knowledge of what and how to do.

· Availability of an appropriate socio-psychological climate.

· Reward participants for changing behavior.

Speaking about the socio-psychological climate, Kirkpatrick refers primarily to the immediate supervisors of the training participants. He identifies five types of “climate”: prohibitive, discouraging, neutral, supportive, demanding. The manager’s position, accordingly, changes from a prohibition on changing behavior to a requirement to change behavior after the end of the training. Kirkpatrick believes that the only way to create a positive climate is to involve leaders in curriculum development.

4. Results

Outcomes include changes that occurred due to participants receiving training. As examples of results, Kirkpatrick cites increased productivity, improved quality, decreased accidents, increased sales, and decreased employee turnover. Results should not be measured in money.

According to Kirkpatrick, assessment at this level is the most difficult and expensive. Here are some practical tips that can help you evaluate your results:

· if possible, use a control group (those who did not receive training),

· carry out the assessment after some time so that the results become noticeable,

Conduct pre- and post-program assessments (if possible),

Conduct the assessment several times during the program,

· compare the value of the information that can be obtained through the assessment and the cost of obtaining this information (the author believes that conducting an assessment at level 4 is not always advisable due to its high cost).

The fundamental disadvantages of this approach include the fact that the selected information blocks are not interconnected; in addition, there are no feedback mechanisms to manage the process of generating and transferring knowledge. Other difficulties are associated with the undeveloped procedural level of information collection and its processing. As a result:

1) generalized numerical characteristics do not objectively reflect states and relationships in the measured empirical system;

2) the measurement techniques used do not have the properties of validity, sensitivity and reliability;

3) the influence of external factors is not taken into account;

4) it is not clear how to isolate the “net effect” of training.”

This article is devoted to a comparative analysis of various models of personnel training effectiveness, including the most common models: Kirkpatrick, Berne, Stufflebeam and Phillips. The models in question were studied on the basis of secondary information, including a study carried out by the British company Embrion, as well as a study by the American company ASTD from a qualitative point of view: the composition, features, limitations and advantages with disadvantages of each of them were analyzed. The comparison results were verified using an expert survey carried out by the authors regarding the accuracy and prevalence of each approach to assessing the effectiveness of training. Based on the results obtained, conclusions were drawn regarding the applicability of each model in the practice of Russian companies. In addition, the authors offer some recommendations regarding the use of these approaches to obtain greater economic benefits.

Philipps model

Bern model

Stufflebeam model

Kirkpatrick model

personnel development

training effectiveness assessment

1. Kirkpatrick D.L., Kirkpatrick D.D. “Four steps to successful training: a practical guide to assessing the effectiveness of training” Moscow 2008. pp. 50-51.

2. Mironov V. Assessment of training according to Kirkpatrick: the test of time [Electronic resource]. – Access mode: http://www.top-personal.ru/issue.html?2185 (date of access: -01.12.14).

3. Oparina N.N. Assessing the effectiveness of training and development of top managers [Electronic resource]. – Access mode: http://istina.msu.ru/media/publications/articles/e4f/7fc/422942/Otsenka_effektivnosti.pdf (access date: 11/10/14).

4. Skiba E. Topic March 06: assessing the effectiveness of training. Digest on nine main assessment models [Electronic resource]. – Access mode: http://www.trainings.ru/library/articles/?id=6328 (access date: 12/5/14).

5. Casebourne I., Downes A. Evaluation of learning – a family affair? [Electronic resource]. – Access mode: http://epiclearninggroup.com/uk/files/2013/10/WP_Evaluation_of_-learning.pdf (access date: 11/22/14).

6. Passmore J., Velez M.J. SOAP-M: A training evaluation model for HR // Industrial & Commercial Training, 2012. Vol.6, No. 44. P. 315-326.

7. Topno N. Evaluation of Training and Development: An Analysis of Various Models // Journal of Business and Management, 2012. Vol. 5, No. 2. P. 16-22.

In modern conditions of globalization and fierce competition in national and international markets, one of the factors for increasing competitiveness is improving the quality of the organization’s personnel. However, the total amount of investment in personnel retraining, or various trainings aimed at developing the qualities of an individual employee necessary for the company, cannot serve as an accurate assessment of the effect obtained. Thus, a significant part of the problem is the question of how to evaluate the effectiveness of investments in personnel development. For this purpose, separate models for assessing the effectiveness of training were developed, which mainly used the Kirkpatrick approach.

Kirkpatrick model

Without assessing the effectiveness of training, it is impossible to build a training and development system that provides the necessary business results. In 1954, Donald Kirkpatrick defended his PhD thesis at the University of Wisconsin (USA) on the topic “Evaluating the Effectiveness of Program Management.” He proposed a short formula to describe the learning cycle: reaction - learning - behavior - results (see Figure 1). This scheme is necessary for effective training of company personnel, as well as for obtaining the necessary business results.

Fig.1 Kirkpatrick model

The first level of learning is reaction. Feedback lets you know what clients think about the program, based on the responses, the trainer needs to make some changes to the program to improve it. The first level assessment, “Reaction”, is very important, the trainer needs to know the participants’ opinions about his program, and the listeners must also be sure that the trainer is not indifferent to their opinion.

Second level: acquired knowledge. It’s good if the participant is satisfied with the training, but this does not mean that they learned anything. To assess the knowledge gained, tests are used to test knowledge of the material studied, and skill checklists are also used. Testing is as follows: some time after completing the course, participants are invited to a conversation with an experienced colleague or supervisor to find out what knowledge they have learned from the training.

Third level: behavior. Kirkpatrick identifies this level as the most important and difficult. It is at this level that an assessment is made of how the participants’ behavior has changed as a result of training, and to what extent the acquired knowledge and skills are applied in the workplace.

Level four: results.

The main thing in assessing the fourth level is to imagine the end result. The trainer's task is to realize the expectations of stakeholders, i.e. payback.

Donald Kirkpatrick's method shows us how to turn the learning process into an effective business tool, and integrate individual training into an organization, making training specifically corporate. It explains in detail how to plan, how

set goals, how to argue with managers for the need for training, and, very importantly, how to eliminate unnecessary training if it does not solve the stated problems.

In practice, however, not all four levels are used, but most often only two, in descending order of importance (based on a study conducted by ASTD - American Society for Training and Development) (see Fig. 2).

Fig. 2 Organizations' use of levels of the Kirkpatrick model

Thus, assessing the effectiveness of the learning process must be approached from the perspective of financial management and quantify the results of such investment. A solution to this problem was proposed in 1997 by Jack Phillips.

Model by Jack Phillips

In the early 90s, the famous HR expert Jack Phillips developed a methodology for assessing the effectiveness of investments in employee training. In his work, published in 1975, Kirkpatrick expressed an unexpected thought: “Under no circumstances should you think of translating the fourth level (business result) into money. It’s not necessary, you can’t.” And in 1991, another person, Jack Phillips, not only stated that this was possible, but also proposed a calculation algorithm. To the four levels already mentioned, another one has been added - return on investment, or calculation of the ratio of the profit from the project and the costs of it.

This indicator allows you to evaluate the effectiveness of training. Phillips' methods work well in companies with regular management.

ROI helps to obtain a number of benefits, in particular for company managers: assess the financial effectiveness of investments in employees; obtain a clear and reliable tool for determining the effectiveness of HR activities; make the “human factor” and its impact on the company’s business results measurable. First of all, an ROI model is necessary to calculate the return on investment in human resources and personnel programs, when management invests in people, they need to understand what the return on this investment is.

Stufflebeam model (CIPP)

The Stufflebeam abbreviation (see Fig. 3) is deciphered as follows:

Rice. 3 Stufflebeam model (CIPP)

The model allows you to evaluate both the results and the process of learning and development. Therefore, this model can be successfully applied for long-term modular training programs for top managers.

Bern model (CIRO)

Similar to the Stufflebeam model is the CIPO scheme (see Fig. 4), developed by Bern. The model includes the following assessment steps:

Rice. 4 Bern model (CIRO)

This model can also be used to assess the effectiveness of training for top managers, especially in the context of launching modular long-term programs.

Advantages and Disadvantages of Effective Teaching Models

	Advantages	Flaws
Kirkpatrick	The presence of four levels of assessment: by the reaction of participants, by the results of final control, by changes in production behavior and by changes in the company’s business indicators. Possibility of choice to evaluate the effectiveness of top manager training.	The difficulty of measuring the third level, the unwillingness of top managers to undergo the assessment procedure or work with coaches. Refusal to evaluate based on financial indicators.
	Development of the Kirkpatrick model. Introduction of the fifth level of training and development effectiveness assessment. The ability to use a financial return on investment in training and development.	Calculation of return on investment is possible only if full-fledged management financial accounting is maintained in the organization. It is mainly used to calculate the effectiveness of those programs that are lengthy, expensive and complex.
Stufflebeam	The ability to evaluate both the process and the results of training and development in close connection with the company's goals.	Unclear methods and procedures for assessing the results of training and development of top managers.
	Development of the Stufflebeam and partly Kirkpatrick model. Setting goals, identifying opportunities, identifying participants’ opinions on training and development, evaluating results.		The difficulty of maintaining a constant procedure for assessing the effectiveness of training and development of top managers, application for long-term programs.

The table provides a comparison of the most popular models and methods for assessing the effectiveness of personnel development and training in terms of advantages and disadvantages, including financial, calculation complexity, and so on.

It is worth noting that today there is no ideal model of effective training that would include all the necessary assessment parameters. Each of the presented models has its pros and cons, which are clearly presented above.

The authors conducted a survey of 7 experts in the field of HR on the most accurate models for assessing the effectiveness of training (each expert had to choose the most accurate models, in his opinion, from the proposed list). The survey data is presented below (see Figure 5):

Figure 5. Survey results on the accuracy of training models

This survey showed that the most accurate learning models are the Kirkpatrick model and the Phillips model, and the least accurate are the Stufflebeam model and the Berne model.

Conclusion

In conclusion, we can say that it is necessary to take into account the differences between Western learning culture and ours. In the West, corporate training has long become an indispensable condition and part of business life; there is no need to explain many things to employees. In Russia, the attitude towards business training has now begun to change towards interest, and the technologies themselves are just beginning to be introduced. Thus, the interviewed respondents showed that mainly Western companies most often use effective training models. The most accurate models are those of Kirkpatrick and Phillips. The least accurate models include the Stufflebeam and Berne models; according to respondents, these models are least often used both in the West and in Russia.

Kirkpatrick's model shows how to turn the training process into an effective business tool and integrate individual training into an organization. Phillips' model helps an organization evaluate the financial results of training and development. The Stufflebeam model allows you to evaluate both the results and the learning process itself. Berne's model helps us determine the goals, opportunities, and opinions of training participants. Thus, we can say that each of the presented models has its own advantages and disadvantages. Currently, there is no ideal model that would include all the parameters for assessing effective training at once.

Reviewers:

Putilov A.V., Doctor of Technical Sciences, Professor, Dean of the Faculty of Management and Economics of High Technologies, Ministry of Education and Science of the Russian Federation, Federal State Autonomous Educational Institution of Higher Professional Education "National Research Nuclear University "MEPhI", Moscow ;

Tupchienko V.A., Doctor of Economics, Professor, Professor of the Department of “Business Project Management”, Ministry of Education and Science of the Russian Federation, Federal State Autonomous Educational Institution of Higher Professional Education “National Research Nuclear University “MEPhI”, Moscow.

Bibliographic link

Udovidchenko R.S., Kireev V.S. COMPARATIVE ANALYSIS OF MODELS FOR ASSESSING THE EFFECTIVENESS OF STAFF TRAINING // Modern problems of science and education. – 2014. – No. 6.;
URL: http://science-education.ru/ru/article/view?id=16909 (date of access: November 24, 2019). We bring to your attention magazines published by the publishing house "Academy of Natural Sciences"

What happened to American Donald Kirkpatrick is quite rare. Half a century ago, he proposed a conceptual approach to evaluating training programs that has become classic. Any evaluator today simply must know the four levels of evaluation “according to Kirkpatrick.”

It is interesting that there is still controversy surrounding the approach proposed by Kirkpatrick, but his training evaluation model remains one of the basic ones, and his books are the most cited. The classic is still alive today, although it has already retired from active business. He had a brilliant career; wrote several books that became bestsellers and brought him worldwide fame; was the president of one of the most authoritative professional associations of trainers and consultants in the world - the American Research and Development Society (ARSD). Today, Donald Kirkpatrick continues to engage in social work and maintains contacts with the professional community. He continues to publish articles and give lectures and seminars at major forums.

So, what did Kirkpatrick suggest?..

Kirkpatrick views assessment as an integral part of the training cycle, which includes 10 stages :

Determining needs.
Setting goals.
Definition of subject content.
Selection of training participants.
Formation of an optimal schedule.
Selection of appropriate premises.
Selection of appropriate teachers.
Preparation of audiovisual media.
Program coordination.
Program evaluation.

Justify the existence of a training department by showing how this department contributes to achieving the goals and objectives of the organization.
Decide whether to continue or terminate the training program.
-Get information on how to improve the training program in the future.

Kirkpatrick himself believes that in most cases the assessment is carried out in order to understand how to increase the effectiveness of training, what ways can it be improved. In this regard, it is proposed to answer the following 8 questions:

To what extent does the training content meet the needs of the participants?
Is the choice of teacher (Kirkpatrick uses the word “leader” here, which has a completely different meaning in Russian) optimal?
Does the teacher use the most effective methods to maintain participants' interest, impart knowledge, and develop skills and attitudes?
Are the training conditions satisfactory?
Are participants comfortable with the class schedule?
Do audiovisual aids improve communication and maintain participant interest?
Was the program coordination satisfactory?
What else can be done to improve the program?

Note that all questions except the first and last are formulated by the author as closed (requiring “yes” or “no” answers). From the point of view of formulating task questions for assessment, this form of questions is not always good. However, we adhere to the author's text.

Kirkpatrick believes that in most cases, evaluation is limited to the use of post-training questionnaires - studying the immediate reaction of participants to the training. He calls these questionnaires “smile-sheets,” meaning that participants most often use the questionnaires to express gratitude. A more complex and in-depth assessment is not carried out because

it is not considered urgent or important,
no one knows how to carry it out,
management doesn't require this
people feel safe and do not see the need to “dig” deeper,
there are many things that are more important to them or that they prefer to do.

Four levels Four levels, according to Kirkpatrick, determine the sequence of assessment of training (training). He writes: "Each level is important and influences the next level. As you move from level to level, the assessment process becomes more difficult and time-consuming, but also provides more valuable information. No level can be skipped simply because to concentrate on what the coach considers most important" (it should be noted that many experts do not agree with this statement by Kirkpatrick). Here are the famous four levels according to the author:

Level 1 – Reaction
Level 2 – Learning
Level 3 – Behavior
Level 4 – Results

Reaction

Evaluation at this level determines how program participants respond to the program. Kirkpatrick himself calls it customer satisfaction assessment. When training is conducted in-house, the response of participants is not always interpreted as customer satisfaction. The fact is that participation in such trainings is mandatory. People simply have no choice. The company's management determines the need for this training and obliges employees to take part in it. It would seem that in this case, we need to talk about the reaction of management. Kirkpatrick emphasizes that in this case too the reaction of the participants is a very important criterion for the success of the training for at least two reasons .

Firstly, people one way or another share their impressions of the training with their management, and this information goes higher. Consequently, it influences decisions about continuing training.
Secondly, if participants do not respond positively, they will not be motivated to learn. According to Kirkpatrick, a positive reaction does not guarantee the successful development of new knowledge, skills and abilities. A negative reaction to training almost certainly means a decrease in the likelihood of learning.

Learning

Learning is determined as changing attitudes, improving knowledge and improving the skills of participants as a result of their completion of the training program. Kirkpatrick argues that changes in participants' behavior as a result of training are only possible when learning occurs (attitudes change, knowledge improves, or skills improve).

Behavior

At this level, an assessment is made of the extent to which the participants' behavior has changed as a result of training. Kirkpartick points out that the lack of changes in participants' behavior does not mean that the training was ineffective. Situations are possible when the reaction to the training was positive, learning occurred, but the behavior of the participants did not change in the future, since the necessary conditions for this were not met. Therefore, the lack of change in the behavior of participants after the training cannot be a reason for making a decision to terminate the program. Kirkpartick recommends that in these cases, in addition to assessing the reaction and learning, check the presence of the following conditions :

Participants' desire to change behavior.
Participants have knowledge of what and how to do.
The presence of an appropriate socio-psychological climate.
Reward participants for behavior change.

Results

Kirkpatrick insists that results should not be measured in money. He believes the changes listed above could, in turn, lead to increased profits. Kirkpatrick writes: "I laugh when I hear that professional trainers must be able to demonstrate the benefit to the client in terms of return on investment in training. I think the same thing about the relationship between training programs and profit. Just imagine all the factors that affect profits! And you can add them to the list of factors that affect return on investment."

According to Kirkpatrick, assessment at this level is the most difficult and expensive. Here are some practical tips that can help you evaluate your results:

if possible, use a control group (no training)
carry out the assessment after some time so that the results become noticeable,
conduct pre- and post-program assessments (if possible),
conduct the assessment several times during the program,
compare the value of the information that can be obtained through the assessment and the cost of obtaining this information (the author believes that conducting an assessment at level 4 is not always advisable due to its high cost).

How Kirkpatrick's Book is Organized Kirkpatrick, D. L. (1998). Evaluating training programs: the four levels. San Francisco: Berrett-Koehler Publishers, Inc.

The book is divided into two unequal parts. The first is a description of the 4-level model, theory (about 70 pages). The second part (about 200 pages) is a detailed analysis of several case studies and examples of tools used to conduct assessments at different levels.

Based on materials from the Process Consulting company newsletter (processconsulting.ru/bulletin1.pdf)