FLS

This is Learning Outcome No. 4 - Review Delivery of Training Session. The following are the Contents and Assessment Criteria of this unit.

CONTENTS:

1. Competency Based Training Review and Evaluation 

2. Pre - Test Posttest Analysis Training Evaluation Analysis

3. Training Session Adjustments

ASSESSMENT CRITERIA:

1. Appropriate Training session evaluation instrument is used.

2. Interpretations are made on the results of training session evaluation.

3. Adjustments on training are made on the results of evaluation.

The review of the training session for purposes of evaluating the success of CBT for purposes of improving the methods and processes involved is the focus of this learning outcome. 

The evaluation of Knowledge, Skills and attitude as a result of the training and the implementation of the training program will be discussed and practiced by going through the learning activities outlined in the course site contents.

Your outputs for this Unit shall be the following:

1. Analysis of a hypothetical data on pre test posttest result.

2. Analysis of hypothetical data for program evaluation

3. sample write-up of a program evaluation report 

Learning Objectives:

After reading this LESSON, YOU MUST be able to:

1. determine the components of training that needs to be evaluated; 

2. enumerate and describe ways of reviewing and evaluating training sessions 

Competency based Training, like any training, should evolve dynamically. The drive for continuous improvement will only be possible if training is reviewed, measured and evaluated. Adjustments can be made based on the results of the evaluation. 

In this lesson you will learn how to evaluate training session.

Training Evaluation

The evaluation of training refers to any attempt to obtain information or feedback on the effects of a training program and to assess the value of the training in the light of that information. The primary purpose evaluation is to improve training by determining which training processes achieved their objectives.

It is important after you deliver a training session that you review your delivery and look at what went well and what might be improved. 

In your review, the following components shall be included: 

a. Knowledge – refers to what we know. It is the cognitive domain of human behavior 

– the acquisition of knowledge is measured through the pretest and post test. A pre-test posttest analysis is a very effective and objective method of analysis.

b. Skills – refers to what we do correctly and accurately. It is also the psychomotor domain of human behavior. 

– the acquisition of skills is measured through the performance test. But since we do not have a grading system, rating the performance of trainees for purposes of program evaluation will not be as effective. An evaluation of how the training enhanced their skills shall be used to analyze the effectivity of the training program.

c. Attitude – refers to what we feel. It is also known as the affective domain of human behavior.

– positive attitude towards work, co – workers and the environment is a very important component of training that needs to be evaluated. A rating system on how the training program developed their positive work attitude maybe helpful in evaluating the effectiveness the training program. 

d. Program Implementation – the approach of implementing the training program.

– CBT implementation review and evaluation will be most effective if the methods on how the following categories of program implementation are analyzed:

1. Program Design and implementation 

2. Course Content 

3. Training Methodology 

4. Program Administration and Management 

5. Trainer’s Competence

We will be using pre-test/post test method to test for knowledge and Questionnaire will be used to gather data for skill, attitude and program implementation.

Feedback

In order to review your training you will need to gather feedback. It is useful to gather this feedback at the end of each session so that you can think about whether you need to make any changes to your next session.

Feedback can be obtained from participants, other people like other trainers or supervisor and yourself. Sometimes you need to report feedback in order to help management make decisions about whether to continue the program as it was, change it or stop it. Feedback can be gathered in a number of ways. The table lists the most commonly used methods, their advantages and disadvantages.

CBT Evaluation Tools

The table below shows some information gathering tools:

Rating Sheets

In preparing a rating sheet to evaluate competency based training, a trainer must recall the characteristics of an ideal competency based training (CBT). Group the characteristics into the following aspects, preparation, delivery, support system and facilities. For ease of analysis a recommended rating scale should be used such as the Likert scale. 

A sample evaluation tool and an analysis follow in the succeeding pages. The rating sheet will however depend on what the trainer need to know or verify about his performance as a trainer or the appropriateness of the methodologies that are used.

Focus Group Discussion

To use a focus group discussion as a form of evaluation, you will have to list down discussion questions, which focuses on the areas you want to evaluate. Examples or questions for discussion are the following:

1. What are the competencies you have acquired from the training that are very useful?

2. Do you think the training prepared you with the competencies required in the workplace?

3. What do you think are the weaknesses and strength of the CBT program?

4. What improvements can you suggest for the program?

Once questions are prepared you can organize a meeting where all respondents will be present to give their opinions and come up with a concession.

Document review

This is done to gather information relating to the average training time of learners to acquire the desired level of competency, to identify the success rates in national and institutional assessment, and to summarize the written comments of participants, supervisor and other stakeholders. Available data such as the individual records of trainees, national assessment results as in RWAC and results of focus group discussion or satisfaction surveys and other related researches that covers a certain period of time.

Self Evaluation 

An important part of the evaluation process is self evaluation. Self evaluation can take place during your training session as you reflect on how things are going. This will help you make any instant changes to your plan in response to the situation. 

Self evaluation can also take place after training session when you ask yourself in-depth questions about whether there are things to be changed or problems to be solved.

The following strategies can be used for evaluating your own training and help you improve:

1. Take a video of the CBT and look for ways to improve yourself

2. Ask for feedback from colleagues and other trainers

3. Compare your training methods and techniques with other trainers.

***View attached image***

Questions to answer

1. Were there any parts of the session which did not run as expected? Why?

2. Did any unexpected problem arise?

3. Were the session outcomes achieved? If not why?

4. Should anything be changed for the next session? 

Learning Objectives:

After reading this LESSON, YOU MUST be able to:

1. explain the advantage of using a pre test / post test in testing knowledge;

2. determine the statistical tool used to analyze pre test and post test scores.

The acquisition of knowledge is one of the most important components of training. To test the effectiveness of a competency - based training program this component should be evaluated. 

In this lesson, you will be introduced to this method of evaluation which is one important aspect of program review and evaluation.

Pre-Test Post Test

Pre-Test Post Test

Tests administered upon an agreed upon "entry point" and "exit point." These tests can be standardized or locally-developed. 

Pre-test is a preliminary test administered to determine a student's baseline knowledge or preparedness for an educational experience, course or qualification.

Posttest is a test given to trainees after completion of the course or qualification and used in conjunction with a pre-test to measure their achievement and the effectiveness of the training program.

Pre-Test/Post test is not limited to written test. Performance test can also be used. But for our purpose, we will use the written test as a measure of knowledge.

To serve our purpose, we shall be using a trainer-made test that is item-analyzed. True or False and Multiple Choice Type of written test is therefore most appropriate.

Advantages:

• Useful method for measuring the "value-added" by a program of study

• The "after-only" design of documenting learning is a weak approach because positive change cannot necessarily be attributed to the effectiveness of a program.

• Pre-tests serve several purposes: knowledge of the current status of a group may provide guidance for future activities as well as the basis of comparison for a post-test results; administering a test of entry behavior can determine whether assumed prerequisites have been achieved.

Disadvantages:

• Hard to discern if the positive change charted in a pre-post test is due to learning in the workshop or simply natural maturation.

• Due to trainee’s dropping out, the post-test results may be higher because those who remain are more successful or persistent.

• Problems with statistics: if the control group scored so low that they can only go up, or the control group that scored so high little improvement will be indicated in the post-test scores.

• If using the same test for both the pre- and post-test, some argue that trainees will absorb knowledge just from taking the test and will attend more readily to the content. To avoid this, we shall be using other evaluation instruments to back-up our analysis.

• Concentrates on value-added rather than outcomes assessment.

• Tendency to teach to the post-test

Statistical Analysis

The statistical tool which is most appropriate to analyze data on pretest and post test shall be paired t-test.

T-test is an inferential test that determines if there is a significant difference between the means of two data sets. 

t-test (For Paired Samples) - Use this test to compare two small sets of quantitative data when data in each sample set are related in a special way.

Criteria

• The number of points in each data set must be the same, and they must be organized in pairs, in which there is a definite relationship between each pair of data points

• If the data were taken as random samples, you must use the independent test even if the number of data points in each set is the  Even if data are related in pairs, sometimes the paired t is still inappropriate 

• Here's a simple rule to determine if the paired t must not be used - if a given data point in group one could be paired with any data point in group two, you cannot use a paired t test Since statistical software are now ready available, we shall be using the computer to compute for the p-value which shall be basis of our analysis.

Since majority of the trainers are using Microsoft Office Applications, we shall be using Microsoft Excel Analysis Toolpak to analyze our data. At this point, you shall now install the application on your computer.

Please have your computer around while you do the following steps:

1. Open the Microsoft Excel App in a personal computer running a Windows 10 environment.

2. In the homepage of Microsoft Excel, Click on Options located at the lower left corner of the window.

 3. Click on Add - Ins.

4. Click on Analysis Toolpak then click on Go.

4. In the Add - Ins dialogue box, select the Analysis Toolpak check box and then click OK.

5. If you are prompted that the Analysis ToolPak is not currently installed on your computer, click Yes to install it. After installing Analysis Toolpak you will see the data analysis icon on your Data Toolbar.

Tip If Analysis ToolPak is not listed in the Add-Ins available box, click Browse to locate it.

Note The steps listed herein are applicable for most Microsoft Excel app versions, from 2016 onwards to Office 365 users.

The Statistical Table

The analysis table of the pre-test and post-test using paired t-test will look like the table below:

t-Test: Paired Two Sample for Means

If your statistic is higher than the critical value from the table:

• Your finding is significant.

• You reject the null hypothesis.

• The probability is small that the difference or relationship happened by chance, and p is less than the critical alpha level (p < alpha ).

If your statistic is lower than the critical value from the table:

• Your finding is not significant.

• You fail to reject the null hypothesis.

The probability is high that the difference or relationship happened by chance, and p is greater than the critical alpha level (p > alpha ).

In our example above, the average for post test is equal to 60.37 and 53.74 for pre-test. The t-statistic is equal to 4.24 and t-crical value = 2.55. Since t-statistic is greater than t-critical value, we reject the null hypothesis.

In this case the null hypothesis is “the are no significant differences between pre-test and posttest scores” and the alternate hypothesis is “there are significant differences between pre-test and post test scores”. Stated in layman’s language the null hypothesis means that pre-test score is equal to the posttest scores or the scores did not increase. The alternate hypothesis on the other hand, means that pre-test scores are not equal to the post test scores or the scores increased.

Analyzing the result of the above example, since the null hypothesis is rejected, we accept the alternate hypothesis which is “there are significant differences between the pre-test and post test scores”. Conclusion is the scores increased.

Instead of comparing the t-critical and t-statistical values to determine significant difference, you may also compare the alpha level and p-values. In our example, because the p-value is less than the alpha level, the alternate hypothesis is accepted. However, if the p-value was greater than the alpha level, p>α, the null hypothesis would be retained. If your alpha is .01 and your p-value = 0.00025 there are significant differences between treatment means at .01 level of significance.

An alpha level represents the number of times out of 100 you are willing to be incorrect if you reject the null hypothesis. If you choose an alpha level of 0.05, 5 times out of 100 you will be incorrect if you reject the null hypothesis. Those five times, both means would equal, but that's about it. 95 times out of 100, you will be correct because it is more likely that the means are not equal.

Sample Analysis

Presented below is a hypothetical data of the pretest and posttest scores which will be the basis of our discussions and example for analyzing the results of a pretest and post test.

***View attached image***

From the pretest and post test scores above, the graph is generated. The blue line represents the pretest scores of the trainees and the red line represents posttest scores. Generally, the red line is higher that the blue line which means that post test scores are generally higher than the pretest scores although some students did not show an increase in score like in the case of trainee 15, 16 and 17.

To establish that there is an increase in scores statistically, we test the null hypothesis “there are no significant differences between the pre and post test scores” using paired t-test.

Note: An average, say 25 and 26 may obviously be different but may not differ statistically. There is a need to statistically analyze data, to test whether test scores are statistically different.

The Statistical Table

Using Excel to analyze data

The Office App Microsoft Excel provides statistical tools to do these kinds of analysis.

1. Encode your data in Microsoft Excel.

To perform paired t-test, select Data

1. Click on the icon Data Analysis (the dialog box on Analysis Tools will appear)

2. Click t-Test: paired Two sample for means, then click OK

3. Click on the black, upward arrow icon beside the blank area for Variable 1 Range.

4. Select the range of scores for posttest, then click on the arrow again, to confirm selected range.

4. Click on the black, upward arrow icon beside the blank area for Variable 2 Range, select the range for pretest scores, then click on the black arrow to go back to the dialog box and confirm the selection.

5. On the blank area for Hypothesized Mean Difference type 0.

6. Select the output range (or the range where the table will appear). 

7. Then click OK. The result will appear on the area you have previously selected on Step 6.

• Note: In case you cannot do this on your own ask the assistance of your trainer. 

Learning Objectives:

After reading this LESSON, YOU MUST be able to:

1. describe how to use the 5 point Likert scale in analyzing data from a questionnaire;

2. enumerate the data of a training evaluation that can be measured by the questionnaire suggested.

3. explain how the acquisition of skills and attitudes maybe measured using a questionnaire

In the previous Lesson we discussed how to analyze the effectivity of the training program in terms of the knowledge domain using the pre - test / posttest analysis. 

In this lesson we will discuss how to use a questionnaire in measuring the effectivity of a program in training skills and attitude. The training program evaluation form prepared herein also measures how CBT was implemented.

Training Evaluation

Training evaluation is designed to determine the effectiveness of the training in achieving its objectives. Among the importance of having a training evaluation are the following:

1. Training evaluation tells us about the effectiveness or quality of training in terms of achieving its effects, outcomes and impacts.

2. Training evaluation tells us if a training activity needs to be continued, redesigned, or discontinued.

3. Knowing the effects of training is very important in objective setting

4. The expected effects found in the objectives help us decide on the choice of method, design and tool to use in evaluating training

Training Evaluation Objectives

Training evaluation objective is a statement of what changed in the trainees are expected to be achieved and the cause of such change. The training objective should be SMART(Specific, Measurable, Achievable, Realistic and Timebound). It should also possess the “cause and effect”. The cause maybe one of the indicators of the input(budget), activity(training) or output(trained staff), and the effort may be one of the effects like knowledge, attitude or skills. 

The practical purpose of training evaluation is to verify if the training objectives were achieved at the end of a training activity. The training evaluation objective may use the following verbs:

• To determine + cause + effect + subject

• To find out + cause + effect + subject 

• To assess + cause + effect + subject 

• To evaluate + cause + effect + subject 

• To investigate + cause + effect + subject 

Example:

1. To determine if CBT approach effectively enhance the skills of the trainees in Computer Hardware servicing

2. To assess if the use of computer based learning materials effectively enhance the knowledge of trainees in Trainers methodology

Parts of a Training Evaluation Report

1. Title of the Report

The title should be explicit and concise. It should give the general idea about what the training evaluation is all about. The title should give interest on the content of the report.

2. Executive summary

This part of the report is a summary of the objectives, methods used, results of the evaluation, conclusions and the recommendations.

3. Rationale

The purpose of evaluating the training program is written on this part of the report.

4. Objectives

The general objective and the specific objectives should contain this part of the report.

5. Methodology

The research method used in evaluating the training program is usually discussed here. The number of participants and how they were selected should also be included in the report.

6. Results and discussion

This is the body of the report. It should contain the following parts:

a. Data interpretation – data that are presented in tables and graphs are interpreted. Interpretation is a verbal documentation of the numerical and graphical data.

b. Data analysis – The results of the statistical treatment are interpreted and explained.

c. Conclusion

The statements of the results which answer the objectives of the training evaluation are summarized here.

7. Recommendation

As a result of the conclusions derived from the evaluation, recommendations are then clearly stated. These are the course of actions that arises from the results and conclusions.

Training Evaluation Instrument

The attached training program evaluation form was developed by the National TVET Trainers’ Academy (NTTA) as a part of their Training Program Evaluation for purposes of improving the training programs using the CBT approach.

This instrument aims to evaluate how effectively the training program enhanced the capability of the trainer participants in Trainer’s Methodology I in terms of the following:

1. The perception of the trainees on the effectiveness of the training program in enhancing their:

     a. skill

     b. work attitude

2. How well the programs were implemented was divided into the following program components:

     A. Program Design and Organization

     B. Course Content

     C. Training Methodology

     D. Program Administration and Management

     E. Facilitator/Trainer

A sample questionnaire that follows is recommended but revisions and enhancement is allowed depending on the training programs offered and on the objectives of your training evaluation. 

Learning Objectives:

After reading the LESSON, YOU MUST be able to:

1. enumerate the sources of data for decisions in training adjustments.

In the previous lessons, you learned to evaluate training sessions through different methods such as the self evaluation, focus group discussions, Training Program Evaluation and pretest posttest.

Training Session Adjustment

In the previous lessons, you learned to evaluate training sessions through different methods such as the self evaluation, focus group discussions, Training Program Evaluation and pretest posttest. 

The results from these methodologies are legitimate sources of information and basis for adjusting training sessions. Self assessment and self reflections, if properly recorded, in the provided space in the session plan would give the trainer an idea what to adjust to improve the attainment of knowledge and skills.

Trainee’s satisfaction on the training session, activities that are unsuccessful, feedback from trainees among others are sources of important information which will lead to a successful adjustment in the training sessions. 

Proper documentation and analysis of these important data should be done to serve as a basis for more scientific modifications in training. 

Documentations should be submitted to proper authorities to serve as a basis for decision making about training methodologies and curriculum.

These documents maybe very useful to the following:

1. Other instructors

2. The research team

3. The Vocational Instruction Supervisor

4. The administrators

5. Industry partners