Evaluation

In the previous chapters we explained all the steps and outlines necessary for the development of Problem-based learning in Leprosy or TB. If the course is to be relevant, it must mirror the needs of the community. For that purpose a situation and task analysis of the community is necessary. The result of this will be insight in the health needs of that community and a description of the work that each category of health worker is expected to do; (job descriptions). After these, the tasks of the different health workers can be analysed to find out what students need to learn in order to perform the task well. Therefore much hard work has to be done if its general objectives are to be formulated, and it's learning programmes to be designed. For example problem cases have to be developed to give the students opportunities for self-directed learning. The training of tutors and the development of adequate assessment methods also have to be planned. During this stage many of people are involved, and critical discussions took place about the content and the different approaches for learning that can be used for students to learn and be content in its learning.

When the programme starts it is essential to ask whether it meets the expectation of the community, and of the different responsible teachers and managers. To give a clear and credible answer to this question, the programme must be properly evaluated: this chapter therefore discusses the most important questions about evaluation.

In general, evaluation means the collection of information about the different components of the programme itself, the participation of the students, the role of the tutors, the material and resources available and much else also. The information that is collected must be credible enough to make it potentially useful for decision-makers to change the programme when necessary. Credible enough! This means that scientific research design is not essential to this kind of information. Tutors and other responsible teachers can collect data from students, they can study the different learning materials and discuss these in teacher meetings.

The evaluation of a course or programme in a more comprehensive must be planned step by step. In general information must be collected from different stages during the development of the programme. For evaluation purposes we can make a distinction between 3 stages.

• The plan. In this stage the needs of a community are assessed, the course objectives are formulated, and course or programme is planned
• Formative evaluation. When the plan is implemented the programme must be allowed to get thoroughly established until it is functioning day to day as effectively as possible. Information can then be collected about the functioning of the course and this information can be used to strengthen areas, which are obviously weak. The programme is also monitored for possible calamities. This kind of evaluation is focused on the processes of the programme, how the things are going, and how they may be changed.
• Summative evaluation. This refers to evaluation at to end of the course. The course is finished and now it must be evaluated for failure or success. The criteria used for this evaluation usually test the efficiency, effectiveness, the acceptability of the course, and its relevance to the real the needs of the community. What is the real product of the course? What are its outcomes.

To illustrate the relation between these aspects of evaluation see Figure 3.

Figure 3. process of Evaluation

Click on the figure to enlarge!

If one looks at figure 3 in more detail the change of the purpose of the evaluation is seen. At the beginning evaluation is to be expected to be a help- for the teacher and those responsible for the course so that they can define the objectives and plan the course as well as possible. Later, evaluation is used to study how the course is functioning in practice and thus to use the data to improve the course. The teaching staff itself, perhaps with the help of an educator can perform such evaluation. Finally, there is the very formal evaluation, which is done by internal and external evaluators. The last ones are not involved in teaching in the course. This evaluation is designed to study the level of the courses compared with other similar courses. Also political and economical factors are taken into account. In addition, if there are adverse reports from the community about those trained by the course, the reasons for these reports maybe revealed by the external evaluation.

Also other factors play a role in summative evaluation. For example, when epidemiological studies show that in some areas detection rates for leprosy are less as 1 or 2 per 10.000 a year, questions can be asked-"Is something wrong with the training? Have the students been adequately trained to recognise early signs of leprosy and to distinguish its lesions of those of other common skin lesions in that area? Is the course deficient because it is too restricted to leprosy and does not put leprosy into the context of the prevalent diseases of the area.?

And, finally, how cost/effective is the course if all important factors are considered.

We stress these differences in the purpose of evaluation. Administrators and responsible teachers must know exactly what their purpose is of the evaluation.

• Is the information meant to develop a programme as adequate as possible?
• Is the information meant to help the programme to improve?
• Is the information meant to discuss the ongoing process of the programme?

If these different questions are not considered, some new courses may well be judged too tough and too rigid. Every new course needs some years to mature and any evaluation must take the age of the course into account.

We can compare this process with the training of students. Plans are made for training, then courses are given to the student, but students need help in their study, when weaknesses are seen in their competence(formative). At the end of the course students are tested on the minimal acceptable level of competence. (summative).

In the next paragraphs we will discuss the most important questions concerning the different stages in evaluation. Of course the lists are not exhaustive but they illustrate the concerns in the plan, the formative and summative evaluation of the course itself.

The Plan, or the design of the course

Evaluation questions have to do with the needs of the community, the course and educational objectives, the training programme, the training of the tutors/facilitators, the preparation of the students, and the relationship between the different parts of the programme.

Asking some of the following questions will help to evaluate this stage;

• Are the trainees appropriate for the course? What can be said about the selection of the trainees?
• What is really known about the community- its cultural and anthropological environment and its socio-economic status, and the epidemiology of different related diseases.
• Are job and task descriptions of the different health workers available?
• Are the goals of the course appropriate? Can the educational objectives be reached within the total time available. Is it possible to estimate the study burden on the students.
• Is the PBL philosophy clear to the staff and the trainees when courses start?
• Are the tutors (facilitators) well trained before the programme starts?
• Are the cases and problems well constructed and in harmony with the goals of the course? Is there a plan for revision of the cases when this is necessary?
• Are the facilities for study, library, books and Audio-visual aids sufficient and easily available for the students?
• How well are the tutorials organised?
• What is the balance between PBL (self-study) and other teaching methods?
• What can be said about the balance between formal education, and self-study?
• What are the lodging and meal facilities for the students, how can they use their spare time? And are there any facilities for them so that they can have good recreational activities?

Formative evaluation

When courses run, the responsible people can observe the progress of the course and of the students and their results. When a course is new the information gathered about it is designed to help the programme and the students and the facilitators. The target is to improve the programme in order to allow the programme to meet its goals as well as possible.

After a period of functioning in the evaluation one can raise specific questions which also have to do with the comparison with the original intentions, goals and objectives and the actual practice in education. The most important questions to be raised in this stage are:

• How are the tutorial groups functioning? What is the climate in the groups?
• How are the tutors performing? Are they active in stimulating students in independent learning? How well do they help the small groups to function
• What can be said about the study-burden on students, their progress and their results?
• Are the cases and problems valid for self study and are the quality of self-studies in harmony with the course objectives?
• What is the balance between teacher-oriented and student-oriented learning methods? For example, lectures versus tutorial group work and self-study.
• Are there problems with study materials and facilities-problem-booklets, library, the bedside teaching?
• What are the attitudes of the tutor/facilitators and students towards problem-based learning?
• Are the students kept informed about their progress?(feedback), so that they can correct any problems in their learning. Are there problems in communication between the students and the facilitators so that feedback is not effective..
• Are the examinations really assessing the desired skills and knowledge? Are the methods used for assessment adequate and appropriate?(reliable, valid and practical)
• Is their adequate administrative support for the course? Is the record keeping accurate? Is the budget adequate?

Summative evaluation

As is pointed out before, summative evaluation is a kind of evaluation that must be done when responsible teachers like to study the relevance of the programme when the course got on its feet. Now responsible teachers have to raise the question to what extent the goals, activities and outcomes of the educational programme indeed are a response to the need of the community. Prevalence and incidence rates can change in the community, so responsible people have to change the course content as well. Also questions about the combination of leprosy with TB or dermatology in training programmes must be answered after this sort of evaluation. Since summative evaluation is a very formal kind of evaluation, in which not only responsible teachers but also administrators and political agencies must participate, we will not discuss this very extensively.

However, summative evaluation is a hard fact from time to time !