Research in Education

Types of Educational Research to conduct in Schools

• Action learning
• Action Research http://en.wikipedia.org/wiki/Action_research
• Appreciative Inquiry
• Lesson study
  • http://www.tc.columbia.edu/lessonstudy/readings.html
• Participatory research
• Peer research
• Phronetic social science
• Praxis intervention
• Reflective practice

About Educational Research

• Wisconsin Center of Education Research http://www.wcer.wisc.edu/
• Teachers College Record http://www.tcrecord.org/
• http://www.lab.brown.edu/pubs/themes_ed/act_research.pdf
• Education Research Association of Singapore http://www.eras.org.sg/
• Center for Collaborative Action Research http://cadres.pepperdine.edu/ccar/projects.school.html - Action Research with Students
• An Action Research Project http://cadres.pepperdine.edu/ar/c7/King/APR%20Home.htm

Outlines

Educational Research

From Wikipedia

• What is it?
  • Variety of methods in which individuals evaluate different aspects of education: student learning, teaching methods, teacher training, and classroom dynamics
  • must be conducted in a rigorous and systematic way
  • findings need to be interpreted within the context in which they wrere discovered as they may not be applicable in every time or place.[3]
• 10 Characteristics: Gary Anderson [2]
  • Educational research attempts to solve a problem.
  • Research involves gathering new data from primary or first-hand sources or using existing data for a new purpose.
  • Research is based upon observable experience or empirical evidence.
  • Research demands accurate observation and description.
  • Research generally employs carefully designed procedures and rigorous analysis.
  • Research emphasizes the development of generalizations, principles or theories that will help in understanding, prediction and/or control.
  • Research requires expertise—familiarity with the field; competence in methodology; technical skill in collecting and analyzing the data.
  • Research attempts to find an objective, unbiased solution to the problem and takes great pains to validate the procedures employed.
  • Research is a deliberate and unhurried activity which is directional but often refines the problem or questions as the research progresses.
  • Research is carefully recorded and reported to other persons interested in the problem.
• 2 Approaches
  • Basic, or academic research
    • focuses on the search for truth [2] or the development of educational theory.[1]
    • “design studies that can test, refine, modify, or develop theories”.[1]
    • researchers are affiliated with an academic institution and are performing this research as part of their graduate or doctoral work.
  • Applied or contractual Research approach
    • The pursuit of information that can be directly applied to practice
    • Researchers find solutions to existing educational problems.
    • Utilitarian in nature: strives to find information that will directly influence practice.
    • Applied researchers are commissioned by a sponsor and are responsible for addressing the needs presented by this employer.[2]
    • The goal of this research is “to determine the applicability of educational theory and principles by testing hypotheses within specific settings”.[1]
• Methodology
  • Qualitative research
    • uses data which is descriptive in nature.
    • Tools include: observations, conducting interviews, conducting document analysis, and analyzing participant products such as journals, diaries, images or blogs,.[1]
    • Types of qualitative research
      • Case study [1][2]
      • Ethnography [1][2]
      • Phenomenological Research [1][2]
      • Narrative Research [1]
      • Historical Research [2]
  • Quantitative research
    • uses data that is numerical and is based on the assumption that the numbers will describe a single reality.[1]
    • Statistics are often applied to find relationships between variables.
    • Types of quantitative research
      • Descriptive Survey Research [1]
      • Experimental Research [1]
      • Single - Subject Research [1]
      • Causal - Comparative Research [1]
      • Correlational Research [1][2]
      • Meta-analysis [1]
  • Combination methods
    • Derivatives of scientific method are far too reductionistic in nature,.[5]
    • In social research this phenomenon is referred to as triangulation (social science).[7]
    • Barrow
      • "Since educational issues are of many different kinds and logical types, it is to be expected that quite different types of research should be brought into play on different occasions. The question therefore is not whether research into teaching should be conducted by means of quantitative measures (on some such grounds as that they are more ‘objective’) or qualitative measures (on some such grounds as that they are more ‘insightful’), but what kind of research can sensibly be utilized to look into this particular aspect of teaching as opposed to that." [8]
    • Types of combined methods
      • Action Research[1]
      • Program Evaluation[1][2]

Research Methods

From Teachers College Record

• Qualitative Methods
• Survey Research
• Experimental Research
• Writing for Publication
• Libraries

Action Research

From Brown University

http://www.lab.brown.edu/pubs/themes_ed/act_research.pdf

• What is Action Research
  • Process which participants examine their own educational practice systematically and carefully using techniques of research
  • Disciplined inquiry done by a techer with intent that the research will inform and change his or her practice in future
  • Cycle of posing questions, gathering data, reflection and deciding on a course of action: which leads to change the school environment, resulting in different set of circumstances with different problems posed which require a new look
  • Assumptions (Watts 1985 p118)
    • For teachers and principals:
      • work best on problems they have identified for themselves
      • Become more effective when encourgaed to examine and assess their own work and then consider ways of working differently
      • help each other by working collaboratively
    • Working with colleagus help teachers and principals in their professional development
• What is Not Action Research?
  • Not a library project where teachers learn more about a topic that interests us.
  • Not problem-solving in the sense of finding out what is wrong but rather a quist for knowledge about how to improve
  • Not research on or about people
  • Not finding all available information on a topic looking for correct answers
  • Not learning why we do certain things, but how we can do things better
• Types of Action Research (Diagram)
  • Individual Teacher Research
    • Single issue in the classroom: classroom management, instructional strategies, use of materials, student learning
    • Support from supervisor, instructor or parents
    • Adressed on an individual basis by collecting data
    • Drawbacks: teachers may not share with others
  • Collaborative ACtion Research
    • Teachers who are interested in addressing a classroom or departmental issue.
    • Issue may involve one classroom or common problem shared by many classroom.
    • Support by individuals outside the school such as university or community partner
  • School-Wide Research
    • Issues common to all
    • E.g. lack of parental involvement in activities
    • Organisational and decision-making structures
    • areas of need for improvement and plan of action to improve student performance
  • Custer or Group wide research
    • Organisational, community-based, performance-based or processes for decision making.
    • Downside: Documentation requirements and ability to keep process in motion. Commitment from staff for effort and meet deadlines of assignments
    • Real School reform and change can take hold based on a common understanding through inquiry, and involvement of many lend energy to the process and create an environment of stakeholders
• History of Action Research
• Steps in Action Research (Diagram)
  • 5 cyclical Phases of Inquiry
    • Identification of Problem Area
      • Question should be
        • a higher-order question
        • stated in common language (no jargon)
        • concise
        • meaningful
        • not already have an answer
      • Guidelines in choosing questions
        • Is it something which the teacher has influence?
        • Is it something of interest and worth the tme and effort that will be spent?
      • Pitfalls
        • Discomfort or tension between colleagues in the classroom
        • e.g. Teacher uses latest fashionable teaching strategy but not really know or understand what or how kids are learning
    • Collection and Organisation of Data
      • Vehicles for data collection
        • Interviews
        • Portfolios
        • Diaries and journals
        • field notes
        • Recordings: audio, photos, video
        • questionnaires
        • focus groups
        • anecdotal records
        • checklists
        • individual files
        • logs of meetings
        • case studies
        • surveys
        • records - tests, report cards, attendance
        • self-assessment
        • samples of student work, projects, performances
      • Considerations
        • Which data is most appropriate for issue being researched
        • Is data easy to collect
        • which resources are readily available for use
        • How structures and systematic will collection be
      • About Data:
        • at least 3 sources for triangulation of data for basis of actions
        • Organise data that makes it useful to identify trends and themes
    • Interpretation of Data
      • Analyse and Identify major themes
      • Use classroom, individual or subgroup data.
      • Quantifiable data:
        • analyzed without use of statistics or technical assistance
        • Opinions, attitudes or checklists summarized in table form
      • Non quantifiable data
        • reviewed holistically and note important theme or elements
    • Action based on Data and Evidence
      • Using information from data collection and review of current literature: Design plan of action to make a change and to study change
    • Reflection and Evaluation of Results
      • Assess effects of intervention to determine if improvement has occurred.
      • If there is improvement, do data clearly provide supporting evidence?
      • If not, what changes can be made to the action s to elicit better results?
    • Next Steps:
      • As result of action research project, identify additional questions raised by the data and plan for additional improvements, revisions and next steps
• Benefits of Action Research
  • Worthwhile pursuit for excellent educators who are looking for ways to expand upon their existing knowledge
  • Focus on School Issue, problem or area of collective interest
    • Research done on the teachers' students helps to confer relevance and validity to a disciplined study
    • Allows teachers to pick up threads suggested in academic circles and eave them into their own classroom
    • Comforting for parents or education administrators outside school to know that a teacher is not just blindly following what the latest study seems to suggest but transforming knowledge into something meaningful
  • Form of Teacher Professional Development
    • Research and reflection allow teachers to grow and gain confidence in their work.
    • Influences thinking skills, sense of efficacy, willingness to share and communicate, and attitudes toward the process of change.
    • Teachers learn about themselves, students, colleagues and determine ways to continually improve
  • Collegial Interactions
    • Action research in pairs or teams of teachers allows time to talk with others about teaching and teaching strategies.
    • By working in teams, teachers must describe their own teaching styles and strategies and share their thoughts with others
    • Examine various instructional strategies, learning activities and curricular materials used in the classroom.
    • Develop stronger relationships through discussions
    • Practice of action research becomes part of school culture, leading to increased sharing and collaboration across departments, disciplines, grade levels and schools
  • Potential to impact school change
    • Teachers become more apt to look at questions that address school and ministry concerns rather that questions that affect individual teacher
    • New patterns of collegiality, communication and sharing
    • Contribution to body of knowledge about teaching and learning
    • Development of priorities for school-wide planning and assessment efforts with potential to motivate change for improvement
  • Reflect on own Practice
    • Opportunities for teachers to evaluate themselves in schools in a structured manner
    • Educators can investigate what effect their teaching is having on their students, how they could work better with other teachers or ways of changing the whole school for the better
  • Improved Communications
    • Team work within the school or district brings individuals together for a shared purpose.
    • Educators become more flexible in their thinking and more open to new ideas (Pine, 1981)
    • Positive changes in patterns of collegiality, communication and networking (Little, 1981)
• Stories from the Field
  • "What can we provide for effective reading instruction for third and fourth grade english language learners who are limited readers or non-readers
• Q&A
  • Why should schools engage in action research?
    • Johnson, 1995:
      • Promote personal and professional growth
      • Improve practice to enhance student learning
      • Advance teaching profession
  • How does Action Research benefits teachers?
    • Teachers learn what it is that they are able to influence and make changes that produce results that show change.
    • Provides opportunity to work with others and to learn from sharing of ideas.
  • How does action research benefit students in the classroom?
    • improve the teaching and learning process by reinforcing, modifying or changing perceptions based on informal data and non-systematic observations

From Wikipedia

• What is it?
  • reflective process of progressive problem solving
  • led by individuals working with others in teams or as part of a "community of practice"
  • to improve the way they address issues and solve problems.
  • undertaken by larger organizations or institutions, assisted or guided by professional researchers, with the aim of improving their strategies, practices, and knowledge of the environments within which they practice.
  • As designers and stakeholders, researchers work with others to propose a new course of action to help their community improve its work practices.
  • Kurt Lewin, 1946 paper “Action Research and Minority Problems” a comparative research on the conditions and effects of various forms of social action and research leading to social action” that uses “a spiral of steps, each of which is composed of a circle of planning, action, and fact-finding about the result of the action”.
  • (Reason & Bradbury, 2002: an interactive inquiry process that balances problem solving actions implemented in a collaborative context with data-driven collaborative analysis or research to understand underlying causes enabling future predictions about personal and organizational change
    • researcher’s agenda Vs participants;
    • those motivated primarily by instrumental goal attainment and by the aim of personal, organizational, or societal transformation; and
    • 1st-, to 2nd-, to 3rd-person research, that is, my research on my own action, aimed primarily at personal change; our research on our group (family/team), aimed primarily at improving the group; and ‘scholarly’ research aimed primarily at theoretical generalization and/or large scale change.
  • challenges traditional social science, by moving beyond reflective knowledge created by outside experts sampling variables to an active moment-to-moment theorizing, data collecting, and inquiry occurring in the midst of emergent structure.
  • Torbert 2002 “Knowledge is always gained through action and for action. From this starting point, to question the validity of social knowledge is to question, not how to develop a reflective science about action, but how to develop genuinely well-informed action — how to conduct an action science”
• Theories
  • Chris Argyris' Action Science
    • Study of how human beings design their actions in difficult situations. Humans design their actions to achieve intended consequences and are governed by a set of environment variables.
    • How those governing variables are treated in designing actions are the key differences between single loop learning and double loop learning.
      • When actions are designed to achieve the intended consequences and to suppress conflict about the governing variables, a single loop learning cycle usually ensues.
      • On the other hand, when actions are taken, not only to achieve the intended consequences, but also to openly inquire about conflict and to possibly transform the governing variables, both single loop and double loop learning cycles usually ensue. (Argyris applies single loop and double loop learning concepts not only to personal behaviors but also to organizational behaviors in his models.)
    • This is different from experimental research in which environmental variables are controlled and researchers try to find out cause and effect in an isolated environment.
  • John Heron and Peter Reason's Cooperative Inquiry or collaborative inquiry
    • The major idea of cooperative inquiry is to “research ‘with’ rather than ‘on’ people.” and all active participants are fully involved in research decisions as co-researchers.
    • Cooperative inquiry creates a research cycle among four different types of knowledge:
      • propositional knowing (as in contemporary science),
      • practical knowing (the knowledge that comes with actually doing what you propose),
      • experiential knowing (the feedback we get in real time about our interaction with the larger world) and
      • presentational knowing (the artistic rehearsal process through which we craft new practices).
    • The research process includes these four stages at each cycle with deepening experience and knowledge of the initial proposition, or of new propositions, at every cycle.
  • Paulo Freire's Participatory action research (PAR)
    • significant methodology for intervention, development and change within communities and groups, promoted and implemented by many international development agencies and university programs, as well as countless local community organizations around the world.
    • Paulo Freire: a response to the traditional formal models of education where the “teacher” stands at the front and “imparts” information to the “students” who are passive recipients. This was further developed in "adult education" models throughout Latin America.
  • William Torbert’s Developmental Action Inquiry
    • is a “way of simultaneously conducting action and inquiry as a disciplined leadership practice that increases the wider effectiveness of our actions.
    • Torbert, 2004: helps individuals, teams, organizations become more capable of self-transformation and thus more creative, more aware, more just and more sustainable”
    • challenges our attention to span four different territories of experience (at the personal, group, or organizational scales) in the midst of actions.
    • promotes timeliness – learning with moment to moment intentional awareness – among individuals and with regard to the outside world of nature and human institutions.
  • Jack Whitehead's living educational theory and Jean McNiff's Action Research approaches
    • Whitehead and McNiff (2006): individuals generate explanations of their educational influences in their own learning, in the learning of others and in the learning of social formations.
    • They generate the explanations from experiencing themselves as living contradictions in enquiries of the kind, 'How do I improve what I am doing?'
    • They use action reflection cycles of expressing concerns, (saying why you are concerned in relation to values), imagining possibilities in developing action plans, acting and gathering data, evaluating the influences of action, modifying concerns, ideas and action in the light of the evaluations. The explanations include life-affirming, energy-flowing values as explanatory principles.
• Action research in organization development
  • Wendell L. French and Cecil Bell define organization development (OD) at one point as "organization improvement through action research".[1]
  • OD's underlying philosophy, it would be action research as it was conceptualized by Kurt Lewin believed that the motivation to change was strongly related to action: If people are active in decisions affecting them, they are more likely to adopt new ways. "Rational social management", he said, "proceeds in a spiral of steps, each of which is composed of a circle of planning, action, and fact-finding about the result of action
  • process of change involves three steps:
    • Unfreezing or problem awareness:
      • Faced with a dilemma or disconfirmation, the individual or group becomes aware of a need to change.
    • Changing:
      • The situation is diagnosed and new models of behavior are explored and tested.
      • period of changing, trying out new forms of behavior in an effort to understand and cope with the system's problems.
    • Refreezing:
      • Application of new behavior is evaluated, and if reinforcing, adopted.
      • new behaviors are tried out on the job and, if successful and reinforcing, become a part of the system's repertoire of problem-solving behavior
  • steps and processes involved in planned change through cyclical process of change.
    • Input Phase - The cycle begins
      • a series of planning actions initiated by the client and the change agent working together.
      • The principal elements of this stage include a preliminary diagnosis, data gathering, feedback of results, and joint action planning.
      • Based on systems theory, this is the input phase, in which the client system becomes aware of problems as yet unidentified, realizes it may need outside help to effect changes, and shares with the consultant the process of problem diagnosis.
    • action, or transformation, phase.
      • actions relating to learning processes and to planning and executing behavioral changes in the client organization.
      • effect of altering previous planning to bring the learning activities of the client system into better alignment with change objectives.
      • Included in this stage is action-planning activity carried out jointly by the consultant and members of the client system.
      • Following the workshop or learning sessions, these action steps are carried out on the job as part of the transformation stage.
    • Output, or results, phase.
      • Actual changes in behavior (if any) resulting from corrective action steps taken following the second stage. Data are again gathered from the client system so that progress can be determined and necessary adjustments in learning activities can be made.
      • Minor adjustments of this nature can be made in learning activities
      • Major adjustments and reevaluations would return the OD project to the first, or planning, stage for basic changes in the program.
  • Action research is
    • problem centered, client centered, and action oriented.
    • involves client system in a diagnostic, active-learning, problem-finding, and problem-solving process.
    • Data not returned in the form of a written report but fed back in open joint sessions, and the client and the change agent collaborate in identifying and ranking specific problems, in devising methods for finding their real causes, and in developing plans for coping with them realistically and practically.
    • Applies scientific method in the form of data gathering, forming hypotheses, testing hypotheses, and measuring results,
    • Sets in motion a long-range, cyclical, self-correcting mechanism for maintaining and enhancing the effectiveness of the client's system by leaving the system with practical and useful tools for self-analysis and self-renewal

Program Evaluation

From Wikipedia

• What is it?
  • a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs,[1] particularly about their effectiveness and efficiency.
  • In both the public and private sectors, stakeholders will want to know if the programs they are funding, implementing, voting for, receiving or objecting to are actually having the intended effect, and answering this question is the job of an evaluator.
• Conducting an Evaluation
  • Rossi, Lipsey and Freeman (2004) kinds of assessment, which may be appropriate at different stages:
    • Assessment of the program's cost and efficiency
    • Assessment of the program's outcome or impact (i.e., what it has actually achieved)
    • Assessment of how the program is being implemented (i.e., is it being implemented according to plan?)
    • Assessment of program design and logic/theory
    • Assessment of the need for the program
  • Assessing needs/ needs assessment
    • examines the population that the program intends to target, to see whether the need as conceptualised in the program actually exists in the population; whether it is, in fact, a problem; and if so, how it might best be dealt with.
    • This includes identifying and diagnosing the actual problem the program is trying to address, who or what is affected by the problem, how widespread the problem is, and what are the measurable effects that are caused by the problem.
    • Rossi, Lipsey and Freeman (2004) caution against undertaking an intervention without properly assessing the need for one, because this might result in a great deal of wasted funds if the need did not exist or was misconceived.
    • Needs assessment
      • involves the processes or methods used by evaluators to describe and diagnose social needs[4]
      • need to identify whether programmes are effective and they cannot do this unless they have identified what the problem/need is.
      • Programmes that do not do a needs assessment can have the illusion that they have eradicated the problem/need when in fact there was no need in the first place.
      • involves research and regular consultation with community stakeholders and with the people that will benefit from the project before the program can be developed and implemented.
      • Bottom-up approach: In this way potential problems can be realized early because the process would have involved the community in identifying the need and thereby allowed the opportunity to identify potential barriers.
  • Importance of Programe Evaluator
    • 1. Construct a precise definition of what the problem is.[4] Evaluators need to first identify the problem/need.
    • 2. Assess the extent of the problem.[4] They need to answer the ‘where’ and ‘how big’ questions. Evaluators need to work out where the problem is located and how big it is.
      • Evaluators need to also answer the ’how’ and ‘what’ questions[5]
      • The ‘how’ question requires that evaluators determine how the need will be addressed. Having identified the need and having familiarised oneself with the community evaluators should conduct a performance analysis to identify whether the proposed plan in the programme will actually be able to eliminate the need.
      • The ‘what’ question requires that evaluators conduct a task analysis to find out what the best way to perform would be.
      • For example whether the job performance standards are set by an organization or whether some governmental rules need to be considered when undertaking the task.[5]
    • 3. Define and identify the target of interventions and accurately describe the nature of the service needs of that population[4] It is important to know what/who the target population is/are – it might be individuals, groups, communities, etc. There are three units of the population: population at risk, population in need and population in demand[4]
      • Population at risk: are people with a significant probability of developing the risk
      • Population in need: are people with the condition that the programme seeks to address;
      • Population in demand: that part of the population in need that agrees to be having the need and are willing to take part in what the programme has to offer
      • Being able to specify what/who the target are will assist in establishing appropriate boundaries, so that interventions can correctly address the target population and be feasible to apply<[4]
• Areas for Assessment
  • 5 Tiered Approach to Assessments (Jacobs, 2003).
    • Tier 1: Needs Assessment
    • Tier 2: Monitoring and Accountability
    • Tier 3: Quality Review and Program Clarification
    • Tier 4: Achieving Outcomes
    • Tier 5: Establishing Impact
  • Assessing Needs: The Conduct of a needs assessment:[6]
    • 1. Perform a ‘gap’ analyses:
      • Evaluators need to compare current situation to the desired or necessary situation. The difference or the gap between the two situations will help identify the need, purpose and aims of the programme.
    • 2. Identify priorities and importance
      • n the first step above, evaluators would have identified a number of interventions that could potentially address the need e.g. training and development, organization development etc.
      • These must now be examined in view of their significance to the programme’s goals and constraints. This must be done by considering the following factors: cost effectiveness (consider the budget of the programme, assess cost/benefit ratio), executive pressure (whether top management expects a solution) and population (whether many key people are involved).
    • 3. Identify causes of performance problems and/or opportunities
      • When the needs have been prioritized the next step is to identify specific problem areas within the need to be addressed. And to also assess the skills of the people that will be carrying out the interventions.
    • 4. Identify possible solutions and growth opportunities
      • Compare the consequences of the interventions if it was to be implemented or not.
      • Needs analysis is hence a very crucial step in evaluating programmes because the effectiveness of a programme cannot be assessed unless we know what the problem was in the first place.
  • Assessing program theory
    • The program theory, also called a logic model or impact pathway,[7] is an assumption, implicit in the way the program is designed, about how the program's actions are supposed to achieve the outcomes it intends.
    • This 'logic model' is often not stated expicitly by people who run programs, it is simply assumed, and so an evaluator will need to draw out from the program staff how exactly the program is supposed to achieve its aims and assess whether this logic is plausible.
    • It is important to read research that has been done in the area. Explicating this logic can also reveal unintended or unforeseen consequences of a program, both positive and negative.
    • The program theory drives the hypotheses to test for impact evaluation. Developing a logic model can also build common understanding amongst program staff and stakeholders about what the program is actually supposed to do and how it is supposed to do it, which is often lacking(see Participatory Impact Pathways Analysis).
  • Assessing implementation
    • Process analysis looks beyond the theory of what the program is supposed to do and instead evaluates how the program is being implemented.
    • This evaluation determines
      • whether the components identified as critical to the success of the program are being implemented or whether target populations are being reached, people are receiving the intended services, staff are adequately qualified, etc.
    • an ongoing process in which repeated measures may be used to evaluate whether the program is being implemented effectively.
  • Assessing the impact (effectiveness)
    • The impact evaluation determines the causal effects of the program. This involves trying to measure if the program has achieved its intended outcomes. This can involve using sophisticated statistical techniques in order to measure the effect of the program and to find causal relationship between the program and the various outcomes.
  • Assessing efficiency
    • Cost-benefit or cost-effectiveness analysis assesses the efficiency of a program. Evaluators outline the benefits and cost of the program for comparison. An efficient program has a lower cost-benefit ratio.
• Reliability, validity and sensitivity in program evaluation
  • Ensure that the instruments (for example, tests, questionnaires, etc) used in program evaluation are as reliable, valid and sensitive as possible.
  • Reliability
    • The reliability of a measurement instrument is the 'extent to which the measure produces the same results when used repeatedly to measure the same thing' (Rossi et al., 2004, p. 218).[4]
    • The more reliable a measure is, the greater its statistical power and the more credible its findings. If a measuring instrument is unreliable, it may dilute and obscure the real effects of a program, and the program will 'appear to be less effective than it actually is' (Rossi et al., 2004, p. 219).[4] Hence, it is important to ensure the evaluation is as reliable as possible.
  • Validity
    • The validity of a measurement instrument is 'the extent to which it measures what it is intended to measure' (Rossi et al., 2004, p. 219).[4]
    • This concept can be difficult to accurately measure: in general use in evaluations, an instrument may be deemed valid if accepted as valid by the stakeholders (stakeholders may include, for example, funders, program administrators, et cetera).
  • Sensitivity
    • The principal purpose of the evaluation process is to measure whether the program has an effect on the social problem it seeks to redress; hence, the measurement instrument must be sensitive enough to discern these potential changes (Rossi et al., 2004).[4]
    • A measurement instrument may be insensitive if it contains items measuring outcomes which the program couldn't possibly effect, or if the instrument was originally developed for applications to individuals (for example standardised psychological measures) rather than to a group setting (Rossi et al., 2004).[4]
    • These factors may result in 'noise' which may obscure any effect the program may have had.
• Methodological constraints and challenges
  • The shoestring approach
    • designed to assist evaluators operating under limited budget , limited access or availability of data and limited turnaround time, to conduct effective evaluations that are methodologically rigorous
    • (Bamberger, Rugh, Church & Fort, 2004).[10]
    • Many programs do not build an evaluation procedure into their design or budget. Hence, many evaluation processes do not begin until the program is already underway, which can result in time, budget or data constraints for the evaluators, which in turn can affect the reliability, validity or sensitivity of the evaluation. > The shoestring approach helps to ensure that the maximum possible methodological rigour is achieved under these constraints.
  • Budget constraints
    • Issues
      • Most original projects do not include a budget to conduct an evaluation (Bamberger et al., 2004).
      • Evaluations being allocated smaller budgets that are inadequate for a rigorous evaluation. It might be difficult to effectively apply the most appropriate methodological instruments.
      • Consequently affect the time available in which to do the evaluation (Bamberger et al., 2004).[10]
    • Budget constraints may be addressed by simplifying the evaluation design, revising the sample size, exploring economical data collection methods (such as using volunteers to collect data, shortening surveys, or using focus groups and key informants) or looking for reliable secondary data (Bamberger et al., 2004).[10]
  • Time constraints
    • Evaluator maybe summoned to conduct an evaluation when a project is already underway if they are given limited time to do the evaluation compared to the life of the study, or if they are not given enough time for adequate planning.
    • Evaluator is not familiar with the area or country in which the program is situated (Bamberger et al., 2004).[10]
    • Time constraints can be addressed by the methods listed under budget constraints as above, and also by careful planning to ensure effective data collection and analysis within the limited time space.
  • Data constraints
    • Issues
      • If the evaluation is initiated late in the program, there may be no baseline data on the conditions of the target group before the intervention began (Bamberger et al., 2004).[10]
      • data have been collected by program staff and contain systematic reporting biases or poor record keeping standards and is subsequently of little use (Bamberger et al., 2004).[10]
      • the target group are difficult to reach to collect data from - for example homeless people, drug addicts, migrant workers, et cetera (Bamberger et al., 2004).[10]
    • Mitigations
      • reconstructing baseline data from secondary data or through the use of multiple methods, such as the combination of qualitative and quantitative data can increase validity through triangulation and save time and money.
      • through careful planning and consultation with program stakeholders. By clearly identifying and understanding client needs ahead of the evaluation, costs and time of the evaluative process can be streamlined and reduced, while still maintaining credibility.
• Methodological challenges (In many developing countries)
  • the major sponsors of evaluation are donor agencies from the developed world, and these agencies require regular evaluation reports in order to maintain accountability and control of resources, as well as generate evidence for the program’s success or failure (Bamberger, 2000).[12]
  • [13] Some of the issues include differences in culture, attitudes, language and political process (Ebbutt, 1998, Smith, 1990).[14]
  • Culture
    • defined by Ebbutt (1998, p. 416) as a “constellation of both written and unwritten expectations, values, norms, rules, laws, artifacts, rituals and behaviours that permeate a society and influence how people behave socially”.
    • Culture can influence many facets of the evaluation process, including data collection, evaluation program implementation and the analysis and understanding of the results of the evaluation (Ebbutt, 1998).
    • In particular, instruments which are traditionally used to collect data such as questionnaires and semi-structured interviews need to be sensitive to differences in culture, if they were originally developed in a different cultural context (Bulmer & Warwick, 1993).[15]
    • The understanding and meaning of constructs which the evaluator is attempting to measure may not be shared between the evaluator and the sample population and thus the transference of concepts is an important notion, as this will influence the quality of the data collection carried out by evaluators as well as the analysis and results generated by the data (ibid).
  • Language
    • language is tied closely to culture (ibid). Language can be a major barrier to communicating concepts which the evaluator is trying to access, and translation is often required (Ebbutt, 1998).
    • Problems in translation, including the loss of meaning as well as the exaggeration or enhancement of meaning by translators (ibid).
    • data collection instruments need to take meaning into account as the subject matter may not be considered sensitive in a particular context might prove to be sensitive in the context in which the evaluation is taking place (Bulmer & Warwick, 1993).
    • two important concepts when administering data collection tools: lexical equivalence and conceptual equivalence (ibid).
      • Lexical equivalence asks the question: how does one phrase a question in two languages using the same words? This is a difficult task to accomplish, and uses of techniques such as back-translation may aid the evaluator but may not result in perfect transference of meaning (ibid).
      • conceptual equivalence. It is not a common occurrence for concepts to transfer unambiguously from one culture to another (ibid). Data collection instruments which have not undergone adequate testing and piloting may therefore render results which are not useful as the concepts which are measured by the instrument may have taken on a different meaning and thus rendered the instrument unreliable and invalid (ibid).
• Utilization of evaluation results
  • 3 Conventional Uses
    • Persuasive utilization
      • Enlistment of evaluation results in an effort to persuade an audience to either support an agenda or to oppose it. Unless the 'persuader' is the same person that ran the evaluation, this form of utilization is not of much interest to evaluators as they often cannot foresee possible future efforts of persuasion.[4]
    • Direct (instrumental) utilization
      • Evaluators often tailor their evaluations to produce results that can have a direct influence in the improvement of the structure, or on the process, of a program.
      • For example, the evaluation of a novel educational intervention may produce results that indicate no improvement in students' marks.
      • This may be due to the intervention not having a sound theoretical background, or it may be that the intervention is not run according to the way it was created to run.
      • The results of the evaluation would hopefully lead to the creators of the intervention going back to the drawing board and re-creating the core structure of the intervention, or even changing the implementation processes.[4]
    • Conceptual utilization
      • Evaluation results may be used to conscientize people with regards to the issues that form part of the concerns of the program.
      • Example: the results can also be used to inform educators and students about the different barriers that may influence students' learning difficulties. A number of studies on these barriers may then be initiated by this new information.[4]
  • Variables affecting utilization
    • relevance
    • communication between the evaluators and the users of the results
    • information processing by the users
    • plausibility of the results
    • level of involvement or advocacy of the users.[4]
  • Guidelines for maximizing utilization (Rossi et al. (2004, p. 416).[4])
    • Evaluators must understand the cognitive styles of decision makers
    • Evaluation results must be timely and available when needed
    • Evaluations must respect stakeholders' program commitments
    • Utilization and dissemination plans should be part of the evaluation design
    • Evaluations should include an assessment of utilization
• Internal versus external program evaluators (Division of oversight services, 2004).
  • Internal evaluators
    • Advantages
      • May have better overall knowledge of the program and possess informal knowledge of the program
      • Less threatening as already familiar with staff
      • Less costly
    • Disadvantages
      • May be less objective
      • May be more preocuppied with other activities of the program and not give the evaluation complete attention
      • May not be adequately trained as an evaluator.
  • External evaluators
    • Advantages
      • More objective of the process, offers new perspectives, different angles to observe and critique the process
      • May be able to dedicate greater amount of time and attention to the evaluation
      • May have greater expertise and evaluation brain
    • Disadvantages
      • May be more costly and require more time for the contract, monitoring, negotiations etc.
      • May be unfamiliar with program staff and create anxiety about being evaluated
      • May be unfamiliar with organization policies, certain constraints affecting the program.
• 3 Paradigms in program evaluation Potter (2006)
  • Positivist
    • evaluation can only occur where there are “objective”, observable and measurable aspects of a program, requiring predominantly quantitative evidence.
    • includes evaluation dimensions such as needs assessment, assessment of program theory, assessment of program process, impact assessment and efficiency assessment (Rossi, Lipsey and Freeman, 2004).[17]
    • A detailed example: study conducted by the Public Policy Institute of California report titled "Evaluating Academic Programs in California's Community Colleges", in which the evaluators examine measurable activities (i.e. enrollment data) and conduct quantitive assessments like factor analysis.[18]
  • Interpretive
    • it is essential that the evaluator develops an understanding of the perspective, experiences and expectations of all stakeholders.
    • lead to a better understanding of the various meanings and needs held by stakeholders, which is crucial before one is able to make judgments about the merit or value of a program.
    • The evaluator’s contact with the program is often over an extended period of time and, although there is no standardized method, observation, interviews and focus groups are commonly used.
    • A report commissioned by the World Bank details 8 approaches in which qualitative and quantitative methods can be integrated and perhaps yield insights not achievable through only one method.[19]
  • Critical-emancipatory
    • based on action research for the purposes of social transformation.
    • This type of approach is much more ideological and often includes a greater degree of social activism on the part of the evaluator.
    • Appropriate for qualitative and participative evaluations.
    • Because of its critical focus on societal power structures and its emphasis on participation and empowerment, Potter argues this type of evaluation can be particularly useful in developing countries.