“The controversy over the best evaluation design continues to rage. During the late 1970s and early 80s, many debates were held at professional evaluation meetings about the value of experimental design in evaluation. Many evaluators thought the issue had been laid to rest, i.e., there was an acknowledgement of the usefulness and appropriateness of experimental designs--sometimes, but a consensus that such an approach does not constitute a gold standard, and not even the most frequently used design.
For many evaluators, especially in education but also in other fields, this prior understanding has been turned on its head by the US Government endorsement of randomized clinical trials as the sine qua non approach in evaluation (followed by quasi experimental and regression discontinuity designs). There has been considerable reaction to this methodological/ideological turn, with strong objections by AERA, AEA, and NEA.”
(communication Dec. 5, 2005 on The Role of RCTs in Evaluation, AEA, http://weblogs.elearning.ubc.ca/mathison/archives/2005/12/the_role_of_rct.html)
During the past decade, ushered in by an administration that endorsed NCLB, the evaluation profession has once again seen pressure to pursue RCTs and funding has been tied to these requirements. Many federal agencies, among them SAMHSA and the Department of Education, ushered forward a series of judgments regarding the programs they would fund. “Best practices,” “Model programs,” “promising programs,” became the nomenclature of the 2000s and inspired another round of debates within the evaluation field as to the appropriateness of these terms, the limitations of RCTs and other derivations of the experimental model of evaluation when applied to the diverse contexts and communities in which programs and services are delivered.
Evaluators were split around the idea of taking a stand against RCTs or objecting to them as the “gold standard” of evaluation. AEA issued a position statement (see reference below as required reading) that was protested within the ranks but supported by many. In 2006, many widely known evaluation experts gathered at the Claremont Graduate University Stauffer Symposium to answer the question, “What constitutes credible evidence in evaluation and applied research?” (required viewing and reading). In the Fall of 2005 and 2006 the Nevada Evaluation Association and the Center for Program Evaluation co-sponsored two one-day conferences and facilitated dialogues under their Evaluation Training Institute that further explored the question. The 2005 conference examined the question in relation to SAMHSA-funded projects and the aims of its “Service to Science” initiative focusing on substance abuse prevention among youth. Local evaluators offered workshops on their projects and their dilemmas. The Fall 2006 facilitated dialogue featured invited speakers, Judith Ottoson and Lawrence Green to present on and discuss the topic: “If we want more evidence-based practice we need more practice-based evidence: Issues of neglected external validity in evaluations of the public health field.”
The debate and response to the question has reverberated among the ranks of evaluators, their clients, and their funders, but has not systematically been addressed or followed through to some resolution by key stakeholders in Nevada: state agencies and public institutions, who often are the messengers of federal pass-through dollars and their local evaluators, who must translate those messages and walk a fine balance between the needs of their clients and the demands of their funders. This Learning Circle will offer an opportunity to present responses and solutions or positions to the question, and encourage participants to incorporate diverse perspectives while achieving some form of consensus for guiding the field of evaluation stakeholders in Nevada.
1) Encourage greater self-awareness among professional evaluators regarding their personal philosophies and orientations and strategies for negotiating the tensions presented in mandated evaluation designs and approaches.
2) Develop products and position statements related to the evaluation profession in Nevada that address the question, “what constitutes credible evidence in health, human service and educational contexts?”
3) Strengthen in-state evaluation networks and capacity to respond to requests for evaluation in a timely and consistent fashion.
Note: The question is very broad and the group will be invited to refine the question to suit their interests, applications and time commitments so that it lends itself to both individual and group projects.
Six to eight emerging and mid-career evaluators residing and/or working in Northern Nevada who are members of the Nevada Evaluation Association and/or are associated with public institutions in Nevada. Recruitment of evaluators through NEA’s on-line membership directory (N = 25) and website, as well as email notification/invitation to evaluation staff at Division of Health Services, Department of Education, local education agencies, Department of Child and Family Services (DCFS), Cooperative Extension. This Learning Circle will encompass the entire state of Nevada, but initially draw upon only those working in Northern Nevada (Carson City is the capitol and there are statewide evaluators positioned there). Ideally the group will consist of professional evaluators from diverse private and public organizations, with no more than half representing internal evaluators for public agencies. LC members will be recruited using a posting on the NEA website and a message sent to members by the organization. In addition, a Learning Circle Website/Google page will be established with an overview of the opportunity, the commitments and expectations, and an application for interested evaluators. This link will be emailed to various sources, including the university, the state’s department of education, health and human services division and local school districts as well as NEA members, private consultants, and non-profit agencies.
The communication will involve face-to-face, real-time and static distance-based communication technologies. This LC program will take place over a 6 month – 8 week period (either May-mid-July or September-mid-November, 2009). The Learning Circle will be launched with a one-half to full-day, face-to-face session, during which members will have the opportunity to bond, build trust and get grounded in the debate among expert evaluators in response to the LC question, providing a common base of knowledge and experience from which to launch individual investigations.
In addition, to live communication the LC will communicate asynchronously on an ongoing basis using a “Google Site” as the primary mechanism for sharing information and providing feedback to group members.
The completed product of this LC process will be selected and co-constructed by members, and may draw upon the following options for consideration:
Pre Start –
Week 2 May 21- Sharing of Reflections and Possibilities
Week 3 May 28- 6-7 p.m.
describes a initial activity on voice threads for the participants that precedes a face to face
We are well underway--having met now three times, and although there are fits and starts, I am still confident that we will find great meaning and learning in coming together. I've attached some of the documents I used to get started and three of the reflections from each meeting:
What is a Learning Circle -- this is 5 powerpoint slides defining a learning circle
Template is the plan posted above
TPI-NEA LC 5/21 agenda has a reflection from the first meeting;
Reflection on TPI is the second reflection;
Reflection notes from LC meeting of June 11th is the third.
Enjoy our lessons learned and themes as they emerge.