“The controversy over the best evaluation design continues to rage. During the late 1970s and early 80s, many debates were held at professional evaluation meetings about the value of experimental design in evaluation. Many evaluators thought the issue had been laid to rest, i.e., there was an acknowledgement of the usefulness and appropriateness of experimental designs--sometimes, but a consensus that such an approach does not constitute a gold standard, and not even the most frequently used design.
For many evaluators, especially in education but also in other fields, this prior understanding has been turned on its head by the US Government endorsement of randomized clinical trials as the sine qua non approach in evaluation (followed by quasi experimental and regression discontinuity designs). There has been considerable reaction to this methodological/ideological turn, with strong objections by AERA, AEA, and NEA.”
(communication Dec. 5, 2005 on The Role of RCTs in Evaluation, AEA, http://weblogs.elearning.ubc.ca/mathison/archives/2005/12/the_role_of_rct.html)
During the past decade, ushered in by an administration that endorsed NCLB, the evaluation profession has once again seen pressure to pursue RCTs and funding has been tied to these requirements. Many federal agencies, among them SAMHSA and the Department of Education, ushered forward a series of judgments regarding the programs they would fund. “Best practices,” “Model programs,” “promising programs,” became the nomenclature of the 2000s and inspired another round of debates within the evaluation field as to the appropriateness of these terms, the limitations of RCTs and other derivations of the experimental model of evaluation when applied to the diverse contexts and communities in which programs and services are delivered.
Evaluators were split around the idea of taking a stand against RCTs or objecting to them as the “gold standard” of evaluation. AEA issued a position statement (see reference below as required reading) that was protested within the ranks but supported by many. In 2006, many widely known evaluation experts gathered at the Claremont Graduate University Stauffer Symposium to answer the question, “What constitutes credible evidence in evaluation and applied research?” (required viewing and reading). In the Fall of 2005 and 2006 the Nevada Evaluation Association and the Center for Program Evaluation co-sponsored two one-day conferences and facilitated dialogues under their Evaluation Training Institute that further explored the question. The 2005 conference examined the question in relation to SAMHSA-funded projects and the aims of its “Service to Science” initiative focusing on substance abuse prevention among youth. Local evaluators offered workshops on their projects and their dilemmas. The Fall 2006 facilitated dialogue featured invited speakers, Judith Ottoson and Lawrence Green to present on and discuss the topic: “If we want more evidence-based practice we need more practice-based evidence: Issues of neglected external validity in evaluations of the public health field.”
The debate and response to the question has reverberated among the ranks of evaluators, their clients, and their funders, but has not systematically been addressed or followed through to some resolution by key stakeholders in Nevada: state agencies and public institutions, who often are the messengers of federal pass-through dollars and their local evaluators, who must translate those messages and walk a fine balance between the needs of their clients and the demands of their funders. This Learning Circle will offer an opportunity to present responses and solutions or positions to the question, and encourage participants to incorporate diverse perspectives while achieving some form of consensus for guiding the field of evaluation stakeholders in Nevada.
1) Encourage greater self-awareness among professional evaluators regarding their personal philosophies and orientations and strategies for negotiating the tensions presented in mandated evaluation designs and approaches.
2) Develop products and position statements related to the evaluation profession in Nevada that address the question, “what constitutes credible evidence in health, human service and educational contexts?”
3) Strengthen in-state evaluation networks and capacity to respond to requests for evaluation in a timely and consistent fashion.
Note: The question is very broad and the group will be invited to refine the question to suit their interests, applications and time commitments so that it lends itself to both individual and group projects.
Six to eight emerging and mid-career evaluators residing and/or working in Northern Nevada who are members of the Nevada Evaluation Association and/or are associated with public institutions in Nevada. Recruitment of evaluators through NEA’s on-line membership directory (N = 25) and website, as well as email notification/invitation to evaluation staff at Division of Health Services, Department of Education, local education agencies, Department of Child and Family Services (DCFS), Cooperative Extension. This Learning Circle will encompass the entire state of Nevada, but initially draw upon only those working in Northern Nevada (Carson City is the capitol and there are statewide evaluators positioned there). Ideally the group will consist of professional evaluators from diverse private and public organizations, with no more than half representing internal evaluators for public agencies. LC members will be recruited using a posting on the NEA website and a message sent to members by the organization. In addition, a Learning Circle Website/Google page will be established with an overview of the opportunity, the commitments and expectations, and an application for interested evaluators. This link will be emailed to various sources, including the university, the state’s department of education, health and human services division and local school districts as well as NEA members, private consultants, and non-profit agencies.
The communication will involve face-to-face, real-time and static distance-based communication technologies. This LC program will take place over a 6 month – 8 week period (either May-mid-July or September-mid-November, 2009). The Learning Circle will be launched with a one-half to full-day, face-to-face session, during which members will have the opportunity to bond, build trust and get grounded in the debate among expert evaluators in response to the LC question, providing a common base of knowledge and experience from which to launch individual investigations.
In addition, to live communication the LC will communicate asynchronously on an ongoing basis using a “Google Site” as the primary mechanism for sharing information and providing feedback to group members.
The completed product of this LC process will be selected and co-constructed by members, and may draw upon the following options for consideration:
a collaborative position statement or product on the group’s response to the question, “What constitutes credible evidence in evaluation?” to put forth for critical debate and endorsement by the broader field of Nevada’s evaluation stakeholders.
a series of products (3 to 4) providing stories or case perspectives that support the position statement and illustrate how the each author chose to investigate and address a sub-question within the overarching theme.
a personal development plan aligning one’s evaluation philosophy/orientation with the group’s response or on advocating for a broader reach in evaluation, based on their position and affiliation.
Pre Start –
Pre-reading of: The Role of RCTs in Evaluation, AEA, http://weblogs.elearning.ubc.ca/mathison/archives/2005/12/the_role_of_rct.html) and one article in New Directions for Evaluation Volume 2007 Issue 113, Special Issue: Informing Federal Policies on Evaluation Methodology: Building the Evidence Base for Method Choice in Government Sponsored Evaluation Published Online: 21 Mar 2007Copyright © 2008 Wiley Periodicals, Inc., A Wiley Company
Voicethread introduction (to be developed)
On-line dissemination of “Living the Goal-Free Life” bonding activity to be completed in advance and brought to the one-day worksho.
Session 1 Full-day workshop in Reno, NV—
May 14, 2009 at Redfield Campus (smart classroom) 9-3 p.m.
Orientation to the Learning Circle concept
Establish group ground rules and expectations
Bonding activity: Living the Goal-Free Life
Presentation and discussion of select videos from the Claremont Graduate University Stauffer Symposium 2006.
Presentation of a guiding template for considering the question and investigating areas of professional interest Identification of individual questions and potential cases
Posting and review of on-line Homework assignment: Reflection on one’s personal philosophy and: 1) where the align with responses to the question; 2) inherent tensions they have experienced in their current role as relates to this alignment (or mis-alignment); and 3) possibilities for rethinking their dilemmas and supporting the evaluation field from within.
Week 2 May 21- Sharing of Reflections and Possibilities
Develop draft LC topic list and preliminary guiding principles as they emerge in the discussion of the question.
Week 3 May 28- 6-7 p.m.
Defining Audiences and Purposes to which the LC projects/products might be addressed
Week 4 June 4- 6-7 p.m.
Drafts of LC products
Week 5-9 June 11-July 16, 2009 6-7 p.m.
Sharing and collaboration of drafts, methods and writing.
Week 10 July 23 6-7 p.m.
Final Presentation of LC products to group
Circle Opening Message:
This is the welcome message that was sent to the first participants of this learning circle. It
describes a initial activity on voice threads for the participants that precedes a face to face
session.
Welcome to the NEA-TPI Learning Circle!
There are six of us signed up so far for this Learning Circle.
We can use this site as a place to upload and share documents
we are working on. We can also use it as a discussion board.
When you come into this site, please take a moment to go to your
own page and add your bio and photo to it. Simply click on your
name above and then hit edit to make any changes. The instructions
for how to do this once you get to the page editor are in the body
of your page. Delete the instructions once you have added your
photo and bio.
Here are the first assignments for you to complete before we meet on May 14th:
Listen to the Jing file:Introductory Activity
Read the articles attached below on the AEA and non-AEA response
to the question: What constitutes credible evidence in evaluation?
Then go to the link below to hear how to reflect on this reading and add
your comments and images to this voicethread. If possible, also bring
a copy to our meeting of your thoughts or send it to me in advance so
that we can look at it together during our session.
http://voicethread.com/share/369073/
When you are done with all of these activities, send us all an email to
let us know we can check the website. If you want to be automatically
notified of any changes to the website, go to More Actions on the
menu bar above this page and add your preferences. You can either
be notified of changes to this page or the overall website. This will
automatically send you an email.
I'm looking forward to your responses.
Deborah
Spirit in Action
Deborah Loesch-Griffin, Ph.D.
Turning Point, Inc.
www.turningpointnevada.com
6/26/09
We are well underway--having met now three times, and although there are fits and starts, I am still confident that we will find great meaning and learning in coming together. I've attached some of the documents I used to get started and three of the reflections from each meeting:
What is a Learning Circle -- this is 5 powerpoint slides defining a learning circle
Template is the plan posted above
TPI-NEA LC 5/21 agenda has a reflection from the first meeting;
Reflection on TPI is the second reflection;
Reflection notes from LC meeting of June 11th is the third.
Enjoy our lessons learned and themes as they emerge.
Deborah