Rationale for WISR's Methods of Evaluation

WISR's Philosophy of Transformative Evaluation

In the field of evaluation, most people make the important distinction between Summative Evaluation and Formative Evaluation. "Summative Evaluation" answers the question of whether or not an individual or a program meets a certain level of standards--in conventional colleges, students receive letter grades, and in some nontraditional programs, like WISR, students are graded pass/no pass. These are both summative approaches to evaluation. With regard to a program, such as, for example, one of WISR's degree programs, faculty, advisory committee members, or outsider evaluators (e.g., the State or an accrediting agency) might evaluate the degree program in terms of whether it meets certain standards--either WISR's own, stated standards, or the standards of some outside agency. This type of evaluation is important in determining whether a program, or a student, for example, is performing "good enough."

"Formative Evaluation" is concerned with using evaluation to determine how an individual or a program can improve. Based on the evidence obtained in evaluating the individual or program, we use the evaluation process to come up with some evidenced-based suggestions, or strategies, that will help the individual, or the program, to improve. In these cases, evaluation is used to help students make better progress in their studies, learn more effectively and more deeply in terms of some stated outcomes or goals, or improve in the pursuit of their own learning goals and interests. Similarly, formative evaluation, for example of one of WISR's degree programs, gives faculty and students insights into strategies and improvements that will help the degree programs themselves to better achieve their stated objectives. As we will briefly discuss further in the section on Grading and Documentation, one of the challenges to maximizing learning is that sometimes the need to grade, to perform a summative evaluation, distracts from learning. There are many well-documented cases of how grading can detract from learning, go back at least to Becker's, Geer's and Hughes' study, Making the Grade [Howard S. Becker, Blanche Geer, Everett C. Hughes. Making the Grade. John Wiley and Sons, 1968.]

The sometimes, contradictory purposes of formative and summative evaluation, are related to what is also known as "Learning-Certification Contradictions. Specifically, the need to do summative evaluations as part of the credit-granting and degree-granting process sometimes results in students and faculty paying attention to the demands of certification, of the summative evaluations, rather than only focusing on learning and formative evaluation. At WISR, we try to consciously confront the tensions between learning and certification, between summative and formative evaluation, in our approaches to working with students and evaluating student learning. Being mindful of this tension, and some of the contradictions involved, does not completely remove the problem, but it does enable students and faculty to put as much attention as possible on the pursuit of learning and the role of valuable, formative evaluation. One of WISR's co-founders first became aware of these contradictions, as well as some approaches to addressing the contradictions, when he, and a faculty colleague initiatied an innovative individualized learning program in the College of Community Services at the University of Cincinnati in 1972. They wrote about these insights in several publications, and one such article is available here, with the permission of the University of Alabama Press.[Bilorusky, J. & Butler, H. (1975) Beyond contract curricula to improvisational learning. In N. E. Berte (Ed.), Individualizing education through contract learning (pp. 144-172). Tuscaloosa, AL: University of Alabama Press.]

This leads to the discussion of a third concept of evaluation, developed here at WISR during the process of our founding in 1975, and one that we have discussed and paid attention to over the years--"Transformative Evaluation." Transformative Evaluation is the pursuit of an inquiring process of evaluation that attempts to not be "paradigm-bound"--that is the evaluation process is to include questioning of the meta-assumptions that underlie what is being evaluated, in this case, the learning at WISR. As noted above, one of the underlying dynamics is the inherent contradictions between learning and certification. This means that at WISR, we attempt to evaluate the impact of our academic requirements, of the structure of our courses and programmatic curricula, and of our academic methods of evaluating student performance. In this way, we can try to minimize the negative impact of certification on learning. Our efforts to accomplish this include continually asking our students about the ways in which our academic requirements both impede and facilitate their learning. As one might expect, we sometimes get different answers from different students, and we try to take into account this complexity in making decisions about our overall curriculum, as well as about how to work with individual students.

In addition, unlike summative and formative evaluation, transformative evaluation involves a conscious re-evaluation of our mission and program learning outcomes, rather than only addressing how well we are living up to our mission, and pursuing and achieving our intended outcomes.

Transformative evaluation also enabled us to realize, after more than 20 years, that an important, but not initially stated, part of our mission has always been to help our students to "build bridges to the next important things they want to do in their lives." We elaborated our articulation of this new, explicitly stated part of our mission, as follows: "We believe it is important to consciously and continually help students to design learning activities—action projects, research, and writings—that help to build bridges to the student’s desired career and life paths, oftentimes while also working toward a more sustainable and just future. We believe that people should not have their visions limited by the definitions of existing jobs and careers, and that they can, and should, be encouraged to be both visionary and realistic in pursuing a life path that makes sense to them. Consequently, WISR’s educational programs are suited for learners with many different types of future goals, including but not limited to: changing careers, pursuing advancement in one’s existing career, becoming more capable and more meaningfully engaged in one’s existing job or career niche, writing books and articles, organizing people and networks for social change, or creating new organizations and programs."

Evaluation as the Discovery of Grounded Theory

Related to the value we place on transformative evaluation, we have learned the merits of those approaches to research which develop concepts, theories, questions, tentative courses of action, and working hypotheses from specific examples and stories. Specifically, we have learned much from sociologist, Herbert Blumer’s book, Symbolic Interactionism [Blumer, H. (1986). Symbolic interactionism: Perspective and method. Berkeley: University of California Press], and medical anthropologists, Glaser and Strauss’ The Discovery of Grounded Theory [Glaser, B. & Strauss, A. (1967). The discovery of grounded theory. Chicago: Aldine]. Of special note is Blumer’s notion of “sensitizing concept”—where a “concept” is always in the process of being re-evaluated and reformulated in light of emerging evidence and experience. Such concepts are not merely stated in abstract terms, but are also coupled with a rich variety of examples that illustrate the complexity, situational variability and nuances of the concept. In line with Glaser and Strauss’ research approach, new experience and evidence provides new examples, and counter-examples, so that the sensitizing concept can be refined and revised over time. For example, the application of this approach to the field of social work has been discussed by Butler, Davis & Kukkonen (1979) [ Butler, H., Davis, I. & Kukkonen, R. (1979). The logic of case comparison. Social Work Research and Abstracts. Vol. 15, No. 3].

We state our learning outcomes in terms that are measurable, in ways that we are able to observe whether or not a student has accomplished or achieved a particular outcome. Over time, we pay attention to the variety of specific examples and cases that will necessarily be both distinctive and also illustrative or indicative of the stated outcome. We do not use narrowly defined, quantitative indicators, nor do we settle for vague, global abstractions. Instead, our evaluation process is a continual inquiry into how well evidence, drawn from the written papers and observations of the learning and efforts of different students and in different circumstances, help to build an accurate, albeit complex and nuanced, picture of how well, and in what ways are students are (or aren't) learning. This approach to evaluation, and its related approach to learning--as emergent and growing out of the experiences of the learner is very much in line with John Dewey's philosophy [e.g., John Dewey, Experience and Education. Kappa Delta Pi, 1938].

WISR’s Learning Methods, Learning Outcomes and Assessment of Outcomes: Relevant Theories and Epistemology on the Development of Expert Knowledge and Practice in the Real World

As part of our ongoing process of reflecting on, and articulating, our methods of learning and evaluation, WISR faculty recently revisited our ideas about "expert knowledge and practice"--as a way both to educate further our students, as well as to guide our methods of evaluating student learning and engaging in continual review of our academic programs. The following as the abstract of the discussion paper written by WISR President, John Bilorusky, in May 2018, as part of this dialogue. The paper may be read in its entirety, here. The synopsis follows:

My aim in this paper is to clarify how and why WISR’s approach to learning makes sense as a sound way to help learners become more expert scholars, practitioners and activists—people who are able to contribute, as leaders, in the professional fields and communities with whom they are committed and engaged. Our insights about our learning approach—how to frame our learning goals and outcomes, the importance of action-research, and the value of learner-centered education that promotes self-directed learning--have been informed not only by our practice and experience over the past 40+ years, but also by a few especially valuable theoretical perspectives and insights developed by others.

According to Kuhn, science is a history of relatively stable “plateaus” of little change accompanied with illuminating puzzle-solving, and then, punctuated with dramatic periods of crisis and revolution, fueled by “anomalies” or data that are “exceptions to the rule.” An important implication of Kuhn’s work for WISR learners is to direct attention to the importance of learning about and then doing action-research--to identify and study “anomalies” or situational “exceptions to the rule” that are not well explained by theories in psychology, the social sciences and their professional fields. By making this a conscious and intentional part of their action-research projects at WISR, our learners are able to critique and refine the theories and strategies of practice that they have been learning. In addition, Kuhn and others have shown that “science” is made up of emotional and political conflicts among scientists, as well as intuition, engaged curiosity, and many other human dynamics that are quite varied and full of the complexities and uncertainties of life. This reality is contrasted with the seemingly mechanistic, precise, dispassionate, and emotionally detached methods of observation that are presented as essential and unbreakable rules and qualities of scientific inquiry to novices reading books on research methods in the social sciences.

As we will see when we consider in some detail the insights of the Dreyfus brothers, that we can sometimes be impeded by our emotions is no reason for us to aim to rid ourselves of our emotions—first, because that is impossible and to do, and otherwise we would be kidding ourselves, and second, because sometimes emotions, as well as our idiosyncratic experiences, may actually aid the development of knowledge and expertise and be invaluable qualities of human intelligence.

The late Hubert Dreyfus (Professor of Philosophy, UC Berkeley) and his brother, Stuart Dreyfus (Professor of Applied Mathematics and Operations Research, UC Berkeley) have formulated 5 stages of expert skill development, a model that has been of great interest to others in many fields over the past several decades. The stages are novice, advanced beginner, competent, proficient and expert. The Dreyfus brothers discuss how people can progress though these stages toward greater expertise, and the changing roles of practical involvement, personal experience, and instruction in the learning processes at each stage, as well as the evolving, complex interplay of emotional engagement, perception and cognitive analysis.

Our affinity for the Dreyfus model is not simply because it is consistent with what we “say”, but more because it resonates with what we have experienced, and have come to appreciate and sense quite deeply, based on our experience with dozens and dozens of students and colleagues from many walks of life, and with different interests and passions. For some of us at WISR, these experiences cover decades of experience as faculty who have been engaged in guiding and supporting learners in their efforts to develop greater expertise. The Dreyfus model is a very good “public knowledge” articulation that approximates what has come to be the personal knowledge of a number of us at WISR. The Dreyfus model does not suggest that a person is either novice, or competent or expert, in general, but rather, only in a particular area or domain of endeavor. So, for all of our students at WISR, we are dealing with at least two or three overlapping domains of practice and knowledge—the broad field of study in which the degree is offered, and typically one or two main areas of specialization which are fueling the student’s passion for learning.

We all know that we have strengths and limitations, and it is a regular part of learning at WISR for students to build on their strengths in the same activities in which they are simultaneously addressing areas of weakness or special challenge. Vygotsky’s theory of development and his concept of “Zone of Proximal Development” are very valuable in understanding how and why “readiness” to develop further should be combined with intentional guidance and support. For example, in a particular course, a student may design, with faculty guidance an action-research project, related to a student’s area of strength (e.g, further developing expertise and knowledge as a somatic therapist) while also addressing a limitation (e.g., gaining insights and experience as an educator of practicing therapists). In this regard, the student’s area of greater strength and expertise becomes a sort of bridge for the student to develop in an area where they have less expertise. An inquiring, self-aware and self-directing learner may take their experience with the stage of their greatest expertise in any domain, and then use their “sense” of that realm of expertise (let’s say “proficiency”) to guide them in domains where they are less expert, especially if they can be engaged in learning activities that include both domains of knowledge and practice.

At WISR, some of us on the faculty have long recognized the value of the Dreyfus perspective on stages of expert development; however, only recently have we come to a deeper appreciation of how closely these stages approximate our objectives for the learning progress of WISR students. Here are the main highlights of how the Dreyfus stages contribute to learning objectives at WISR:

• In each degree program, our intention is for students to achieve a level of expertise in both the broad field of study and at least one area of specialization appropriate to that degree—for BS students, advanced beginner; for MS students, competent; for EdD students, proficient.

• We have as a further guiding intention, to help students in each degree program, embark on the transition toward the next level

• For all WISR students we intend for them: 1) to develop a deeper and richer appreciation and understanding of how to incorporate perspectives on multiculturalism and social change and justice into their everyday functioning, 2) to develop, or further develop, their abilities as a self-directed learner, and 3) to learn and practice methods of participatory action-research, and more specifically to use these methods to help them to continue to progress through the Dreyfus stages, while at WISR and beyond graduation.

Specific learning outcomes for Masters’ students, specific learning outcomes in the student’s major field and area(s) of specialization include: 1) learning how to do conscious and deliberate planning and critically reflective comparison of alternative courses of action; 2) critically examining theories and principles of practice with an increased awareness of uncertainty, complexity and subtlety in using such theories and principles. For doctoral students, specific outcomes include: 1) the development of a holistic perspective in their field and areas of expertise; 2) a well-informed familiarity with a variety of theoretical perspectives and principles of practice; 3) the ability to make critically informed decisions in the face of many variables and changing situations; and 4) active, capable, and confident engagement in action-inquiry and experimentation to develop further their proficiency, and 5) contribute to creating new knowledge and practices in their field and area(s) of specialization.

Elements of WISR’s Distinctive Learning Approach, and Development from Novice to Expert:

• Personalized learning leads to experiences that are of great significance and worth to each student, and it promotes better learning not only by motivating students, but by creating the emotional involvement that is essential to the development of greater expertise in the Dreyfus model, and confirmed by other research.

• With WISR’s emphasis on improvisational, transformative, self-directed learning, students become in engaged in not only evaluating their learning, but also in evaluating the direction of their learning. In this regard, learning becomes “transformative.”

• WISR faculty guide students to meaningful direct action and practice, and from the perspective of the Dreyfus model and stages of expert development, emotional involvement and practical experiences are essential in progressing through the higher-level stages.

• WISR gives great importance to students learning and using methods of participating action-research. These methods help students to evaluate and critically reflect on their own efforts and the endeavors of others, as well as in formulating new alternatives and plans of action. Beyond the methods of data gathering, students learn how to judge and synthesize evidence, and formulate, re-evaluate and re-formulate concepts in their areas of interest.

• WISR students read key theories in their fields of study as well as “bigger picture” philosophical works, and also, quite importantly, readings that convey the tangible and nuanced details of the practical and lived experiences of others.

• Faculty provide students with the challenge and support that enable them to make transitions from one stage or domain of expertise to the next, in line with Vygotsky’s idea about the Zone of Proximal Development.

• Student self-assessments, oral exams and ongoing dialogue with faculty enhance student self-awareness and critical reflection on their learning processes and outcomes.

WISR’s learning outcomes and methods have grown out of over 45 years of conscious and highly committed educational experimentation. In this paper, I’ve discussed intellectual traditions and theoretical insights that have aided our efforts. WISR has been fortunate through these many years to attract continually dedicated faculty, each of whom has brought their own distinctive, and very valuable, experiences, insights and wisdom to our collegial processes. Over the years, faculty discussions have continually led to improvements and refinements. We are proud of our students and alumni for their many successes and important work and contributions to the lives of others, and as faculty, we are pleased to have been supporting members of the cast in this drama. The ideas articulated here are only an update at one point in time, regarding some of the history of the evolution of our approach to learning, and the current formulation of learning outcomes.