MetaAnalysis

The Why, How, and Lessons Learned From Meta-Analysis of Action Research

Abstract

The purpose of this symposium is to compare and investigate emerging themes across three ongoing meta-analyses of action research (AR) studies. Two of the studies focus on action research conducted by master’s degree candidates within universities, and the other is an international study, focused on published action research in education, business, and healthcare. While each researcher applied meta-analytic coding tools germane to her or his particular study, all attempted to synthesize action research findings across many studies. Each researcher gained important new insights from the metaanalysis of action research data but encountered challenges the research team believes are inherent in any analysis across large bodies of action research data that have been collected, analyzed and written up by diverse groups of people. The interactive symposium closes with an examination of how their combined meta-analytic approach led to new insights within a sample drawn from their respective data sets.

Symposium Overview

A panel discussion will commence with each participant discussing his or her particular study, analytic framework, findings, and methodological challenges. The panel will offer an overview of emerging themes, identifying key similarities and differences and share the additional insights gained by applying colleagues’ analytical framework to a subset of their respective study data.

Discussant Jean McNiff invited questions and discussion among those in attendance.

Margaret Riel, with her colleague Kathleen Lepori, analyzed the final reflection of 25 action researchers whose work is published on the Center for Collaborative Action Research (cadres.pepperdine.edu/ccar). With a focus on the reflective nature of action research (Schön, 1983, Coghlan and Brannick, 2005) and on the transformational change that is often a part of this process (Whitehead & McNiff, 2006), they sought to understand this change. They started with the action research question, “If we better understand what action researchers say about the changes they experiences, will this suggest better strategies for supporting action research process?”

They began with the assumptions that change occurred in knowledge, practice and identity. However, as they examined the reflective essays, they also noticed that students described the outcomes of action research on three levels: personal, organizational and scholarly.

• The personal level addresses changes to the person (knowledge, practice and identity in the professional context)

• The organizational is acquiring a deeper understanding of the process of change in the workplace -- attention to

community roles, knowledge and practice and how they change

• The scholarly level -- this is the development of knowledge, practices and identity of the researcher. Attention to

methods, metacognitive and findings.

Sentences were coded first for discussion of personal, professional or scholarly change and then for change in knowledge, practice and identity. Sentences that included more than one topic were coded for the most salient topic. They found that half of the reflective writing was focused on transformations of their personal knowledge, practices and identity in their professional setting. The rest of the reflections were evenly divided into insights on organizational learning and change and reflections on their knowledge, practices or identity as researchers. The reflective statements were also coded by knowledge, practices and identity. Looking across the three levels, students were more likely to reflect on how knowledge had changed, and then with equal focus on change in practices and identity.

An outcome of this research has been the development of different reflective practices where students spend time thinking about their knowledge, practice and identity as professionals and as action researchers. This is separated from their reflections on how others are changing in terms of their knowledge, practice and roles and identities. This direction is helping students move past descriptive analysis to a deeper focus on the how action research affects the way they develop skill and expertise as professionals. The goal of the shared session is to examine how this approach might change when extended to other data and how forms of analysis might see this data in a different frame.

Joseph M. Shosh studied the action research practices of 22 Pennsylvania and New Jersey teachers who met weekly during the fall semester of 2010 to share their evolving research practice and findings within their teacher inquiry support groups (Hammerman, 1999; Shosh & Zales, 2005, 2007). Here teachers developed and enacted their self-designed data collection plans to document their practice in a field log to make their familiar practices strange to themselves (Geertz, 1973). Within the groups, they discussed the analytic memos they penned to examine data through a series of progressive (Dewey, 1997), dialogical (Freire, 2003), social constructivist (Vygotsky, 1978), and sociolinguistic (Delpit & Dowdy, 2002) lenses. They wrote to learn about their data along the way, utilizing a wide array of narrative conventions, including anecdotes, vignettes, layered stories, pastiches, dramas, and poems (Ely, 2007; Richardson & St Pierre, 2005). Data were coded (Saldaña, 2009; Bogdan & Biklen, 2003) and then categorized in graphic organizers. Theme statements emerged as teacher researchers explained how the field log data contained within each category related to the research questions. Through his analysis of survey data, inquiry support group observations, and digitally recorded teacher interviews, Shosh identified the actions that teacher researchers took within their diverse teaching environments to

(1) support student achievement,

(2) develop collaboration among learners,

(3) differentiate instruction,

(4) encourage active student engagement,

(5) promote student ownership,

(6) facilitate student metacognition, and

(7) address inevitable challenges as teachers and researchers.

http://www.actionresearch.gr/home.

To download individual teacher research studies, visit http://home.moravian.edu/public/educ/eddept/mEd/thesis/2011.htm.

E. Alana James examined 150 published action research (AR) studies as reported in journals of action research, organizational development, education, healthcare, professional development, and public administration. Six additional studies of student work, published by their universities were included for the AERA symposium. Articles were chosen by convenience from webb as edsources, purposively representing all the major configurations as to how the methodology is used. The author makes an assumption that in some form AR is iterative and includes discovery, measurable actions and reflections by the practitioner or group involved in the study (Coghlan & Brannick, 2005; Dick, 1998; Elliott, 1991; James, Slater, & Bucknam, 2012; Kemmis, 2006; McNiff, 1993; McTaggart, 1989; Noffke & Brennan, 1997; Reason & Bradbury, 2008; Sagor, 2005; Somekh, 2003; Stringer, 2007; Tomal, 2005; Whitehead & McNiff, 2006; Zeichner, 2003; Zuber-Skerritt, 1992). Rather than focus on the processes involved, this meta-analysis studied published outcomes asking whether and to what extent they lived up to ideals presented in the writings of Kemmis (2006) and the early work of C. Argyris (1990), or Chris Argyris and Schön (1978). Each article was rated across several continuums of professional to emancipatory purposes as well as whether the work, as reported, focused on personal transformation or the actionable development of practice. As an example: a one was given to all articles that whose purpose was clearly focused on improving some kind of efficiency, a three was given if the article enhanced a critical view or understanding of the issue and a two if there were elements of both improvement of an efficient process while also developing a critical understanding of the issues involved. This meta-analysis leads to potential future discussion within the academy. If only 26% of the articles to day show clear actionable outcomes should we holding ourselves to a different standard? If and to the extent it has become clear that authors curb the way in which AR is presented to meet the standards of the journal, should AR journals have a prescribed ratio of what type of articles are published? What and how are AR articles being used outside of those who use the methodology? What effect is our work having and are we living up to the “brave new hope” ideals of our early authors who saw AR as a means to advance practice within the constraints of research?

Through their respective meta-analyses, James, Shosh & Riel noted that the studies examined share their respective findings in a multiplicity of ways, which sometimes makes it difficult to identify each theme clearly for coding purposes. Theme statements are often complex and may arguably fit multiple coding categories. Some theme statements extol the virtue of the action research methodology without articulating clearly what was learned through the methodology. Some findings statements tend to be outliers or negative cases worthy of separate examination. Researcher stances philosophically ground the methodology and the findings within individual studies, but stances across studies are not necessarily compatible with one another. The meta-themes provide a summary of research findings that bring a collective view of a body of studies into focus, while necessarily obscuring the significant details of each particular study. Moving past simple findings from meta-data into valid conclusions proves difficult because of the evolutionary nature of thought. Given that the data analysis happens over a long period of time, to what extent does the data analysis allow backtracking to include studies read and responded to earlier in the process? In an attempt to apply their respective meta-analytic frameworks to one-another’s data sets, the researchers encountered expected problems in superimposing coding systems derived inductively from one body of work onto another.

Because the individual researchers had initially coded data in response to their own research questions, the application of those coding schemes to alternate data sets revealed new insights about how the work of other action researchers was similar to and different from their own. The fact that the schemes largely worked suggests to us that there are indeed common social practices occurring across school, business, and healthcare action research settings that warrant further meta-analytic review. Action research is now about 50 years old, and, as such it is reaching middle age as a research methodology, so we believe it is appropriate to look back over a body of work to establish what has been done and to use that as a basis for improved practice in the future. By publicly sharing these findings, James, Shosh, & Riel hope both to engage in new dialogue with other education stakeholders about effective teaching and practitioner research practices and make the published accounts of their research participants more accessible to the action research community. In an era in which the value of graduate teacher education is increasingly called into question (Levine, 2005, 2006, 2007; Chingosa & Peterson, 2011), teacher action research holds tremendous promise in helping teachers share with one another what they’ve learned about “adding value” to classroom learning in ways that are far more authentic and hold much more promise than do current positivistic value added measures, and in a similar fashion, action research provides an effective process for continuous learning for practitioners and the possibility of transforming workplaces into learning organizations.

References

Argyris, C. (1990). Overcoming organizational defenses. Facilitating organizational learning. Boston: Allyn and Bacon.

Argyris, C., & Schön, D. A. (1978). Organizational learning. Reading, Mass.: Addison-Wesley Pub. Co.

Bogdan, R. C., & Biklen, S. K. (2003). Qualitative research for education: An introduction to theory and methods (4th ed.). Boston: Allyn and Bacon.

Chingosa, M. M., and Peterson, P.E. It’s easier to pick a good teacher than to train one: Familiar and new results on the correlates of teacher effectiveness.

Economics of Education Review, 30 (3), 449-465.

Coghlan, D., & Brannick, T. (2005). Doing action research in your own organization. London: Sage Publishing.

Delpit, L., & Dowdy, J. K. (Eds.) (2002). The skin that we speak: Thoughts on language and culture in the classroom. New York: The New Press.

Dick, B. (1998). Cycles within cycles. Occasional pieces in action research methodology, # 8. . Retrieved July 23, 2006, 2006, from http://www.scu.edu.au/

schools/gcm/ar/arm/op008.html

Elliott, J. (1991). Action research for educational change. Milton Keynes, England: Open University Press.

Ely, M. (2007). In-forming representations. In D.J. Clandinin (Ed.)., Handbook of narrative inquiry: Mapping a methodology. Thousand Oaks, CA: Sage.

Freire, P. (2003). Pedagogy of the oppressed (M. B. Ramos, Trans., 30th Anniversary ed.). New York: Continuum. (Original work published 1970)

Geertz, C. (1973). The interpretation of cultures. New York: Basic Books.

Hammerman, J. K. (1999). Teacher inquiry groups: Collaborative explorations of changing practice. In M. Z. Solomon (Ed.), The diagnostic teacher:

Constructing new approaches to professional development (pp. 187-200). New York: Teachers College Press.

James, E. A., Slater, T., & Bucknam, A. (2012). Action research for business, non-profits and public administration : A tool for complex times. Thousand Oaks,

CA, USA: Sage Publishing.

Kemmis, S. (2006). Participatory action research and the public sphere. Educational Action Research, 14(4), 459-476.

Levine, A. (2005). Educating school leaders. Washington D.C.: Education Schools Project.

Levine, A. (2006). Educating school teachers. Washington D.C.: Education Schools Project.

Levine, A. (2007). Educating researchers. Washington D.C.: Education Schools Project.

McNiff, J. (1993). Teaching as learning: An action research approach. New York: Rouledge.

McTaggart, R. (1989, September 3 – 9, 1989). Sixteen tenets of participatory action research. Paper presented at the 3er Encuentro Mundial Investigacion

Participatva (The Third World Encounter on Participatory Research) Managua, Nicaragua.

Noffke, S. E., & Brennan, M. (1997). Reconstructing the politics of action in action research. In S. Hollingsworth (Ed.), International action research: A casebook

for educational reform. London: Falmer.

Reason, P., & Bradbury, H. (2008). The Sage handbook of action research : participative inquiry and practice (2nd ed.). London ; Thousand Oaks, Calif.: SAGE

Publications.

Richardson, L., & St. Pierre, E. A. (2005). Writing: A method of inquiry. In N. K. Denzin, & Y. S. Lincoln (Eds.), The Sage handbook of qualitative research

(3rd ed., pp. 959–978). Thousand Oaks, CA: Sage.

Sagor, R. (2005). The action research guidebook: A four step process for educators and school teams. Thousand Oaks, CA: Corwin Press.

Saldaña, J. (2009). The coding manual for qualitative researchers. Thousand Oaks, CA: Sage.

Schön D., (1983). The Reflective Practitioner Basic Books, New York.

Shosh, J. M., & Zales, C. R. (2005). Daring to teach writing authentically K-12 and beyond. English Journal 95(2), 77-81.

Shosh, J. M., & Zales, C. R. (2007). Graduate teacher education as inquiry: A case study. Teaching Education 18(3), 257-275.

Somekh, B. (2003). Theory and passion in action research. Educational Action Research, 11(2), 247-264.

Stringer, E. T. (2007). Action research (3rd ed.). Los Angeles: Sage Publications.

Tomal. (2005). Action research for educators. Lanham: Rowman & Littlefield Education.

Whitehead, J., & McNiff, J. (2006). Action research: Living theory. London: Sage.

Vygotsky, L. S. (1978). Mind in Society: The Development of Higher Psychological Processes. Cambridge: Harvard University Press.

Zeichner, K. (2003). Teacher research as professional development for P-12 educators in the USA. Educational Action Research, 11(2), 301-326.

Zuber-Skerritt, O. (1992). Professional development in higher education: A theoretical framework for action research. London: Kogan Page.