What Threatens The Validity of Beason’s Study?
Entisar Elsherif, Libya
Indiana University of Pennsylvania
Introduction
Many studies investigated business people’s response to errors in writing to emphasize the importance of ‘error-free writing’. One of these is Larry Beason’s (2001) study that is reported in an article titled Ethos and Errors: How Business People React to Errors. This study is a survey that was conducted to explore error gravity and non-academics’ reactions to errors. In this survey, fourteen business people completed questionnaires and participated in interviews. The results indicated that there were a variety of reactions to errors. It was also revealed that readers made negative judgments on the writers. Readers’ response to errors threatened writers’ credibility by creating adverse decisions on the writer as a writer, a business person, and as a representative. This paper is an attempt to analyze Beason’s methodology in studying response to error and to explore how successful was the researcher in investigating response to errors. Beason was efficacious in classifying his research results into three categories; however, subject- selection, mortality, and exposure are considered to be threats to the validity of his research.
The first question that is raised: why was Beason interested in response to errors if there is a reasonable amount of research in this area? Previous research looked at business people’s reactions to errors by studying the seriousness of errors. They looked at how much errors were bothersome without looking at the causes of disturbance or readers’ decisions on writers’ characters as writers and employees. Beason stated that teachers and students needed to understand the severe impact of errors on “comprehension” and the “writer’s credibility” (p. 35). He tried to illustrate the seriousness of errors and its interference with meaning resulting as unsuccessful communication. By studying readers’ reactions to errors, errors study changed from exploring simple textual issues to focusing on greater issues in rhetorical matters of meaning construction.
Summary of Beason’s study
In order to study non-academics’ reactions to errors, Beason concentrated on a certain number of individuals with the aim of examining “a highly individualistic process” (p. 35). His research questions were: (1) how non-academics such as business people respond to errors? And (2) what elements constitute a person’s reaction to error? Beason studied fourteen business people’s reactions to errors and investigated what created these reactions. Beason’s aim was to raise awareness of effective error-free writing by concentrating on readers’ judgments on the writers of the text. He found that there was not any similarity in error gravity because there were no ‘idiosyncratic’ readers’ interpretations of error. The study had shown that readers’ elements of interpretation were repeated. Reactions to errors were not about the same error to the same degree, but they were common. Beason synthesized individual responses to specify that there were not “quantitative formulas for anticipating a reaction to error” (36). However, people’s negative reactions to errors in business writing seemed to have major qualities.
Beason chose survey research because surveys are usually used to explore “people’s attitudes, beliefs, values, demographic facts, behavior, opinions, habits, desires, and other types of information” (McMillan & Schumacher, 1989). Since he is investigating business people’s opinions about errors and their reactions to it, survey research is the appropriate methodology to look at response to error. Questionnaires and interviews are widely used in surveys to collect information on the topic of interest (Mertens, 2011). Beason used questionnaires and interviews, to gather data that would provide answers to his research questions. Questionnaires were used to gather information related to error gravity whereas interviews were used to look at the subjects’ reactions in depth.
The procedure of the study was in two phases. The participants were asked to complete questionnaires, and then they were interviewed. During the first phase, the participants were asked to rank twenty boldfaced errors to scale error gravity “using 1-4 scale” starting from 1 as “the least bothersome” error (p. 38). The stimulus in the questions was that he led the subjects to be bothered by boldfaced errors. He did not give the participants the chance to identify the errors by themselves and scale how it bothered them. In the second phase, the participants were interviewed to explain their reactions to errors. He found that not only the reader but also what is being read influenced error gravity. In other words, error gravity depends on who is reading the text as well as the text itself. To illustrate, there was inconsistency when scoring the most to the least bothered misspellings, such as “they’re” and “their." For some subjects, a misspelling seemed as just a misspelling, for others it was looked as whether it “sounded odd” when it was read aloud or not (p. 42). The inconsistencies in responses about errors were results of two variables: “textual and extra-textual features of discourse” (p. 46).
Beason specified that the effects of errors on readers and their assumptions about writers are greater than the writers’ or employees’ expectations or realization. He stated that subjects indicated that the credibility of people as writers or employees can be affected by their errors in their written texts. Accordingly, he represented the specific image problems that the subjects noticed into three main categories and eleven subcategories. The first category is related to the writer’s credibility as a writer. In this category, errors exemplify the writer as a hasty writer, a careless writer, an uncaring writer, and an uninformed writer. The second category is about the writer as a business person in which errors influence the person’s image and characterize him/her as a faulty thinker, not a detail person, poor oral communicator, a poorly educated person, and a sarcastic, pretentious, aggressive writer. The third category presents the writer as an organizational representative in which he/she represents the company to customers and/or represents the company in court. Those three categories and their underlying subcategories demonstrate how readers’ response to errors affects writers’ images and credibility.
Subject-selection as a threat
Subject-selection is a significant stage to ensure successful accessibility to information related to the research question through questionnaires. Beason chose fourteen subjects as the sample of his population. They were business people from “one ethnic group, white Americans from two regions in the US” (p. 36). The researcher did not specify his reasons behind choosing business people. Hairston (1981), who conducted a similar research on business people’s response to error, explained her choice of business people as non-academics because they are the ones who hire college students and graduates that fail to write comprehensible reports or letters. This made those non-academics complain from their errors and blame the teachers as not being able to doing their jobs effectually. Correspondingly, Beason indicated that if one of the teachers’ goals of teaching writing is to prepare students to write efficiently outside the classroom to the workplace, non-academics’ response to error need to be considered. By conducting such research, teachers as well as students will be able to recognize the importance and influence of errors on readers and to signify the importance of error-free writing in business writing specifically.
The researcher used purposeful sampling, also known as purposive sampling, to choose the subjects. Purposeful sampling is one of the nonprobability approaches (Weisberg, Krosnick & Bowen, 1996) that is used to select a small number of participants (Mertens, 2011). Despite the fact that generalization was not intended in this research, the fourteen subjects are really a little number for study. Alreck & Settle (1995, p. 59) revealed that small samples will have sample errors, and this will lower reliability. Accordingly, fourteen subjects as a sample is certainly small and will lessen the reliability of the research findings. This is because this limited number does not represent all the business people’s thoughts and reactions to error in business writing. They might have opinions that are different or not as similar as the whole population. It is known that the larger the sample the higher the probability of the representation of the results of the whole studied group, which is business people. In addition, the way the researcher chose his subjects appears to be limited to a particular number of professions and to certain limited regions. The subjects represented eight managers from different sectors, one Computer Program analyst, one senior agent, one accountant, one Digital Graphics specialist, one Human Resource partner, and one assistant administrator. The managers were representing Investments branch manager, District Service banking manager, automobile club manager, a gold-mining company’s lab manager, three regional banking managers, and a real estate manager. So, choosing only one or two subjects from each profession would not provide results that can represent the group or can be generalized. As a matter of fact, one person from one profession does not represent a group or an organization. Including business people from just two regions in America meant limiting the research results, as well. There is a high probability that the researcher neglected some other business people. The probability of avoiding including other business people is regarded as a disadvantage of purposeful sampling.
Looking at the research design, it seems that there are some limitations that question the findings of the study. As explained earlier, the researcher chose questionnaires and interviews to investigate error gravity and readers’ reactions to error. According to Mertens (2011, 185), “serious bias can occur in the interpretation of the results from a survey that does not make explicit its sampling and the characteristics of the respondents." Looking at the way the subjects were selected and their professions and regions, it is clear that the researcher did not provide satisfactory details to clarify his sampling strategies and respondents’ characteristics. It seemed difficult to find evidence that the researcher carefully chose elite business people. The chosen fourteen subjects are not an adequate number to represent the population of the business people. In addition, there was lack of diversity of the subjects’ profession. In brief, what made the subject-selection seemed as a limitation is the way they were chosen and their small number that might not represent business people’s thoughts and reactions to errors.
Mortality as a threat
Beason indicated that seven of the participants dropped out of the study. Three of them dropped out because reading and writing was not practiced in their daily jobs as used to be. The other four dropped out because they did not have enough time to participate in the study. Consequently, those seven recommended their colleagues who were willing to participate as replacements. Beason indicated that they met the criteria, and therefore, they were replaced. He did not specify the way he used to make sure they met the criteria. This drop out is known as mortality and is regarded as a threat to the validity. This change might have influenced research. However, the researcher did not provide information about how this change did not affect the findings.
Exposure as a threat
Beason chose to differentiate the errors in the sentences of the questionnaire by writing them in boldface. His reason was to show the participants that his attention was to get their opinions on such errors, which is the main focus of the study. Thus, he designed the questionnaire using a specific wording. According to Gray (2004, p. 207), the questions' wording can influence the questionnaire’s validity. Leading questions are one of the pitfalls of the questionnaire’s design. Even though Beason tried to provide reasons for using special typed font features to distinguish errors from the rest of the sentences, boldfaced errors seemed as a leading question.
According to Mertens (2011, p. 189), researchers should “avoid leading questions." It appears that Beason did not. The wording of the questions made them seem as leading questions. This is a threat to the validity of the questionnaire. Although Beason knew that readers’ natural response could be altered because of the boldfaced wording, he thought that this way would help the reader to understand his research’s focus. To avoid this, he could have used a transmittal letter. To illustrate, Hairston (1981) used sentences that included errors in the questionnaires and asked the participants to scale how the errors bothered them. She (Hairston, 1981) didn’t differentiate errors, but she explained the focus of her research in a letter that she phrased carefully. She (Hairston, 1981) explained that her focus was on what kinds of grammatical errors that were mostly significant in their everyday work related writing and provided them with guidelines about the questionnaire. In the same letter, she (Hairston, 1981) encouraged them to answer honestly and not to be threatened by her questions by thinking as if they were tested by an English professor. By having such a letter the researcher could have explained his attentions and prevented using boldfaced leading questions that threaten the validity of the questionnaire. This threat is known as exposure to the questions and lowers the validity of the results.
Conclusion
This paper attempted to look at Beason’s study and discuss three points that threatened the validity of the study. Subject selection, mortality, and exposure are believed to threaten the validity of Beason’s results. The ideas presented are driven from my understanding of Merten’s (2011) explanations. It seemed that the researcher failed to explain his intent by choosing business people’s reactions towards errors. The researcher did not provide specific reasons for his choice but indicated that the findings of the study would enable teachers and students to understand the importance of errors and how errors influence readers. Besides, the researcher chose to include a limited number of subjects that made the sampling inadequate. The smaller the sample the more the sampling error is a reason for not to depend on a small number of subjects. It was also noticed that the researcher relied on subjects from certain limited professions and regions to gather his data, which can be seen as a limitation. Those subjects do not represent the population. Dropout was another noticeable limitation that the researcher did not explain. He just indicated that those who withdrew from the study suggested replacements that met the criteria. He did not provide explanations about how he made sure that they met the criteria. He also did not provide details about how he made sure that this drop did not affect the research findings. Further, the wording of the questions is another limitation. Boldfaced errors stimulated subjects’ reaction to errors, which raises questions on the validity of his classifications. By taking Mertens’ (2010) points about threats to the research validity into the account, it seemed that Beason’s research might be invalidated by subject-selection, mortality, and exposure. The final question is raised here: what would have the results been if the researcher had a larger sample that is chosen randomly? Randomization and the larger number of sampling would increase the validity of the study.
This study inspired me to investigate academic journals’ editors’ reactions to second language scholars’ errors and supervisors’ reactions to second language graduates’ errors. By looking at editors’ and supervisors’ reactions to errors, the results will reveal vital information to second language scholars as well as graduates that will cause a change in their perceptions of effective writing.
References
Alreck, P. L. & Settle, R. (1995). The Survey Research Handbook. Boston, Massachusetts: Irwin McGraw-Hill.
Beason, L. (2001). Ethos and Error: How business people react to errors. College Composition and Communication. 53(1), 33-64.
Gray, D. E. (2004). Doing Research in the Real World. London: SAGE.
Hairston, M. (1981). Not All Errors Created Equal: Nonacademic Readers in the professions Respond to lapses in Usage. College English, 43(8), 794-806.
McMillan, J. H. & Schumacher, S. (1989). Research in Education A Conceptual Introduction (2nd ed.). Glenview, Illinois: Scott, Foresman and Company.
Mertens, D. M. (2010). Research and Evaluation in Education and Psychology (3rd ed.). Thousand Oaks, CA: SAGE.
Weisberg, H. F., Krosnick, J. A. & Bowen, B. D. (1996). An Introduction to Survey Research, Polling, and Data Analysis (3rd ed.). Thousand Oaks: SAGE Public.