Scenarios

QRP #5: Misrepresenting study importance in the proposal

• Scenario: John has been told by his Chair that in order to have a stronger tenure portfolio to be able to get promoted, he needs to apply for external funding and ideally get a grant. John has very limited experience writing up grant proposals. He sends a first draft to two senior colleagues to get some feedback, and they both advise him to emphasize the importance of the implications of the project. John isn’t clear on how to do so, as the project, while interesting to him, doesn’t seem to have clear implications for either teaching or research. As he really values the advice from his senior colleagues, and as he feels some pressure to write a strong one to help his tenure case, he adds some language that he feels exaggerates the impact and value of his proposal to society. He sends it back to his senior colleagues who both give it a thumbs up. He is hesitant to submit this revised version, as it does not feel like he is accurately representing the project. Should he submit it?

QRP #7: Not producing the promised project outcomes due to project mismanagement

• Scenario: Nina is working on a grant application with a team of researchers at different stages of their careers. As she is the primary investigator (PI), she’s in charge of drafting a budget. She reaches out to two of the team members who have a high success rate at getting grant funding to ask for advice on how much to apply for. Her draft budget is $110,000, as she is thinking that $100,000 would cover the salary costs, given everyone’s salaries and months devoted to the project, and that $10,000 would be enough to cover travel costs for conferences for the duration of the project. The team members tell her to add at least 20% to that, as, according to them, grant agencies often cut budgets. Nina is highly uncomfortable doing so, as she feels that the higher predicted costs would be unwarranted, but she doesn’t want to go against the advice of her team members. Should she take her colleagues’ advice?

QRP #10: Having an unnecessarily long/burdensome data collection for participants

• Scenario: Lin is conducting a study on intelligibility and comprehensibility of authentic audio materials in the listening classroom (Study 1). While designing the study, Lin realized that the target participant population overlapped with another research study on learner beliefs that they had wanted to conduct but had not yet started (Study 2). Knowing that participant recruitment can be challenging and time-consuming, Lin decided to include instruments from Study 2 as part of the data collection for Study 1 so that they could have additional data to analyze separately. When Lin piloted this data collection, they found that it took participants about an hour to complete the Study 1 procedures with an additional 20 minutes to complete the Study 2 instruments. Some pilot participants noted that the study felt long, while others found it interesting and did not have a problem with the amount of time it took. Should Lin go ahead with this plan for an expanded data collection?

QRP #11: Recruiting participants to join a study in a way that makes refusal difficult or uncomfortable

• Scenario: Fatma is taking a research methods course and is required to conduct a complete research project within a single semester. Fatma is ready to gather data for her project and has asked people in her department if it would be possible to visit their classes and recruit participants. Fatma has visited three classes already and things went mostly fine. She went to the final few minutes of class and read her IRB approved recruitment script. She then hung up a sheet with a QR code that students could use to sign up for the study as they left the classroom.  Today, Fatma visited Agustín’s classroom and followed the same routine as before. However, near the end of Fatma’s pitch, Agustín added a few words to help motivate the students. He mentioned how important research was, how students at this campus need to support each other, and that Fatma might not pass her class without the help of the students. In the first three classes that Fatma visited, around 1 in 4 students agreed to take part in the study. In Agustín’s class, nearly 3 out of 4 agreed to do so.

QRP #1: Removing whole items/cases knowingly/purposefully to obtain favorable results

• Scenario: Kai is conducting a study looking at changes in cultural understandings and attitudes held by first year language students. As part of this semester long project, Kai is conducting observations in four classrooms along with a repeated survey (4 times) that attempts to track changes in attitudes among the students. While analyzing the survey data, Kai noticed that four of the 63 participants appeared to not take the survey task seriously. These participants selected 1 or 6 (the anchor points) for all questions in each of the four surveys. In looking back at their observation notes, Kai realized that these four students were absent from the classes at higher rates than other students and did not engage with class discussion unless directly called upon. Prior to running any statistical tests, Kai decided to remove the data of these four students since they felt that the data was unreliable. Kai later re-ran the tests including the removed participants and found that including the participants changed the outcome of the statistical tests in ways that did not align with Kai's hypotheses.

QRP #4: Being ambiguous about whether an exploratory vs. confirmatory analysis was employed

• Scenario: Ren is interested in what makes students successful language learners. His previous research had shown that students whose background included having good language teachers in high school, living in an area where the second language was often used, having a desire to consume media in the L2, and seeing a future usage of the language resulted in participants rating their motivation to learn language higher than students who lacked more than one of those components. In trying to design a follow up study, Ren decided to send a survey to students in 400 level language courses and asked them about their past and future selves in terms of learning and using their language. The survey included items such as “Knowing how I will use my language in the future is important” or “Watching movies in my L2 is a waste of time”. The results of the survey showed almost exactly what Ren thought it would. Most of the students mentioned having positive language experiences prior to going to college and they all had plans for how they wanted to use the language in the future. Since the data was so robust, Ren put forward an argument about the need to strengthen language opportunities in high school and to include career readiness components in all levels of language courses to help motivate students.

QRP #5: HARKing (i.e., hypothesizing after results are known)

• Scenario: Finally, it was done. Noel had spent several months brainstorming a research idea, fought for weeks with the IRB, tried four different methods to recruit enough participants, and sent out hundreds of email reminders. And now, Noel was looking at his glorious spreadsheet full of information. Noel had originally wanted to investigate if the text color used for subtitles would have an effect on the amount of material comprehended by participants. He used five different font colors (the only ones available on his media player) and found that light hues (gold and orange) had significantly higher levels of comprehension than did dark hues (green, black, and purple). In his write up, Noel hypothesized that lighter hues were more likely to draw attention without interfering with the onscreen images. His null hypothesis was that there would be no difference between the two hues, which was rejected by the t-test he ran.

QRP #14: Interpreting statistical results inappropriately

• Scenario (Note: can also be used for Write-up/Dissemination #9): Julio is interested in the relationship between the correlation between two individual difference variables hypothesized to be linked to second language proficiency. He, therefore, administers an online survey to measure those variables along with a background questionnaire to gather demographic data on his sample of Chinese learners of English. Thanks to connections with scholars in China who kindly forward his invitation to participate, Julio collects a fairly large sample (N = 650). The correlations between his two variables of interest turns out to be very small (r = .03). Due to the large sample, however, the correlation is statistically significant. Julio highlights the statistical significance of this finding in the abstract and throughout the results and discussion, only mentioning once the observed correlation of .03.

QRP #7: Not sharing scripts used to analyze the results

• Scenario: Jean recently published a study that involved the important methodological component of creating a collocation/concordancing program in Python for corpus linguistic research. This program was capable of: (1) transforming an enormous corpus into a readable dataset that the program could process in a matter of seconds, (2) processing user-input for collocate searches, and (3) organizing output into an Excel document for the user. At the completion of her project, she was asked to share this Python script on a public platform so that other researchers could use it. As only an intermediate-level programmer, Samantha was hesitant to share the Python script because, in her view, the code was written rather crudely. She questioned whether publicly sharing the script would hurt her reputation in the larger research community as a programmer, and whether the script would actually be useful to other researchers in the field who were potentially more adept at programming. As she knew of a handful of researchers with programming skills, she reasoned that there were other researchers who could likely write a much more efficient, user-friendly program with the same capabilities. For these reasons, she opted not to share her Python script. Was this a reasonable decision to make?

QRP #9: Not attempting to publish results in a timely manner

• Scenario (Note: also can be used for Data Analysis/Interpretation, QRP #14): Julio is interested in the relationship between the correlation between two individual difference variables hypothesized to be linked to second language proficiency. He, therefore, administers an online survey to measure those variables along with a background questionnaire to gather demographic data on his sample of Chinese learners of English. Thanks to connections with scholars in China who kindly forward his invitation to participate, Julio collects a fairly large sample (N = 650). The correlations between his two variables of interest turns out to be very small (r = .03). Due to the large sample, however, the correlation is statistically significant. Julio highlights the statistical significance of this finding in the abstract and throughout the results and discussion, only mentioning once the observed correlation of .03.

QRP #11: Salami publication

• Scenario: Cynthia and Sam are a pair of graduate students who have spent the better part of a year collecting data for a study investigating the effect that the gender and age of the interlocutors in the TOEFL iBT spoken exam has on test scores. It’s time to start the write-up, and Cynthia and Sam must decide what content to include in their paper. They have two options: (1) they can write one paper that presents the results and analysis of the gender/scores relationship and the age/scores relationship, or (2) they can write two shorter papers, one focused on the relationship between age and test scores, and one focused on gender and test scores. If they choose option 1, they will have a difficult decision to make as to which student will be the first author for the project. If they choose option 2, each of the graduate students will have a first authored paper. While they feel that they could easily fit the results and analysis into a single paper, they have put equal work into the project over the course of the past year, and would both benefit greatly from a first authored paper as they enter the academic job market. Which option should they choose? (adapted from scenario provided by the Center for Taiwan Academic Research Ethics Education: https://ethics.moe.edu.tw/resource/case/detail/32/)

QRP #14: Employing excessive self-citation

• Scenario: How many self-citations are too many? Imagine your co-author sends you the most recent version of a paper you are both working on. Your co-author has a fair bit of experience in the topic of the paper. However, it seems that he has inserted citations of his own work in places where it doesn’t seem especially relevant and/or where reference to another work might be more appropriate. Would you bring this up with him? How so?

QRP #17: Inappropriately attributing author roles when listed in publications

• Scenario: Misha is a first-year assistant professor. She’s leading a research project where she’s working with two other researchers: an MA student and a senior scholar. The original plan was for all three to be equally involved in the project, but as time went on, it became clear that most of the work has been done by the MA student and by Misha. The senior scholar, Dr. Marke, has offered to have her name taken off the paper to reflect this, but Misha is hesitant to do so, as she worries that it will be more difficult to publish papers from the project without the name recognition of Dr. Marke. Also, Dr. Marke has been helpful in an advisory role. When it comes time to submit the first paper from the project, their target journal asks the team to detail the contributions of each of the team members. None of the categories to choose from cover the kind of support Dr. Marke has provided, so unless Misha adds something like ‘taking an active role in the write-up’ (the scholar did rewrite one of the paragraphs, she reasons), Dr. Marke’s box will be empty. Should Misha bend the rules a little to keep Dr. Marke listed as an author?

QRP #18: Inappropriate ordering of authors

• Scenario: Harriet and Gloria recently completed a study for a graduate seminar investigating language variation of users posting in political activist groups on different social media platforms. They put in a year of hard work completing the project, which required the extensive process of building a large stratified, representative corpus of 5,000 social media posts, writing various concordancing programs to analyze their data, interpreting the findings, and writing up the manuscript. As they began to look for a journal that was a good fit for their study, Gloria’s mentor, a well-respected researcher in the field, contacted Gloria and Harriet to inform them that he’d been asked to submit a study to a high impact journal for a special edition on a related topic. The professor asked if the three of them could co-author the manuscript for submission. Based on this seemingly great opportunity for the students to publish a prestigious journal, and based somewhat on the pressure felt by Gloria not to turn down the offer from her mentor, Harriet and Gloria agreed. However, following this conversation, the professor, without consulting the students, made minor edits to the methods section of the paper, placed his name as the first author, and submitted the manuscript to the journal. Harriet and Gloria both privately felt that, due to their independent conceptualization of the study, compilation of the corpus, analysis of the data, and write-up of the manuscript, they should have been first and second author on the paper. Due to the power dynamic in this relationship, and the fear of losing the opportunity to publish, they are hesitant to approach Gloria’s mentor. What would you advise them to do?

QRP #22: Irresponsibly co-authoring

• Scenario: Maisy was a research assistant for Dr. Glass and was tasked with transcribing about 50 hours of focus group data. The project had something to do with how L2 French learners talked about trees and outdoor spaces. Honestly, Maisy wasn’t all that sure what the goal of the project was but she did hear a lot about trees. She spent weeks transcribing the data in the format that Dr. Glass had taught her. They had weekly meetings to talk about the transcriptions and Dr. Glass would often ask Maisy if she had noticed anything funny or interesting in the data. Since Maisy struggled to figure out who was talking and when, she had started to notice various cues the participants used to signal that they wanted a chance to speak. Maisy also mentioned the instances when the participants used incorrect French terms as every instance required Maisy to relisten to the same clip numerous times. Maisy kept working on creating the transcript throughout the semester and was surprised several months later when Dr. Glass mentioned that she had just submitted their manuscript. She was proud that Maisy was able to not only transcribe data but also helped with the data analysis.