The full collection of QRP videos can be accessed through our YouTube Channel:
Section 1: Funding
Cherry-picking samples/data/results in favor of the funder
Video: Cherry-picking to favor the funder
Description: Cherry-picking is when a person purposefully picks evidence that supports their claim while omitting data that would potentially nullify their argument. Cherry-picked data will often seem to validate the claim made by an author (they would be bad cherry-pickers otherwise). When it comes to project funders, they might only provide limited access to a dataset or restrict the type of data that could be discussed. Scholars might also only point to positive data to keep their funders happy and willing to provide them with more money. Science and society are the losers here since low quality, unrepresentative data is being published skewing our understanding of how a phenomenon works.
2. Choosing a topic on the grounds that the funder might expect the study to portray them in a positive light
Video: Choosing a topic to please funder
Description: Imagine the publisher of a prominent test or of a textbook series put out a call for grant proposals that involve their products. This is a fairly common situation, actually. Upon seeing the call, a researcher might be tempted to submit a proposal that would likely cast the publisher/company in a positive light. At first glance, this might seem objectively wrong. However, there is certainly a fair bit of gray area here, too. The findings of the study might also portray the company positively simply because of the nature of their products. The key here is to let theoretical and/or practical issues from the literature drive the questions that we ask rather than how we think the findings of our proposed studies might present the funder.
3. Not reporting a conflict of interest (financial/otherwise)
Video: Conflicts of Interest
Description: It's often the case that we have personal and/or professional links to a particular strand of research or even to a commercial product such as a test that generates revenue. These connections can introduce the potential for bias in our research and, in many cases, cannot always be avoided. The important thing is to simply declare any potential conflicts of interest in a grant proposal. That way the reviewers and grant funding agency is aware in advance of funding the project.
4. Misrepresenting researcher qualification/experience in the proposal
Video: Misrepresenting researcher qualification
Description: Grant proposals almost always require a section that describes the qualifications of the team involved. This might include their areas of expertise, publication record, and experience with other funded projects. This information is then used, along with the proposal to evaluate the likely success of the project. In order to provide the grant-funding agency with an accurate view of the team and its potential, it is imperative to describe such qualifications honestly and without embellishment.
5. Misrepresenting study importance in the proposal (e.g., exaggeration of impact and value of proposal to society)
Video: Misrepresenting study importance
Description: Many grant applications will require some kind of impact statement, which asks the author to describe what will be gained from the project in terms of knowledge, outputs, products, and so forth. Some researchers exaggerate in this section, which is not appropriate. In order to avoid stepping into QRP territory, it's critical that we take a reasoned, balanced view on the potential contribution of the studies we seek funding for.
6. Over budgeting (i.e., knowingly asking for more money than is actually needed)
Video: Over budgeting
Description: This QRP should be self-explanatory. The general idea, though, is that we should only request funds that we actually expect to need and use. It can be tempting to pad the budget a bit in order to allow for funding for other projects or, for example, for equipment that isn't strictly needed for the project at hand. This is not appropriate. Furthermore, over-budgeting can also lead to problems related to getting approval to use unspent funds for purposes other than those identified in the proposal.
7. Not producing the promised project outcomes due to project mismanagement (e.g., producing fewer articles than promised)
Video: Not producing promised project outcomes
Description: We of course want to write a grant proposal that is as competitive as possible to the funding agency. An important part of writing a successful proposal is assuring the funder that the money will be put to good use, that is, that the study will yield results that are shared appropriately and widely. While not producing the promised project outcomes could be due to either over-promising or under-performing, it can be considered a 'breach of contract' with the funding agency, and thus a QRP.
8. Using funds for a purpose other than what was stated in the proposal (e.g., for a research assistant instead of for participant-related expenses)
Video: Using funds for different purpose
Description: Grant proposals almost always include a budget. This budget is itemized to show exactly how all the requested funds will be used. For a budget of $8,000, for example, you might allocate $5000 to paying an hourly research assistant, $2000 for compensating participants, and $1000 for purchasing new software you need for the project. It is generally not appropriate to re-allocate these funds for other purposes without approval from the funder. An example of this would be deciding to use some of the money from the $8000 budget to cover conference-related expenses, which were not part of the budget that the funder approved.
9. Misrepresenting literature in a grant proposal (e.g., over-emphasizing previous research that supports the proposal and/or ignoring conflicting evidence)
Video: Misrepresenting literature in a proposal
Description: It often happens that researchers find themselves looking for a reference to support a point they want to make, rather than (more appropriately) basing one's ideas on the theoretical or empirical evidence in the literature. Along these same lines, researchers often highlight the studies that support or point to the need for the research they're hoping to get funding for. This problem may be especially prevalent in the absence of meta-analytic evidence to show overall findings across a given body of work.
10. Making changes in context after proposal submitted (e.g., proposing to carry out funded work in one context/with one population but then actually carrying it out elsewhere/with another population)
Video: Making changes after a proposal is submitted
Description: Any and all major changes made the design of a funded study need to be reported to and cleared with a funder. For example, if you got funding to conduct a study of classroom-based learners of English as a second language, but you had to then change the sample to a smaller set of online learners of English as a foreign language, you would need to ensure that the funder approved this. Very minor changes are sometimes acceptable without approval. As a general rule, though, it's better to check than to assume.
11. Not disclosing impacts that funder directly had on research decisions (e.g., using particular datasets, selection of published outcomes)
Video: Not disclosing funder impact
Description: Grant-funding agencies can sometimes influence a study by encouraging the use of certain instruments or by requesting that the authors publish some of the results instead of others. Sometimes funders even write into grant contracts that they have the right to do so. Such influence is not necessarily inappropriate but it is necessary to declare when this happens in a written report.
Section II: Data Collection and Design
1. Selecting variables out of convenience and/or familiarity when more theoretically grounded variables are available
Video: Selecting variables out of convenience
Description: Some researchers seem to include variables in their designs not because of any strong theoretical or practical reason but, rather, because those variables are familiar or convenient to collect. I've been seeing this lately in SLA with working memory, for example. A more appropriate approach is to think through our variables more thoroughly and to ensure that we're including them in our design because of a hypothesized relation. It's also OK to include exploratory variables in a design but these need to be clearly labeled as such.
Video: Choosing a design that provides convenient access to data
Description: When we decide on a study design, including an instrument type, we want to make sure that we measure what we set out to measure, that is, that we have high validity. It is likely more time-consuming and otherwise costly to prioritize validity over convenience. That is, we're likely tempted to go with an instrument type that provides easy or convenient access to data - or we may even have to for different reasons. So as a field, we'd want to think about when (if ever) it would be more all right to sacrifice validity to some degree in the interest of completing a study.
Video: Data Collection and Design, QRP #3: Defaulting to convention
Description: As creatures (and researchers) of habit, we often choose research designs, instruments, analyses and so forth simply because they are familiar, and not because they necessarily provide the strongest evidence or because they have a strong validity argument behind them, for example. This is a pattern that can lead to inefficiencies, noisy data, and flaws in our results. When setting up a study, it's imperative to question whether we are choosing all aspects of our design and analysis due to the high validity evidence they provide as opposed to defaulting to convention or convenience.
Video: Employing instruments without a strong validity argument
Description: We often have many choices about which measures we use for a particular variable. In deciding which measure to use, researchers sometimes fail to consider whether or not the different choices have a strong validity argument behind them, relying more on other factors such as convenience or simply whether others in the domain have used them. Here's a tip: If you find that measures in your area do not have strong validity arguments behind them, try doing some of the validation work before moving ahead with more studies on unvalidated measures.
Video: Not being transparent about data collection
Description: In a typical study, we make a great deal of decisions in the data collection phase. We may for example decide to exclude certain populations from our sampling frame or choose to make changes to our instrument and therefore start over -- or we may expand our sample after a while if we don't get enough data. Decisions made at this stage are important for readers of our study to know about, and failing to report these steps in a transparent manner could be an issue.
Video: Biasing design for favorable outcomes
Description: Many times we, as researchers, go into running studies with a preference about what we find. We might hope, for example, to find an advantage for one particular treatment over another, or for one variable A to be a stronger predictor of Y than variable B. This is, to some extent, natural and unavoidable. What lands us in the realm of 'questionable', however, is when these preferences lead us to make design-related decisions that are more likely to yield our preferred outcomes. We need to avoid such choices whenever possible. When we're not sure about our objectivity, we might consider consulting with others who have a different point of view and/or disclosing our preferences or biases going into the study.
Video: Data Collection and Design, QRP #7: Not reporting effect of decisions on study outcomes
Description: The research process holds a substantial number of decision points that you will have to make along the way that will affect the outcome of your results. For example, you'll need to make the larger decisions of what type of methodology to select and what type of variables to include, but you'll also need to make smaller decisions that arise unexpectedly throughout the process - for example, how do you handle potential outliers? You may end up having to make these larger decisions based on outside factors that are out of your control. This includes things such as accessibility to the population that you are hoping to collect data from, and journal submission guidelines.
When it comes to reporting our decisions, it’s important to consider which will be the most meaningful and useful to the audience. For example, when in the semester you invited your participants will likely be less meaningful information than who you invited, and why. When making these decisions, it is helpful to consider factors such as audience and journal submission guidelines. Who is the target audience of this paper - practitioners or researchers? What is the word limit for the journal, and how much space will you have to report such decisions? As researchers, it is crucial that we have good reasons for our decisions. Whether or not other researchers agree with your decisions, this information can help inform future studies and are important for replication research. This also helps the research community have more confidence in what you did.
Video: Leaving out known moderator variables without explanation
Description: For a variety of practical reasons, we certainly can't include every variable that matters in every single study. We have to make choices about which variables to include. However, there are sometimes well-known and well-studied covariates or moderating variables that really should be examined and which, when left out, can be considered questionable. An example of this might be a study that looks at reading comprehension only as a function of grammatical knowledge without considering vocabulary knowledge, when both of these variables are known to be highly correlated with reading ability. Including only one of these variables in the study will almost necessarily yield results showing that variable to have an outsized influence on the dependent variable, in this case, reading comprehension.
Video: Fishing for results
Description: We often collect more data than we actually need to address our RQs or that we have a strong theoretical justification for. This is inefficient, but not necessarily unethical or even questionable. We venture into the realm of questionable when we start to analyze data from those variables. For example, it can be tempting to start running correlations for variables not because we hypothesized that they are relevant, but simply because we have the data. This can lead to fishing for results and HARKing (i.e., hypothesizing after results are known).
Video: Unnecessarily burdensome data collection
Description: Data collection is time consuming and can even be expensive. We therefore always want to make sure that we get all the relevant information from our study participants. At times, in order for us to get all the relevant information, we would ideally have participants take part in very time consuming and/or mentally taxing activities. While finding a balance between our research needs and the participants' well-being is not always easy, we would want to make sure that we don't take up too much of our participants' time or energy.
Video: Recruiting in ways that makes refusal uncomfortable
Description: We of course want to try our best to up our chances of getting as many participants as we need for a study. If we fear that we may not reach that number, we may be tempted to try to nudge participants to sign up by making refusal difficult or uncomfortable. We might for example talk about what the consequences might be if we fail to get enough participants (e.g., how students may not be able to graduate). This may well be true and may up our numbers, but where do we draw the line for what recruitment strategies are considered acceptable?
Section III: Data Analysis and Interpretation
1. Removing whole items/cases knowingly/purposefully to obtain favorable results
Video: Removing items to obtain desired results
Description: There are cases when removing items or cases might be justified. For example, items may have proven too difficult to understand for participants and will thus not yield results with high validity, or we may wish to remove cases if we believe that the participant did not take the test/survey seriously. However, we should avoid removing anything from the analysis just to get more favorable results, or results that align with our hypotheses.
2. Using unjustified methods of handling outliers
Video: Unjustified handling of outliers
Description: We have many options for how to handle outlying data points in a given set of data. The same applies to even defining and identifying such data points. Consequently, researchers are often tempted to choose a definition and approach to handling those data points that produces a more favorable outcome. For example, by removing (or not removing) some data points that the researcher considers to be outliers, the analysis may then cross the threshold for statistical significance. Thus, it is imperative to decide how outliers will be identified and dealt with prior to collecting and analyzing a given data set.
3. Not being transparent with regard to the reporting on what steps were taken for data cleaning (e.g., removing cases/items without a stated criterion or justification for doing so)
Video: Not being transparent about data cleaning
Description: Quantitative linguists make a lot of use of language corpora. One’s output files can be enormous, and one often needs to clean the masses of data by applying various screening strategies. It is important to report on these strategies in detail in one’s presentations and publications. One should explain carefully on what grounds examples have been included in the data, and on what grounds they have been excluded from the analyses. Failing to do so would make it difficult for another party to use the results for comparisons. Not being transparent about steps taken in data cleaning can be considered a questionable research practice and something we should not do.
4. Being ambiguous about whether an exploratory vs. confirmatory analysis was employed
Video: Exploratory vs. confirmatory analysis
Description: We use exploratory designs to learn more about a topic that hasn't been researched much. Such studies are hypothesis generating. By contrast, confirmatory studies are hypothesis-testing; that is, we have hypotheses based on theory and previous research that we wish to put to the test on new data, for a new population, or in a new context more broadly. Most confirmatory studies involve some elements of exploration, but we always want to be transparent about what parts of the analysis were and weren't to help future studies.
5. HARKing (i.e., hypothesizing after results are known)
Video: HARKing
Description: HARKing is hypothesizing after you have already analyzed your data and know your results. You want to avoid this because it can lead to issues where you are not really confirming the data that you found, because you're using the data to make up hypotheses. A stronger way to do research is to make hypotheses, and then do tests to see if those hypotheses are true.
6. Cherry-picking data to analyze
Video: Cherry-picking data to analyze
Description: Cherry picking data to analyze can be more or less problematic in different contexts. For example, in many study designs, we are unable to present all the findings - we have to be selective, and present the most relevant findings. In corpus linguistics studies, we tend to get more results than could ever fit in a research article; in more qualitatively oriented research designs, we similarly have to choose what to focus on. Contexts where it could be more problematic include when we choose to disregard some findings to make the results better line up with our argument or hypotheses.
7. Choosing a method of analysis that will likely lead to favorable outcome (e.g., in favor of the researcher’s hypothesis)
Video: Selecting method of analysis to obtain desired results
Description: The same data set can often be analyzed in a variety of different ways to address the same research question(s). Sometimes different analyses will yield essentially the same outcome, but this is not always the case. This situation, which highlights the combined art and science of data analysis, can lead researchers to choose an approach that yields a more favorable outcome, which is problematic and certainly questionable as well.
8. p-hacking (i.e., running analyses in a manner that produces statistical significance)
Video: P-hacking
Description: P-hacking occurs when a researcher continues analyzing their data in different ways until they observe statistically significant findings. This could occur by means of adding new background variables to the model, for example, as new covariates, or by breaking up the sample using different grouping variables. This practice often occurs as a result of researchers' (mistaken) belief that their findings are only important or meaningful if they obtain statistical significance.
9. Ignoring alternate explanations of data
Video: Alternate explanations of data
Description: If we only look at one possible explanation of our data, then we can usually validate what we have found in a number of different ways. If we take a second closer look, however, we might find a different explanation. It is good practice to test your hypothesis to make sure that that explanation has scientific validity. When you are doing your data collection, you want to think of as many possibilities as you can that could take down your hypothesis. If you find that you can't eliminate your hypothesis by any other alternate explanation, and re-tests show that it is the most valid one, you can have increasing confidence that it is the best possible explanation of your data.
10. Using unjustified methods of handling missing data (e.g., imputing / inserting values for missing data that are not justified and/or that are more likely to yield desired outcomes)
Video: Unjustified methods of handling data
Description: Missing data is a reality that we can face in quantitative research for a variety of reasons, such as incomplete survey responses, participant attrition, or technology recording errors. Missing data can impact the power of the statistical model used to analyze the data, so this becomes more critical in research domains where large sample sizes are not the norm and researchers feel that they cannot afford to remove entire cases if one data point is missing. One way of preserving all cases in the data set is to impute or insert values for any missing data points. Yet this can be done using a variety of techniques which may in turn impact the results of our study. Similar to the way we manage outliers, our handling of missing data should be done in a principled and transparent manner. As we encounter missing data, we may want to ask ourselves questions like: What justification do we have for selecting a particular data handling method? How does this method impact our overall results compared to other possible methods for handling missing data? Are we being transparent in reporting the decisions we have made around handling missing data, including their potential impact on our study findings?
11. Categorizing continuous variables without sufficient justification
Video: Categorizing continuous variables without justification
Description: Statistical techniques such as ANOVA can require categorical variables rather than continuous variables. Therefore, if you have a continuous variable and you really want to use ANOVA, you'd presumably turning this variable into a categorical one, by grouping the data points into bands (e.g., 0-4, 5-10, and so on). This is problematic in that we are losing a lot of information and nuance that way - just to be able to use a statistical test. Ideally, we'd want to collect continuous data when possible and then use them as such in our models.
12. Using too many statistics tests without correction (e.g., Bonferroni)
Video: Too many statistical tests without correction
Description: This is actually one of those concepts in statistics that took me quite a while to fully appreciate in different ways and I’ll set up an example of why it took me a while to understand this. Let's imagine a scenario in which I have done 20 different independent samples t-tests. The first 19 of them had results that had a higher p-value than .05, which is what we set as our alpha level. And so, we say those tests were non-significant. I then run a 20th test that actually has a significant p-value, which for us is below .05. And just for sake of argument, let's imagine it has a medium effect size. I found something here, right? The data is showing that there is some sort of a score or relationship that we want to investigate. But the problem here is that I've run 19 other tests and I've added a 20th, for 20 total tests. And so you might be thinking, or at least I was always thinking, “why would it matter?” If I only ran one test right here, or that 20th test by itself, the p-value wouldn't change. The effect size wouldn't change. The confidence intervals wouldn't change. Nothing should change as long as we don't manipulate the data. And so why would it matter if we run it 1 out of 1 times or 1 out of 20? And the problem comes down to confidence, right? If we are confident in our results, that we decrease our number of research degrees of freedom and we say “no, we think this is going to be the correct answer.” The chances that there would be higher errors are much more minimal. The fact that we've run 20 tests, what it basically is doing is increasing our signal to noise ratio. When you're at 20 tests, you might be thinking to yourself. “Yeah, that's not too many”, but imagine you did this at a much larger scale. Let's say that you ran 400 tests, or 1000 tests, and you got one out of 1000 tests to be significant. Again, is it because there's a true and natural reason for why that one test was significant, or was it just sort of a fluke? The numbers might have worked out that way. Maybe we have the right sample size, we have the right test conditions, all that kind of stuff. Because again we wouldn’t expect it to be the case that one out of 100 or one out of 1000 tests is significant. And so, when you're running multiple tests you want to use some sort of correction. That is going to make it much harder to get over, and typically our corrections are having a decreased alpha level, so maybe instead of setting a .05 we set at .001 alpha level that we feel more confident in. Or again, we do fewer tests because that again increases our confidence in the tests that we did run. And so, for me this is always a concept that didn't make a lot of sense until I started thinking about it in that fashion, right? That we're doing too much. And inside of that too much, there's just a better chance we'll get a hit and again hit being a positive result than if we did a much smaller size. So, I hope that helps explain this a little bit better.
13. Using incorrect statistical methods (e.g., tests that are not appropriate for the type of data being analyzed)
Video: Inappropriate statistical methods
Description: Quantitative research often involves application of statistical techniques. As very few of us are trained statisticians, knowing everything we need to know about a technique and how it should (not) be applied is a big ask. However, if we are to be able to trust the results we get from statistical techniques, we need to use them correctly. Questions of whose responsibility it is that the analyses are correct are tricky: are we responsible for checking a co-author's statistical analyses? Is it OK to claim ignorance (i.e., that we did not know that we weren't using the statistical techniques correctly)?
14. Interpreting statistical results inappropriately (e.g., claiming equivalence between groups based on a non-statistically significant difference; undue extrapolation)
Video: Inappropriate interpretation of statistical results
Description: When we use statistical tests, we not only want to make sure they're appropriate for our data (by checking assumptions, etc.), but we also want to make sure we interpret them correctly. For example, we cannot claim equivalence between groups just because we don't find a statistically significant difference between them; all we can say is that we do not have enough evidence to reject the null hypothesis. It is our responsibility as researchers to learn enough about a method such that we are able to interpret the results correctly.
Section IV: Write-up and Dissemination
1. Failing to refer to relevant work by other authors
Video: Failing to refer to relevant work by others
Description: Failing to refer to relevant work by other authors is bad practice, and a potential QRP. Few people would disagree with the statement that we should do our best to find and read relevant studies, and then make sure to refer to them, where relevant. Failing to do so for competitive reasons, or because you don't agree with the author's perspective may give readers the impression that that author's work is not relevant, which may reflect poorly on both you and the other author.
2. Not providing sufficient description of the data analyses or other procedures
Video: Not providing sufficient description of data analyses
Description: Is failing to describe one’s data analyses in sufficient detail a questionable research practice? Well, superficial descriptions of data analyses leave many questions open to a reader or an audience and usually make it difficult to assess the results and the value of the study. One can think of many reasons why data analyses are not reported on accurately. Some things may seem so self-evident to the researcher that one doesn’t come to think of having to spell them out explicitly. Or then there may be a length limit to the paper or a conference talk making it necessary to cut down the details. Be that as it may, we should not be happy with insufficient descriptions of data analyses. They are an example of questionable research practices, and something we should not promote.
3. Not providing sufficient description of the data and the results (e.g., exact p-values, SD)
Video: Not providing sufficient description of data and results
Description: Quantitative studies tend to produce substantial amounts of results in the form of tables, graphs and other numerical information. It is customary to give basic numerical information in one’s publications. Failing to describe the data used for the analyses in sufficient detail, and not to report basic numerical information makes it difficult for readers to grasp the dimensions of a study. For instance, only reporting one’s results in terms of percentages, without giving the corresponding raw figures, usually leaves the reader in the blue about the scope of the study and the value of the results obtained. Indeed, not providing sufficient description of the data and the results is a questionable research practice and something we should not do.
4. Not reporting or publishing results because they are not statistically significant (i.e., the ‘file drawer’ issue)
Video: The 'file drawer' issue
Description: Our field places a lot of emphasis on statistical significance, which results in the tendency to either not report results that are statistically non-significant, or to fail to publish these studies all together. This is known as the file-drawer problem. One reason that this happens is that journals are far less likely to accept studies with statistically non-significant results. Because publishing is important for career advancement, there’s little incentive to attempt to publish statistically non-significant results when the researcher can expect for the study to be rejected. It’s important to remember that statistically non-significant results that come from a methodologically sound study are still valid and help to build our cumulative knowledge. If you fail to report results that are statistically non-significant, you may be withholding important scientific findings that will help to move the field forward.
5. Employing selective reporting of results/instruments
Video: Selective reporting of results
Description: Wouldn’t you wish to have your study confirm the hypotheses you have presented when launching the analyses? It may be tempting to resort to selective reporting of results to achieve this goal. Or then one might opt for analytical instruments that would enhance a certain outcome of the study to the disadvantage of others. However, reporting on the results only selectively so as to be able to draw conclusions to one desired direction is something we need to consider a questionable research practice.
6. Not sharing data when allowable
Video: Not sharing data
Description: When we conduct a study, it’s important for us to share our data with the research community if we’re able to. Sometimes it isn’t possible to share your data if confidentiality or copyright are at play, but if not, we should be putting our data out there for others to access. There are a few reasons people may be reluctant to share their data: perhaps you put lots of hard work and time into the collection of the data and you feel some level of ownership over it. Maybe you are concerned that another researcher may use the data you collected to publish a study that you had intended to conduct yourself. Regardless of your concerns, there are several reasons that data sharing is important. When you make your data accessible, you may be saving someone else a lot of time that they would have spent collecting a dataset that already exists. Sharing data also helps with accountability. You want people to trust you are a researcher, and the results that you publish, and making data accessible is a good way to do that. Finally, data sharing is important for the sake of collaboration and good-will in the field.
7. Not sharing scripts used to analyze the results
Video: Not sharing scripts used to analyze results
Description: Scripts take time to write and may be messier than we would like, so some may be a bit hesitant to share them. However, making sure to share our scripts increases transparency (shows researchers what we actually did in our study) and helps future researchers carry out replication studies and build on our work.
8. Not sharing instruments/coding schemes
Video: Not sharing instruments/coding schemes
Description: There are several reasons why we would want to share instruments and coding schemes. For example, in sharing what we did in a study, we are transparent about all the decisions made, thus enabling others to scrutinize our results and learn from what we did and didn't do. Also, sharing our instruments and coding schemes enables other researchers to use them, thus enabling replication studies and cumulative knowledge building in the field, which makes for results that are more easily comparable. So failing to do so could be problematic for the field in terms of transparency and community-building of research.
9. Not attempting to publish results in a timely manner
Video: Not attempting to publish in a timely manner
Description: This QRP may seem less problematic at first glance as it seems that it would only affect ourselves. If we collect data and do not get around to publishing the study for a year or two, it could negatively affect our chances for tenure promotion, our qualification for a job, or our ability to present in conferences. While it may seem that others will not be negatively affected by this QRP, there are in fact other effects that we don’t always think about. First, if you are working with others on the project, this QRP has the potential to negatively affect your co-authors in the same way that it would affect you. If you are not completing the parts of the paper that you are responsible for, and in doing so, holding up the project, they too may not be able to apply for tenure or have a good shot at job promotion. You also have your own participants to think about. If your study stands to benefit your participants and you do not publish the paper for others to read, you may be impeding the ability for your participants to benefit from the research that they participated in. This is important to consider when you decide to take on new projects and plan your research agenda. It’s a good idea to consider which projects you will have time to complete so that these projects do not sit and collect dust.
10. Presenting misleading figures of data (e.g., displaying a truncated entire y-axis)
Video: Write-up/Dissemination, QRP #10: Misleading data visualization
Description: Data visualization is a key part of how we share our research findings. As we decide how to visualize our data, we need to be mindful of how readers might interpret our data figures. Let’s take a fairly standard design for researching the effects of instruction on second language learning. Imagine a study where one group of learners received an experimental type of teaching and the other group received a traditional type of teaching. The effects of these two teaching approaches were measured by examining learners’ changes in test performance before and after the intervention. Upon analysis, the results showed that the experimental group’s average scores increased from 74% on the pre-test to 77% on the post-test, while the comparison group’s average scores went from 74% to 75%. One way we could present these findings is using a line graph, with time on the x-axis (i.e., pre and post) and average score percentage on the y-axis. For percentage data, we might expect the y-axis to range from 0 to 100. However, this might make it harder to see the changes between the pre- and post-test scores, or we may not feel that this highlights the differences we want our audience to note between the two groups. So we might be tempted to present a graph with the y-axis ranging only from 70% to 80%, which makes the difference between the pre- and post-test scores for the two groups really noticeable. By zooming in like this, however, we are removing important context, namely that the two groups only improved by 3% and 1% out of 100% and not out of 10%. While our y-axis may be labeled accurately, a casual reader might glance at a graph of only 10% of the possible data range and get the impression that our findings were more substantial than they actually were. So as we prepare our figures, it is helpful to ask ourselves how a reader might reasonably interpret the figure and whether misinterpretation is likely.
11. Salami publication (e.g., dividing up the results of a single study into multiple manuscripts in order to publish more)
Video: Salami Publication
Description: We tend to want to have a smaller number of research questions so that we can say more about each one. The problem with salami publications is that you are trying to cut too many pieces out of your dataset, so your questions start to become less meaningful. It's good to use as much information from your dataset as you can and still write a good paper that is comprehensible to your reader. Having a paper that you cut into smaller bits is fine if you can't talk about your whole dataset, but you don't want to cut it down so far that now you are talking about way too detailed of a project. This becomes a questionable practice when it is done for the sake of increasing the number of publications on a CV without actually conducting additional research.
12. Not managing time well for one’s own conference presentations, resulting in less time for other presenters, limiting discussion, and impacting others at the conference
Video: Irresponsible time management at conferences
Description: At academic conferences, we typically view all presentations as having the same level of impact on the field, and thus, there is the expectation that everyone will have an equal amount of time to give their presentation. For this reason, it is important to prepare for a conference presentation in such a way that you will be able to deliver it and take questions within your allotted time. Going over your allotted time will negatively impact the presenter that follows you, potentially causing them to rush through their slides, cut out important parts of their presentation, and harm their overall experience at the conference. This may also negatively impact the audience by causing them to be late for the next presentation they wish to attend. Occasionally, we may go a minute or two over our presentation time, but if we do, we should be prepared to end our Q&A session at the proper time and cede the floor to the next presenter.
13. Presenting same presentation at multiple conferences
Video: Same presentation, multiple conferences
Description: Sometimes there are good justifications to give the same presentation at multiple conferences. It's important to consider who your audience is. If the conference is research-focused, you can expect the attendees to be primarily interested in that aspect of the project, meaning that those interested in pedagogy are less likely to attend. This means that if your study carries important pedagogical implications, those who would be most interested in this information, and most likely to use it, may not ever hear the presentation. This works in reverse - if you are presenting your study at a practitioner-based conference, those who are primarily interested in the research aspect of the project (i.e., methodology, data analysis) are unlikely to attend and will not have access to the information that you are sharing. The location of the conference often has a similar effect on the audience. Local or regional conferences are generally easier to attend than international conferences as they often mean less travel time and lower cost. Conferences that are farther away and more costly to attend will likely attract fewer of the scholars that you intend to present to. In this case, it may be appropriate to give the presentation at a regional conference even if you've already given it at an international one, and vice versa. If you can make an argument that the audience you will be presenting to is vastly different, this practice may not be so questionable. However, if you expect the audiences at the conferences to be highly similar, this looks more like CV padding. In this case, the practice is more questionable.
14. Employing excessive self-citation
Video: Excessive self-citation
Description: What is considered ‘excessive’ can be hard to pin down. Generally, self-citation that appears to be excessive occurs when a researcher is one of a highly limited number, or potentially the only researcher, who conducts a particular type of niche research that is relevant to the paper. If this is truly the case, then this QRP may not be so questionable as it is also not appropriate to purposefully limit self-citation. This becomes problematic when the author does so in order to pad a CV or increase name recognition and readership. In addition to this, if you are failing to cite relevant studies by other authors, this may also be unethical as you are indirectly impeding another researcher’s advancement in the field. Even if you believe that you are one of the few people who conducts your type of research, be certain to do some digging in the literature in order to be certain that you are not failing to cite other authors. Be certain to consider whether a citation is truly necessary based on the language you used, and whether the citation is absolutely relevant to the study.
15. Intentionally omitting relevant work because it does not align with one’s theoretical or methodological approach
Video: Omitting work that does not align with one's approach
Description: We all tend to work within some theoretical or methodological framework. It is therefore not inconceivable that we will disagree with statements or approaches from a competing framework. It gets a little tricky, however, when it comes to the background section of an article - to what extent is it OK to omit relevant work because it doesn't align with your own approach? One could maybe argue that those papers aren't as relevant as ones produced in your framework, but we of course do not want to end up in an echo chamber where we only cite people we agree with.
16. Inappropriately including or excluding authors
Video: Including or excluding authors
Description: Including authors on a paper that have not adequately contributed is known as ‘gifted authorship'. This is particularly problematic when the individual is included in order to reward the individual in some way, return a favor, or pad their CV. This sometimes also happens when lesser-known researchers include a well-regarded, senior scholar on a paper in order to increase readership, citations, or recognition of their own names. This is problematic not just because it gives the impression that the individual has contributed research to the field that they in fact haven’t; this also means that the individual then becomes responsible for the paper, regardless of the quality of the study. Inappropriately excluding authors is also problematic, as the researcher is not getting credit for the work that they have done. It can be really tricky to tease out who deserves authorship, in particular when it comes to the initial ‘idea’ for the study. If the individual who conceptualized the study did not participate in any way in the research process or write-up following the contribution of the idea, including them may feel somewhat like gifted authorship, while leaving them off the paper may feel like ‘stealing’ an idea. It’s important to remember to have these conversations early in the research process, and often.
17. Inappropriately attributing author roles when listed in publications
Video: Inappropriately attributing author roles in publications
Description: Clearly delineating authorship is an important aspect of the publication process, especially in fields where collaborative research and publication are more common. Part of this process is accurately attributing the roles that each author played in the project. A growing number of journals now provide opportunities for authors to report not only that an individual was involved in a project but also in what ways the author was involved. This is a matter of transparency, and it allows readers who may have questions about certain aspects of a study to know which author(s) would best be able to answer these questions. For example, if only one author from a team of three conducted the data analysis, this role should not be attributed to all team members as they may not be equally prepared to explain the nuances of the statistical tests used. This level of transparency can offer a more realistic picture of what collaboration looks like in our research while also serving as a check that everyone listed as an author contributed to the publication in a meaningful way. Publications serve as a record of our scholarly work, so appropriate attribution of author roles helps us ensure that that record is accurate.
18. Inappropriate ordering of authors
Video: Inappropriate ordering of authors
Description: Not everyone will agree on which contributions to a paper are most valuable and should be rewarded by a higher position in author order. We may believe that author order should reflect the amount of time or writing that an individual puts into the paper. Others might feel that the variety of contributions at different stages of the process matter most, or that the individual who emerges as the ‘leader’, scheduling meetings, setting deadlines, etc. deserve a higher position in the author order. We may also be tempted to put well-known or respected scholars, academics working towards tenure, or early-career graduate students who would benefit greatly from being a first author on a paper higher in author order. In the latter two cases, this begins to look like gifted authorship. It is best if this conversation can happen before the research process begins. Have conversations about how each author can best contribute to the study, which pieces of the research they believe they are best equipped to take the lead on, and importantly, what this means for author order.
19. Not giving research assistants due credit in publications
Video: Authorship credit for RAs
Description: This is a difficult QRP because there are many different opinions about the role of a research assistant. Research assistantships often come with benefits such as mentorship and financial compensation. In these situations, some may feel that RAs are already compensated for their work, and that authorship credit should only be awarded to them if they go above and beyond their expected role, for example, participating in the study design conversations or data analysis. Other academics fall into the opposite camp on this issue, feeling that if RAs contribute to the project, they are naturally a part of the research team. Those who take this position feel that RAs should be held to the same standards for authorship as the other researchers on the project are, regardless of their status as a research assistant. This decision, however, is highly dependent upon various other contextual factors, such as the level of the student (e.g., undergraduate students versus Ph.D. students), the nature of the project, the mentor or PI, and the roles that are expected of the research assistant. The best way to handle this is to get ahead of this issue. It’s useful for the mentor or PI to have an open discussion with the research assistant before the project starts in order to determine what will qualify the RA for authorship credit moving forward.
20. Exaggerating the implications and/or importance of findings in order to increase likelihood of publication
Video: Exaggerating implications or importance of findings
Description: When we have worked hard on a project for a long time, we are likely to be (a) excited about the topic and the findings and (b) eager to see it in print. It might therefore be tempting to exaggerate the importance of the findings. This may or may not be an issue in all contexts, in that reviewers and editors tend to be able to see through overly enthusiastic accounts of something, but we should of course always try to be as objective as possible when presenting our findings.
21. Lifting short phrases from others without quoting directly
Video: Lifting phrases without citation
Description: Especially when we start out in the field, we may struggle to know how to avoid having overly cautious citation practices while still not plagiarizing ideas and text from other researchers. To make matters worse, we may need to know the field relatively well to know which concepts and ideas are considered 'common knowledge' and which always require citation. As a rule of thumb, it's of course better to air on the side of caution, and 'overcite' rather than 'undercite'.
22. Irresponsibly co-authoring (e.g., not being involved enough to be able to verify accuracy of analysis)
Video: Irresponsibly co-authoring
Description: In academics, getting credit for publications is the way we get paid (other than the actual money they give us for working). As scholars we use this credit to help us join graduate programs, get jobs, gain tenure, boost our CVs, secure funding, and more. The more academic bling you earn the higher your status in the field tends to be and the more opportunities open to you. So it is important to make sure that all authors listed on a publication deserve to be there. If a person is “gifted” authorship, they unfairly benefit from gaining academic credit without having done the work, meaning they might have a better chance of getting a job or a prestigious grant over someone else who has done the work. From an ethical standpoint, gifting authorship is also problematic. If I conduct research and do a lot of shady things and then list you as an author, you become responsible for the things that I did. You agree to what was discussed in the paper, even if it is questionable work or includes offensive language. Because you are now an author who helped “write” the paper. What counts as authorship can be hard to pin down. The APA says: An author is considered anyone involved with initial research design, data collection and analysis, manuscript drafting, or final approval. But not everything counts. Things like mentorship or being paid in a different way (such as a research assistant) are usually not enough to earn authorship credit. Saying “I wonder if X impacts Y” is not the same thing as sitting down and defining X and Y and then figuring out how to test the impacts. The best way to avoid issues of authorship is with open communication. Talk with people connected to the project and see if they want to be part of a research team and contribute meaningfully to the work. Write up contracts explaining what everyone’s roles and expectations are from the start. Avoid headaches before they start.