MetaAnalysisFAQs

Participant Data Questions


Sample Size Questions

Question: If a study is conducted at the group/team do I also report the number of individuals?
Answer: Ye, please include both the number of individuals and the number of group/teams if applicable. 


Age Mean Value Questions

Question: If a paper does not provide the mean, but does provide the range do we report the range instead?
Answer: Yes, please report the range instead. Remember to report the actual range itself do not calculate the midpoint and use that value.


Country of Origin Questions

Question: Is this the country the study took place?
Answer: No, this should be the study of origin of the participants not where the study was conducted.

**************************************************************************************************************************************************************************


Analysis Data Questions


General Questions

Question: I totally confused by how the data reports it analysis. What do I do? 
Answer: I would just make a note of this in the final comments. Please cite the location of the text i.e. page number and column and paragraph.  We can meet to discuss it later. 

Manipulation Checks Questions

Question: Should I code manipulation checks?
Answer: No. You should not include information related to manipulation checks into the form. They are not apart of the study unless they show up as independent variables. If this is true the information we need would be in the section on independent variables. 

Question: What are manipulation checks anyway? 
Answer: Researchers conducting experimental studies attempt to increase or decrease independent variables (i.e. manipulate). Manipulation checks are performed to ensure that the researchers were successful at increasing and/or decreasing their variables. 

Pilot Study Questions

Question: Should I code pilot studies?
Answer: No. 

Question: What are pilot studies anyway? 
Answer: Researchers conducting experimental studies often run pre-studies to ensure that everything is ready to go for the full study.


Beta Coefficient Questions

Question: What are beta coefficients
Answer: Beta coefficients are the estimates resulting from some type of regression analysis. They can be standardized or unstandardized normally represented in many papers by the symbol β 

Question: What information do I need to record with regards to the beta coefficient
Answer: The beta coefficient, standard error and/or the t-value or p-value associated with the significance of the beta coefficient.  The t-value is determine by the beta coefficient/standard error (i.e. T value = beta coefficient/standard error). Therefore, if we know 2 of the three values we can determine the other. In addition, we can determine the t-value by the p-value associated with the significance of the beta coefficient. 


Correlation Coefficient Questions

Question: What are correlation coefficients?
Answer: Correlation coefficients represent the bivariate correlation between two variables that range from -1 to 1. They are represented in many papers by the symbol r. For example, the correlation between variable X and variable Y is r = .50. 


T and F Test Questions

Question: What is the difference between a T-test and a F-test?
Answer: T-tests are normally used when comparing two means, while F-test are used to compare more than two means. For example, when comparing the mean of the control condition with the mean of treatment 1 condition a T-test is used. However, when comparing the means of the control condition, treatment 1 condition and treatment condition 2 a F-test is used. 


Question: The paper reports both T-tests and F-tests? Which do I report?
Answer: Both. In many cases papers with use a F-test to determine if there are differences between more than two means. For example, when comparing the means of the control condition, treatment 1 condition and treatment condition. Then after discovering that the F-test is significant many studies will compare each mean with another mean using a the T-test 

Ideally, if the paper reports the mean and standard deviation for each condition we can calculate the F and T tests. Unfortunately, many papers do not report  the means and/or standard deviation. When this occurs we can then use the T and F values to determine the effect sizes.  


Question: This study had Bonferroni corrected paired-sample t-tests should I include them as T-tests?
Answer: Yes, you should include the Bonferroni corrected paired-sample t-tests. Bonferroni and/or Tukey are post hoc t-test done after the F-test. You should include them as t-test and note that they are bonferroni corrected in the comments at the end of the survey. 


Question: Some papers are reporting the results of a F-test from an ANOVA by only comparing two means. But, I thought we use F-test when we are comparing more than two means and T-test when comparing just two means.
Answer: Typically yes, but you can also run an ANOVA with only one condition. This produces a F-value that informs you that the one condition is significant. This implies that at least one mean within that condition is significantly different from at least one other mean. However, when your condition only has two means you do not need to determine which means are different from each other. 

You can simply treat this as an F-test with two means. You skip the control condition and input the two means as separate treatments. 

****************************************************************************************************************************************************************************
Study's Setting Questions

Question: How can I determine the study's setting? Typically, the context and task are key indicators followed by the participants.

Context: 
For example, if the study is being conducted in a lab setup as a home it would fall under the home setting. If the study is being done in a hospital it would probably fall under a healthcare setting. If the study was being in a classroom it would probably fall under a education setting. 

Task:
For example, if the task involved a robot helping the participant find a good movie to watch it would probably fall under the home setting. If the task involved a robot helping to get patients motivated to rehab a recovering injury it would probably fall under a healthcare setting. If the task involved helping participants learn math then it would probably fall under an education setting. 

Context and Task:
Often you can use the context and task to determine the study's setting.

For example, if the study is being conducted in a lab setup as a home and the task involved a robot helping the participant find a good movie to watch. You would be more confident that it would fall under the home setting.

For example, if the study is being done in a hospital and the task involved a robot helping to get patients motivated to rehab a recovering injury. You would be more confident that it would fall under a healthcare setting.

For example, if the study was being done in a classroom and the task was helping participants learn math. You would be more confident that it would fall under a education setting. 

Participants:
At times the type of participants can help you determine the study's settings. For example, if the participants were patients recovering from an injury you could conclude it was a healthcare setting. Unfortunately, most studies use students as participants even if there are attempting to study home or work settings. Please see the question below for more details on this issue. Therefore, the context and task are the primary sources of information when determining the study's setting. 

Question: Would a study conducted at a university using students as participants automatically be considered an educational setting?
Answer: No. Almost all the studies will be conducted using students at a university. However, unless the task or context (e.g.  participants are college students being tutored by robots) is related to education or learning then the study does not fall under the category of an "educational" setting.  

*****************************************************************************************************************************************************************************
Questions about the Robot

Type of Robot Control

Question: How can I tell the difference between remote controlled robot operation and Wizard of Oz?
Answer: If the participants controlled the robots and not the researchers during the study it was remote controlled.  However, if the researchers controlled the robot and attempted to hide the fact that the robot was not autonomous it was Wizard of Oz.


Comments