Questions, comments? Don't hesitate to get in touch with us: {tsujish | chbergma | alecristia}@gmail.com
Please also check out the accompanying article.Community-augmented meta-analyses (CAMAs) like the one introduced on this site are a simple tool to significantly facilitate the accumulation and evaluation of previous studies within a specific scientific field. Based on our own experience, we have created a step-by-step tutorial of how to create a CAMA. Please note that we followed the PRISMA statement on structured reviews and meta-analyses for the meta-analytic part of our CAMA. We strongly recommend to refer to the PRISMA checklist and flowchart in addition to our tutorial.
This tutorial is rather lengthy, so you may want to look at specific sections depending on your interests:
Part A: Assembling a set of relevant studies1. Narrow down the topicThe topic should be narrow enough that methods and outcome variables are comparable, but broad enough to address other researchers visiting different questions with comparable techniques, and who can profit from and enlarge the CAMA.Example Narrow enough: InPhonDB and InWordDB both focus on infant speech perception. However, the former focuses on the ability to discriminate speech sounds and the latter on the ability to segment words out of fluent speech, two different abilities that are assessed in distinct basic paradigms. 2. Conduct a literature searchConduct a systematic literature search in one or multiple search engines such as Google Scholar*, Science Direct, Web of Science, or PubMed. Use a keyword combination that is as inclusive as possible (this means you will get a large number of false positives you will need to weed through, but reduce the chances of failing to detect a study). We also set up search alerts to identify us via email in case new research with our keyword combination was published.
Example
We used the keyword combination "ALL (vocabulary infant speech perception longitudinal AND LIMIT-TO (topics, "infant,language development,language acquisition,speech perception")) "infant,language development,language acquisition,speech perception")) for InVarInf (on Science Direct, yielding 567 results), and "{infant|infancy} & {vowel|speech sound|syllable} & discrimination" (on Google Scholar, yielding 1340 results) for InPhonDB. *Note that Google Scholar lists both research that is published in journals and unpublished research (which is available and identifiable online), which is relevant for assessing the possibility of bias in reporting. 3. Converge on final sampleThis step includes both excluding irrelevant studies and adding further relevant studies. In order to select the relevant studies out of the studies retrieved in step 2, we read through the titles/abstracts of each study and accessed all which were potentially relevant*, excluding further studies after reading through the actual articles.Further relevant studies can be identified by looking through the reference list of relevant studies, as well as contacting experts. We defined experts as researchers that are first or last authors in at least two relevant studies, and who are still active in the field or can be contacted otherwise. Of course, you might also know of additional potential experts that are active in a closely related field. The PRISMA flowchart is helpful for keeping an overview of exclusions and additions to the search sample. In addition, we kept excel sheets (click for an example you can copy and use) to keep track of all potentially relevant studies. *Fellow researchers/authors can be contacted where there is no access, and if accessing the articles is still not possible this should be noted in the search protocol.
ExamplePart B: Create a database of relevant variables4. Create coding form and enter study informationHere we code potentially relevant moderator variables (e.g., participant characteristics, experimental manipulation) and outcome variables (means, outcomes of statistical analyses). While the relevant variables might be fixed a priori for a regular, narrowly focused meta-analysis, the primary goal of a CAMA is to provide an exhaustive assembly of data that can serve multiple research interests and purposes. Therefore, it is important to code as many variables as possible. Nevertheless, too many variables can be difficult to overview for users and demotivating to new contributors, and uninformative variables should therefore be skipped.We adopted a two-step procedure to determine the relevant columns to code. In a first step, we coded all moderator and outcome variables per experiment. In a second step, we removed predictor variables that were only reported in a small subset of studies (for instance, size of presentation screen, signal-to-noise ratio), or that turned out to be non-significant predictors of effect size.We entered studies in spreadsheet format, where each row represented one unique measurement sample (e.g., one group of participants measured in one condition), and each column represented a moderator or outcome variable. Each topic requires a different amount of coded variables, as you can see by comparing the spreadsheets for InWordDB, InVarInf and InPhonDB. You may want to start your own spreadsheet from scratch, or take a look at our template spreadsheet containing suggestions for basic columns we consider useful for a wide range of topics. 5. Calculate effect sizes and weightsWith all the study variables coded, the last step to obtaining a 'meta-analyzable' CAMA is the transformation of all coded outcome variables into effect sizes (which provide a comparable metric to quantitatvely express the strength of your effect). You will also need a weight for each of these effect sizes (which indicates how strongly this particular entry should be weighted in relation to other entries). What kind of effect size fits best your data format and how exactly you calculate it varies by the type of data you are looking at. We recommend novel CAMA creators to consult textbooks1 and primers2 for a general overview, and scientific articles3 for more specific questions.You can access our R4 (R core team, 2014) analysis scripts (for InVarInf, InWordDB, InPhonDB) in order to get the code for calculating the effect sizes and accompanying weights we use in our CAMAs (Cohen's d, Hedge's g, Pearson's r). We recommend you to go through the scripts (using our CAMAs or your own brand-new database) in order to get a hands-on understanding of how that works. Complementary to that, we here provide some descriptive examples to give you an idea of how to get from specific outcome variables to an effect size. Example 1: correlational 2 D. Lakens. (2013). Calculating and Reporting Effect Sizes to Facilitate Cumulative Science: A Practical Primer for t-tests and ANOVAs . Frontiers in Psychology 4:863. Material: osf.io/ixgcd/files/ 3 Since textbooks do not cover every possible question that different meta-analysts may encounter, we turned to articles for more specific questions. We found this article useful for considering the comparability of effect sizes from within- and between-partcipant designs: Morris, S. B., & DeShon, R. P. (2002). Combining Effect Size Estimates in Meta-Analysis With Repeated Measures and Independent-Groups Designs. Psychological Methods, 7(1), 1805-125. doi: 10.1037//1082-989X.7.1.105 4 If you are not familiar with R, you can perform the same calculations in programs like excel as well. 6. Meta-analysisHaving put together a spreadsheet that contains moderator variables, effect sizes and their weights, all ingredients for a meta-analysis are in place. The main pieces of information we want a meta-analysis to convey is the size of the overall effect, whether there is heterogeneity in the sample, and if so, which moderator variables can explain parts of this heterogeneity.We refer to the same literature and CAMA scripts as in step 5 to get background knowledge and hands-on experience with conducting meta-analyses. In addition, we again provide a short descriptive illustration. Example Meta-analytic regressions can easily be performed with the metafor (Viechtbauer, 2010) package in R. The rma function is constructed similarly to a regular regression function, for instance: Part C: Share your database online7. Open your database to the public
Finally, you want to make your database available to the research community. We used google sites, since they allow the set-up of a multitude of online tools (questionnaires, forms, downloadable material) quickly and easily.* |