How to Read Outcome Tables

It is good practice to check all tables within a study to ensure you've identified all the outcomes. However, there may be many tables which are of no use to us. 

We are only interested in the tables that report outcomes using includable impact evaluation methods. Therefore, you will likely not find outcomes for our purposes in the following tables:

• Summary statistics/ descriptive statistics / other data summary tables
  (please note: these tables could be referenced along with an outcome's results table if they provide a strong written definition for the outcome).

• Indicators at baseline

• Parallel trends

• First order-conditions / first-stage estimates

• Correlation matrices

• Balance tests / checks at baseline

If you're not too sure what a table is presenting, read how the authors describe the results from the table in the text - do these seem like impact evaluation estimates?
If you're still unsure, you can always comment on the FAQ page in the DEP DEX Tracker.

Identifying outcomes can be difficult. 

Outcome variables may also be referred to as 'dependent variables', 'explained variables' (not to be confused with 'independent variables' or 'explanatory variables')

1. Searching for outcomes within the text is a good starting point

• Often authors will make it very clear what they're investigating. Some authors will explain their outcomes in detail with separate headings – others may describe their outcomes in other areas such as the introduction, data or methods sections. 

• However, sometimes the list of outcomes an author investigates is so numerous that they only mention those they're most interested in within the main body of the text. This is why we always want to check the tables within a study – including in the appendices.

2. For ALL outcomes, we should always reference the outcome table that it appears in

• Whether you have identified an outcome from the text or from a table, you should always reference any table that presents impact estimates for that outcome in your outcome description.

• It's important to check ALL tables within a study to ensure you haven't missed anything

Once you have identified what you believe to be the outcome tables, it can sometimes be difficult to read and understand them.

Steps to identify outcomes from outcome tables:

1. Read the table's title - titles will sometimes make the outcomes very explicit

• From Table 4 directly below, we can see that the table is showing the impact of smart city construction (the intervention in this study) ON eco-efficiency (the outcome in this study).

• Notice that eco-efficiency is not very descriptive and could be measured in many ways. Only using the outcome tables does not give us a lot of detail – therefore it's important to use BOTH the outcome tables and the text to generate your outcome descriptions.

  • • • In this case, eco-efficiency is: calculated using the Super-efficiency SBM model (see equation 1, page 3) (Table 4) (Tingting et al. 2020)

2. Read the Notes at the bottom of the table - notes will sometimes include important information on the outcome/dependent variables

• See from Table 1 directly below, the outcome (dependent variable) of this table is the Gini index, which is determined from the notes field.

• Notice that the table title gives some indication that the outcome is a measurement inequality – however the Gini index is a lot more specific 

• We could also supplement this outcome description by searching "Gini index" within the text and seeing how the authors describe it.

3. If it's not clear from the title or the notes, check the columns in the table

• In Table 4 directly below, the outcomes are clear both from the title AND from the columns (the columns are even clearly labelled here as the dependent variables)

• In the case of this study, TECH and EFFCH does not give us a sufficiently detailed description of the outcome, so we would supplement our outcome description by searching for additional information within the text. Searching TECH and EFFCH in the paper provides the following descriptions.

  • • • TECH: Technical Change Index. TECH represents the technical change index, which measures the change in the best practice gap between the contemporaneous frontier and the global frontier between the two years (Page 5) (Table 4, Page 10) (Zhang et al. 2020).

      • EFFCH: Technical efficiency change index. EFFCH reflects the catching-up effect of a decision-making unit (DMU) approaching the contemporaneous frontier (Page 4) (Table 4, Page 10) (Zhang et al. 2020).

• Also note: TECH and EFFCH appear twice because there are two different treatments being investigated in this study ('time x treat' and 'time x treat1'). In the admin panel, you would need to extract the two treatments as separate intervention arms, and ensure both TECH and EFFCH are listed under both.

4. Sometime a table will present many outcomes on the left-hand side. This can make them difficult to distinguish from the above tables, which include intervention and control variables on the left-hand side

• In all the above examples – the variables on the left-hand side are the intervention and control variables from the authors models. We can identify the variable representing the intervention (otherwise known as the treatment or independent variable) in the above tables as follows:

1. In the first Table: du x dt is a variable taking the value 1 if the city engaged in smart city construction

• From this study, the intervention was smart city construction

2. In the second Table: V represents a VAT adaptation variable

• From this study, the intervention is VAT adaptation

3. In the third Table: time x treat represents the year of implementation of the APCP

• From this study, the intervention is the APCP (Air Pollution Prevention and Control Action Plan)

• You should use your best judgement to determine whether the variable(s) (usually at the top left-hand side of a table) represent the intervention. Seeing an 'X' (representing some form of interaction, between treatment and time in the above cases) or a word such as 'treat/treatment' can also offer a strong indication.

5. When authors present many outcomes on the left-hand side

• Table 2 directly below is an example where outcomes are listed on the left-hand side of a table. We know these are outcomes as there is no variable representing the intervention on the left-hand side.

• All of these many outcomes on the left-hand side would need to be extracted. 

  • • For studies with this many outcomes, it's likely that the authors will not go into detail about how every outcome was measured. We should always check – however if there's no additional detail, we can capture them simply by copying the outcome names from the table over to our outcome description box on the admin panel

    • You must ensure to capture the table number for the outcomes you extract and that you group outcomes according to their lowest level of aggregation

      • • • i.e., we should only have one of each outcome code on the admin panel. Multiple outcomes that apply to the same outcome code from our taxonomy should be listed in the same description box for that outcome).

5. Caveats and exceptions

• Notice in Table 2 directly below, that there are both columns at the top and variables down the left-hand side.

• There appear to be no intervention variables on the left-hand side – therefore we can assume these are outcome variables.

• However we also have columns here. When you see columns and the outcome variables are on the left-hand side, this strongly indicates the sample has been split in some way (there is some form of sub-group analysis).

  • • • Here the authors are doing sub-group analysis for their male and female participants

      • This is very important for our Equity coding! If the sub-group analysis matches one of our equity dimensions, you should ensure to apply the appropriate equity code and include this table in your Equity descriptions.