Area-Of-Effect Placebo Tests

Update: BETA version (v1.1) of Stata programme available (here)

Feedback and bug reports are very much appreciated.

Beta v1.2 will introduce buffer zones and a R-integration.

Project introduction

The emergence of GIS data offers a plethora of analytical approaches to investigate societal phenomena or policies in a spatial context. However, not all policies are implemented on the level of clearly delineated administrative areas. Some interventions might be active in imprecisely specified or only partially known geographic sectors. As a direct consequence, the resulting uncertainty regarding the Area-of-Effect (AoE) impacts on estimates of the effectiveness of a related policy.

Within this project I develop a Stata tool to investigate the robustness of area-specific effectiveness estimates when the observed area might suffer from unknown degrees of misspecification relative to the actual Area-of-Effect of a policy. In this regard, there are several dimensions of uncertainty regarding the observed Area-of-Effect, which relate to a potential misspecification of the observed intervention area in three dimensions, i.e. its position, orientation and size.

The impact of these forms of area misspecification can be assessed in the aoeplacebo programme, either by generating a number of AoE placebo test diagnostics or conducting AoE permutation simulations.

Application background

Starting point for the AoE placebo test is a context where geo-referenced incidents are documented (left panel). In order to control for confounding factors in the vicinity of these incidents, cell specific aggregates of incident occurences (e.g. incident probabilities, right panel) are derived. This then allows to implement high-dimensional fixed effects or to control directly for local conditions.

In the above fictitional scenario, there is a policy measure active in the red polygon. Imagine that when this policy is active, incident occurence is reduced. Identifying grid cells in this area (while it is active) allows to obtain an AoE estimate for this policy. If we assume that we cannot observe the "true" area in which this policy is active but rather an imperfect representation, we could run AoE placebo tests to investigate the implications of misspecification of our applied area.

Programme design features

Starting point for the AoE placebo test is a context where geo-referenced incidents are documented (left panel above).

Using the aoeplacebo programme (developed for Stata) it is possible to calculate AoE placebo estimates for various levels of misspecification of the area's position, orientation and scale. The panel below represents results for the diagnostic design, where potential misspecifications are investigated for each dimension separately.

The diagnostic panels visualise for what levels of dimension-specific misspecification we would obtain AoE estimates similar to those estimates obtained based on the initial area definition. In the context of the utilised artifical data, some caution might be advised since we recover rather comparable AoE estimates for placebo areas which feature a relatively large degree of area misspecification.

If we want to investigate unknown levels of area misspecification occuring across all three dimensions simultaneously, the permutation design would be the way to go. The subsequent graph illustrates the adjustment, based on random variations of an area's dimensions. The final graphical output is a kernel density plot, providing a quick overview of the distribution of AoE permutation estimates.

Programme syntax and functionality

Related outputs

Documentation

Weisser, Reinhard A. (2020): Area-of-Effect placebo tests; Technical Report, January 2020.

Software

AOEPLACEBO.ado - Main programme to conduct AoE placebo test in Stata (version Beta 1.1)

AOEPLACEBO.hlp - Stata help file for the main programme

AOEPLACEBO_EXAMPLES.ado - Auxiliary programme to showcase the main programmes' functionality

Datasets

AOE_TEST.zip - Datasets required to showcase the main programme's functionality

How to use the AOEPLACEBO programme

Step 1: Download the three software components and install them in your ado file directory.

Step 2: Download and extract the datasets. Save these in a place of your choosing, yet preserve the folder name.

Step 3: Open Stata and type 'help aoeplacebo'.

Step 4: To run the examples showcasing the programme's functionality, change your working directory to the unpacked AOE_TEST folder. Then, you can go to the Examples section in the help file and simply hit 'click to run'.

Step 5: Run your own Area-of-Effect diagnostic or permutation tests.

The programme comes with a number of dependencies. AOEPLACEBO will inform you which other programmes are missing. Typically, these can be easily installed from the SSC archive (type 'help ssc' in Stata).