Shifts in correlation

The detection of regime shifts (change points) in the correlation coefficient between two correlated variables x and y is based on the formula for the variance of the sum x + y:

s2x+y = s2x + s2y + 2rsx sy.

Here, s2 denotes sample variance and r is the sample correlation coefficient between x and y. Further, it is assumed that the means ̅x = ̅y = 0 and the variances s2x = s2y = 1, that is, the stepwise trends (regime means) in x and y are removed and the series are normalized by the regime standard deviations. In this case, the above formula can be written as:

s2x+y = 2 (1 + r).

It shows that, as the correlation coefficient increases, the variance also increases from a minimum of 0 at r = -1 to a maximum of 4 at r = 1. To detect abrupt shifts in r, a technique similar to that for the variance in individual series can be applied to a series of x + y. It can also be applied to a series of the difference x - y, since s2x-y = 2 (1 - r).

In STARS, the detection test is performed for both x + y and x - y series. When the correlation coefficient between x and y is high (|r| > 0.6 or so), the detected change points most often are the same for x + y and x - y series. For a smaller |r|, the change points may differ, and the decision which one to select is based on the p-value computed using the Fisher r-to-z-transformation and then testing the null hypothesis using the Student t-test that the correlation coefficients for the adjacent regimes are the same. The change point with the minimum p-value among the competing change points is chosen as the final one.

If you are sure that your data satisfies the assumptions that ̅x = ̅y = 0 and s2x = s2y = 1, then the detection test for the correlation can be run from the “Data” worksheet containing the original data. If not, it is recommended to use the 3-step procedure, described in detail in Rodionov (2015):

1. Remove the stepwise trends from x and y using the shift detection in mean. Use prewhitening to remove the red noise (auto-correlation), since it can significantly affect the detection of change points in r.

2. Normalize the time series by standard deviations for each regime found using the shift detection in variance

3. Run the regime detection test in the correlation for the normalized time series.

When you select all three check boxes in the form (“Mean”, “Variance” and “Correlation”) as shown in the figure below, the software guides you through these steps. Check the "Mean" box first, select the parameters for red noise estimation and then check the "Variance" and "Correlation" boxes. Often, however, it is more convenient to work step by step inspecting the results after each step. If after step 1 (shifts in mean) the results are satisfactory, activate the "ResM" worksheet. In this case when the entry form is open, the Shifts in variance checkbox will be automatically preselected. Similarly, if the program is run when the "NormV" worksheet is active, the Shift in correlation checkbox is preselected. 

Here are three examples illustrating how the software works. In the first example, STARS was applied to a couple of synthetic time series with predefined characteristics. Its purpose is to show all the steps in the 3-step procedure outlined above. In the other two examples, real climate observations were used. These examples show significant structural changes in the climate system in the late 1990s, when a number of well established relationships between climatic variables were broken.

1. Synthetic time series.

2. The Arctic Oscillation.

3. ENSO effect on the northern tropical Atlantic.

4. Relationship between Arctic Oscillation and East Asian summer monsoon.

5. Lagged relationship between Arctic Oscillation and circumpolar vortex.

References

Rodionov, S., 2015: A Sequential Method of Detecting Abrupt Changes in the Correlation Coefficient and Its Application to Bering Sea Climate. Climate, 3, 474-491.