2. correlation models

Following Schneidman et al.\cite{Schneidman2003}, for example, we might say that the uncertainty associated with mth-order marginals for a system of L variables is something like:

In order to consider correlations on more than one level in hierarchical complex systems, we begin with ways to quantify pair and higher-order components of total-correlation (the generalization of mutual-information to more than two subsystems as a special-case of always-positive KL-divergence\cite{Kullback51}) on a single level. 

This may facilitate use of subsystem correlation-information as the natural thermodynamic limit on evolving complexity. As shown in Fig. \ref{Fig1}, for example, a limit on subsystem correlation-information in the John Conway's life adds realism to the process by preventing the steady-state endings to the evolution process.

    ,  

from which the "connected information" of order m for a system with L variables becomes, in terms of both our equations and their equation (6):

    

Total correlation $I_c$ is simply the sum of these positive terms for $m$ running from 2 (pair correlations) up to L.

This unpacking of always-positive total-correlation measures into pair and post-pair components is of special interest to physicists because of the total-correlations connection, as a special case of KL-divergence, to applications for the second-law of thermodynamics. In fact the move to always-positive information-measures, like KL-divergence as the negative of Shannon-Jaynes entropy\cite{Gregory2005}, may signal a pedagogical move from entropy-1st thermodynamics to correlation-1st thermodynamics\cite{Fraundorf2011a} in the decades ahead.

Related references: