A. power law exponents
As a quantitative example of "surprisal tabulation" to start off with, consider a set of electrostatic force measurements (with measurement errors of ±0.2 force units) as a function of radial distance, like for example the following {separation-distance, force} pairs: {{0.8, 1.64204}, {0.9, 0.891087}, {1., 1.0413}, {1.1, 0.898058}, {1.2, 0.917022}, {1.3, 0.762478}, {1.4, 0.465493}, {1.5, 0.571502}, {1.6, 0.601989}, {1.7, 0.431092}, {1.8, 0.256245}, {1.9, 0.331824}, {2., -0.00813074}}. Let's compare the extent (in bits or surprisal) to which an unknown power-law, and a one-over-r-squared power law, are surprised by this data. The maximum-likelihood parameter-estimates for the a/rb model are {a -> 1.01717, b -> 1.76984} while the maximum-likelihood parameter-estimate for the a/r2 model is {a -> 1.01833}.
If we choose parameter-priors using Akaike Information Criterion, the cross-entropy surprisals (in bits) of each model at this particular data-set are:
Quantitative comparison of two models
\Models
observations
dataset 1
unknown power
a/rb
9.61 bits
action at distance
a/r2
8.01 bits
Over a sample of 10,000 such datasets, the average surprisal at the data by the unknown power-law is about 1.31 bits larger than the suprisal of the generating one-over-r-squared power law, in spite of the fact that the extra free-parameter in the unknown power-law always generates a closer (if not identical) fit to the data.
The Occam-factor correction may alternatively be generated using specific models (ala Gregory) for the prior-probability of each model fit-parameter. Adding prior-probability models for each fit parameter, however, may seem a bit like adding parameters even if the effect is less problematic, so the decision whether or not to do this should perhaps be left as a case-by-case decision.
Related references:
Burnham, K. P. and Anderson D. R. (2002) Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, Second Edition (Springer Science, New York) ISBN 978-0-387-95364-9.
Phil C. Gregory (2005) Bayesian logical data analysis for the physical sciences: A comparative approach with Mathematica support (Cambridge U. Press, Cambridge UK) preview.