Robustness measures are a central tool for evaluating lobbying regulations. However, the existing indexes exhibit significant shortcomings. Moreover, if lobbying regulations aim to facilitate public scrutiny, we should be concerned about the lacking accessibility of these measures and widespread misconceptions about the nature of lobbying and lobbying regulation. With this in mind, my Laidlaw Undergraduate Research and Leadership Scholarship project sought to create a valid and accessible robustness index. In addition, I created the online platform lobbymeter.eu, aimed at increasing public knowledge about lobbying regulation.

Constructing a new index

An extensive literature review showed that the eleven existing robustness measures exhibit significant shortcomings, potentially leading to distorted scores. The following were some of the most commonly identified problems:

1. Unsuitability for application in an international context (e.g., Opheim index, Newmark indexes)

2. Following from Crepaz and Chari’s (2018) research: indexes with a very high or very low number of index items strike an inadequate balance between validity and reliability (e.g., Sunlight Foundation index with 5 items, Laboutková et al. measure with 158 items).

3. Equal weighting does not reflect the varying importance of policy provisions (e.g., Holman and Luneburg index).

The first step in constructing the new index was the selection of suitable index items. The OECD Principles for the Transparency and Integrity of Lobbying arguably set the gold standard for lobbying regulations (Crepaz and Chari, 2018). A valid robustness measure should therefore reflect these principles. In line with theoretical work by Babbie (2008), index items were also selected based on the following four criteria:

1. Face validity

2. Unidimensionality

3. Appropriate level of generality/specificity

4. Adequate variance

The final 23 items were divided into six sub-indices to facilitate the evaluation of different dimensions of lobbying regulation. Different weights were applied across these items, to remedy discrepancies in the number of items within each sub-index and to reflect the varying importance of different provisions. The full index can be viewed here.

Finally, the index was assessed in terms of validity. A convergent validity assessment yielded a correlation coefficient of 0.6807 between normalized CPI scores and normalized scores of the new index across 13 jurisdictions worldwide.

The diagram below shows the results of the content validity assessment, which was carried out similar to the analysis by Crepaz and Chari (2018) (i.e., comparing index items to conceptual dimensions defined by the OECD).

Unfortunately, assessing the index’s reliability was outside the scope of this project, but items were selected with potential issues of reliability in mind.

Lobbymeter online resource

A core element of my Laidlaw project was the creation of the online resource Lobbymeter. The website was created with a few goals in mind:

1. Digitize the scoring sheet to facilitate easy application of the index

2. To present robustness scores in an easily understandable format

3. To increase public knowledge about lobbying and lobbying regulation

As such, lobbymeter.eu provides a simple online form that automatically calculates robustness scores based on my newly developed index. Furthermore, the website contains a host of information easily accessible information about the benefits of lobbying regulation, addresses some common misconceptions, and provides detailed country profiles of EU member states’ lobbying regulations.

In sum, the newly developed index seeks to provide a global, valid, and accessible measure of robustness. In addition to providing an overall robustness score, its sub-indexes allow for the evaluation of individual policy dimensions. The online resource lobbymeter.eu makes this research more accessible and facilitates the application of the index through an online scoring sheet. In addition, it could be a useful contribution to increasing public knowledge about transparency policies. The next steps in further developing this work should be to undertake reliability assessments and test how applicable the index is to sub-national lobbying regulations.

References

Babbie, E. (2008) ‘Indexes, Scales, and Typologies’, in The Basics of Social Research. 4th ed. Belmont: Thomson Wadsworth, pp. 168–198.

Crepaz, M. and Chari, R. (2018) ‘Assessing the validity and reliability of measurements when evaluating public policy’, Journal of Public Policy, 38(3), pp. 275–304.

Hahn, J. (2015) How transparent is your state’s lobbying disclosure?, Sunlight Foundation. Available at: https://sunlightfoundation.com/2015/08/12/how-transparent-is-your-states-lobbying-disclosure/ (Accessed: 26 June 2021).

Holman, C. and Luneburg, W. (2012) ‘Lobbying and transparency: A comparative analysis of regulatory reform’, Interest Groups & Advocacy, 1(1), pp. 75–104.

Laboutková, Š., Šimral, V. and Vymětal, P. (2020) Transparent Lobbying and Democracy, Transparent Lobbying and Democracy. Cham: Palgrave Macmillan. doi: 10.1007/978-3-030-36044-3.

Newmark, A. J. (2005) ‘Measuring State Legislative Lobbying Regulation, 1990-2003’, State Politics & Policy Quarterly, 5(2), pp. 182–191.

Newmark, A. J. (2017) ‘Lobbying regulation in the states revisited: What are we trying to measure, and how do we measure it?’, Interest Groups and Advocacy. Palgrave Macmillan

OECD (2010) ‘Recommendation of the Council on Principles for Transparency and Integrity in Lobbying OECD/LEGAL/0379’. Paris: OECD.

Opheim, C. (1991) ‘Explaining the Differences in State Lobby Regulation’, The Western Political Quarterly, 44(2), p. 405.

UK, 6(3), pp. 215–230.