WORKSHOP THEME: 'Expert indicators in the social sciences: Challenges of validity, reliability and legitimacy.'
FINAL PROGRAM WITH PLANNING SCHEDULE: You can download the final program, which contains details about the organization, theme and the planning schedule. Printed copies will be available on registration at the meeting.
Lunchtime roundtable discussion notes are available for download
WHERE: The Hilton Garden Inn, 1100 Arch St, Philadelphia, 19107 (Next to Reading Market)
WHEN: From 8.30am to 7pm on Wednesday 31 August 2016, immediately before the American Political Science Association Annual Meeting.
SPONSORS: The event is co-sponsored by EIP, the Varieties of Democracy project (V-DEM), and the APSA Elections, Public Opinion, and Voting Behavior (EPOVB) organized section.
WORKSHOP CHAIRS: Pippa Norris (EIP, Harvard and Sydney University) and Svend-Erik Skanning (V-DEM, Aarhus University)
WORKSHOP COORDINATOR: Alexandra Kennett (EIP Project Administrator, Sydney University)
REGISTER: Register to reserve your place and specify your meal requirements, if you have not already done do, via the online form.
QUERIES TO: electoralintegrity [at] gmail.com.
Participants will be provided with refreshments, lunch and an evening drinks reception. In addition, paper-givers, chairs and discussants will be invited to a local dinner after the event from 7pm to 9.30pm.
Theme: 'Expert indicators in the social sciences: Challenges of validity, reliability and legitimacy.'
Svend-Erik Skaaning and Pippa Norris
Expert surveys have become increasingly common in comparative social science, in risk analysis by private sector organizations, in evaluation research, and among NGOs and policy makers (Meyer & Booker 1991). Expert surveys are often used to deepen understanding of, among others, the left-right position of political parties and news media outlets, the perceived extent of corruption or bribe-paying, and the quality of democratic governance. Expert surveys increasingly supplement alternative sources of information, such as citizen mass surveys, event analysis of media reports, and official statistics.
This data collection technique has been applied to diverse research topics such as the series of studies on party and policy positioning (Laver and Hunt 1992; Huber and Inglehart 1995; Saiegh 2009; Laver, Benoit, and Sauger 2006; McElroy and Benoit, 2007), the power of prime ministers (O’Malley 2007), evaluations of electoral systems (Bowler, Farrell, and Pettitt 2005); policy constraints horizons (Warwick 2005); campaign communications (Lileker, Steta and Tencher 2015); human rights and democracy (Landman and Carvalho 2010), and the quality of public administration (Teorell, Dahlstrom and Dahlberg 2011). Expert surveys have been widely used in research on corruption - the Corruption Perceptions Index (Transparency International 2013; Global Integrity); measuring democracy since the 1900s -Varieties of Democracy (Coppedge et al. 2011)- and electoral integrity (Norris, 2014; Norris, 2015; Martinez i Coma and Van Ham, 2015). The World Bank Institute Good Governance indicators combine an extensive range of expert perceptual surveys drawn from the public and private sectors. Indeed among the mainstream indicators of democracy, Freedom House’s estimates of political rights and civil liberties, Polity IV’s classification of autocracies and democracies, and the Economist Intelligence Unit’s estimates of democracy are all, in different ways, dependent upon expert judgments.
Expert surveys seem especially useful for measuring complex concepts that require expert knowledge and evaluative judgments; and for measuring phenomena for which alternative sources of information are scarce (Schedler 2012). Yet, expert surveys are not risk free and scholars have pointed out their limitations (Budge, 2000; Mair 2001; Steenbergen and Marks, 2007). Moreover, in contrast to mass social surveys, we still lack a common methodology to construct such surveys, as well as agreed technical standards and codes of good practice. There has been heated debate about the pros and cons of methods used to evaluate the spatial positions of party policies, and about the use of governance indicators more generally, but by contrast there has been remarkably little discussion about the challenges of validity, reliability, and legitimacy facing the construction of expert perceptual surveys. Yet it is critical to consider these issues given the lack of a clear conceptualization and sampling universe of ‘experts’, contrasting selection procedures and reliance upon domestic and international experts, variations in the number of respondents and publication of confidence intervals, and lack of consistent standards in levels of transparency and the provision of technical information. Moreover, more research needs to be done on how to evaluate the consequences of expert and context heterogeneity on the validity of expert judgments (Martinez i Coma and van Ham 2015), for example by using item response models to test and correct for expert heterogeneity (Pemstein et al. 2015), and using techniques such as ‘anchoring vignettes’ (King & Wand 2007) or ‘bridge coders’ (V-Dem) to test and correct for context heterogeneity.
This workshop features sessions and papers covering the following topics:
Benoit, Ken and Michael Laver. 2005. Party Policy in Modern Democracies. London: Routledge.
Bowler, Shaun; David Farrel and Robin Pettitt. 2005. ‘Expert opinion on electoral systems: So which electoral system is best?’ Journal of Elections, Public Opinion and Parties 15(1): 3-19.
Budge, Ian. (2000). ‘Expert judgments of party policy positions: Uses and limitations in political research.’ European Journal of Political Research 37(1): 103–113.
Transparency International. 2013. Corruption Perception Index. http://www.transparency.org/whatwedo/publication/cpi_2013
Huber, John and Inglehart, Ronald. 1995. ‘Expert Interpretations of Party Space and Party Locations in 42 Societies’, Party Politics 1:73-111.
King,G. &Wand, J. (2007). Comparing incomparable survey responses: Evaluating and selecting anchoring vignettes. Political Analysis 15(1): 46–66.
Landman, Todd and Edzia Carvalho. 2010. Measuring Human Rights. London: Routledge.
Laver, Michael and Ben Hunt, B. (1992). Party and Policy Competition. London: Routledge.
Laver, Michael, Kenneth Benoit, and Nicolas Sauger. 2006. ‘Policy Competition in the 2002 French Legislative and Presidential Elections.’ European Journal of Political Research 45: 667-697.
Lilleker, Darren., Stetka, V. and Tenscher, J., 2015. Towards hypermedia campaigning? Perceptions of new media's importance for campaigning by party strategists in comparative perspective. Information, Communication and Society, 18 (7), 747-765.
Mair, Peter. (2001). ‘Searching for the position of political actors: A review of approaches and a critical evaluation of expert surveys.’ In M. Laver (ed.), Estimating the policy positions of political actors. London: Routledge.
Martinez i Coma, Ferran; Van Ham, Carolien . 2015. ‘Can experts judge elections? Testing the validity of expert judgments for measuring election integrity.’ European Journal of Political Research 54 (2): 305-325.
McElroy, Gail and Kenneth Benoit. 2007. ‘Party groups and policy positions in the European Parliament.’ Party Politics 13:5-28. Meyer, M. & Booker, J. (1991). Eliciting and analyzing expert judgment: A practical guide. London: Academic Press.
O’Malley, Eoin. 2007. The Power of Prime Ministers: Results of an Expert Survey’ International Political Science Review 28(1):7-27.
Pemstein, D.; Tzelgov, E.; Wang, Y. 2015. “Evaluating and Improving Item Response Theory Models for Cross-National Expert Surveys”. Varieties of Democracy Institute: Working Paper Series No. 1.
Saiegh, Sebastian. 2009. ‘Recovering a basic space from elite surveys: Evidence from Latin America.’ Legislative Studies Quarterly 34(1):117-145.
Schedler, Andreas. 2012. ‘Judgment and Measurement in Political Science’ Perspectives on Politics 10(1):21-36.
Steenbergen, Marco R. and Gary Marks. 2007. ‘Evaluating expert judgments.’ European Journal of Political Research 46: 347–366.
Teorell, Jan. Carl Dahlström & Stefan Dahlberg. 2011. The QoG Expert Survey Dataset. University of Gothenburg: The Quality of Government Institute. http://www.qog.pol.gu.se
Warwick, Paul. 2005. ‘Do Policy Horizons Structure the Formation of Parliamentary Governments?: The Evidence from an Expert Survey’ American Journal of Political Science 49(2):373-387.
Panel chairs are asked to keep speakers running to timer and the event on schedule. Speakers are asked to present each paper for no more than 10 minutes. Discussants are also each asked to offer comments for around 10 minutes. This allows the remainder of the time for interactive Q&A with all event participants. A laptop with PowerPoint projection facilities will be available in each room.
Please email your presentation to the Workshop Coordinator, firstname.lastname@example.org, at least 24 hours before the workshop or hand it to her on a stick when registering. We anticipate around 70 participants and attendees in total.
All registered participants will be offered morning and afternoon refreshments, a buffet lunch, and an early evening cocktail reception. After the end of the reception, we will walk over to the restaurant (3 minutes away) with invited participants for the workshop dinner.
After the workshop, all papers will be reviewed by the organizers for possible publication in either an edited book or a special issue of a journal. Please let us know if you have already placed your paper elsewhere so that you would not want to be considered for this process.
Travel grant recipients must email their University of Sydney Vendor Form, flight and accommodation invoices to email@example.com for reimbursement following the event.
Pippa Norris (Harvard and Sydney University)
Staffan Lindberg (V-DEM and University of Gothenburg)
Chair: Alessandro Nai (University of Sydney)
Discussant: Jeffrey Karp (Exeter University)
1.1 Pippa Norris (Harvard and Sydney University)
The pragmatic case for electoral assistance: The impact of regional organizations on electoral reform
1.2 Josephine Boyle (Rider University), Michael Brogan (Rider University) and Frank Rusciano (Rider University)
1.3 David Carroll (The Carter Center) and Obehi Okojie (Georgetown University)
Assessing Electoral Integrity: Comparing Indicators and Developing an Overall Assessment Framework
1.4 Kirill Kalinin (University of Michigan)
Signaling Games of Election Fraud
*** EIP/International IDEA essay competition winner 2016 ***
Chair: Pippa Norris (Harvard and University of Sydney)
Discussant: Frank Rusciano (Rider University)
2.1 Masaaki Higashijima (Tohoku University) and Eric C. C. Chang (Michigan State University)
The Choice of Electoral Systems in Dictatorship
2.2 Eleanor Hill (University of Manchester)
Biraderi, Political Machines and Postal Voting on Demand in Great Britain
2.3 Shane Singh (University of Georgia)
Compulsory Voting and Electoral Integrity
2.4 Víctor A. Hernandez-Huerta (University of the Andes)
Rejecting election results now to be elected in the future: evidence from sub-national elections in Mexico.
Chair: Holly-Ann Garnett (McGill University)
Discussant: Alessandro Nai (University of Sydney)
3.1 Mike Omilusi (Ekiti State University)
From Convenient Hibernation to Circumstantial Desperation: Hate Speech, Party Political Communication and Nigeria’s 2015 General Elections.
3.2 Sharon Lean (Wayne State University) and Matthew Lacouture (Wayne State University) (copies are available on request from the authors (firstname.lastname@example.org)
Resources, repertoire and context: what determines the quality of domestic election monitoring?
3.3 Max Grömping (University of Sydney)
Explaining news attention to domestic election monitoring initiatives
3.4 Nicholas Kerr (University of Alabama) and Anna Lührmann (University of Gothenburg and V-DEM)
Public Trust in Manipulated Elections: The Role of Election Administration and Media