Events‎ > ‎

Philadelphia Aug 2016


Contents

  1. 1 Logistics
  2. 2 Summary Schedule
  3. 3 Theme: 'Expert indicators in the social sciences: Challenges of validity, reliability and legitimacy.'
    1. 3.1 References
  4. 4 Maps: 
    1. 4.1 Getting to the Hilton Garden Inn, 1100 Arch St, Philadelphia 
    2. 4.2 The location of meeting rooms in the Hilton Garden Inn
  5. 5 Guidelines for participants
  6. 6 Full Schedule
    1. 6.1 8:30 – 9:00 Arrival, registration, and buffet continental breakfast (Foyer)
    2. 6.2 9:00 – 9:30  Welcome to the Workshop (Rittenhouse I&II)
    3. 6.3 9:30 – 11:00 Panel 1: (EIP) Assessing and improving electoral integrity (Rittenhouse I&II)
    4. 6.4 9:30 – 11:00 V-DEM) Closed Organizational Meeting (Executive Boardroom)
    5. 6.5 11:00 – 11:30 Coffee/Tea Break (Foyer)
    6. 6.6 11:30 – 1:00 Panel 2: (EIP) The rules of the game: electoral systems and integrity (Franklin 1&2)
    7. 6.7 11:30 – 1:00 Panel 3: Information, communication and legitimacy (Garden Room)
    8. 6.8 11:30 – 1:00 (V-DEM) Closed Organizational Meeting (Executive Boardroom)
    9. 6.9 1:00 – 2:00 Buffet Lunch and Workgroups: (Joint) Best practices for the use of expert indicators (Rittenhouse I&II)
    10. 6.10 2:00 – 3:00 Panel 4: (Joint) Judging the judges: evaluating the quality and validity of expert assessments (Franklin 1&2)
    11. 6.11 2:00 – 3:30 Panel 5: (Joint) Issues and techniques of expert assessments (Garden Room)
    12. 6.12 3:30 – 4:00 Coffee/Tea Break (Foyer)
    13. 6.13 4:00 – 6:00 Plenary Roundtable: (Joint) Lessons for best practices in expert indicators (Rittenhouse I&II)
    14. 6.14 6:00 – 7:00 Drinks Reception, (Garden Room)
    15. 6.15 7:15 – 9:00 Dinner for invited participants

Logistics

WORKSHOP THEME: 'Expert indicators in the social sciences: Challenges of validity, reliability and legitimacy.'

FINAL PROGRAM WITH PLANNING SCHEDULE: You can download the final program, which contains details about the organization, theme and the planning schedule.  Printed copies will be available on registration at the meeting.

Lunchtime roundtable discussion notes are available for download 
 
WHERE: The Hilton Garden Inn,  1100 Arch St, Philadelphia, 19107 (Next to Reading Market)

WHEN: From 8.30am to 7pm on Wednesday 31 August 2016, immediately before the American Political Science Association Annual Meeting

SPONSORS: The event is co-sponsored by EIP,  the Varieties of Democracy project (V-DEM), and the APSA Elections, Public Opinion, and Voting Behavior (EPOVB) organized section.

WORKSHOP CHAIRS: Pippa Norris (EIP, Harvard and Sydney University) and Svend-Erik Skanning (V-DEM, Aarhus University)

WORKSHOP COORDINATOR: Alexandra Kennett (EIP Project Administrator, Sydney University)

REGISTER: Register to reserve your place and specify your meal requirements, if you have not already done do, via the online form.


Summary Schedule

Participants will be provided with refreshments, lunch and an evening drinks reception. In addition, paper-givers,  chairs and discussants will be invited to a local dinner after the event from 7pm to 9.30pm.
 

8.30-9.00

Arrival, registration and continental breakfast

Foyer

9.00-9.30

Welcome

Rittenhouse I&II

9.30-11.00

Panel 1: (EIP) Assessing and improving electoral integrity

Rittenhouse I&II

9.30-11.00

 (V-Dem) Closed organizational meeting

Executive Boardroom

11.00-11.30

Coffee/tea Break

Foyer

11.30-1.00 

Panel 2: (EIP) The rules of the game: electoral systems and integrity

Franklin 1&2

11:30-1:00

Panel 3: (EIP) Information, communication and legitimacy

Garden Room

11.30-1.00

(V-Dem) Closed organizational meeting

Executive Boardroom

1.00-2.00

Lunch and Workgroups 5: (Joint) Best practices for the use of expert indicators

Rittenhouse I&II

2.00-3.30

Panel 4: (Joint) Judging the judges: evaluating the quality and validity of expert assessments

Franklin 1&2

2.00-3.30

Panel 5: (Joint) Issues and techniques of expert assessments

Garden Room

3.30-4.00

Coffee/tea Break

Foyer

4.00-6.00

Plenary session: (Joint) Lessons for best practices in expert indicators

Rittenhouse I&II

6.00-7.00

Drinks reception

Garden Room

7.30-9.00

Dinner for invited participants and discussants

Barbuzzo Medittanean Kitchen & Bar

Theme: 'Expert indicators in the social sciences: Challenges of validity, reliability and legitimacy.'

Svend-Erik Skaaning and Pippa Norris  

Expert surveys have become increasingly common in comparative social science, in risk analysis by private sector organizations, in evaluation research, and among NGOs and policy makers (Meyer & Booker 1991). Expert surveys are often used to deepen understanding of, among others, the left-right position of political parties and news media outlets, the  perceived extent of corruption or bribe-paying, and the quality of democratic governance. Expert surveys increasingly supplement alternative sources of information, such as citizen mass surveys, event analysis of media reports, and official statistics.

This data collection technique has been applied to diverse research topics such as the series of studies on party and policy positioning (Laver and Hunt 1992; Huber and Inglehart 1995; Saiegh 2009; Laver, Benoit, and Sauger 2006; McElroy and Benoit, 2007), the power of prime ministers (O’Malley 2007), evaluations of electoral systems (Bowler, Farrell, and Pettitt 2005); policy constraints horizons (Warwick 2005); campaign communications (Lileker, Steta and Tencher 2015); human rights and democracy (Landman and Carvalho 2010), and the quality of public administration (Teorell, Dahlstrom and Dahlberg 2011). Expert surveys have been widely used in research on corruption - the Corruption Perceptions Index (Transparency International 2013; Global Integrity); measuring democracy since the 1900s -Varieties of Democracy (Coppedge et al. 2011)- and electoral integrity (Norris, 2014; Norris, 2015; Martinez i Coma and Van Ham, 2015).  The World Bank Institute Good Governance indicators combine an extensive range of expert perceptual surveys drawn from the public and private sectors. Indeed among the mainstream indicators of democracy, Freedom House’s estimates of political rights and civil liberties, Polity IV’s classification of autocracies and democracies, and the Economist Intelligence Unit’s estimates of democracy are all, in different ways, dependent upon expert judgments.

Expert surveys seem especially useful for measuring complex concepts that require expert knowledge and evaluative judgments; and for measuring phenomena for which alternative sources of information are scarce (Schedler 2012). Yet, expert surveys are not risk free and scholars have pointed out their limitations (Budge, 2000; Mair 2001; Steenbergen and Marks, 2007). Moreover, in contrast to mass social surveys, we still lack a common methodology to construct such surveys, as well as agreed technical standards and codes of good practice. There has been heated debate about the pros and cons of methods used to evaluate the spatial positions of party policies, and about the use of governance indicators more generally, but by contrast there has been remarkably little discussion about the challenges of validity, reliability, and legitimacy facing the construction of expert perceptual surveys. Yet it is critical to consider these issues given the lack of a clear conceptualization and sampling universe of ‘experts’, contrasting selection procedures and reliance upon domestic and international experts, variations in the number of respondents and publication of confidence intervals, and lack of consistent standards in levels of transparency and the provision of technical information. Moreover, more research needs to be done on how to evaluate the consequences of expert and context heterogeneity on the validity of expert judgments (Martinez i Coma and van Ham 2015), for example by using item response models to test and correct for expert heterogeneity (Pemstein et al. 2015), and using techniques such as ‘anchoring vignettes’ (King & Wand 2007) or ‘bridge coders’ (V-Dem) to test and correct for context heterogeneity.

This workshop features sessions and papers covering the following topics: 

  1. Use expert surveys in comparative politics and related social sciences to develop new ways of testing the validity of expert survey data; 
  2. Use meta-analysis to compare expert perceptual surveys standards, methods, and techniques in terms of validity, reliability and legitimacy to other sources of information; or 
  3. Propose codes of conduct and good practice for conducting expert surveys.

References

Benoit, Ken and Michael Laver. 2005. Party Policy in Modern Democracies. London: Routledge.

Bowler, Shaun; David Farrel and Robin Pettitt. 2005. ‘Expert opinion on electoral systems: So which electoral system is best?’ Journal of Elections, Public Opinion and Parties 15(1): 3-19.

Budge, Ian. (2000). ‘Expert judgments of party policy positions: Uses and limitations in political research.’ European Journal of Political Research 37(1): 103–113.

Transparency International. 2013. Corruption Perception Index. http://www.transparency.org/whatwedo/publication/cpi_2013

Huber, John and Inglehart, Ronald. 1995. ‘Expert Interpretations of Party Space and Party Locations in 42 Societies’, Party Politics 1:73-111.

King,G. &Wand, J. (2007). Comparing incomparable survey responses: Evaluating and selecting anchoring vignettes. Political Analysis 15(1): 46–66.

Landman, Todd and Edzia Carvalho. 2010. Measuring Human Rights. London: Routledge.

Laver, Michael and Ben Hunt, B. (1992). Party and Policy Competition. London: Routledge.

Laver, Michael, Kenneth Benoit, and Nicolas Sauger. 2006. ‘Policy Competition in the 2002 French Legislative and Presidential Elections.’ European Journal of Political Research 45: 667-697.

Lilleker, Darren., Stetka, V. and Tenscher, J., 2015. Towards hypermedia campaigning? Perceptions of new media's importance for campaigning by party strategists in comparative perspective. Information, Communication and Society, 18 (7), 747-765.

Mair, Peter. (2001). ‘Searching for the position of political actors: A review of approaches and a critical evaluation of expert surveys.’ In M. Laver (ed.), Estimating the policy positions of political actors. London: Routledge.

Martinez i Coma, Ferran; Van Ham, Carolien . 2015. ‘Can experts judge elections? Testing the validity of expert judgments for measuring election integrity.’ European Journal of Political Research 54 (2): 305-325.

McElroy, Gail and Kenneth Benoit. 2007. ‘Party groups and policy positions in the European Parliament.’ Party Politics 13:5-28. Meyer, M. & Booker, J. (1991). Eliciting and analyzing expert judgment: A practical guide. London: Academic Press.

O’Malley, Eoin. 2007. The Power of Prime Ministers: Results of an Expert Survey’ International Political Science Review 28(1):7-27.

Pemstein, D.; Tzelgov, E.; Wang, Y. 2015. “Evaluating and Improving Item Response Theory Models for Cross-National Expert Surveys”. Varieties of Democracy Institute: Working Paper Series No. 1.

Saiegh, Sebastian. 2009. ‘Recovering a basic space from elite surveys: Evidence from Latin America.’ Legislative Studies Quarterly 34(1):117-145.

Schedler, Andreas. 2012. ‘Judgment and Measurement in Political Science’ Perspectives on Politics 10(1):21-36.

Steenbergen, Marco R.  and Gary Marks. 2007.  ‘Evaluating expert judgments.’ European Journal of Political Research 46: 347–366.

Teorell, Jan. Carl Dahlström & Stefan Dahlberg. 2011. The QoG Expert Survey Dataset. University of Gothenburg: The Quality of Government Institute. http://www.qog.pol.gu.se

Warwick, Paul. 2005. ‘Do Policy Horizons Structure the Formation of Parliamentary Governments?: The Evidence from an Expert Survey’ American Journal of Political Science 49(2):373-387.

Maps: 

Getting to the Hilton Garden Inn, 1100 Arch St, Philadelphia 


The location of meeting rooms in the Hilton Garden Inn



Guidelines for participants

The workshop panels are designed to present written papers, which are roughly 8,000-10,000 words in length. Electronic copies of papers are available online for download from the workshop website.

Panel chairs are asked to keep speakers running to timer and the event on schedule. Speakers are asked to present each paper for no more than 10 minutes. Discussants are also each asked to offer comments for around 10 minutes. This allows the remainder of the time for interactive Q&A with all event participants. A laptop with PowerPoint projection facilities will be available in each room.

Please email your presentation to the Workshop Coordinator, alexandra.kennett@sydney.edu.au, at least 24 hours before the workshop or hand it to her on a stick when registering. We anticipate around 70 participants and attendees in total.

All registered participants will be offered morning and afternoon refreshments, a buffet lunch, and an early evening cocktail reception. After the end of the reception, we will walk over to the restaurant (3 minutes away) with invited participants for the workshop dinner.

After the workshop, all papers will be reviewed by the organizers for possible publication in either an edited book or a special issue of a journal. Please let us know if you have already placed your paper elsewhere so that you would not want to be considered for this process.

Travel grant recipients must email their University of Sydney Vendor Form, flight and accommodation invoices to alexandra.kennett@sydney.edu.au for reimbursement following the event.

Full Schedule

Papers will be posted here after 15 August 2016

8:30 – 9:00 Arrival, registration, and buffet continental breakfast (Foyer)


9:00 – 9:30  Welcome to the Workshop (Rittenhouse I&II)


Pippa Norris (Harvard and Sydney University)

Staffan Lindberg (V-DEM and University of Gothenburg)



9:30 – 11:00 Panel 1: (EIP) Assessing and improving electoral integrity (Rittenhouse I&II)


Chair: Alessandro Nai (University of Sydney)

Discussant: Jeffrey Karp (Exeter University)


1.1 Pippa Norris (Harvard and Sydney University)

The pragmatic case for electoral assistance: The impact of regional organizations on electoral reform


1.2 Josephine Boyle (Rider University), Michael Brogan (Rider University) and Frank Rusciano (Rider University)




1.3 David Carroll (The Carter Center) and Obehi Okojie (Georgetown University)

Assessing Electoral Integrity: Comparing Indicators and Developing an Overall Assessment Framework


1.4 Kirill Kalinin (University of Michigan)

Signaling Games of Election Fraud 
*** EIP/International IDEA essay competition winner 2016 ***
 


9:30 – 11:00 V-DEM) Closed Organizational Meeting (Executive Boardroom)


11:00 – 11:30 Coffee/Tea Break (Foyer)



11:30 – 1:00 Panel 2: (EIP) The rules of the game: electoral systems and integrity (Franklin 1&2)


Chair: Pippa Norris (Harvard and University of Sydney)

Discussant: Frank Rusciano (Rider University)


2.1 Masaaki Higashijima (Tohoku University) and Eric C. C. Chang (Michigan State University) 

The Choice of Electoral Systems in Dictatorship


2.2 Eleanor Hill (University of Manchester)

Biraderi, Political Machines and Postal Voting on Demand in Great Britain


2.3 Shane Singh (University of Georgia)

Compulsory Voting and Electoral Integrity


2.4 Víctor A. Hernandez-Huerta (University of the Andes)

Rejecting election results now to be elected in the future: evidence from sub-national elections in Mexico.


OR

11:30 – 1:00 Panel 3: Information, communication and legitimacy (Garden Room)


Chair: Holly-Ann Garnett (McGill University)

Discussant: Alessandro Nai (University of Sydney)


3.1 Mike Omilusi (Ekiti State University)

From Convenient Hibernation to Circumstantial Desperation: Hate Speech, Party Political Communication and Nigeria’s 2015 General Elections.


3.2 Sharon Lean (Wayne State University) and Matthew Lacouture (Wayne State University) (copies are available on request from the authors (sflean@wayne.edu)

Resources, repertoire and context: what determines the quality of domestic election monitoring?


3.3 Max Grömping (University of Sydney)

Explaining news attention to domestic election monitoring initiatives


3.4 Nicholas Kerr (University of Alabama) and Anna Lührmann (University of Gothenburg and V-DEM)

Public Trust in Manipulated Elections: The Role of Election Administration and Media



11:30 – 1:00 (V-DEM) Closed Organizational Meeting (Executive Boardroom)


1:00 – 2:00 Buffet Lunch and Workgroups: (Joint) Best practices for the use of expert indicators (Rittenhouse I&II)


Code of standards and ethical conduct governing best practices in mass surveys have been agreed by many market research companies, survey research organizations, and professional bodies. Members endorse these standards and agree to abide by them in their conduct. Yet similar conventions have not yet been adapted and agreed to promote best ethical practices in the generation of expert-based indicators and surveys.

In this breakout session, led by a moderator, groups at each table will be asked to discuss two questions: (i) In your experience, what are the most important challenges to maintain appropriate standards and practices when generating and using expert indicators? (ii) What codes of conduct would you suggest to address these challenges?

Each table group will select a rapporteur to report back the key points during the final 15-20 minutes.

Chair: Pippa Norris

Table discussion moderators

1. Sarah Lister (UNDP)

2. Gary Marks (UNC Chapel Hill)

3. Monty Marshall (Polity IV)

4. Annika Werner (Griffith University, MARPOR)

5. Sarah Repucci (Freedom House)

6. Sabine Donner (BTI Index)

7. Jennifer Dunha (Freedom House) 




2:00 – 3:00 Panel 4: (Joint) Judging the judges: evaluating the quality and validity of expert assessments (Franklin 1&2)


Chair: Gary Marks (UNC Chapel Hill)

Discussant: Sarah Repucci (Freedom House)


4.1 Kyle Marquardt (University of Gothenburg and V-DEM Institute), Daniel Pemstein (North Dakota State University), Brigitte Seim (UNC Chapel Hill) and Yi-ting Wang (National Cheng Kung University)



4.2 Kelly McMann (Case Western Reserve), Brigitte Seim (UNC, Chapel Hill), Jan Teorell (Lund University), Daniel Pemstein (North Dakota State University)
and Staffan Lindberg (University of Gothenburg)




4.3 Jan Teorell (Lund University), Michael Coppedge (University of Notre Dame), Svend-Erik Skaaning (University of Gothenburg and V-DEM and Staffan I. Lindberg (University of Gothenburg and V-Dem)




OR

2:00 – 3:30 Panel 5: (Joint) Issues and techniques of expert assessments (Garden Room)

Chair: Svend-Erik Skaaning (V-DEM and Århus University)

Discussant: Annika Werner (Griffith University)

5.1 Daniel Pemstein (North Dakota State University), Brigitte Seim (UNC Chapel Hill) and Staffan Lindberg (University of Gothenburg)



5.2 Holly Ann Garnett (McGill University) 

Measuring the Quality of Election Management: Perceptions, Design and Capacity 


5.3 Carolien van Ham (University of New South Wales) and Ferran Martínez i Coma (University of Sydney)

Do experts judge elections differently in different contexts? The cross-national comparability of experts

To access a copy of the paper please email the authors.

3:30 – 4:00 Coffee/Tea Break (Foyer)



4:00 – 6:00 Plenary Roundtable: (Joint) Lessons for best practices in expert indicators (Rittenhouse I&II)


During this interactive Q&A session, we will return to the build upon the lunchtime discussion. Panelists will each be asked to discuss their experience of the challenges of generating standards and codes governing best practices in generating expert indicators designed to measure social phenomena.

The discussion will open with two questions where each panelists is asked to address their remarks for 4-5 minutes: (i) in your experience, what are the most important 2-3 challenges to standard and ethics when undertaking expert surveys? (ii) What best practices and codes of conduct would you suggest to address these challenges?

The Q&A, moderated by the chair, will pick up these points interactively. The panel aims to stimulate discussion and explore whether there is a consensus around some minimal best practices for generating expert-based indicators. We will conclude by considering how best to build agreement around these issues.

Chair: Pippa Norris (Harvard and University of Sydney)

§ Sarah Lister (UNDP Oslo Governance Center)

§ Gary Marks (University of North Carolina-Chapel Hill, Chapel-Hill Expert Survey)

§ Monty Marshall (Societal-Systems Research, Polity IV)

§ Annika Werner (Griffith University, Manifesto Project)


§ Sabine Donner (BTI Transformation Index)

§ Staffan I. Lindberg (V-Dem)


6:00 – 7:00 Drinks Reception, (Garden Room)


7:15 – 9:00 Dinner for invited participants

Barbuzzo Medittanean Kitchen & Bar, 110 South 13th Street, Philadelphia, 19107 (215-546 9300) http://www.barbuzzo.com/ (9-minutes’ walk from the Hilton)







Comments