MO 410
Designing for Collective Intelligence
Professor Scott E Page
Winter 2020
Ross
T TH 1:00 - 2:30
Professor Scott E Page
Winter 2020
Ross
T TH 1:00 - 2:30
In this course, we study collective intelligence. We learn what it is, how to measure it, and how to comprise teams, design institutions, and grow organizational cultures that achieve it. Collective intelligence takes two forms: superadditive and emergent. The former occurs when a group of people demonstrates an ability, capacity, or accuracy that surpasses its best (or average member) Achieving this type of collective intelligence requires combining, interrogating, and refining the skills and contributions of team members. Examples include the wisdom of crowds phenomenon in which the average prediction from a group of forecasters is more accurate than any individual, diversity trumps ability results in which teams of diverse problem solvers find better solutions than any individual problem solver, and crowdsourcing innovations which tap into large populations. Increasingly, collective intelligence combines human intelligence with artificial intelligence. By combining their internal Sherlock Holmes’ with Watson, organizations can become what Thomas Malone calls Superminds.
Collective intelligence of the second type, emergent collective intelligence arises when the collective exhibits functionalities not present in the repertoires of its members. Emergent collective intelligence arises when a crew guides a ship into port, a surgical team performs an operation, a soccer team executes an elaborate offense, or when the members of a jazz quartet respond and adjust to one another to produce a produces a collective sound. Emergent collective intelligence can also be found in the natural world. It arises a network of neurons and axons produces cognition, in colonies of bees and ants, and when diverse species manage an ecosystem.
Central to our analysis will be how organizational structures and culture facilitate both types of collective intelligence. We will learn how collective intelligence the necessity of diversity, interdependence, and connectedness. Achieving the right balance among the three depends on choosing the appropriated institutional structure and building the right culture. We will cover five types of institutional structures: self organization, markets, organizations, democracies and algorithms. Examples of the last include Google’s PageRank and traffic routing programs such as Waze.
The concept of collective intelligence can be found in a variety of disciplines including economics, organizational theory, political science, and ecology.
Economics: The peculiar character of the problem of a rational economic order is determined precisely by the fact that the knowledge of the circumstances of which we must make use never exists in concentrated or integrated form but solely as the dispersed bits of incomplete and frequently contradictory knowledge which all the separate individuals possess.
- Friedrich Hayek
”The Use of Knowledge in Society”, American Economic Review, September 1945
Organizational Theory: All of these, and others, fit the general rubric of ‘bounded rational- ity’, and it is now clear that the elaborate organizations that human beings have constructed in the modern world to carry out the work of production and government can only be under- stood as machinery for coping with the limits of man?s abilities to comprehend and compute in the face of complexity and uncertainty.
- Herbert Simon
Nobel Prize Lecture, Stockholm 1978
Politics: There are good theoretical reasons to believe that when it comes to epistemic reliability, under some reasonable assumptions, the rule of the many is likely to outperform any version of the rule of the few, at least if we assume that politics is akin to a complex and long enough maze, the knowledge of which cannot reside with any individual in particular or even just a few of them.
-Helene Landemore
Democratic Reason: Politics, Collective Intelligence, and the Rule of the Many, Princeton University Press, 2013
Ecology: Many animal groups regulate their movement without a leader, such as bird flocks that turn in the sky, or fish schools that swerve to avoid predators. Social insects live in colonies, and simple cues, mostly chemical, regulate how colonies forage, maintain their nests, and reproduce.
-Deborah Gordon
”The Ecology of Collective Behavior”, PLOS: Biology 2014
These four accounts reveal the building blocks of collective intelligence: diversity, interdepen- dence, connectedness, and adaptation. First, collective intelligence relies on diversity. Col- lectively intelligent groups of people (or birds) differ in their information, knowledge, or skills. Sometimes treating each member of the collective as equal produces the best outcomes. In other cases, differential weighting and influence by either ability or relative diversity improve the collective?s performance.
Second, interdependence is necessary for the whole to be more than the parts. Creating positive interactions requires task appropriate depth of communication and coordination. The depth of dialogue required to intersect information sets when collectively identifying the truth requires deep communication. Averaging numerical predictions requires almost none.
Third, collective intelligence requires a network structure that balances exploration and ex- ploitation. A network can be too connected and produce over exploitation, or not connected enough and result in too much exploration. Finally, strategic and adaptive behavior by individuals can amplify collective intelligence and they can also undermine it if they reduce relevant diversity.
Thomas Malone and Michael Bernstien (2015) Handbook of Collective Intelligence, MIT Press.
Mulgan, Geoff (2017) Big Mind: How Collective Intelligence Can Change Our World Princeton University Press
Course Requirements: In addition to exams, students will be asked to participate in class discussions and group exercises, design organizational structures and classroom exercises.
Grades: Grades will be based on class participation (10%) in class exercises (30%), an organizational design project (20%), a midterm (20%), and a final exam (20%).
Participation: This will be measured by the quality of student comments. All students will be expected to provide constructive and generative insights and feedback when asked and also to offer ideas that contribute to the intellectual environment.
In Class Exercises: A portion of the in class exercises will include evaluations. Some exercises will take the form of a competition and the individuals and grades will be determined by score. Others will be collaborative. For some of those exercises, students will be asked to evaluate the exercise, and discuss the implications and learnings.
Organizational Design Project: In the final two weeks of the course, students be assigned to groups and apply the ideas, tools, and measures, that we have developed in class to evaluate an organization’s ability to produce collective intelligence. The group will then propose changes to the organizational design to improve collective intelligence. The proposed changes may, but need not, include combining human and artificial intelligence.
Midterm Examination: The midterm will test basic knowledge of how collective intelligence arises specific tasks such as problem solving, predicting, and brainstorming. The exam will also cover how institutions facilitate and frustrate the achievement of collective intelligence.
Final Examination: The final exam will demand more nuanced application of the ideas, tools, and insights developed in the class. Students will be expected to analyze the performance of institutions, i.e does the the stock market produce accurate valuations?, and to defend or critique existing institutional choices, such as the use of voting to decide on guilt or innocence.
1. The Wisdom of Crowds and The Diversity Prediction Theorem
We introduce the concept of the collective intelligence using the Diversity Prediction Theo- rem. Students will see how to apply decision making tools to make accurate estimates and learn how the accuracy of a forecasting team depends in equal measure on average ability and collective diversity.
In Class Exercise: Prediction Contest
Readings: Page, Scott E, The Difference Chapter 7. Suroweicki, James, The Wisdom of Crowds. Chapter 1.
2. Problem Solving and Heuristics
We learn what an heursitic is and the power of combining heuristics. Students participate in an exercise in which they first develop heuristics and then learn to communicate them effectively to others (4.5)
In Class Exercise: Traffic Jam and Shapes Reading: Page, Scott E, The Difference Chapter 8.
Video: What are Heuristics? https://www.youtube.com/watch?v=ZINodNt-33g
Technical Readings: Marcolino, L. , A. Jiang, M. Tambe (2013)“Multiagent team formation - Diversity beats strength’ Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI).
Kleinberg, Jon and Maithra Raghu (2015) “Team Performance with Test Scores” Cornell University School of Information.
Bachrach, Yoram, Thore Graepel, Gjergji Kasneci, Michal Kosinski, and Jurgen Van Gael (2012) “Crowd IQ - Aggregating Opinions to Boost Performance” AAMS proceedings
3. Collective Intelligence in Brainstorming
We introduce the concept of brainstorming. Students learn how to measure individual cre- ativity in the alternative uses tests and then learn Shapley Value as a way to measure an individual’s contribution to a group.
In Class Exercise: The Alternative Uses Test
Readings: Page, Scott E, The Model Thinker Chapter 9, The Diversity Bonus Chapter 2.
Video: The Alternative Uses Test https://www.youtube.com/watch?v=U6xXz2Namyw
Technical Readings: Guilford, J.P. (1971). The nature of human intelligence. New York: McGraw-Hill. Chapter 3.
4. Democracy: The Condorcet Jury Theorem
The Condorcet Jury Theorem underpins our use of juries as well as democracy writ large. Students learn the core concept and then engage in an in class design case to leverage insights from the theorem to improve
In Class Design Case: US Pharmocopeia: Redisigning Ballots Reading: Page, Scott E, The Model Thinker Chapter 1.
5. Algorithm: Random Forests
The Human Zillow Competition Project: Students will design a random forest procedure for a home pricing competition that will take place in the next class. In the competition, groups will be given fifty houses or apartments currently for sale in a major US City. They will have the class period to classify each property as “likely” to sell or “not likely” to sell. The time constraint will force reliance on a decentralized algorithm, such as a random forest. Teams will be scored on their collective accuracy and compared against a Zillow based threshold.
Video Random Forest - Fun and Easy Machine Learning https://www.youtube.com/watch?v=D ̇2LkhMJcfY
6. Organization + Algorithm:
The class will be divided into teams to develop a procedure to find the best educational video for a topic related to collective intelligence. Teams will then have communicate their procedure to another team who will carry out the procedure.
7. The Human Zillow Competition
In this class, the student groups will make their predictions.
8. Self Organization and Control
In this class students will learn Wolfram’s Four Classes of Outcomes: equilibrium, periodic, random, and complex. They will then learn a tool Lyapunov Functions that give one set of conditions for equilibria to occur. They will then use the concept of Lyaponuv functions to consider the problem of self organization in some domain – it could be a city, an amusement park, Mecca, or a college campus. Groups will be tasked with coming up with a mini case where the model applies.
In Class Exercise: The Self Organized BLANK
Reading: ‘How analytics enhance the guest experience at Walt Disney World” https:// higherlogicdownload.s3.amazonaws.com/INFORMS/7b76599e-e044-4a8a-b759-63aa43d67d19/ UploadedImages/Roundtable-revised ̇ORMS3902 ̇April2012.pdf
Technical Reading: Schelling, Thomas. 1978. Micromotives and Macrobehavior. New York: W. W. Norton. Chapter 4.
9. The Efficient Market (and Basketball) Hypothesis
The efficient market hypothesis is a core concept from finance. Here, we view it through the lens of collective intelligence and also consider the efficient basketball hypothesis – namely that scoring in basketball is random for similar reasons.
Readings: Lo, Andrew (2017) Adaptive Markets Chapter 1. Page, Scott E, (2018) The Model Thinker Chapter 12.
10. The Adaptive Market
What would a theory be without a contrary view? Here, we consider Andrew Lo’s critical take on the efficient market hypothesis. Students will be asked to take sides on the question of whether markets are efficient or not.
In Class Exercise: Students will participate market experiment that can, but does not neces-
sarily, produce bubbles. In the first experiment, an asset pays a dividend either $2 or nothing
with probability 1 each period for 40 periods. Thus, the asset should be worth K when there 2
are K periods left. In the second experiment, the probability that the asset pays a dividend will be unknown to the participants.
Reading: Lo, Andrew (2017) Adaptive Markets Chapter 9.
11. This Time it’s Different: The Madness of Crowds From Tulips to 2008
In some instances, markets appear to lack collective intelligence entirely. They become mad. In this class, we study the breakdown of collective intelligence
In Class Exercise: Information Cascades
Readings: “Tulip Mania: the classic story of a Dutch financial bubble is mostly wrong.” The
Conversation 2018
http://theconversation.com/tulip- mania- the- classic- story- of- a- dutch- financial- bubble- is- mostly- wrong- 91413
Technical Readings: Bikhchandani, S., Hirshleifer, D., and Welch, I. (1992), “A Theory of Fads, Fashion, Custom, and Cultural Change as Informational Cascades,” Journal of Political Economy, Volume 100, Issue 5, pp. pp. 992-1026.
Young-Jin Lee, Kartik Hosanagar, Yong Tan (2015) “Do I Follow My Friends or the Crowd? Information Cascades in Online Movie Ratings.” Management Science 61: 9.
Asch S (1951) Effects of group pressure upon the modification and distortion of judgment. In: Guetzkow H (ed) Groups, leadership and men. Pittsburgh: Carnegie Press.
12. The Firm: Decomposability
Firms provide a structure with which to create collective intelligence. In this class, we will compare firms to markets.
In Class Design Project Why Firms?
Readings: Davis, Jerry (2017) “Post-Corporate: the Disappearing Corporation in the New Economy” thirdway.org, Simon, Herbert (1991) “Organizations and Markets.” Journal of Economic Perspectives pp 25-44.
Technical Readings: Nickerson, Jackson and Todd Zenger (2004) “A Knowledge-based Theory of the Firm: The Problem-Solving Perspective, ” Organization Science 15: pp 617- 632.
13. Emergent Collective Intelligence: Bees and Ants
We turn to ecology to see how superorganisms such as bees and ants produce collective intelligence without a central planner.
Video: “Honey Bee Decision Making” https://www.youtube.com/watch?v=sX8B135Ypq8
Technical Reading: Seeley, T. S. Camazine, J. Sneyd (1991) “Collective decision-making in honey bees: how colonies choose among nectar sources.” Behavioral Ecology and Socio- biology 28 (4), 277-290
14. Bees, Lone Wolfs and High Performing Teams
We investigate in novel classroom experiment how bees communicate effectively using a waggle dance (4.1) Students will be divided into teams and will have to find the best solution to a problem. Each team will consist either of bees, who can only communicate by “waggle dances’, lone wolfs who can communicate their own actions and values but cannot coordinate explicitly with others, or be a high performing team which will be able to coordinate actions across members.
In Class Exercise: The Dance of the Bees
15. In Class Midterm
16: Democracy and Voting Rules
We return to democracy as an institutional mechanism for producing collective intelligence and explore the strengths and weaknesses of different voting rules.
Group Project: A Case of Democracy: Teams will find a case where an organization uses one voting rule and propose an alternative rule that would produce better outcomes.
Reading: The Mathematics of Voting http://web.math.princeton.edu/math ̇alive/Voting/Lab1.shtml
17. Designing Prediction: Superforecasting
We will study the work on Phil Tetlock on how to construct superforecasting teams.
Readings: “Philip Tetlock: Ten Commandments for Aspiring Superforecasters” Farnham Street.
https://fs.blog/2015/12/ten-commandments-for-superforecasters/
Technical Readings: Tetlock, P. and D Gardner (2015) Superforecasting: The Art and Science of Prediction Crown Books, Lamberson PJ and S.. Page SE (2011) “Optimal Fore- casting Groups.” Management Science. Mannes, A. E., Soll, J. B., and Larrick, R. P. (2014). “The wisdom of select crowds.” Journal of Personality and Social Psychology, 107, 276-299, Hong L, Page, and M. Riolo (2012)“Incentives, Information, and Emergent Collective Accuracy” Managerial and Decision Economics 33:5 pp 323 - 334.
18. A Case of Democracy Presentations
Students will present their cases for democratic mechanisms within organizations.
19. Collective Truth Telling
Students will learn how collective truthing can be seen as an inverse of creativity and that the same measures can be applied to determine individual and contextual contributions.
In Class Exercise: The Collaborative LSAT in which student work together to solve LSAT questions but communication is costly.
Reading: Page, Scott E, (2017) The Diversity Bonus Chapter 4.
19. Designing Problem Solving
Students will be given two problems, an example might be the Sunday New York Times Crossword and a cubic brain teaser. They will be given a fixed amount of time toe develop a team problem solving approach to the two problems and then compete to solve the two problems (5.3).
Readings: Page, Scott The Difference Chapter 10.
Technical Readings: Barkoczi, D. and Galesic, M. (2016). Social learning strategies modify the effect of network structure on group performance. Nature Communications, 13109. Mason and Watts, (2012) “Collaborative Learning in Networks?? PNAS, Lazer D. and A. Friedman, “The network structure of exploration and exploitation.” Administrative Science Quarterly, December 2007.
20. The Limits and Limitlessness of Crowdsourcing
We will discuss DARPA’s red balloon challenge, the polymath project, Innocentive, Threadless and a variety of other attempts at crowdsourcing (some successful some not).
Group Project: Teams will create a category within the large set of crowdsourcing attempts and then identify best practices and lessons from within that category.
Technical Readings: Lakhani, Karim R., and Lars Bo Jeppesen. (2007) “Getting Unusual Suspects to Solve R&D Puzzles.” Forethought. Harvard Business Review 85, no. 5. Jeppe- sen, Lars Bo, and Karim R. Lakhani. (2010) “Marginality and Problem-Solving Effectiveness in Broadcast Search.”Organization Science 21. pp 1016-1033. Lifshitz-Assaf, Hila (2013) “From Problem Solvers to Solution Seekers: The Co-evolving Knowledge boundary and Pro- fessional identity work of R&D organizational members at NASA ” Working Paper, Harvard University. Howe, Jeff (2008), Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business, Random House, New York
21. Collaborative Filtering
Students will learn the concept of collaborative filtering and learn to apply it to real world problems.
Reading: “Introduction to Recommender Systems” https://hackernoon.com/introduction-to-recom
22. Crowdsourcing Cases: Presentations
Students will present their group projects in which they apply collaborative filtering tools to crowdsourcing examples.
22. Artificial Intelligence
We will cover the basics of artificial intelligence and its impact on organizations. Reading: “AI Fueled Organizations.”
https://www2.deloitte.com/insights/us/en/focus/tech-trends/2019/driving-ai-potential-o html?id=us:2ps:3gl:confidence:eng:cons:11619:em:tt19:U9TbeUVx:1135060868:339149874941: b:Tech ̇Trends ̇Cognitive:AI ̇Fueled ̇Organizations ̇BMM:nb
Technical Reading: Agrawal, Ajay, Joshua Gans and Avi Goldfarb (2019)“Exploring the Impact of Artificial Intelligence: Prediction versus Judgment.” Information Economics and Policy (forthcoming).
23. Superminds: Artificial Intelligence + People
We will cover how A1 plus people affect the ability of markets, hierarchies, democracies, algorithms, and self organizing groups to achieve collective intelligence.
Reading: Scott E Page, “The Choice” mimeo. Malone, Thomas (2018) Superminds: The Surprising Power of People and Computers Thinking Together Little Brown and Company
Group Project Supermind Case Construction: Market, Hierarchy, Democracy, Algorithm, and Ecology
24. Big and Thin: AI + People
We cover cases of how to combine people and AI.
Cases: Public Services
Reading: Ang, Yuen Yuen (2019) “Integrating Big Data and Thick Data to Transform Public
Services Delivery” IBM Center for The Business of Government 13
26. Human Zillow Competition: Results
We compare and evaluate our predictions from the Human Zillow Competition.
27. Supermind Case Presentations
28. Supermind Case Presentations