Philip Tetlock

• Pavel Atanasov, Jens Witkowski, Barbara Mellers, Philip Tetlock (2024), Crowd prediction systems: Markets, polls, and elite forecasters, International Journal of Forecasting. Abstract

  What systems should we use to elicit and aggregate judgmental forecasts? Who should be asked to make such forecasts? We address these questions by assessing two widely-used crowd prediction systems: prediction markets and prediction polls. Our main test compares a prediction market against team-based prediction polls, using data from a large, multi-year forecasting competition. Each of these two systems uses inputs from either a large, sub-elite or a small, elite crowd. We find that small, elite crowds outperform larger ones, whereas the two systems are statistically tied. In addition to this main research question, we examine two complementary questions. First, we compare two market structures, continuous double auction (CDA) markets and logarithmic market scoring rule (LMSR) markets, and find that the LMSR market produces more accurate forecasts than the CDA market, especially on low-activity questions. Second, given the importance of elite forecasters, we compare the talent-spotting properties of the two systems, and find that markets and polls are equally effective at identifying elite forecasters. Overall, the performance benefits of “superforecasting” hold across systems. Managers should move towards identifying and deploying small, select crowds to maximize forecasting performance.

• Philip Tetlock, Christopher Karvetski, Ville Satopää, Kevin Chen (2024), Long-range subjective-probability forecasts of slow-motion variables in world politics: Exploring limits on expert judgment, Futures and Foresight Science. Abstract

  Skeptics see long-range geopolitical forecasting as quixotic. A more nuanced view is that although predictability tends to decline over time, its rate of descent is variable. The current study gives geopolitical forecasters a sporting chance by focusing on slow-motion variables with low base rates of change. Analyses of 5, 10 and 25-year cumulative-risk judgments made in 1988 and 1997 revealed: (a) Specialists beat generalists at predicting nuclear proliferation but not shifting nation-state boundaries; (b) Some counterfactual interventions—e.g., Iran gets the bomb before 2022—boosted experts’ edge but others—e.g., nuclear war before 2022—eliminated it; (c) accuracy fell faster on topics where expertise conferred no edge in shorter-range forecasts. To accelerate scientific progress, we propose adversarial collaborations in which clashing schools of thought strike Bayesian reputational bets on divisive issues and use Lakatosian scorecards to incentivize the honoring of bets.

• Ben Powell, Ville Satopää, Niall MacKay, Philip Tetlock (2024), Skew-Adjusted Extremized-Mean: A Simple Method for Identifying and Learning From Contrarian Minorities in Groups of Forecasters, Decision, 11 (1), pp. 173-193. Abstract

  Recent work in forecast aggregation has demonstrated that paying attention to contrarian minorities among larger groups of forecasters can improve aggregated probabilistic forecasts. In those papers, the minorities are identified using `meta-questions’ that ask forecasters about their forecasting abilities or those of others. In the current paper, we explain how contrarian minorities can be identified without the meta-questions by inspecting the skewness of the distribution of the forecasts. Inspired by this observation, we introduce a new forecast aggregation tool called Skew-Adjusted Extremized-Mean and demonstrate its superior predictive power on a large set of geopolitical and general knowledge forecasting data.

• Philip Tetlock and G. Mitchell, “Stretching the limits of science: Was the implicit-racism debate a ‘bridge too far’ for social psychology?”. In The Cambridge handbook of implicit bias and racism, edited by J. A. Krosnick, T. H. Stark & A. L. Scott, (Cambridge: Cambridge University Press, 2024), pp. 147-165
• Philip Tetlock, Christopher Karvetski, Ville Satopää, Kevin Chen (2023), Exploring the limits on Meliorism: A commentary on Tetlock et al, Futures and Foresight Science.
• Pavel Atanasov, J. Witkowski, Barbara Mellers, Philip Tetlock (Under Review), The person-situation debate revisited: Forecasting skill matters more than elicitation method.
• Philip Tetlock, Lu Yunzi, Barbara Mellers (2022), False Dichotomy Alert: Improving Subjective-Probability Estimates vs. Raising Awareness of Systemic Risk, International Journal of Forecasting.
• E. Karger, J.T Monrad, Barbara Mellers, Philip Tetlock (Under Review), Reciprocal scoring: A method for forecasting unanswerable questions.
• Ike Silver, Barbara Mellers, Philip Tetlock (2021), Wise teamwork: Collective confidence calibration predicts the effectiveness of group discussion, Journal of Experimental Social Psychology .
• Ville Satopää, Marat Salikhov, Philip Tetlock, Barbara Mellers (2021), Decomposing the Effects of Crowd-Wisdom Aggregators: The Bias-Information-Noise (BIN) Model, International Journal of Forecasting.
• All Research from Philip Tetlock »

Past Courses

• MGMT9330 - Psych Found MGMT

  This seminar-based course, with active discussion and analysis, is required of all first-year doctoral students in Management and open to other Penn students with instructor permission. The purpose of this course is to examine and understand basics in the theory and empirical research in the field of micro organizational behavior and to build an understanding of people's behavior in organizations and across organizations. The course covers a blend of classic and contemporary literature so that we can appreciate the prevailing theories and findings in various areas of organizational behavior. This course covers micro-organizational behavior, focused on topics such as influence/status, virtual teams, job design, organizational culture and socialization, identity in organizations and overall look on where the field of micro-organizational behavior is going.

• PPE2355 - Political Psychology

  This course will explore psychological approaches to understanding political beliefs, attitudes, and actions at the levels of both individual citizens and national leaders. It will also explore the possibility that psychological science itself is not immune to the political debates swirling around it. Specific topics will include: the workings of belief systems (and their power to shape what we "see"), cognitive biases (and their power to cause miscalculations), sacred values and their role in stabilizing belief systems and social interaction, personality and ideology (the linkages between the personal and the political), and clashing conceptions of morality and distributive and corrective justice (striking variations among people in what they consider to be fair). We shall also explore some topics that have sparked controversy in the psychological research literature and that tend to polarize opinion along political lines, including work on intelligence and unconscious bias.

• PSYC2355 - Political Psychology

  This course will explore psychological approaches to understanding political beliefs, attitudes, and actions at the levels of both individual citizens and national leaders. It will also explore the possibility that psychological science itself is not immune to the political debates swirling around it. Specific topics will include: the workings of belief systems (and their power to shape what we "see"), cognitive biases (and their power to cause miscalculations), sacred values and their role in stabilizing belief systems and social interaction, personality and ideology (the linkages between the personal and the political), and clashing conceptions of morality and distributive and corrective justice (striking variations among people in what they consider to be fair). We shall also explore some topics that have sparked controversy in the psychological research literature and that tend to polarize opinion along political lines, including work on intelligence and unconscious bias.

• PSYC4998 - Mentored Research

  Mentored research involving data collection. Students do independent empirical work under the supervision of a faculty member, leading to a written paper. Normally taken in the junior or senior year.

• PSYC4999 - Honors Mentored Research

  The Honors Program has been developed to recognize excellence in psychology among Penn undergraduates and to enhance skills related to psychological research. The 4998 credit signifies an Honors Independent Study, completed as part of the Honors Program. The honors program involves: (a) completing a year-long empirical research project in your senior year under the supervision of a faculty member (for a letter grade). This earns 2 cu's. (b) completing a second term of statistics (for a letter grade) before graduation. (c) participating in the year-long Senior Honors seminar (for a letter grade). This seminar is designed especially for Psychology Honors majors; this receives a total of 1 cu. (d) participating in the Undergraduate Psychology Research Fair in the Spring semester, at which honors students present a poster and give a 15-minute talk about their research. (e) a total of 15 cu's in psychology is required. Students will be selected to be part of the Honors Program in the Spring of their junior year (see application process online)

• PSYC9999 - Independent Study

  Individual Study and Research

• Fellow, American Philosophical Society, 2019
• Thomas Schelling Award, Kennedy School of Government, Harvard University, 2017
• Fellow, National Academy of Public Administration, 2016
• Fellow, American Academy of Political and Social Science, 2016
• Andrew Carnegie Fellow, 2015
• Fellow, American Academy of Arts and Sciences, 2009
• Harold Lasswell Award for Distinguished Scientific Contribution in the Field of Political Psychology, 2008
• Grawemeyer World Order Prize, 2007
• Woodrow Wilson Award for best book published on government, politics, or international affairs, 2006
• Robert E. Lane Award for best book in political psychology, American Political Science Association, 2006
• National Academy of Sciences Award for Behavioral Research Relevant to the Prevention of War, 1999
• MacArthur Fellow in International Security and Conflict Resolution, 1999
• Nevitt Sanford Award for Distinguished Professional Contributions to Political Psychology, International Society of Political Psychology, 1997
• Philip Converse Book Award for outstanding book in the field published five or more years ago, American Political Science Association (for co-authored book, Reasoning and choice: Explorations in political psychology, 1992
• Woodrow Wilson Book Award, American Political Science Association (co-recipient with P. Sniderman & R. Brody, for Reasoning and choice: Explorations in political psychology), 1992
• American Association for the Advancement of Science Prize for Behavioral Science Research, 1988
• Fellow of Division 8 of the American Psychological Association, 1987
• Fellow at the Center for Advanced Study in the Behavioral Sciences, 1987
• Erik H. Erikson Award of the International Society of Political Psychology, 1987
• MacArthur Fellow in International Security and Conflict Resolution, 1987
• Distinguished Scientific Award for Early Career Contribution to Social Psychology, American Psychological Association, 1986
• Canada Council Doctoral Fellowship, 1977
• Yale University Fellowship, 1976
• Governor-General’s Gold Medal, Award for Undergraduate Academic Excellence, 1975
• British Columbia Psychological Association Gold Medal, 1975

• The Psychology of Holiday Gift Giving: The Good, the Bad, the Irrational - 12/22/2011
• Don't Mention It: How 'Undiscussables' Can Undermine an Organiztion - 12/20/2011
• Research Goundup: Improving Intelligence Forecasts, Vertically Integrated Health Care, and 'Worrisome' Health Care Costs - 12/20/2011

Knowledge @ Wharton

• How Dynamic Electricity Pricing Can Improve Market Efficiency, Knowledge @ Wharton - 11/12/2024
• Is Algorithmic Management Too Controlling?, Knowledge @ Wharton - 11/12/2024
• The Hidden Cost of AI Energy Consumption, Knowledge @ Wharton - 11/12/2024
• Charitable and Political Giving | Pinar Yildirim, Knowledge @ Wharton - 11/12/2024
• The Future of Banking, Knowledge @ Wharton - 11/11/2024
• Election Statistics, Knowledge @ Wharton - 11/06/2024
• Robots Are Taking Over Low-skilled Jobs — and Changing Votes, Knowledge @ Wharton - 11/05/2024
• Crypto Politics: Liberals’ and Conservatives’ Confidence in Cryptocurrencies, Knowledge @ Wharton - 11/05/2024
• Leadership and Elections | Mike Useem, Knowledge @ Wharton - 11/05/2024
• Automation Is Harming Low-skilled Workers and Possibly Their Voting Preferences, Knowledge @ Wharton - 11/01/2024