Prospects for using hypothetical nudges to approximate real behavior change

Hypothetical scenarios provide an essential alternative to field experiments for scholars interested in nudging behavior change, comprising a substantial proportion of the literature. Yet the conditions under which hypotheticals more or less accurately estimate real-world treatment effects is not well understood. To investigate, we identified five recent field studies of real-world nudges in distinct domains and designed four styles of hypothetical scenarios to approximate each of those five studies. This setup allows clear comparison of old field data with new hypothetical data. Across our 20 pre-registered experiments (N=16,071, n>200 per cell), we find that hypothetical scenarios accurately estimated the direction of treatment effects, but varied widely in estimating the magnitudes of those effects. None of our four designs reliably reduced estimation error. Instead, hypotheticals appeared most calibrated when real-world treatment effects were extremely small, a promising direction for future study.