Abstract

How can selection of innovation projects be structured to reduce false positives (investments that should not have been made) and false negatives (investments that should have been made but weren’t)? Our understanding of drivers of these errors in real organizations has been limited due to difficulties in collecting decision and outcome data on a complete set of proposed projects. We collected qualitative data on a mobile application accelerator to understand how it implemented three different selection regimes over time. Using data on all 3,580 submissions to the accelerator, we then collected the outcomes for all funded and rejected projects to measure false positives and false negatives at the project-level before empirically evaluating the effectiveness of the three selection regimes in avoiding both types of selection errors. We find limited effects of changing selection regimes on the likelihood of selection errors. Our findings suggest that in the last regime, which was designed to increase the quality of submissions through additional layers of screening, considering differences in the pools of projects submitted for selection, evaluators were more likely to make false positive and false negative decisions. Additional analyses suggest adverse selection and over-weighting of applicant track record as likely mechanisms.