Research

Forecasts for key variables in a variety of operational decisions may be generated by human experts or by artificial intelligence (AI) technologies. A decision maker can benefit from the distinct advantages that each source may offer by providing AI assistance to the experts, allowing them to augment the information contained in the AI forecast by incorporating their own knowledge about the uncertain target variable. When multiple experts are available, accuracy can be further improved by utilizing the wisdom of crowds, forming a consensus by averaging each of their AI- assisted forecasts. However, the potential accuracy of a crowd of AI-assisted forecasters may be limited by two structural characteristics. First, because the AI assistance is valuable to each expert at an individual level, the opinion of the AI can end up being overrepresented in the crowd’s consensus. Second, the experts may fail to appropriately utilize the AI assistance when forming their forecasts by underemphasizing the information it provides. Using a stylized Bayesian model of information aggregation and a two-step forecast elicitation procedure, we develop a prescriptive method that can recover the most accurate consensus forecast given all information collectively observed by the AI and every expert in the crowd. This procedure works by pivoting the crowd’s average AI-assisted forecast either toward or away from the crowd’s average initial forecast, elicited before the forecasters have observed the AI advice. We develop a robust, data-driven method to determine the size and direction of the pivoting adjustment, which takes into account both the crowd’s aggregate AI usage efficiency and the strength of the collective evidence supporting such an adjustment. We test the predictive performance of this method in three laboratory experiments. Across 3,188 participants and three AI accuracy treatments, we find that the pivoted forecasts match, and in many cases outperform, the AI-assisted crowd, the AI advice itself, and the unassisted crowd of forecasters. The proposed forecasting method also proves to be empirically robust—the procedure defaults to the unadjusted AI-assisted crowd forecast whenever there is insufficient evidence to warrant an adjustment.

Making accurate order quantity decisions for new products with uncertain demand is an important problem faced by inventory managers. When products are heterogeneous, algorithmic solutions to the newsvendor problem developed using feature information and historical demand data can provide useful recommendations to the manager. When such recommendations are interpretable, existing research shows that the quality of algorithm-assisted operational decisions improves significantly. We study a class of interpretable, decision tree-based prescriptive algorithms that organize products into transparent and operationally meaningful categories, and assign a unique order quantity to each group. Using the method of "honest" recursive partitioning, we propose an algorithm which minimizes the empirical newsvendor risk associated with the tree-based policy while accounting for adaptive overfitting to the observed data. We analyze the total regret associated to the honest tree algorithm, and characterize its complex and equivocal relationship with model complexity and sample size. We summarize this relationship by expressing the optimal model complexity as a function of training sample size, and show that the optimal degree of segmentation of the product space increases sublinearly with training data. These results highlight that hard-coding interpretability as an attribute of a prescriptive newsvendor model can enable human managers to make better judgmental adjustments to the algorithm's recommendation, with limited cost to the prescriptive performance of the algorithm. Finally, we demonstrate the application and performance of our honest tree algorithm using real-world sales data from two different product categories. 

We study how a decision maker should aggregate repeated forecasts when a crowd of forecasters are dynamically exposed to useful but partially overlapping information. We demonstrate that while the most recent predictions obtained from forecasters may be superior in terms of forecast accuracy, it does not necessarily aggregate the information efficiently. This issue arises because common and idiosyncratic information have different values at the individual and crowd levels, and individual forecasters maximizing their own accuracy may be acting in a fashion that is detrimental to the crowd's overall accuracy. We characterize the globally optimal aggregate, show how it can be recovered by dynamically pivoting across forecast histories, and study a one-shot setting in which the latent information structure can be inferred from the forecast panel itself without using realized outcomes or historical training data.