Poster Session II 

Time and venue: May 16th, 2:30 p.m. – 3:30 p.m., DDS Atrium.

Title: An Adaptive Algorithm for Hyperparameter Optimization with Application to Neural Network

Presenter: DuoDuo (Danny) Ying    

Abstract: Deep learning algorithms are increasingly popular for complex prediction and classification tasks, and hyperparameter configurations play an important role in algorithm performance. However, the best hyperparameter tuning strategy remains unresolved. We propose an adaptive algorithm to improve performance and reduce experimental cost. Our algorithm incorporates the Efficient Global Optimization procedure and intelligently adjusts the region of interest in each iteration. Our algorithm outperforms existing methods in a variety of test functions. It is applied to optimize the neural network hyperparameters for two popular machine learning datasets, MNIST and CIFAR.

Title: Designing Experiments Toward Shrinkage Estimation 

Presenter: Evan T. R. Rosenman     

Abstract: How can increasingly available observational data be used to improve the design of randomized controlled trials (RCTs)? We seek to design a prospective RCT, with the intent of using an Empirical Bayes estimator to shrink the causal estimates from our trial toward causal estimates obtained from an observational study. We ask: how might we design the experiment to better complement the observational study in this setting? We show that the risk of such shrinkage estimators can be computed efficiently via numerical integration. We then propose three algorithms for determining the best allocation of units to strata given the estimator's plannned use: Neyman allocation; a 'naive' design assuming no unmeasured confounding in the observational study; and a robust design accounting for the imperfect parameter estimates we would obtain from the observational study with unmeasured confounding. We propose guardrails on the designs, so that our experiment could be reasonably analyzed without shrinkage if desired.

Title: Bayesian Optimization with Noise-free Observations: Improved Regret Bounds via Random Exploration 

Presenter: Hwanwoo Kim  

Abstract: This paper studies Bayesian optimization with noise-free observations. We introduce new algorithms rooted in scattered data approximation that rely on a random exploration step to ensure that the fill-distance of query points decays at a near-optimal rate. Our algorithms retain the ease of implementation of the classical GP-UCB algorithm and satisfy cumulative regret bounds that nearly match those conjectured in arXiv:2002.05096, hence solving a COLT open problem. Furthermore, the new algorithms outperform GP-UCB and other popular Bayesian optimization strategies in several examples.

Title: A General Equivalence Theorem for Crossover Designs under Generalized Linear Models    

Presenter: Jeevan Jankar   

Abstract: With the help of Generalized Estimating Equations, we identify locally D-optimal crossover designs for generalized linear models. We adopt the variance of parameters of interest as the objective function, which is minimized using constrained optimization to obtain optimal crossover designs. In this case, the traditional general equivalence theorem could not be used directly to check the optimality of obtained designs. In this manuscript, we derive a corresponding general equivalence theorem for crossover designs under generalized linear models. 

Title: Designing Experiments for Operation of a Dynamic Cloth Media Wastewater Treatment Filter

Presenter: Madison De Boer

Abstract: Effective operation of wastewater treatment (WWT) processes is crucial to achieve high effluent water quality and low energy consumption, both of which have human and environmental impacts. In this study, we use response surface methodology (RSM) to optimize the operation of a cloth media primary clarifying filter. Machine learning algorithms, such as reinforcement learning, are becoming increasingly popular, but WWT processes are complex, and many facilities do not have the automation necessary to implement sophisticated control strategies. Thus, we test a traditional approach to optimization as stepping stone to using more advanced algorithms. In designing the experiments necessary for this study, the selection of input and output variables posed significant challenges due to the intricacies of the system. Ultimately, effluent total suspended solids (TSS) and the number of backwashes per hour of the filter were selected as the targets for optimization. These correspond to effluent water quality and energy consumption. RSM was used to select settings of the input variables, and the system was run for one day under each combination of setpoints. Extensive analysis of the resulting data showed that summarizing results over the day masked key input-output relationships, so hourly averages were calculated instead, and the new models based on hourly averages were consistent with our collaborators' expectations. This process of soliciting input and output features, collecting experimental data, analyzing the data to identify key relationships, and building models based on the appropriate temporal frequency illustrates the complexity of optimizing operation for even one step of a WWT train.

Title: Orthogonalized Moment Aberration for Multi-Stratum Factorial Designs   

Presenter: Ming-Chung Chang        

Abstract: Multi-stratum factorial designs are prevalent in industrial and agricultural applications. However, the experimental units in these designs are subject to multiple

error sources, posing a challenge in designing optimal multi-stratum factorial designs. This poster presentation introduces an orthogonalized moment aberration criterion for

multi-stratum factorial designs. This row-based criterion enables fast computation. Additionally, we establish that the minimum orthogonalized moment aberration

designs are optimal under the criterion proposed by Chang and Cheng [Ann. Stat. 46 (2018) 1779-1806].

Title: Training Design for Statistical Evaluation of Seismic Foundation Models

Presenter: Samuel Myren 

Abstract: Foundation models, which are massive models pretrained on big data through self-supervision that are quickly fine-tuned to accomplish different downstream tasks, have been honed though generative natural language models and are actively being developed for other scientific disciplines such as seismology. However, statistical evaluation of scientific FMs that considers multiple tasks, model interpretability, learning speed, and unbiased train/validation/test splits, presents a significant challenge. This research addresses these challenges for seismic FMs through analysis of deep learning phase picking models. We evaluate and compare four models, one trained using the Stanford Earthquake Dataset (STEAD) (Model 0a) and one trained using a subset of the STEAD data (Model 0b), one trained using the Italian seismic dataset for machine learning (INSTANCE) (Model 1), and one initialized with weights from Model 0b and fine-tuned using INSTANCE data (Model 2). We carefully curate the train/validation/test splits to limit geographical mixing to reduce leakage of signal characteristics between the train and test phases. We also evaluate the effect of the amount of training data used on performance. Beyond this, by grouping sets of the data based on geographic locations and training the model on randomly selected subsets as well as using model snapshots (i.e. ensembling), we obtain uncertainty in predictions and in summary metrics arising from aleatoric and epistemic uncertainty. We utilize multiple seismic test sets to evaluate model performance on in and out of distribution data. To explore interpretability, we use metrics like Brier score and calibration scores. Performing such a wholistic evaluation provides a strong statistical evaluation approach to be applied widely to scientific foundation models.

Title: Understanding Differential Evolution Hyperparameter Response Surface for Constructing Space-Filling Designs     

Presenter: Samuel Onyambu

Abstract: Differential Evolution (DE) is a powerful optimization algorithm inspired by the principles of natural evolution. It belongs to the class of evolutionary algorithms and has gained popularity for its simplicity, robustness, and effectiveness in solving complex optimization problems. We study how the hyperparameters affect the performance of a discrete DE algorithm introduced by Stokes, Wong and Xu (2023) for constructing space-filling designs. Using various designs, we aim at understanding the surface structure of the involved hyperparameters, the importance of each hyperparameter, and thereby provide a guideline on optimal parameter settings under different setups.

Title: Planning Reliability Assurance Tests for Autonomous Vehicles

Presenter: Simin Zheng      

Abstract: Artificial intelligence (AI) technology has become increasingly prevalent and transforms our everyday life. One important application of AI technology is the development of autonomous vehicles (AV). However, the reliability of an AV needs to be carefully demonstrated via an assurance test so that the product can be used with confidence in the field. To plan for an assurance test, one needs to determine how many AVs need to be tested for how many miles and the standard for passing the test. Existing research has made great efforts in developing reliability demonstration tests in the other fields of applications for product development and assessment. However, statistical methods have not been utilized in AV test planning. This paper aims to fill in this gap by developing statistical methods for planning AV reliability assurance tests based on recurrent events data. We explore the relationship between multiple criteria of interest in the context of planning AV reliability assurance tests. Specifically, we develop two test planning strategies based on homogeneous and non-homogeneous Poisson processes while balancing multiple objectives with the Pareto front approach. We also offer recommendations for practical use. The disengagement events data from the California Department of Motor Vehicles AV testing program is used to illustrate the proposed assurance test planning methods. Key Words: Bayesian Analysis; Recurrent Events; Multiple Objectives; Pareto Front Optimization; Reliability Growth Model; Weibull Model.

Title: d-QPSO: an R Package for Searching for Efficient Designs under Generalized Linear Models      

Presenter: Sloka Sudhin     

Abstract: Lukemire et. al. (2019, https://doi.org/10.1080/00401706.2018.1439405) developed the d-QPSO algorithm, a quantum-behaved particle swarm optimization technique for searching for D-optimal designs for experiments with outcomes modeled under logistic regressions. As a nature-inspired algorithm, d-QPSO mimics the behavior of a flock of birds as they scan an area for food, with each bird (“particle”) representing a possible solution, and the amount of food at any given position representing the quality of the corresponding solution. The d-QPSO approach is particularly useful due to its ability to search for designs for experiments with mixed discrete and continuous factors, as well as its ability to search for both approximate and exact designs. In this research, we are developing an R library optimizing the existing algorithm, extending its capabilities to generalized linear models (GLMs) beyond logistic regression, and creating thorough documentation, with the ultimate goal of releasing the library on CRAN for public access. This presentation will outline the essential properties of the d-QPSO approach, illustrate its ability to find designs under several experimental settings, and demonstrate the utilization of the corresponding R library.

Title: Generating Higher Resolution Sky Maps using a Deep Gaussian Process Poisson Model  

Presenter: Steven D. Barnett

Abstract: The Interstellar Boundary Explorer (IBEX) satellite was launched in 2008 in an effort to learn more about the heliosphere, which sits at the boundary between our solar system and interstellar space. IBEX detects energetic neutral atoms (ENAs) emanating from the heliosphere to create sky maps describing their rate of emission. These maps are used by physicists to inform their theoretical models about the heliosphere. However, the data collected by IBEX are both noisy and irregular. Multiple tools have been developed to smooth out this data to produce higher resolution sky maps. We propose a deep Gaussian process Poisson model for the rate of energetic neutral atoms (ENAs) emanating from the heliosphere. We believe our deep Gaussian process model constitutes a more cohesive model than those developed previously. Additionally, deep Gaussian processes have shown a greater ability to learn complex surfaces, while maintaining a simpler covariance function. We have developed a Markov chain Monte Carlo algorithm utilizing elliptical slice sampling and the Vecchia approximation to learn the underlying latent deep Gaussian process.

Title: A Sequential Approach to Obtain Optimal Designs for Non-linear Models using Closed-form Solutions for Faster Convergence

Presenter: Suvrojit Ghosh 

Abstract: Nonlinear experiments involve response and regressors that are connected through a nonlinear model, where the Fisher information associated is generally a complex nonlinear function of the unknown parameter of interest. It brings us to a complicated situation since designing an efficient experiment will require knowledge of the parameter, but the purpose of the experiment is to generate data to yield parameter estimates. The Chaudhuri-Mykland procedure have addressed this problem by choosing the design points sequentially, optimizing a D-optimality criterion and using parameter estimates based on available data, following by updating the parameter estimates. To avoid the optimization of the D-optimality criterion for obtaining the next design point at every step, we introduce a new method where we aim at plugging in the available parameter estimates to existing closed form solutions for several non-linear models, provided the solutions exist. This will save computational time and cost and take advantage of many novel closed-form solutions derived by researchers for specific models that have not been utilized to their fullest potential. Two sets of numerical experiments have been conducted—one involving the standard nonlinear regression model and the other involving the Generalized Linear Model (GLM). The latter constitutes a factorial experiment with two levels and a binary response. The purpose is to explore the utilization of closed-form solutions, aiming to compare our procedure with the one proposed by Chaudhuri-Mykland, hence providing a more transparent understanding of their advantages in terms of computational time and efficiency.

Title: Active Learning for Screening Generalized Order of Addition Experiments        

Presenter: Xiaotian Zhang 

Abstract: Order-of-addition experiments have recently drawn much attention in scientific research, particularly in combinatorial drug studies. In several such experiments, practical considerations are required, including the fact that not all treatments can be administered to every subject and the time intervals between consecutive orders may vary. Additionally, the response may depend not only on the order in which treatments are administered, but also on the quantities of the factors (e.g., doses of different drugs). Traditional order-of-addition designs struggle with such complexity. In this paper, we develop a novel active learning procedure for optimization under various practical constraints. It can efficiently conduct variable screening, identify the optimal quantities of treatment variables, and determine the best timing of administration concurrently. Our method is also capable of handling both equality and inequality constraints, such as minimum time gaps between treatment applications. Numerical results demonstrate the superiority of our proposed approach.

Title: Local Transfer Learning Gaussian Process with Application to Surrogate Modeling and Analysis  

Presenter: Xinming Wang

Abstract: A critical bottleneck for scientific progress is the costly nature of computer simulations for complex systems. Surrogate models provide an appealing solution: such models are trained on simulator evaluations, then used to emulate and quantify uncertainty on the expensive simulator at unexplored inputs. In many applications, one often has available data on related systems. For example, in designing a new jet turbine, there may be existing studies on turbines with similar configurations or geometries. A key question is how to transfer information from such ``source'' systems for effective surrogate training on the ``target'' system. We thus propose a new LOcal transfer Learning Gaussian Process (LOL-GP) model, which leverages a carefully-designed Gaussian process to transfer such information for surrogate modeling. The key novelty of the LOL-GP is the use of a data-learned regularization model, which learns regions where transfer should be performed and when regions where it should be avoided. This ``local transfer'' property is desirable in scientific systems: for certain parameters, the systems may behave similarly and thus transfer is beneficial; for others, the systems may be considerably different and thus transfer is detrimental. In accounting for local transfer, the LOL-GP can remedy an important limitation of ``negative transfer'' in existing transfer learning models, where the transfer of information worsens predictive performance. We derive a Gibbs sampling algorithm for efficient posterior predictive sampling on the LOL-GP. We then show, via a suite of numerical experiments and an application to jet turbine design, the improved surrogate performance of the LOL-GP over the existing state-of-the-art.

Title: Robust and Efficient Assessment of Potency (REAP) as a quantitative tool for dose-response curve estimation

Presenter: Xinying Fang      

Abstract: The median-effect equation has been widely used to describe the dose-response relationship and identify compounds that activate or inhibit specific disease targets in contemporary drug discovery. However, the experimental data often contain extreme responses, which may significantly impair the estimation accuracy and impede valid quantitative assessment in the standard estimation procedure. To improve the quantitative estimation of the dose-response relationship, we introduce a novel approach based on robust beta regression. Substantive simulation studies under various scenarios demonstrate solid evidence that the proposed approach consistently provides robust estimation for the median-effect equation, particularly when there are extreme outcome observations. Moreover, simulation studies illustrate that the proposed approach also provides a narrower confidence interval, suggesting a higher power in statistical testing. Finally, to efficiently and conveniently perform common lab data analyses, we develop a freely accessible web-based analytic tool to facilitate the quantitative implementation of the proposed approach for the scientific community.

Title: Integer Programming for Maximin Order-of-addition Designs

Presenter: Yen-Chun Liu    

Abstract: The objective of solving an order-of-addition (OofA) problem is to identify the optimal sequence for a set of components, leading to an optimal value in a black-box function. General OofA problems can be divided into two stages, experiment designs and model fitting. In the aspect of constructing space-filling designs, existing studies mainly rely on heuristic algorithms based on component exchanges and are restricted permutation cases. As for model fitting, the prevailing focus is on determining the optimal permutation under the parametric model assumption for the black-box function. Furthermore, most existing OofA problems are geared toward low-dimensional scenarios, which may not be suitable for computer experiments where the data typically exhibit high dimensionality. In this work, we propose to utilize integer programming (IP) models to generate maximin OofA designs and expand the scope of OofA problems to include repetitive orders. The method is powerful in the sense that it requires no model assumption and benefits from the well-established optimization structure inherent in IP models. We demonstrate that by transforming OofA sequences into binary matrices, not only does it guarantee the optimality of maximin designs, but it also eliminates the need for exhaustive searching in subsequent active learning procedures. We present a robotic application to illustrate the practicality of our approach.