DAE 2024 - Session VI

Session VI (May 16, 10:30am-12:00pm): Covering Arrays and Combinatorial Testing, organized by Ryan Lekivetz

Title: BayesFLo: Bayesian Fault Localization for Software Testing
Speaker: Irene Ji, Duke University
Abstract: Software testing is an essential step for robust software development. There has been promising recent work on applying combinatorial testing methods such as covering arrays for software testing, which can greatly reduce testing cost while increasing the chance of detecting failures. After such tests, however, a tester then has to use the obtained data to guide their search for potential root causes for bugs in the software. In this work, we propose a novel Bayesian fault localization method called "BayesFlo", yielding a principled and probabilistic ranking of suspicious input combinations for identifying the root causes of failures. Our model leverages recent developments on structured Bayesian prior specification and graph-theoretic algorithms to calculate the posterior root cause probabilities, which can then be used in risk-based sequential test case design.

Title: Systematic Training and Testing for Machine Learning Using Combinatorial Interaction Testing
Speaker: Laura Freeman, Virginia Tech
Abstract: The systematic use of combinatorial coverage provides a defensible basis for selecting and characterizing test and training sets for machine learning models. This presentation adapts combinatorial interaction testing, which has been successfully leveraged in identifying faults in software testing, to characterize data used in machine learning. The several benchmark data sets demonstrate that combinatorial coverage can be used to select test sets that stress machine learning model performance, to select training sets that lead to robust model performance, and to select data for fine-tuning models to new domains. Thus, the results posit combinatorial coverage as a holistic approach to training and testing for machine learning. In contrast to prior work which has focused on the use of coverage in regard to the internal of neural networks, this talk considers coverage over simple features derived from inputs and outputs. Thus, this talk addresses the case where the supplier of test and training sets for machine learning models does not have intellectual property rights to the models themselves. Finally, the talk concludes with a framework for systematic inclusion and exclusion of features to identify the most likely features to cause differences in performance in ML and thus be included in coverage metrics.

Title: Design-based Hyperparameter Optimization for Machine Learning Models
Speaker: Rakhi Singh, Binghamton University
Abstract: Hyperparameters significantly influence the learning process of any machine learning model and significantly impact the corresponding prediction and classification accuracies. Tuning hyperparameters is critical due to its pivotal role in enhancing model performance, but it often requires extensive manual tuning and experimentation. Existing methods for hyperparameter optimization, like grid search or Bayesian optimization, are effective but are usually very computationally expensive. A design-based approach for automating hyperparameter optimization for neural networks was proposed in Shi et al., 2023, NEJSDS. In this talk, I will first add to their comparisons by including covering arrays as designs and random forests and gradient-boosted trees as modeling methods. From these comparisons, we conclude that covering arrays provides excellent competition to the other compared designs. Finally, I will introduce new design selection criteria to find LASSO-based screening designs that usually perform even better than covering arrays.

Title: Design and Analysis of Repeated-Tuple Testing Experiments
Speaker: Frederick Kin Hing Phoa, Academia Sinica
Abstract: Testing experiments via covering array has been popular in hardware and software experiments due to its cost-efficiency to handle large number of inputs, but some tuple combinations among factor columns may measure only once and it makes the experiment vulnerable to outlying measurements caused by machine or human errors. In this work, we introduce a useful class of cost-efficient designs for multi-factor experiments. It provides guaranteed repeated measurements on all 𝑡-tuples from any 𝑡 factor columns where 𝑡≤3, and the number of minimum repetitions is adjusted by the experimenters. Given the number of factors of interest, it utilizes less amount of resources than an orthogonal array while its repeated measurement provides a resistance towards outliers that a covering array fails to achieve. We develop a systematic method to construct families of these designs with small runs under different number of factors and number of repetitions. We further provide a model for the analysis of testing experiments that uses our designs, showing its nice properties on the outlier resistance and reduction in variations.