Session VI (May 16, 10:30am-12:00pm): Covering Arrays and Combinatorial Testing, organized by Ryan Lekivetz 

• Title: Systematic Training and Testing for Machine Learning Using Combinatorial Interaction Testing 

• Speaker: Laura Freeman, Virginia Tech

• Abstract: The systematic use of combinatorial coverage provides a defensible basis for selecting and characterizing test and training sets for machine learning models. This presentation adapts combinatorial interaction testing, which has been successfully leveraged in identifying faults in software testing, to characterize data used in machine learning. The several benchmark data sets demonstrate that combinatorial coverage can be used to select test sets that stress machine learning model performance, to select training sets that lead to robust model performance, and to select data for fine-tuning models to new domains. Thus, the results posit combinatorial coverage as a holistic approach to training and testing for machine learning. In contrast to prior work which has focused on the use of coverage in regard to the internal of neural networks, this talk considers coverage over simple features derived from inputs and outputs. Thus, this talk addresses the case where the supplier of test and training sets for machine learning models does not have intellectual property rights to the models themselves. Finally, the talk concludes with a framework for systematic inclusion and exclusion of features to identify the most likely features to cause differences in performance in ML and thus be included in coverage metrics.