From Code Changes to Quality Gains: An Empirical Study in Python ML Systems with PyQu

Table of Content

Abstract

In an era shaped by Generative Artificial Intelligence for code generation and the rising adoption of Python-based Machine Learning systems (MLS), software quality has emerged as a major concern. As these systems grow in complexity and importance, a key obstacle lies in understanding exactly how specific code changes affect overall quality—a shortfall aggravated by the lack of quality assessment tools and a clear mapping between ML systems code changes and their quality effects. Although prior work has explored code changes in MLS, they mostly stop on what the changes are, leaving a gap in our knowledge on the relationship between code changes and the MLS quality.

To address this gap, we conducted a large-scale empirical study of 3,340 open-source Python ML projects, encompassing more than 3.7 million commits and 2.7 trillion lines of code. We introduce PyQu, a novel tool that leverages low-level software metrics to identify quality-enhancing commits with average accuracy, precision, and recall of 0.84 and 0.85 of average F1 score. Using our tool and a thematic analysis, we identified 61 code changes, each demonstrating a direct impact on enhancing software quality, and we classified them into 13 categories based on contextual characteristics. 41% of the changes are newly discovered by our study and have not been identified by state-of-the-art Python changes detection tools. Our work offers a vital foundation for researchers, practitioners, educators, and tool developers, advancing the quest for automated quality assessment and best practices in Python-based ML software.

Research Methodology

RQ1 & RQ2: How effective is PyQu in detecting quality-enhancing commits in MLS? and How well does PyQu generalize to unseen code changes?

Explored hyperparameter for all Machine learning classifiers

The following table summarizes the hyperparameter configurations explored for each machine learning classifier used in our study. These configurations were systematically tested to identify the optimal settings for quality prediction across five software quality attributes. All models were evaluated using stratified train-test splits, with fixed random seeds for reproducibility. 

Optimal Classification Model and Hyperparameters per Quality Attribute

Best performance metrics for All quality attributes 

RQ3: What are the quality-enhancing code changes in MLS?

Quality-Enhancing Code Changes Taxonomy 

Labelled Dataset 

PyQu Source Code