Tuan Ngo, Abid Hassan, Saad Shafiq, and Nenad Medvidović
University of Southern California, USA
Published in International Conference on Software Engineering (ICSE) 2026
Pre-print: https://arxiv.org/abs/2411.01074
Abstract
Deep Neural Networks (DNNs) tend to accrue technical debt and suffer from significant retraining costs when adapting to evolving requirements. Modularizing DNNs offers the promise of improving their reusability. Previous work has proposed techniques to decompose DNN models into modules both during and after training. However, these strategies yield several shortcomings, including significant weight overlaps and accuracy losses across modules, restricted focus on convolutional layers only, and added complexity and training time by introducing auxiliary masks to control modularity.
In this work, we propose MODA, an activation-driven modular training approach. MODA promotes inherent modularity within a DNN model by directly regulating the activation outputs of its layers based on three modular objectives: intra-class affinity, inter-class dispersion, and compactness. MODA is evaluated using three well-known DNN models and five datasets with varying sizes. This evaluation indicates that, compared to the existing state-of-the-art, using MODA yields several advantages: (1) MODA accomplishes modularization with 22% less training time; (2) the resultant modules generated by MODA comprise up to 24x fewer weights and 37x less weight overlap while (3) preserving the original model’s accuracy without additional fine-tuning; in module replacement scenarios, (4) MODA improves the accuracy of a target class by 12% on average while ensuring minimal impact on the accuracy of other classes.
MODA aims to decompose an n-class classification model into n corresponding modules, each containing a subset of weights extracted from the original DNN model that are necessary for predicting the respective class. MODA achieves this via two steps: (1) Activation-driven training: MODA aims to simultaneously reduce the model’s prediction errors while enhancing its modularity through the three principal objectives of intra-class affinity, inter-class dispersion, and compactness. During the training, in each layer, the distribution of activation outputs is gradually refined by promoting specialization where only a limited set of units (e.g., neurons) is involved in making predictions for any given class. (2) Structured decomposition: Once the modular model is trained, MODA identifies the layers’ units that are frequently activated while classifying samples of each class. These units, along with their associated weights, are extracted to form distinct modules.
RQ1: How well do the DNN modules decomposed by MODA perform in terms of module reuse and replacement?
RQ2: How effective is MODA in training modular DNNs?
RQ3: How do MODA’s underlying design choices impact its performance?
Summary: MODA can generate accurate yet minimal-sized reusable modules across model architectures and dataset scales. The composed models from MODA achieve comparable accuracy to the standard model across sub-tasks without retraining, while exhibiting significantly fewer weights than those from MwT.
*Note that reuse accuracy is considered as the primary metric for evaluating the effectiveness of DNN modularization, rather than the composed model size/overlaps/FLOPs. If the gap between reuse accuracy versus original model accuracy is significant, the benefits of the size and FLOPs diminishes.
*Note that INCITE, GradSplitter, and the retraining variant of MwT are excluded at ImageNet scale due to the prohibitively high computational cost.
Model: VGG16 - Dataset: SVHN
-----------------------------------
Model: VGG16 - Dataset: CIFAR10
-----------------------------------
Model: VGG16 - Dataset: CIFAR100
-----------------------------------
Model: VGG16 - Dataset: ImageNet100R
-----------------------------------
Model: VGG16 - Dataset: ImageNet100D
-----------------------------------
Model: ResNet18 - Dataset: SVHN
-----------------------------------
Model: ResNet18 - Dataset: CIFAR10
-----------------------------------
Model: ResNet18 - Dataset: CIFAR100
-----------------------------------
Model: ResNet18 - Dataset: ImageNet100R
-----------------------------------
Model: ResNet18 - Dataset: ImageNet100D
-----------------------------------
Model: MobileNet - Dataset: SVHN
-----------------------------------
Model: MobileNet - Dataset: CIFAR10
-----------------------------------
Model: MobileNet - Dataset: CIFAR100
-----------------------------------
Model: MobileNet - Dataset: ImageNet100R
-----------------------------------
Model: MobileNet - Dataset: ImageNet100D
-----------------------------------
Summary: In a series of module replacement experiments, MODA demonstrates an average improvement of 12.04% in accuracy for the target class, with an average increase of 3.52% the accuracy of other classes. In contrast, the previous replacement approach, CNNSplitter, shows a modest 0.91% improvement for target classes with the average increase of 1.99% in accuracy for other classes.
Summary: MODA can promote modularity in DNN models while achieving classification accuracy on par with standard training and outperforming the state-of-the-art MwT. Moreover, MODA demonstrates faster training time compared to MwT across all cases, with an average reduction of 22%, even though MwT only promotes modularity within convolutional layers.
Imact of modularization objectives: MODA−, a variant of MODA that includes only intra-class affinity and inter-class dispersion (but without compactness), produces modules with average sizes of only 33.4% and 30.2% in terms of average module size and overlap, while their reuse accuracy matches that of the standard model. Furthermore, incorporating the compactness loss function enables MODA to produce even lighter modules than MODA−, reducing size by 17.8% and weight overlap by 16.5%, with negligible impact on reuse accuracy.
Impact of hyper-parameters: MODA can be tailored to the desired goals by configuring its hyper-parameters reflecting intra-class affinity, inter-class dispersion, and compactness. While hyper-parameters of intra-class affinity and inter-class dispersion should be configured together, compactness has negligible influence on other losses, enabling independent control of its hyper-parameter.
Impact of alpha (weight factor for intra-class affinity loss)
Impact of beta (weight factor for inter-class dispersion loss)
Impact of gamma (weight factor for compactness loss)
Impact of tau (modularizing threshold)
*Source of all the figures above: https://github.com/dnn-modularization/MODA/tree/main/paper_figures
MODA: https://github.com/dnn-modularization/MODA
Forked MwT: https://github.com/dnn-modularization/forked_MwT
Forked GradSplitter: https://github.com/dnn-modularization/forked_GradSplitter
Forked INCITE: https://github.com/dnn-modularization/forked_INCITE