METaL SEMINAR

Ambrogio Maria Bernardelli

Optimazing AI: MILPs for discrete Neural Networks

Training neural networks is typically done using gradient-based methods, which often require large datasets, significant computational resources, and careful hyperparameter tuning. In this presentation, we explore an alternative approach based on Mixed-Integer Linear Programming (MILP) to train discrete neural networks exactly, particularly in low-data settings. The focus is on few-bit neural networks, including Binarized Neural Networks (BNNs), whose weights are restricted to +1 and −1, and Integer-Valued Neural Networks (INNs), whose weights lie within a limited integer range {−P, ..., P}. These models are especially attractive because of their lightweight architecture and their ability to run on low-power devices, where computations can be implemented using simple Boolean or integer operations. A new multiobjective ensemble method, called BeMi, is introduced. Instead of training a single network to distinguish all classes, the approach trains one network for each pair of classes and combines their predictions using a majority voting scheme. The training process simultaneously optimizes accuracy, robustness to small input perturbations, and sparsity, reducing the number of active weights in the network. Experimental results on the MNIST dataset show significant improvements over previous solver-based approaches. While earlier methods achieved an average accuracy of 51.1%, the proposed ensemble method reaches 68.4% accuracy when trained with 10 images per class and 81.8% accuracy with 40 images per class. At the same time, it removes up to 75.3% of the network connections, producing simpler and more efficient models.