How can deep neural networks perform so well on unseen examples? In this work package we study the fundamental building blocks of deep learning.

Participants

• PI:  Marelie Davel, North-West University, NITheCS Associate

• Stefan Lotz, SANSA (Associate)

• Katherine Malan, UNISA (Associate)

• Jonathan Shock, University of Cape Town (Associate)

• Terence van Zyl, University of Johannesburg (Associate)

• Anna Bosman, University of Pretoria

• Devon Jarvis, University of Witwatersrand

• Coenraad Mouton, University of Kiel, Germany

• Randle Rabie, North-West University

• Tian Theunissen, North-West University

Objective To understand and predict the ability of deep neural networks to perform well or poorly on unseen inputs.

Deep learning refers to a collection of machine learning tools, initially inspired by an early understanding of how the brain functions. These artificial neural networks have given rise to a plethora of complex, layered models able to process large amounts of data effectively. They are particularly effective in modelling large data sets of high-dimensional variables, interacting in complex ways, and form the driving force behind current AI innovations. While practically important, many open questions remain with regard to the underlying principles allowing these models to generalize to unseen inputs. Understanding these principles could lead to better models, less expensive training procedures, and a better understanding of where these models will perform well or poorly. 

Deep learning holds tremendous potential for generating solutions to issues that are important and relevant in the South African context. This work package develops researchers with the skills and expertise required to enable them to design and adapt deep learning techniques, in collaboration with the international research community. 

Activities and Methodology

Main activities: 

• Collaborative research

• Training of postgraduate students and postdoctoral fellows

• Hosting an annual ‘Deep Learning’ bootcamp supported via the North Node

• Participation in and organisation of relevant workshops and conferences

• Presenting results at high-impact venues

Building on existing research at collaborating institutions [1-5], this work package will focus on empirical approaches to investigating the characteristics of  these networks. They will be probed from two perspectives: the characteristics of the input-output mapping, and characteristics of the internal representations created by inner structures of the models. The latter means that, rather than studying a layer representation (sample activations at a layer) as a single high-dimensional vector, individual units are analysed individually or in combination. Analysis approaches include sub-predictor interaction, clustering of representations and information theoretic approaches to the flow of information among units. Such approaches will be used to study specific phenomena, including models with different generalisation abilities, the double descent phenomenon and the smoothness bias.

1. Davel et al., ‘DNNs as layers of cooperating classifiers’, AAAI 2020.

2. Theunissen et al. ‘The missing margin: How sample corruption affects distance to the boundary in ANNs’, SACAIR 2021.

3. Bosman, Engelbrecht & Helbig, ‘Empirical Loss Landscape Analysis of Neural Network Activation Functions, GECCO Companion Conference 2023.

4. Mouton et al., ‘Is fragmentation a useful complexity measure’, Sci4DL workshop, NeurIPS 2024.

5. Mouton, Theunissen & Davel, ‘Input margins can predict generalization too’, AAAI 2024.

Analysis of complex multi-dimensional data is a fundamental skill across many scientific fields in this age of data excess. In this work package we investigate new statistical methods for the analysis of complex systems represented by multi-dimensional, heterogeneous datasets.

Participants

• PI:  Priyanka Nagar (Stellenbosch University, NITheCS Associate)

• Andriette Bekker (University of Pretoria, NITheCS Associate)

• Antonio Punzo (University of Catania, Italy)

• Cristina Tortora (San Jose State University, United States)

In this work package, we develop advanced statistical frameworks to uncover hidden structures in complex data. By focusing on two core areas—directional statistics and model-based clustering—we tackle challenges posed by non-Euclidean and heterogeneous datasets, revealing patterns that standard methods cannot capture.

Objectives

The primary objective of this work package is to develop robust statistical frameworks for analysing complex data structures. Specifically, we aim to:

• Construct frameworks for directional data on non-Euclidean manifolds, enabling accurate representation of cyclic and directional characteristics.

• Advance model-based clustering techniques to address heterogeneity and anomalies in multi-dimensional datasets.

• Integrate these approaches to produce flexible, interpretable, and scalable methodologies applicable across diverse scientific and industrial domains.

Activities and Methodology 

This work package develops theoretical frameworks for modelling complex data structures. Due to the inherent complexity of nonlinear manifold datasets, our approach combines methodologies from two distinct subfields: directional statistics and model-based clustering.

In directional statistics, the focus is on data where observations represent directions or angles—a common occurrence in natural and physical sciences. These datasets lie on curved manifolds, making linear (traditional) techniques inadequate. We aim to construct statistical models that accurately capture the cyclic nature and directional characteristics, enabling robust inference for non-Euclidean data.

In model-based clustering, the emphasis is on heterogeneous, multi-dimensional datasets that often exhibit asymmetry, atypical, and missing values. We develop clustering techniques grounded in mixture models, which assume that observed data arises from a combination of underlying distributions. This approach yields interpretable cluster structures and robust parameter estimates, while also, in some cases, accounting for the uncertainty arising from incomplete data. By avoiding restrictive distributional assumptions and incorporating flexible modelling strategies, we aim to improve reliability and scalability in clustering complex systems.

Through these activities, the work package promotes methodological innovation, fosters research collaborations, and supports postgraduate training, ensuring that the resulting frameworks are applicable across diverse scientific and industrial domains.