Mathematical Challenges from AI Ethics

We still need theories, techniques, tools and tradecraft

A letter published in SOTA Letters VII

Context

As AI systems become widespread in business, industry, government and education, concerns have rightly been raised over ethical issues: whether such systems meet expectations such as fairness, privacy, and respect for personal and property rights. Various sets of requirements have been proposed but there are considerable gaps between any such requirements and the theories, techniques, tools and tradecraft needed to meet ethical and related legal, regulatory and professional requirements (Krsto Pandza & Paul Ellwood, Strategic and ethical foundations for responsible innovation (2013) http://dx.doi.org/10.1016/j.respol.2013.02.007).

In summary then, as Yoshua Bengio wrote in 2016, quoting Daniel Lemire (https://www.facebook.com/yoshua.bengio/posts/835243206580621)

Theory lags practice

Furthermore, this lag are widening as AI systems are developed and implemented at pace in response to government and business ambitions. A recent report of the international group under the auspices of the UN (Governing AI for Humanity (2024) https://www.un.org/sites/un2.un.org/files/governing_ai_for_humanity_final_report_en.pdf) offers a starting point and sets out limits to current computer science.

Thesis

I believe it is therefore important and timely to identify areas where the mathematical sciences can contribute to underpinning ethical-by-design AI, support existing work in AI safety and security, and provide principled reasoning and quantification about the behaviour and interactions of existing systems. Conversely, reasoning and quantification are pre-eminently the domain of mathematics.

Furthermore, I believe that these challenges can and should be elicited by bringing the mathematical sciences community in the UK together with AI developers and practioners and with those working in ethics and regulation. Such activity would benefit the development and application of AI and provide the mathematical sciences with a welcome fresh perspective on problems and theory.

What I don’t want to do in this Letter is to discuss precisely what is or is not ethical for any given AI implementation and application. Such matters have been amply and ably discussed elsewhere. I’m concerned with what a developer or implementer can actually do when confronted with such requirements as “Explain why this system chose person A rather than person B for a loan / parole / keyhole surgery / insurance / …”; “Show that this system does not use a protected characteristic characteristic in making decisions, directly or by proxy”; “Show that this system does not make use of this item of personal data / intellectual property from this system or else remove it”; “Explain why you are confident in giving personal / professional endorsement to this output”. It’s my contention that in general the tools and techniques to meet these requirements do not exist.

Example

A challenge:

Consider a decision engine with a training data set T.  Let E denote the engine trained on the data set T; 

let E’ be the engine trained on the data set T’ consisting of T in reverse order.

Show that E and E’ are functionally the same.

Of course this is not as important as questions of practical governance – although I suggest that it is not entirely unimportant either. My point is that the theoretical understanding of a decision engine required to answer this challenge is a prerequisite for making principled claims about the practical behaviour of such a system.

Some points that a successful answer would need to address:

• analysis of the optimisation algorithm implicit in training and the nature of the loss landscape;

• notions of distance between sets of weights;

• descriptions of functional behaviour, including notions of similarity;

• the relation between distance between weights and similarity of behaviours;

•   the effects of implementation in hardware arithmetic.
  
  

My experience presenting this challenge at a recent conference (doi:10.13140/RG.2.2.31688.64000 ) was that almost nobody understood the point I was making: in fact, most people who commented at all seemed to think that the question was trivial, and that the answer was obvious: although there was disagreement as to what this “obvious” answer was.

It was therefore surprising that a group at MIT (Arora et al, On the Impossibility of Retrain Equivalence in Machine Unlearning (2025) arxiv:2510.16629) proved that for a class of models, the statement is false: the order in which data is presented can affect the behaviour of a model during the unlearning function, the removal of the effect of a training data item (“the right to be forgotten”). It seems then that the answer was not at all obvious. It was perhaps even more surprising that another group described an unlearning technique independent of any access to the training data set (Güler et al, A Certified Unlearning Approach without Access to Source Data (2025) arXiv:2506.06486). The apparent contradiction arises from different assumptions about the nature of the data and different descriptions of the unlearning function. This illustrates clearly the need for the ethical implementor and user to have theories to express properties of data and implementations, and tools and techniques to validate any assumptions about those properties.

I posed this particular problem because it does not depend explicitly on assumptions about the data, but instead is about the behaviour of a class of non-linear optimisation problems and algorithms.

The art and science of reasoning and quantifying uncertainty around decisions on data with only “black-box” access to a decision engine is already part of the domain of data science and mathematical statistics – this is not the direction I wanted to take this particular discussion.

Benefits

The immediate attraction of principled reasoning and quantification is of course as an essential platform for ethical deployment of AI, on issues such as fairness, safety, security, privacy and sustainability. Beyond these we may expect improved efficiency and accuracy from a better theoretical understanding of the underlying optimisation problems; from quantification of information content; and ultimately radically new and more effective engines.

In the reverse direction, I note the unhelpful divisions within the mathematical sciences as practised in the UK (the unhelpful antagonism between “pure” and “applied”), and between mathematics and computer science, as described in two international reports to EPSRC (An International Review of UK Research in Mathematics (2004) https://ima.org.uk/1042/an-international-review-of-uk-research-in-mathematics/ ; International Review of Mathematical Sciences (2010) https://www.lms.ac.uk/sites/default/files/Mathematics/IRM%20draft%20report.pdf ). A novel and well-supported source of new challenges and new connexions will be of benefit to the intellectual health of research in mathematics.

The professional lens

The point is often made that doctors trust an X-ray machine or an MRI scanner to present a true and useful image without understanding the physics of radiation or the mathematics of the Radon transform: why should they then not equally accept an AI decision on the image presented? I think this is a fair point and the answer is that there is an eco-system supporting the medical use of the MRI scanner – a connected series of peer-reviewed scientific publications developing the science and supporting the technology, a community of professionals validating the use and a regulatory system based on that science prescribing the limits within which the technology is considered reliable. Overlaying these structures there is a social compact encouraging these interacting communities of practice to trust and to be trustworthy.

It is not too much to say that in most AI applications some if not all of the elements of this eco-system and social compact have yet to appear – perhaps most importantly in trust across the human/machine interface (Spiegelhalter Should we trust algorithms? (2020) Harvard Data Science Review 2.1 doi:10.1162/99608f92.cb91a35a). I suggest that professionals, especially in those areas where the professional code is regulated by law, will need to think carefully about whether they are fulfilling their obligations if they hand off decision-making to a technology that does not come with that support system.

What is to be done?

Much is of course being done, but it seems to me that the running is being made by people elucidating requirements rather than developing tools and techniques to meet them. I propose therefore that there should be an effort to actively bring the various communities together and work collaboratively to elicit a series of explicit mathematical challenges that can form the basis of a research project at the right scale. Success would include

• identifying research gaps and formulating sufficiently clear deep mathematical questions to lead into a formal research agenda (feeding into research projects and grant applications);

• identifying known results and techniques which have been sufficiently clearly articulated to stimulate development into standard techniques (feeding into curriculum design and standards development) and software tools (feeding into implementation and product);

• stimulating the development of a network of researchers, developers, practitioners and policy makers in this area to build on the progress made.
  
  

… by whom?

The landscape of the mathematical sciences community in the UK is complicated, not to say confusing – see the section below for some of the national resources. Knowledge Exchange is recognised explicitly at most universities and at national hubs.

I am proposing a purely informal network of people interested in bringing the right people together to take forward the process of using some plausible ethical requirement to develop mathematical research questions likely to be attractive to researchers and funders – not to carry out the work so much as to identify how and where it should be done.  This site is currently that network.

or email ai.ethics@chalcedon.co.uk to engage.

… where and when?

I’ll be saying all this and more at a KE Hub Working Group on AI in the Mathematical Sciences at Loughborough University on 26 February. See you there!

Resources

Readers who want to engage with the mathematical sciences community may find the resources page useful.

Acknowledgements

This Letter is based on work carried out by myself and Thomas King in drafting proposals for a workshop in this area, and I also acknowledge the contributions of the participants in a discussion session at the IMA AI Congress in Birmingham in September 2024.

Afterword

This Letter is my personal position. I don’t claim any special standing or expertise in this area – I am simply responding to what seems to me gaps sufficiently serious to be worth devoting some time and effort to remedying. If I’m wrong – that the theories, tool and techniques already exist, or that people are already uniting to fill these gaps – then I would be only to pleased to hear about it.

Richard Pinch

rgep@chalcedon.co.uk

23 January 2026