Data & AI Ethics

Abstract: Biomedical data practices increasingly rely on quantification and predictive modeling, often sidelining the lived, fluctuating experiences of chronically ill patients. This talk explores how such practices generate epistemic harm through survivor bias and the exclusion of bodily doubt. Drawing on auto-ethnographic reflection, I examine how chronic illness disrupts bodily certainty and exposes the limitations of biomedical epistemologies that privilege measurable data over subjective testimony. I argue that survivor bias in predictive systems constitutes a form of epistemic disadvantage (Goldstein 2022), systematically excluding symptoms that resist population-level categorization. This exclusion not only undermines patient agency but also perpetuates bodily doubt—an uncertainty sustained by the absence of data and the fragility of clinical interpretation. Building on Havi Carel’s (2013; 2015) concept of bodily doubt and Margaret Price’s (2015) bodymind lens, I explore how chronic illness challenges the rationalist assumptions of biomedical data ethics and calls for more integrative, interdisciplinary approaches to care. This work invites data ethicists to reconsider the epistemic foundations of medical AI and evidence-based care, advocating for models that account for uncertainty, testimony, and the ethical complexity of lived experience.

References

Carel, Havi. 2013. “Bodily Doubt.” Journal of Consciousness Studies 20 (7–8): 178–97.
Carel, Havi. 2025. “A Taste of Your Own Medicine: The Experience of a Transplant.” The Lancet 405 (10486): 1220–21. https://doi.org/10.1016/s0140-6736(25)00691-9.

Goldstein, Rena Beatrice. 2022. “Epistemic Disadvantage.” Philosophia 50 (January):861–1878. https://doi.org/10.1007/s11406-021-00465-w.

Price, Margaret. 2015. “The Bodymind Problem and the Possibilities of Pain.” Hypatia 30 (1): 268–84. https://doi.org/10.1111/hypa.12127.


Rena Beatrice Alcalay is a postdoctoral researcher at the Technical University of Munich’s School of Social Sciences and Technology. She is a member of the Philosophy of Open Science research group, funded by the European Research Council, which investigates the conditions necessary for fostering fair and equitable research practices across diverse contexts. Her expertise lies in social and applied epistemology, with a focus on the biomedical sciences. Her research also includes exegetical studies of Ludwig Wittgenstein’s works with a special emphasis on questions emerging from hinge epistemology and social psychology. Her primary interest is in clarifying the nature and potential normativity of hinges, shedding light on the phenomenon of groundless prejudices and stereotypes, and exploring their implications for medicine and education.

Abstract: The ascent of tech billionaires—and, depending on the market, soon trillionaires—signals more than a shift in global economic structures; it marks a transformation in the moral and cultural conditions under which democratic life is sustained. This contribution offers a communitarian critique of Big Tech’s influence, grounded in the philosophical frameworks of Charles Taylor, Michael Sandel, and virtue ethicist Shannon Vallor, and further supported by public goods theory and economic insights from Paul Samuelson and Joseph Stiglitz, with Elinor Ostrom’s work emphasizing the civic importance of collective stewardship. It contends that the challenge to democracy posed by concentrated digital power is not merely institutional, economic, or ethical, but a disruption of the very conditions for democratic citizenship.


Thomas Meier is a Science Manager at the Munich Center for Machine Learning at LMU Munich and a Lecturer in the Ethics of Technology, with a special focus on AI-related topics, Ethics, and Political Philosophy, at LMU Munich. He is an invited professor at the Universidad Iberoamericana in Mexico City. 
Kristina Khutsishvili is a postdoctoral researcher in AI Ethics and Public Sector Decision-Making in Connected Places at the University of Cambridge (Department of Engineering and AI@Cam). She previously served as a work package leader and researcher in the CommuniCity Horizon Europe project and worked at the University of Amsterdam.

Abstract: We present the Ethical Data Initiative (EDI), a non-partisan platform that draws on the history, philosophy, and social studies of science to foster open discussions on all forms of data-work. One of our key educational activities is the ‘data clinic’, an interactive problem-solving and teaching format designed to bridge theory and practice in the realm of data governance and ethics. Clinics bring together small groups of TUM students from across a range of disciplines to collaborate with partner organisations on real-world ethical challenges and decision-making processes. The goal is to empower participants, both students and partners, with actionable insights to reflect and handle complex issues such as data privacy, discrimination, transparency, and accountability, and thus foster responsible cultures of data work. Most recently, the EDI team and TUM Masters students worked with the CERTA Foundation of Rwanda on issues around transparency and fairness in AI use by fintech providers across Africa. 

Abstract: Predictive benchmarking, the evaluation of machine learning models based on predictive performance and competitive ranking, is a cornerstone of machine learning research and an increasingly prominent method for scientific inquiry. Yet, benchmark results in themselves only measure model performance relative to a holdout dataset and a concrete learning problem like ImageNet. Drawing substantial scientific inferences from the results, say about general tasks like image classification, requires additional, often implicit, assumptions about the learning problems, evaluation functions, and data distributions. In this paper, we make these assumptions explicit by adapting validity conditions from psychological measurement theory for the context of machine learning. We show how these assumptions can be examined and justified in practice in three case studies: ImageNet in computer vision, WeatherBench in meteorology, and Fragile Families in social science. Our framework clarifies how and when benchmark results support valid scientific claims, bringing into light predictive benchmarking as an independent epistemological practice.

Abstract: Data is a constitutive element of almost all software systems—not only of artificial intelligence. Yet, ethical discussions often isolate data from the broader software context in which it is embedded. In this talk, we argue that to create ethically informed and normatively desirable software artifacts, one must address the entire socio-technical system, including its data collection, processing, and analysis components, as well as its interfaces, purposes, and usage environments. Drawing on our forthcoming book Introduction to Ethical Software Development, we propose the concepts of structural and techno-generic values as epistemic tools to identify and analyze ethical issues in software development. Structural values refer to the organizational, procedural, and societal dimensions that shape software production, while techno-generic values capture the normative features inherent to digital technologies themselves, such as transparency, privacy, and controllability. By applying this dual perspective, data scientists and developers can better understand how ethical challenges emerge in the interplay between data, code, and context—and how to address them systematically throughout the development process.

Abstract: Integrating AI into critical domains such as healthcare holds immense promise. Nevertheless, significant challenges must be addressed to avoid harm, promote the well-being of individuals and societies, and ensure ethically sound and socially just technology development. Innovative approaches like Embedded Ethics, which refers to integrating ethics and social science into technology development based on interdisciplinary collaboration, are emerging to address issues of bias, transparency, misrepresentation, and more. In this workshop contribution, I will outline the Embedded Ethics and Social Science approach and its core principles. Then, drawing on practical experience from embedding ethics in AI-related healthcare consortia, I will discuss how a specific set of methods has proven helpful for embedding ethical and social science inquiry into technology development: stakeholder analyses, literature reviews, ethnographic approaches, peer-to-peer interviews, focus groups, interviews with affected groups and external stakeholders, bias analyses, workshops, and interdisciplinary results dissemination. I am looking forward to discussing how Embedded Ethics can be applied across diverse fields of machine learning with the workshop participants after the presentation.

Abstract: Ethical data practices are crucial in the field of health-related data, where data governance directly affects population well-being. „Health data justice“ is an approach to data governance that aims to redress the exclusions structurally marginalized communities face in the context of increasingly data-intensive medicine. Health data justice centers the perspectives of groups most at risk of data injustices and aims to translate them into actionable policy recommendations. Incorporating these different dimensions requires an interdisciplinary approach. Building from the ongoing work of the “ADJUST - Advancing Health Data Justice: A comparative study of health-related data governance in Canada, Germany, and the United Kingdom” project team, we explore how graphic art informed interdisciplinary research might be a bridge for approaching these concerns. Incorporating graphic art in the research process can support the translation of complex topics, finding a common language, stimulating reflection, and opening up a space for exchanging personal experiences. Drawing on case examples and preliminary results, we discuss how incorporating art-based methods enhances the research process within an interdisciplinary team and with external interlocutors.

Abstract: Data collection enables a wide range of (machine) learning (ML) approaches, with supervised learning having been the dominant paradigm for decades. At its core, supervised learning relies on labels or annotations, which provide contextual information and reflect what is judged to be represented in the collected data (e.g., text, images) as, e.g., a discrete class assignment. These labels not only guide how models are trained and evaluated but also shape the kinds of patterns they can reproduce and their ability to generalize. Consequently, the knowledge and perspectives embedded in labels are central to what an algorithm ultimately learns. In practice, annotations are often treated as a single, unambiguous “ground truth”. Yet, across many tasks, including those commonly perceived as objective, human annotations frequently diverge. While part of this variation can be attributed to noise or annotation error, much of it reflects meaningful differences in perspective, referred to as human label variation (HLV). Conventional practices for label aggregation, such as majority voting, however, may systematically exclude minority viewpoints, raising serious ethical concerns when ML models trained in this way are deployed for decision-making. Although advances such as the use of soft labels to encode uncertainty or disagreement may relax the assumption of a single ground truth, fundamental questions remain: How can we disentangle meaningful signal from noise in observed label variation? How should we conceptualize “truth” in annotation, given differing notions of reality? How can we best elicit and represent the complex knowledge that annotators bring to the task? And ultimately, are current ML models learning in ways that are fair and justifiable? Addressing these challenges requires an interdisciplinary perspective, drawing on fields such as philosophy, cognitive science, computational linguistics, and statistics

Abstract: Ethical understanding is a crucial part of data and AI literacy because it ensures that individuals not only comprehend statistical concepts but also apply them responsibly and with awareness of their societal implications. We raise the question of whether it is possible to use AI in self-learning to learn about AI ethics itself. At first, this may sound paradoxical. Yet we argue that such an approach can be a promising way to embed ethics by design into workflows: from data collection (e.g. consent, inclusivity, bias checks), through analysis (e.g. transparent methods, reproducibility), to communication (e.g. honest reporting, accessible explanations). This work introduces a self-learning tool designed to promote ethical reflection in statistical and AI-related contexts, fostering a comprehensive competency model that integrates (1) knowledge, (2) skills, and (3) attitudes and values [1], supporting learners and professionals in moving beyond technical expertise toward responsible practice. It was co-developed with the Education Working Group of the International Statistical Institute in Statistics to make ethical data and AI standards and declarations actionable and relatable. The tool encourages users to engage critically with ethical questions throughout the lifecycle of statistical and AI projects, emphasizing the principle of "asking why before how." [2]. By integrating real-world case studies, interactive scenarios, and documentation tools, it trains users to reflect critically, document decisions transparently, and include value sets in project design and analysis. This approach not only strengthens teaching in data ethics but also provides practical support for multidisciplinary teams navigating complex ethical landscapes and demonstrates how responsible data science can become teachable, actionable, and scalable across disciplines.

References

M. Piacentini, M. Barrett, V. Mansilla, D. Deardorff, und H.-W. Lee, Preparing our Youth for an Inclusive and Sustainable World: The OECD PISA Global Competence Framework. 2018.

J. Utts, „Enhancing Data Science Ethics Through Statistical Education and Practice“, Int. Stat. Rev., Bd. 89, Nr. 1, S. 1–17, 2021, doi: 10.1111/insr.12446.

Abstract: Under which conditions is it morally permissible to infer (unobserved) information about an individual based on observed information about this individual? At which point is the individual's privacy infringed, or its privacy rights violated? I will explore this question based on recent literature from the field of ethics and discuss potential ways to investigate this problem from a public acceptance perspective via empirical research.

Abstract: Trust in statistics depends on quality and integrity. Producers (e.g. statisticians and statistical institutions) are responsible for the quality of statistics. The integrity of statistics, however, depends on the framework conditions for official statistics, which are defined by political actors.

Trust is built on knowledge and experience, rather than blind faith. Trust in democratic institutions, such as public statistics, is characterised by a mutually reinforcing relationship between Wertschöpfung (value generation) and Wertschätzung (value appreciation). The task of public statistics is to create Wert (value) in the form of informational products and related services. Whether and how well this can be achieved depends on structural preconditions such as governance, budget and competencies. Users' appreciation of these products and services is influenced by concrete experiences, as well as by opinions, attitudes, values, and, not least, (statistical) literacy. For public statistics, danger arises when the conditions for Wertschöpfung (value generation) or the factors influencing Wertschätzung (value appreciation) are unfavourable. As with all mutually influential relationships, these factors can reinforce each other in both positive and negative ways. This provides a starting point for political forces seeking to undermine trustworthiness, equate good and bad information quality, and relativise credible facts.

Concerns have been raised about the fragility of scientific freedom and of conditions allowing work to be carried out professionally due to the numerous measures that have been taken by US authorities in recent months. In a supposedly mature democracy, this is all the more worrying. Preserving the integrity of statistics is one of the challenges in such situations. This is particularly alarming when coupled with policies that harm individuals and groups in terms of fair treatment, both nationally and internationally. Statistics underpin international scientific cooperation in fields ranging from vaccine innovation to climate research and sustainable development by providing accessible and shared sources of evidence. When their integrity is undermined, mistrust can spread across borders. An attack on statistics and science in one country can quickly become a political export, replicated elsewhere by actors with similar objectives.

Statisticians and data scientists must understand this relationship and familiarise themselves with the potential risks associated with their professional activities, whether in research or various application fields. To be considered trustworthy, they must adhere strictly to the principles of professional statistical ethics in their work. This is a prerequisite, as defined by Onara O'Neill. For professional ethics guidelines to be effective, they must be taken seriously in statistical practice and in the training of statisticians, and implemented through application-oriented measures.

However, to counteract or at least mitigate the threats to integrity posed by external actors, it is also necessary to organise the representation of interests across the field of statistics. In this regard, interaction between institutions with defined roles in national and international governance structures, and interest groups such as specific communities or NGOs, fulfils an important function. Experience has shown that establishing a positive data culture requires laying the groundwork early on and for the long term. A package of measures relating to empowerment (e.g. prevention and effective statistics training), evidence (e.g. the clear communication of statistics, public relations and certification) and enforcement (e.g. measures in the event of violations of ethical rules) should be implemented.