AI Recommendation is Undermining Human Autonomy

A Policy Brief

 Dec 2025 (~20 Hrs)

AI Recommendations are Undermining Individual’s Human Autonomy

Executive Summary:

Modern AI systems often involve directly or indirectly recommending options for users. Many of these systems are built in opaque ways that encourage overreliance on the system’s output. This in turn undermines the user’s autonomy in multiple dimensions and has a strong potential to decrease mental and social wellbeing. While current frameworks excel in promoting transparency and explainability, there are several weak areas surrounding human autonomy. This brief works towards proposing the following recommendations for modern policy frameworks which seek to manage AI systems:

-           Frameworks should shift structure towards being centered around social disruptiveness instead of risk.

-           Tools must be developed and deployed for organizations to quickly and iteratively assess the impact of their tools on mental and social wellbeing.

Problem Scope:

The development of generalized AI tools is driving a broad push for personalized recommendations in almost every context, far exceeding what simple algorithms were capable of before. This rapid change has highlighted the negative impacts of recommending technologies. Current AI recommendation systems encourage overreliance on opaque suggestions, which removes human autonomy from users' lives.

In this brief, I specifically use human autonomy as a term referring to three dimensions of autonomy proposed by Prunkl in “Human Autonomy at Risk? An analysis of the challenges from AI”.

-           Agency: User’s ability to carry out the choices they desire

-           Authenticity: User's choices as being reflective of their true inner desires and motivations

-           Status: How the user’s desires are ranked in relevance compared to another system’s

Democratic societies foundationally rely on an individual's ability to act as autonomous agents, taking actions that are in line with internal beliefs, and seeing results from those actions. When recommendations lack diversity, individuals may experience adaptive preference formation, where their preferences are molded by the information that is widely available to them[1] (Prunkl, 2024). In addition, it has been shown that self-reported autonomy is correlated with emotional well-being, which AI recommendation systems might undermine (Ryan & Deci, 2017).

  [1] This is an especially prevalent problem when one recommendation algorithm serves an extremely large number of people. While my suggestions in this brief are meant to represent recommendation systems broadly, it is important to note that most recommendations are served by a few large companies, where even if their models have a small bias, millions of users experience the same bias.

Current Policy Options:

When reviewing current policy implementations, I choose to look at one option from three independent layers –

1.  Societal – EU AI Act

2.  Organizational – NIST AI Risk Management Framework (RMF)

3.  Technical & Design Implementation – IEEE 7000 Series Standards

– to get a sense of how popular options at different levels address human autonomy.

As (Prunkl, 2024) mentions, there is not a current concrete definition of autonomy used in policy, thus while reviewing frameworks I evaluated them based on three dimensions of autonomy discussed by Prunkl.

Starting at the highest level, the EU AI Act centers itself around classifying AI products and tools according to their risk levels. The act prohibits systems with unacceptable risk, places restrictions on high risk and general-purpose AI (GPAI) systems, and places extremely minor restrictions on “limited risk” systems.

Unacceptable risk systems in this act include “subliminal, manipulative, or deceptive techniques to distort behaviour and impair informed decision-making, causing significant harm” (Regulation 2024/1689). This categorization clearly relates strongly to human autonomy, which is an extremely positive step towards increasing awareness of harm towards decision making. However, this restriction towards systems which harm decision making uses vague language that is difficult to apply to specific systems (Noggle, 2018). In addition, the restriction only applies to systems that cause “significant harm”. While restricting highly harmful systems is certainly a high priority, this policy places small restrictions on autonomy-undermining systems that are deemed limited risk (only requiring to notify the user that they are interacting with AI), and no restrictions on low risk systems. Others have also noted that systems may exist that are not high risk but still have high social disruption over a large population (Marchiori et al., 2025), which the act does not account for. Outside of risk classification, the act does a good job of promoting human in the loop design for high risk systems requiring “their high risk AI system to allow deployers to implement human oversight.” (Regulation 2024/1689)

At the next level there exist frameworks like the NIST AI RMF which aim to provide corporations with broad guidelines to follow for managing systems. This framework specifically frames the validity and reliability of an AI system as being necessary for other characteristics of trustworthy systems to exist, including explainability, privacy-enhanced, fairness, etc. This strongly promotes user agency being built on reliable systems, instead of on systems which may inadvertently mislead users about what their options for action might be.

The MAP function helps companies address and establish humans’ status in relation to AI systems, suggesting strong documentation of a system’s knowledge limits, and how the output of the system can be utilized and overseen by humans. If implemented properly, this may help systems and humans recognize where trust should be placed in the context of usage, lessening overuse in potentially opaque scenarios. The MAP function also encourages us to consider “potential costs [... from] realized AI errors” (Dotan et al., 2024) which may encourage non-paternalistic design, where system does not automatically override human intention.

Throughout the RMF, there is heavy emphasis on ensuring transparent and explainable systems, focusing on ensuring that this layer of interaction with the user is meaningful and relevant in the given context. They assert that developers of AI systems should work closely with deployers to ensure these transparency tools are relevant for their users and used as intended.

While the RMF does a great job of addressing specific issues in AI systems, its voluntary structure naturally positions it to think in terms of risk from the perspective of a company rather than defining specific situations where certain restrictions should be imposed, citing that several factors should be “based on the organization’s risk tolerance” (Dotan et al., 2024).

At the lowest level of technical implementation are standards like IEEE 7000 Series Standards, which work to propose specific and measurable methods of implementation to address specific issues. The 7000 Series provides many resources for issues surrounding Autonomous and Intelligent Systems (AIS), with only a few relevant to human autonomy. Similar to the NIST AI RMF, IEEE provides comprehensive support for enhancing transparency and explainability of AIS with the 7001 IEEE Standard for Transparency of Autonomous Systems, 2022. The specific categorizable framework for the transparency of a system gives developers a scale with specific metrics to aim for. These requirements encourage explainability appropriate to the user, recommending non-expert users to be given high level natural language explanations, while an expert (e.g. developer) might be given a full breakdown of the processing going on by the AIS. The 7001 Std recognizes that transparency is a key factor not only in reducing physical risks of the system, but also psychological harm.

The 7010 IEEE Recommended Practice for Assessing the Impact of Autonomous and Intelligent Systems on Human Well-Being, 2020, which focuses on assessing human well-being in AIS contexts, provides a human-first framework for organizations to base the performance of their AIS systems during periodic audits. This type of measuring system, if put in place, will enable organizations to better understand how human autonomy is being affected by their systems as they evolve. However, the standard in its current form does not provide activities which explicitly support human autonomy.

Policy Recommendation:

While current frameworks do a fantastic job of encouraging transparent and explainable systems, which contribute largely to enabling increased user autonomy, many flaws still exist. Here I want to recommend two distinct changes which I believe will push current frameworks further in enabling increased human autonomy when interacting with AI Recommendation systems.

Current frameworks are structured around the saliency of risk, but impact is still possible across a society without much salient risk being involved. Therefore, across all levels, frameworks should be restructured to include consideration for social disruption, creating a more whole, societal-centric view on systems. Marchiori et al., 2025 suggests several dimensions for assessing this social impact including ethical salience of impact, pace of change, reversibility of impact, etc.

Another large concern is that regulation only exists in most places at the highest level with policy like the EU AI Act, which simply requires organizations to establish an internal risk management system. Frameworks such as the NIST AI RMF and IEEE 7000 Series Standards currently serve as voluntary frameworks for organizations to structure these internal policies on. While we do not know enough about the implications of advancing systems to prescribe lower level regulations on all systems, we can certainly implement requirements for organizations with high potential for social disruption to easily measure the impact of their systems on human autonomy and iteratively act to minimize risks their systems may be creating. While IEEE’s 7010 Std provides high level activities for organizations to follow, Working Group members have encouraged the creation of a more structured tool for helping organizations measure their impact. They do note that developers of these tools should be wary of the potential “greenwashing” effect a metric could have.

         As AI Recommendation systems become more prevalent, changes to current regulation frameworks are sorely needed. We should look to implement changes like these quickly to ensure human autonomy is not put at risk by modern systems.

References:

Dotan, R., Blili-Hamelin, B., Madhavan, R., Matthews, J., & Scarpino, J. (2024). Evolving AI Risk Management: A Maturity Model based on the NIST AI Risk Management Framework (No. arXiv:2401.15229). arXiv. https://doi.org/10.48550/arXiv.2401.15229

IEEE Recommended Practice for Assessing the Impact of Autonomous and Intelligent Systems on Human Well-Being. (2020). IEEE. https://doi.org/10.1109/IEEESTD.2020.9084219

IEEE Standard for Transparency of Autonomous Systems. (2022). IEEE. https://doi.org/10.1109/IEEESTD.2022.9726144

Marchiori, S., Hopster, J. K. G., Puzio, A., Riemsdijk, M. B. V., Kraaijeveld, S. R., Lundgren, B., Viehoff, J., & Frank, L. E. (2025). A Social Disruptiveness-Based Approach to AI Governance: Complementing the Risk-Based Approach of the AI Act. Science and Engineering Ethics, 31(5), 25. https://doi.org/10.1007/s11948-025-00545-0

Noggle, R. (2018). The Ethics of Manipulation. https://plato.stanford.edu/archives/fall2025/entries/ethics-manipulation/

Prunkl, C. (2024). Human Autonomy at Risk? An Analysis of the Challenges from AI. Minds and Machines, 34(3), 26. https://doi.org/10.1007/s11023-024-09665-1

Ryan, R. M., & Deci, E. L. (Eds.). (2017). Self-Determination Theory: Basic Psychological Needs in Motivation, Development, and Wellness. Guilford Press. https://doi.org/10.1521/978.14625/28806

Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024    laying down harmonised rules on artificial intelligence. (2024). Official Journal of the European Union, L, 2024/1689. https://eur-lex.europa.eu/eli/reg/2024/1689/oj