SELF-AWARENESS, CREATIVITY, AND LANGUAGE IN ARTIFICIAL AND NATURAL SYSTEMS

https://u-tokyo-ac-jp.zoom.us/rec/share/z73Oa9dsd5x0DKIR9gA04kTMs1HtYhJBQhQBxnKJz20lw0OKlfD49vK1MeS5EATz.8psFnRv8NVj5Tn5i

SESSION I. CREATIVITY AND AWARENESS

9:00 Welcome

9:15 Luc Steels (Royal Academy of Belgium - KVAB) 

CAN LLMS BE A PATH TO AWARENESS AND CREATIVITY? 

What is awareness for and how can it be achieved? I will assume that the brain is a highly distributed complex dynamical system and argue that awareness has two functions: (1) It is crucial for achieving and maintaining coherence despite the distributed nature of the mind, specifically to ensure steadiness in action until a goal is achieved, coherence in dialog both syntactically and semantically, and consistency as a persona with continuity in opinion and perspective. (2) It is crucial for supporting insight learning. This is a form of learning based on computational reflection and strategies for repairing an impasse. It contrasts with statistical induction from a large data set which is the core of Deep Learning and many other learning methods. I will also argue that some form of language, in the form of an inner voice, is the key towards awareness. Recently it has become possible with LLMs, more concretely with chain-of-thought mechanisms and re-entrance, to operationalize awareness in a novel way and to systematically test the role of an inner voice in achieving coherence and insight learning. 

- Steels, L. (2003) Language re-entrance and the 'inner voice'. Journal of Consciousness Studies (4-5):174-185. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=3e6946915dfc471d85cb29d8e0c7e8bd271b5b21

- Garcia-Casademont, E. and L. Steels (2016) Insight Grammar Learning. Journal of Cognitive Science, 2016, 17(1):27-62. doi:10.17791/jcs.2016.17.1.27 https://www.kci.go.kr/kciportal/landing/article.kci?arti_id=ART002096570

9:45 Takashi Ikegami (University of Tokyo) 

CAN CONSCIOUSNESS RESIDE IN AN LLM? FEEDBACK FROM EMBODIMENT AND INTROSPECTIVE PROCESSES

This presentation introduces experiments conducted with ALTER3, an android powered by compressed air with 43 axes of movement, equipped with cameras in its eyes and capable of hearing voices. By integrating recent Large Language Models (LLMs) to provide language understanding and generation capabilities, we have confirmed that ALTER3 can generate appropriate behaviors in response to our verbal commands in a zero-shot manner. Furthermore, ALTER3 has demonstrated the ability to spontaneously generate actions on its own. Does this LLM possess a sense of "self"? We aim to address this challenge and discuss the potential for self-awareness in ALTER3, exploring the philosophical implications of consciousness in artificial systems that combine physical embodiment with advanced language capabilities.

- Takahide Yoshida, Suzune Baba, Atsushi Masumori, Takashi Ikegami, Minimal Self in Humanoid Robot "Alter3" Driven by Large Language Model, Proceedings of the 2024 Artificial Life Conference (2024)

- Takahide Yoshida, Atsushi Masumori and Takashi Ikegami: From Text to Motion: Grounding GPT-4 in a Humanoid Robot "Alter3" (2023) https://arxiv.org/abs/2312.06571

- Atsushi Masumori, Norihiro Maruyama and Takashi Ikegami: Personogenesis through Imitating Human Behaviors in a Humanoid Robot "Alter3". Frontiers in Robotics and AI7 p.165 (2020).

BIO: Takashi Ikegami has been working on the field of artificial life for more than 20 years. Open-ended evolution in artificial life models, self-moving droplet experiments are the main targets of his research. Recently, he became interested in constructing artificial life in the real world. To realize concepts such as "autonomy", "enaction", "sustainability", and "evolvability", he has started several experimental and conceptual works, using an android (called Alter3), a large scale Boids model, and a new theory of collective intelligence. https://www.sacral.c.u-tokyo.ac.jp

10:15 Mizuki Oka (Tsukuba University) 

TOWARDS OPEN-ENDED EVOLUTION IN ARTIFICIAL SYSTEMS

Open-ended evolution is a subfield of artificial life that focuses on realizing continuous evolution in artificial systems. Researchers in this field employ computational methods to simulate and understand life-like systems with the aim of potentially creating life itself and gaining new insights into our understanding of life through a constructive approach. Experimental subjects in open-ended evolution studies are often virtual organisms, and various computational algorithms, including novelty search and quality diversity algorithms, have been developed to facilitate this research. In this presentation, I will outline the key concepts and findings in the field of open-ended evolution. The applications of the technology generated through open-ended evolution research are diverse, ranging from artificial computer systems, such as the creation of game characters and environments, to biological systems, including the synthesis of biological robots like Xenobot. Other potential applications include the continuous discovery of new drugs or materials.

BIO: Mizuki Oka is an artificial life researcher and computer scientist based in Japan. She investigates complex systems, social networks, and web science through the lens of artificial life and computational models. Her research focuses on understanding emergent phenomena in online social systems, open-ended evolution, and the dynamics of information flow in networks. Oka is currently an Associate Professor at the University of Tsukuba, where she has served since 2013 after a postdoc position at the University of Tokyo. She earned her Ph.D. in Engineering from the University of Tsukuba in 2008, specializing in Computer Science.

11:00 Coffee break 

SESSION II. COGNITIVE EMBODIMENT

11:30 Kazuo Okanoya (Teikyo University)

SUBJECTIVITY IN THE RAT: TIME PERCEPTION, EMOTION CONTAGION, AND META-COGNITION 

Rats are well established experimental animals, but their cognitive world is underestimated. I present three studies that show the cognitive capabilities, especially reporting their own internal state. Time perception: rats were trained to report by level press when they felt time elapsed 25 sec from the last press. They successfully leaned the task and neural activities in the posterior cingulate cortex, related with the 2nd person view, was activated. Emotion contagion: rats were trained to discriminate between the positive-outcome stimulus and the negative ones. When they were exposed to emotional calls associated with pleasant or unpleasant calls of other rats prior to the experiments, the subject rats showed biased response according to the emotion. Meta-cognition: rats were trained to report the place of nose-poke hole that was previously presented as the target location. They learned to press the `cancel lever' that cancels the trial when they have low confidence of memory, indicating that they could perceive their own internal memory states. Taken together, these experiments support existence of the subjectivity in the rats.   

- Yuki, S., & Okanoya, K. (2017). Rats show adaptive choice in a metacognitive task with high uncertainty. Journal of Experimental Psychology: Animal Learning and Cognition, 43(1), 109. https://psycnet.apa.org/doiLanding?doi=10.1037%2Fxan0000130

- Saito, Y., Yuki, S., Seki, Y., Kagawa, H., & Okanoya, K. (2016). Cognitive bias in rats evoked by ultrasonic vocalizations suggests emotional contagion. Behavioural processes, 132, 5-11. https://www.sciencedirect.com/science/article/pii/S037663571630198X

- Yuki, S., Nakatani, H., Nakai, T., Okanoya, K., & Tachibana, R. O. (2019). Regulation of action selection based on metacognition in humans via a ventral and dorsal medial prefrontal cortical network. Cortex, 119, 336-349. https://www.sciencedirect.com/science/article/pii/S0010945219301911

BIO: Kazuo Okanoya obtained a Ph.D. in Biopsychology from University of Maryland, USA in 1989. He was an associate professor of cognitive science at Chiba University, a group leader of Biolinguistics laboratory at RIKEN brain science institute, and a professor of the cognitive and behavioral sciences. From 2022, he is a professor at ACRO, Teikyo University where he studies how communicative behavior gives rise to subjective experience. 

12:00 Michael Spranger (Sony AI, Tokyo)

GROUNDING COGNITION THROUGH EVOLUTIONARY LANGUAGE GAMES ON HUMANOID ROBOTS

I will survey a number of experiments on grounding that took place more than a decade ago with the Sony QRIO humanoid robots. One group of experiments was concerned with the body and sensori-motor intelligence, a second group was about the development of spatial concepts and language. These experiments all used the paradigm of language games and showed how a group of autonomous robotic agents can possibly bootstrap shared bodily and conceptual models pushed by the goal of self-organizing a shared communication and conceptualization system. 

- Spranger, M. The evolution of grounded spatial language. Language Science Press, Berlin https://langsci-press.org/catalog/book/53 (open access) 

- Steels, L. and M. Spranger (2008) The robot in the mirror. Connection Science 20:4,337- 358. doi:10.1080/09540090802413186 https://www.researchgate.net/publication/220233571_The_robot_in_the_mirror

BIO: Michael Spranger is the Senior Executive Director of Sony Research Inc., a wholly owned subsidiary of Sony Group Corporation established in April 2023, and President of Sony AI, which is a division of Sony Research. He is a roboticist by training with extensive research experience in fields such as natural language processing, robotics, and foundations of Artificial Intelligence. Michael Spranger has published more than 100 papers at top AI conferences such as IJCAI, NeurIPS and others. Concurrent to Sony AI, he also holds a Senior Researcher position at Sony Computer Science Laboratories, Inc., and is actively contributing to Sony's overall AI ethics strategy. 

12:30 Discussion

12:45 Lunch

SESSION III. EMERGENT COMMUNICATION

13:45 Hiroyuki Iizuka (Hokkaido University)

ENBODIMENT OF LARGE LANGUAGE MODELS AS ILLUSTRATED BY SOUND SYMBOLISM 

We explored the embodied nature of Large Language Models (LLMs) through the lens of sound symbolism - the phenomenon where phonetic properties of words evoke specific sensory impressions. By leveraging an evolving image generation system based on the CONRAD algorithm, we compared how humans and Vision Language Models (VLMs) respond to sound-symbolic associations between pseudo-words and visual stimuli. Our experiments revealed remarkable similarities between human and VLM sensibilities, particularly in how both associate certain phonetic paeerns with specific visual characteristics. Notably, VLMs demonstrated sound-symbolic sensibilities comparable to humans even for novel pseudo-words we generated, suggesting they have developed implicit understanding of cross-modal correspondences between sounds and visual properties.

BIO: Hiroyuki Iizuka received the Ph.D. degree in multi-disciplinary sciences from The University of Tokyo, Japan, in 2004. In 2005 and 2006, he was a Visiting Research Fellow with the Centre for Computational Neuroscience and Robotics (CCNR), University of Sussex. He was an Assistant Professor with the Human Information Engineering Laboratory, Osaka University (2008-2013). He was an Associate Professor with the Autonomous Systems Engineering Laboratory, Hokkaido University, Japan (2013-2023). He is Associate Professor with the Center for Human Nature, Artificial Intelligence, and Neuroscience, since 2023. His research focuses on embodied and enactive cognition, complex adaptive systems, deep learning, swarm behavior, the origin of life.

14:15 Takashi Hashimoto (JAIST)

CREATIVE ASPECTS OF LINGUISTIC COMMUNICATION

One distinctive feature of human linguistic communication is that the meaning of utterances is often internally created rather than simply referring to external situations or expressing emotions. Even with external situations, there is considerable freedom in how they are perceived, and speakers communicate their conceptualization of the situation, meaning the way the speaker internally represents and interprets it. The remarkable characteristic that utterances are hierarchically structured through recursively combined meaningful elements suggests the internal construction of complex concepts. In communication, fruitully created meanings can be shared through inference by both sender and receiver, promoting further concept cocreation. Recipients strive to reconstruct through inference the meaningful content that the speaker has constructed. Speakers infer how to provide cues that help recipients successfully infer the constructed meaning. Thus, language is considered a means of thinking and creating, which can also be used for communication. This inference involves abductive reasoning - that is, the generation and selection of hypotheses. This presentation reports on two cognitive experiments related to this point. Recursive combination contributes to the diversification of generated concepts and interpretations, and in communication, it may help diversify hypotheses about others' mental contents. The first experiment empirically investigated this point. Additionally, generating hypotheses imposes cognitive load, and inferred intention is likely adjusted according to this load and the cognitive effects of the content being conveyed. The second study reports on the effects of this adjustment. Finally, including these experimental results, I would like to discuss the role of abduction - an expansive and generative form of inference - in linguistic communication and creativity.

- Hashimoto, T. (2020) The emergent constructive approach to evolinguistics: considering hierarchy and intention sharing in linguistic communication, Journal of Systems Science and Systems Engineering. Vol. 29, pp. 675-696. https://doi.org/10.1007/s11518-020-5469-x

BIO: Takashi Hashimoto is a professor at Japan Advanced Institute of Science and Technology (JAIST). He received his Ph.D. degree in 1996 from the Graduate School of Arts and Sciences, the University of Tokyo. His research focuses on the origins and evolution of language, the dynamics of communication, and the design of social institutions from a complex systems perspective, and aims to establish knowledge science for creation, sharing, and utilization of knowledge.

14:45 Tadahiro Taniguchi (Kyoto University)

COLLECTIVE PREDICTIVE CODING HYPOTHESIS AND GENERATIVE EMERGENT COMMUNICATION

Humans have formed current language (i.e., symbol) systems through adaptation to the real world environment and communication between people (i.e., agents). In recent years, large-scale language models that achieve language understanding and generation by learning extensively from linguistic corpora through predictive learning have attracted attention. However, the computational theory of intelligence that forms language itself as an extension of real-world environmental adaptation is not well understood. Recently, the speaker proposed the collective predictive coding (CPC) hypothesis as a computational model of symbol emergence systems and the view of generative emergent communication. The hypothesis argues that our symbol systems, in a broad sense, are formed by predictive coding that is performed collectively by a society that is viewed as a subject of predictive coding. The CPC framework can also be seen as a society-wide free energy principle. From this perspective, symbol emergence can be viewed as decentralized Bayesian inference embodied as language games in a multi-agent system. This talk will introduce the concepts of symbol emergence systems and collective predictive coding, discuss recent research results related to symbol emergence, the connection to LLMs, and AI alignment to envision a future society where humans and AI robots coexist.

- Taniguchi, T. (2024). Collective predictive coding hypothesis: Symbol emergence as decentralized bayesian inference. Frontiers in Robotics and AI, 11, 1353870. https://www.frontiersin.org/journals/robotics-and-ai/articles/10.3389/frobt.2024.1353870/full

- Taniguchi, T., Ueda, R., Nakamura, T., Suzuki, M., & Taniguchi, A. (2024). Generative Emergent Communication: Large Language Model is a Collective World Model. arXiv preprint arXiv:2501.00226. 

BIO: Tadahiro Taniguchi received the M.E. and Ph.D. degrees from Kyoto University in 2003 and 2006, respectively. From April 2005 to March 2006, he was a Japan Society for the Promotion of Science (JSPS) Research Fellow (DC2) at the Department of Mechanical Engineering and Science, Graduate School of Engineering, Kyoto University. From April 2006 to March 2007, he served as a JSPS Research Fellow (PD) in the same department. From April 2007 to March 2008, he continued his role as a JSPS Research Fellow at the Graduate School of Informatics, Kyoto University. 

From April 2008 to March 2010, he held the position of Assistant Professor at the Department of Human and Computer Intelligence, Ritsumeikan University. He was promoted to Associate Professor in the same department, a position he held from April 2010 to March 2017. During this period, he also spent a year as a Visiting Associate Professor at Imperial College London, from September 2015 to September 2016.

From April 2017 to March 2024, Taniguchi was promoted to the position of professor at the Department of Information and Engineering at Ritsumeikan University. At the same time, he served as a visiting general chief scientist at Panasonic (Holdings) Cooperation. Since April 2024, he has assumed the role of Professor at the Graduate School of Informatics, Kyoto University, and also serves as an Affiliate Professor at the Research Organization of Science and Technology, Ritsumeikan University. Throughout his career, Taniguchi has been deeply involved in research on machine learning, emergent systems, cognitive robotics, and symbol emergence, and has made significant contributions to these fields.

15:15 Discussion

15:30 Coffee Break

SESSION IV. QUALIA AND THE HARD PROBLEM OF CONSCIOUSNESS

16:00 Riccardo Manzotti (Theoretical Philosophy, IULM, Milano)

ARTIFICIAL CONSCIOUSNESS IS A MIRROR WHERE I CAN SEE THAT QUALIA ARE NOTHING

Many AI experts and philosophers wonder whether Large Language Models (LLMs) could develop consciousness [1]. AI linguistic abilities match or even surpass those of conscious humans. Since language in humans is seen as an expression of thought and possibly of consciousness, this question seems natural. However, these connections remain poorly understood. Nobody knows how thought produces language (or vice versa), nor does anyone truly understand what consciousness is or where it resides.

I argue that thoughts and qualia are not only absent in LLM, but also do not exist in our heads or nervous systems either. Current models of consciousness are leftover of animism disguised as scientific models. Inside a machine or inside a brain there is only machine or brain stuff which is causally and ontologically complete. No additional entity or property has ever been detected. If so, what sets humans apart from a rock? The commonly accepted misconception is the wrong assumption that there is something phenomenal (called either mind or consciousness) located within the body. Such an assumption endorses wrongly an inner magical ability to generate thought, consciousness, or qualia.

Discarding this premise (as well as all kinds of dualism, idealism, and panpsychism), qualia can be set aside as an unnecessary shadow of the world. Rather consciousness is the subset of physical properties of the world as they take place relative to an intelligent body's causal structure. To have a presence like we do, LLMs do not need to generate an internal spark of consciousness or stage a Cartesian theater. They only need to be embedded in a body that allows them to interact causally with the world, just as we do. This is what consciousness truly is according to the Mind-Object Identity hypothesis, or MOI [2, 3]. 

1. Lenharo, M., Consciousness. The future of an embattled field. Nature, 2024. 625: p. 438-440.

2. Byrne, A. and R. Manzotti, Hallucination and Its Objects. The Philosophical Review, 2022. 131(3): p. 327-359.

3. Manzotti, R., The Spread Mind. Why Consciousness and the World Are One. 2017, New York: OR Books.

BIO: Born in Parma, Italy, in 1969, Manzotti received his PhD from the University of Genova in 2001, and is currently full professor of theoretical philosophy at IULM University (Milan). He has been Fulbright Visiting Scholar at MIT (Boston). Manzotti originally specialized in robotics and AI where he started to wonder how matter can have experience of the surrounding world. Eventually he has been a psychologist from 2004 to 2015 and then has become a full time philosopher. His current research focuses on the issue of consciousness and the structure of reality: What is the relationship between experience and the physical world? What is consciousness? Is there a separation between our experience of the world and the world? Does the present have a fixed time span? Can we design and build a conscious machine? What ethical questions do consciousness and technology raise in the 21st century?

In 2014, at MIT, Riccardo Manzotti presented the Spread Mind Theory (elsewhere dubbed the Mind-Object Identity Theory) that addresses the hard problem of consciousness in a completely radical and new way. Over the last few years, Manzotti has continued to develop and test this hypothesis interacting with the international scientific and philosophical community. Published in 2018, Manzotti's 'The Spread Mind' has outlined a radical change in the way we conceive us and the world. Based on empirical evidence from physics and neuroscience, the book develops and verify the astonishing hypothesis that our conscious experience is indeed one and the same with the external world. The book revisits familiar notions about dreams, illusions, and hallucinations. The book has been translated in many languages such as Chinese, Italian, Turkish.

In 2019 Riccardo Manzotti returned together with the acclaimed novelist Tim Parks with Dialogues on Consciousness, an engaging and humorous dialogue about the nature of consciousness and our everyday life. Prof. Manzotti lectures around the world on the topics explored in his books and articles, and has written for publications such as The New York Review of Books, Doppio Zero. He also offers his knowledge and time to various organizations and audiences on a voluntary basis.

16:30 Tom Froese (Okinawa Institute of Science and Technology) 

LINGUISTIC SENSE-MAKING WITH OR WITHOUT HUMAN EXPERIENCE? ADVANCES IN AI IMPLY THAT COMPETENCE CAN BE DISENTANGLED FROM CONSCIOUSNESS

Large Language Models (LLMs) demonstrate a kind of linguistic competence that theories of embodied and enactive cognition have argued would require sense-making, i.e., a lived perspective of a living subject. This creates a dilemma: (a) either LLMs are sense-makers despite lacking biological embodiment, or (b) linguistic competence does not necessarily require subjective involvement. In their treatment of recent AI advances, Frank, Thompson, and Gleiser (2024) dismiss (a), the possibility of disembodied sense-makers, on theoretical grounds. This suggests that (b), the possibility of subjectless competence, deserves closer examination. Indeed, phenomenological approaches have long highlighted how even in the case of human language production, speech gestures spontaneously self-assemble around communicative intentions in a largely unconsciously manner. Taken together, these considerations suggest the need for a conceptual loosening of the relationship between competence and consciousness. 

BIO: Dr. Tom Froese is Associate Professor at the Okinawa Institute of Science and Technology Graduate University (OIST), where he heads the Embodied Cognitive Science Unit (ECSU). Before 2019 he was faculty at the National Autonomous University of Mexico (UNAM), where he headed the 4E Cognition Group. Froese completed his Doctorate in Cognitive Science at the University of Sussex in 2010, and his Master's in Computer Science and Cybernetics at the University of Reading in 2004. Froese's research encompasses theoretical, computational, and experimental approaches to life, mind, and sociality.

17:00 Ken Mogi (Sony CSL, Tokyo)

CONSCIOUSNESS AND AI ALIGNMENT

The rapid development of AIs, generative AIs in particular, has raised issues related to alignment with humans. Here I describe a framework where the alignment between AI and humans are analogous to the alignment between conscious and unconscious processes in the brain. The divisions of labor between explicit and implicit, cognitive and embodied, and conscious and unconscious processes are discussed. Goodhart's law, which states that "when a measure becomes a target, it is no longer a good measure", is discussed in relation to consciousness and alignment. Finally, ikigai is discussed in the context of AI alignment contributing to the well-being of humans in the age of advanced technology.

- Mogi, K. (2024) Artificial Intelligence, Human Cognition, and Conscious Supremacy. Frontiers in Psychology, 15, 1364714. https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2024.1364714/full

- Ishikawa, T., Toshima, M., & Mogi, K. (2019). How and when? Metacognition and solution timing characterize an aha experience of object recognition in hidden figures. Frontiers in Psychology, 10, 1023. https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2019.01023/full?utm_source=Email_to_authors_

BIO: Ken Mogi is a senior researcher at Sony Computer Science Laboratories, and a visiting and project professor at the University of Tokyo. He leads the Collective Intelligence Research Laboratory (CIRL) at the Komaba campus of the University of Tokyo, together with Takashi Ikegami. Ken Mogi is the headmaster of Yakushima Ohzora High School, a correspondence-based institution with 13000 enrolled students.

17:30 Discussion and Conclusions

18:00 Closing

Acknowledgement: Medical LLM CIT project, JSPS KAKENHI: Exploring the Qualia Structure with Large Language Model and a Humanoid Robot. 24H01546.