Protecting the copyright of large language models (LLMs) has become crucial due to their resource-intensive training and accompanying carefully designed licenses. However, identifying the original base model of an LLM is challenging due to potential parameter alterations through fine-tuning or continued pretraining. In this study, we introduce HuRef, a human-readable fingerprint for LLMs that uniquely identifies the base model without exposing model parameters or interfering with training. We first observe that the vector direction of LLM parameters remains stable after the model has converged during pretraining, showing negligible perturbations through subsequent training steps, including continued pretraining, supervised fine-tuning (SFT), and RLHF, which makes it a sufficient condition to identify the base model. The necessity is validated by continuing to train an LLM with an extra term to drive away the model parameters' direction and the model becomes damaged. 

However, this direction is vulnerable to simple attacks like dimension permutation or matrix rotation, which significantly changes it without affecting performance. To address this, leveraging the Transformer structure, we systematically analyze potential attacks and define three invariant terms that identify an LLM's base model. We make these invariant terms human-readable by mapping them to a Gaussian vector using a convolutional encoder and then converting it into a natural image with StyleGAN2. The encoder discriminates between invariants from different base models and ensures Gaussian output through adversarial training, while StyleGAN2 transforms Gaussian vectors into dog images. Consequently, our method generates a dog image as an identity fingerprint for an LLM, where the dog's appearance strongly indicates the LLM's base model. Specifically, if the LLM is adapted from another base model, the generated dog highly resembles that model; otherwise if trained independently from scratch, it exhibits a unique dog image distinct from other models. Experimental results across various LLMs demonstrate the effectiveness of our method, the generated dog image remains invariant to different training steps, including SFT, RLHF, or even continued pretraining with augmented vocabulary in a new language.

In the coming years, diverse ecologies of AI systems are envisioned to rapidly and complexly interact with each other and with humans. Collaborative industrial robots will work on factory floors alongside labourers, care robots will assist human health workers, and personal AI assistants will help with scheduling, albeit in an elementary way. Therefore, it is essential to develop AI systems that can effectively and reliably collaborate with humans in various contexts. This gives rise to the agent alignment problem: how do we create agents that behave in accordance with the user's intentions and possess the ability to adapt to changing circumstances? In this talk, I will provide an overview of the problem and discuss our attempts in human-centred coordination and alignment, as well as collaboration in social context.

Speech Outline:

1. Research status and trends of intelligent decision-making

2. Challenges and difficulties faced by large decision-making models

3. Technical exploration based on large model of embodied intelligent decision-making for robots

Large language models, a recent hot topic, have achieved versatility across various tasks. However, from the perspective of reinforcement learning, current LLM can't tackle problems of decision intelligence. Unfortunately, reinforcement learning is still stuck at one-model-solving-one-task level, posing a significant challenge to the generalizability.To address this issue, we try to introduce pre-trained models to decision tasks. Now we focus on multi-agent systems and pay attention to the competitive and cooperative elements among agents. By extracting representations of different tasks, we train a decision model. Subsequently, through fine-tuning to learn task-specific representations, tasks can be swiftly accomplished using the pre-trained model. We have conducted experiments in various environments, yielding promising results.

In this work, we focus on the concrete form in which to represent embodied foundation priors and propose an intuitive and effective set of the priors that consist of foundation policy, foundation value, and foundation success reward. We name such framework as Foundation Reinforcement Learning (FRL), since it completely relies on embodied foundation priors to explore, learn and reinforce. To verify its effectiveness, we instantiate an actor-critic method with the assistance of the priors, called Foundation Actor-Critic (FAC). The benefits of FRL are threefold: sample efficient learning, robust to noisy priors, and minimal human intervention, which is a novel and powerful learning paradigm, towards achieving an embodied generalist agent.

Building agents with large language models (LLMs) for computer control is a burgeoning research area, where the agent receives computer states and performs actions to complete complex tasks. In this talk, we will review the transition of agent research in the LLM era and discuss the future development of foundation agents. We will also introduce Synapse, a computer agent featuring three key components: state abstraction, trajectory-as-exemplar prompting, and exemplar memory.

Large language models (LLMs) typically employ sampling or beam search, accompanied by prompts such as Chain-of-Thought (CoT), to boost reasoning and decoding ability. Recent work like Tree-of-Thought (ToT) and Reasoning via Planning (RAP) aim to augment the reasoning capabilities of LLMs by utilizing tree-search algorithms to guide multi-step reasoning. These methods mainly focus on LLMs' reasoning ability during inference and heavily rely on human-designed prompts to activate LLM as a value function, thus lacking general applicability and scalability. To address these limitations, we present an AlphaZero-like tree-search learning framework for LLMs (termed TS-LLM), systematically showing how tree-search with a learned value function can guide LLMs' decoding ability. TS-LLM distinguishes itself in two key ways: (1) Leveraging a learned value function, our approach can be generally applied to different tasks beyond reasoning (such as RLHF alignment), and LLMs of any size, without prompting advanced, large-scale models. (2) It can guide LLM's decoding during both inference and training. Empirical evaluations across reasoning, planning, and RLHF alignment tasks validate the effectiveness of TS-LLM, even on trees with a depth of 64.