Md Mohaiminul Islam, Tushar Nagarajan, Huiyu Wang, Fu-Jen Chu,
Kris Kitani, Gedas Bertasius, Xitong Yang
Meta AI and UNC Chapel Hill
Accepted by ECCV 2024 (Oral)
We introduce VidAssist, an integrated framework designed for goal-oriented planning in instructional videos. VidAssist leverages large language models (LLMs) as both the knowledge base and the assessment tool for generating and evaluating action plans, thus overcoming the challenges of acquiring procedural knowledge from small-scale, low-diversity datasets. Moreover, VidAssist employs a breadth-first search algorithm for optimal plan generation, in which a composite of value functions designed for goal-oriented planning is utilized to assess the predicted actions at each step. Extensive experiments demonstrate that VidAssist offers a unified framework for different goal-oriented planning tasks, and achieves remarkable performance in zero-shot and few-shot setups.
Goal-oriented planning aims to generate action plans to achieve a given goal based on the current observations. Depending on the specifications of the current observation and the desired goal, the task can be instantiated in two different setups: Visual Planning for Assistance (left), where the current observation involves an untrimmed video history of the user’s progress and the goal is described in natural language, and Procedural Planning (right), where the initial and goal states are presented with images.
We follow the Socratic model approach to represent visual observations and goals for LLM-based planning. The main idea is to convert non-textual modalities (e.g., images and videos) into text for downstream LLMs. Specifically, for visual observations presented as video history, we predict the action category of each clip using an action recognition model. For visual observations/goals specified with images, we use an image-based step classifier to represent the initial and goal states in a textual description. Afterward, we prompt a state-of-the-art LLM (e.g., Llama-2) to predict the most probable subsequent action, taking advantage of the procedural knowledge embedded within the pretrained LLM. Rather than generating the entire action plan in one run, we employ an autoregressive approach to generate subsequent actions step-by-step until the given goal is achieved.
While the LLM baseline method performs well for short-term planning, its performance diminishes significantly for planning over a long horizon. We hypothesize that deliberate planning mechanisms to explore a combinatorial space of partial solutions are essential to finding the optimal action plan for the procedural tasks. Therefore, we introduce VidAssist, a search-based framework for generating optimal action plans. As shown in Figure, we propose several possible actions at each step by sampling multiple times from an LLM to account for the uncertainty in procedural tasks. Each proposed action is evaluated using a composite of value functions tailored for goal-oriented planning. This assessment process is crucial to our framework, as it facilitates informed and deliberate decision-making and extends beyond the conventional token-level generation process typically used in LLM inference. Finally, we employ a breadth-first search (BFS) algorithm to identify the optimal action plan according to the assessment scores.
We employ the following assessment functions to evaluate the generated partial plans at each step. Please refer to our paper for more details on the assessment functions.
Firstly, VidAssist demonstrates superior performance across all metrics in zero- and few-shot setups.
Secondly, our model consistently outperforms the LLM baseline by a considerable margin in both tasks, demonstrating the inadequacy of the simple LLM prompt-based techniques for this task.
Finally, few-shot VidAssist outperforms the state-of-the-art fully supervised methods in both tasks, especially for longer horizon predictions. This shows the significance of the deliberate search-based planning mechanism of our approach.
The LLM-based partial plan evaluation function achieves the highest success rate when only one value function is used, highlighting the efficacy of utilizing LLM as the assessment tool for evaluating action plans in goal-oriented tasks (left).
Combining all four value functions leads to the highest overall performance, demonstrating the importance of each proposed value function for goal-oriented planning tasks (left).
We observe a consistent increase in performance with the increase in the number of training examples (right).
VidAssist successfully predicts future action steps, while the LLM baseline fails in both examples. This demonstrates the effectiveness of propose, assess, and search techniques of VidAssist for goal-oriented planning in instructional videos. For brevity and clarity, we only show three generated actions at each step.
@article{islampropose,
title={Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videos},
author={Islam, Md Mohaiminul and Nagarajan, Tushar and Wang, Huiyu and Chu, Fu-Jen and Kitani, Kris and Bertasius, Gedas and Yang, Xitong}
}