Project Title: Development of Large-scale Language and Multimodal Models for Dynamic and Sustainable Food Planning in East and Southeast Asia
Project Duration: April 2025 - March 2028 (FY2025 - 2027)
Project Overview:
This collaborative research project aims to develop cutting-edge large-scale language models (LLMs) and multimodal models (LMMs) to analyze diverse food-related data and respond to the dynamically changing food landscape in East and Southeast Asia due to climate change. By leveraging real-time data from local news, social media, and satellite imagery, the project will provide deeper insights into food security and sustainability in the region.
The Japanese team will focus on collecting and analyzing food-related data, including environmental impacts based on meal records. They will also develop an app that visualizes the environmental footprint of food choices. On the other hand, the Singapore team will specialize in building the LLM and LMM, using the Japanese team's data to enhance the models' capabilities in food-related analysis.
Through this joint effort, the project aims to help redesign food production and distribution plans, allowing for more flexible and timely adjustments to the challenges posed by climate change.
Research Leaders:
Japan-side
PI: YAMAKATA, Yoko (Professor, Information Technology Center, The University of Tokyo)
Co-PI: YAMASUE, Eiji (Professor, College of Science and Engineering, Ritsumeikan University)
Singapore-side:
PI: Tat-Seng CHUA (Professor, School of Computing, National University of Singapore)
Co-PI: See-Kiong Ng (Professor, School of Computing, National University of Singapore)
Research Members:
UTokyo:
Program site: https://www.jst.go.jp/aspire/nexus/en/project/singapore_ai.html
Key Dates and Activities (Singapore Visit):
Date: September 8–10, 2025
Venue: NUS School of Computing, National University of Singapore (NUS)
Participants: Faculty members and students from The University of Tokyo and Ritsumeikan University, Japan.
Objectives:
Foster collaboration between Japanese and Singaporean researchers
Develop innovative language and multimodal models for food analysis
Provide tools for dynamic and sustainable food planning in East and Southeast Asia
Build a stronger academic and research partnership between The University of Tokyo, Ritsumeikan University, and NUS
Schedule (tentative):
Opening Session (9:00 - 12:00):
Project overview
Prof. Yamakata (UTokyo).
Prof. Tat-Seng Chua (NUS).
Prof. Yamasue (Ritsumeikan University)
Invited talk
Prof. Chong-Wah Ngo (SMU)
Prof. Bin Zhu (SMU)
Prof. Na Zhao (SUTD)
Get-together Lunch
Student Lightning Talks (15:00 - 17:00):
Each student will briefly present their own research.
Please include one self-introduction slide.
Day 2 & Day 3 (Morning) – Hackathon: Building a Food TMR Dataset
We will create a dataset of Total Material Requirements (TMR) for foods consumed in Singapore.
What is TMR?
TMR is an environmental index that represents the total amount of raw extracted materials needed to produce a product. A higher TMR means greater impact on mineral resources.
On the morning of Day 2, Prof. Yamasue will give a lecture on TMR.
Team Setup:
Students will be divided into three teams, each including at least one student from NUS, UTokyo, and Ritsumeikan.
The goal is to calculate TMR values for as many food items as possible.
Game Rules:
Teams receive a list of representative food items commonly used for nutrition assessment in Singapore.
For each food item:
Determine the countries of origin and import ratios for Singapore.
Find the TMR value for each source country.
Calculate the weighted TMR for Singapore based on import ratios.
All calculation steps must be documented in the dataset.
Accuracy check: the Ritsumeikan student in each team will verify results (or Prof. Yamasue if needed).
The team with the highest number of correctly calculated TMR values wins.
Tools Allowed:
Any resources, including ChatGPT or your own LLMs.
A GPU server with 2×H100 GPUs will be available for free use during the hackathon.
Prizes: NUS, UTokyo, and Ritsumeikan goods
UTokyo: Kyosuke NISHISHITA (M1), Liang Yufan (M1), Ryota Okumura (M1)
Ritsumeikan: Prof. Shoki KOSAI (Assoc. Prof.), Akihiro MIKAMI (B4), Hikaru IKUTA (B4), Ayumu TAMAKI (B4), Atsuto TOKUHARA (B4)
NUS: TBD
Guests: Elisa YAMASUE, Chi-na FUKUSHIMA, Kanta ADACHI
Significance for the NEXUS Project
Calculating TMR for foods in Singapore requires collecting diverse statistical data, often in multiple languages.
This dataset will be central to our NEXUS project’s goal: developing LLMs for multi-step reasoning in food production and distribution.
We expect the work to continue beyond the workshop to complete the dataset.
Once compiled, it will be used to train LLMs and to submit results to an international conference within the year.