Agentic Understanding and Retrieval Architecture

We investigate contextual information fusion schemes for enhancing LLM performance. Our aims are to explore mechanisms for injecting contextual signals (e.g., activity, location, environment) into LLMs for improved alignment and reliability, addressing limitations of existing fusion methods and enabling applications such as restaurant recommendation, safety alerts, and user assistance based on weather.

1. Motivation
- Needs for contextual information

• Recent advances in NLP have led to the proposal of multi-modal models such as VisionLLM and OpenFlamingo

• Currently there is a lack of LLMs specifically designed to deeply understand user context

2. Research goal and issue
- Goal : Develop a mechanism to provide contextual information
- Issue

• Human Activity Recognition / Location-Based System : the classification model used is relatively old and has lower activity recognition performance. LBS systems have low accuracy, not applicable in situations indoor/outdoor mixed environments

• Information fusion : CAAFE-information fusion method is laborious or inaccurate

3. Approach
- Develop a pipeline for injecting contextual information

• Data collection and develop baseline model

• Fusion techniques of optimal descriptive formants for improving LLM response

- Scenarios

• Restaurant Recommendation

• Safety Alerts

• User Aid based on weather

4. Result on scenario1

- Retrieve restaurant information near the user's current GPS coordinates using a map API, and filter this information based on the user's specific query
- Scroll 15 restaurant based on current location
- Distance is derived based on User's GPS coordinates between restuarant location
- Use hte Maps API to search for nearby restaurant information and convert their data into contextual data to enhance the responsiveness of your LLM model
- Context information is crucial for improving the accuracy and relevance of language model responses
- The utilization of context information is necessary for answering difficult questions, such as understanding the specific food a user wnats to better meet their needs
- To prevent hallucinations from unclear details, responses should more effectively utilize exact restaurant information such as distance, address, and restaurant category.