這個計畫在探討AI在人機介面上的應用與未來發展趨勢，我們將以大型語言模型為核心，結合視覺模型、語音輸入、以及電腦操作等其他工具，實現複雜任務自動化。從命令列的互動(Gorilla CLI), 瀏覽器的代理操作(WebVoyager), 以及桌面上跨應用程式的操作(Claude Computer Use), 近期頂級會議上發表的相關研究可以看到未來的AI PC發展方向。透過此計畫我們希望創造四個Agentic AI系統。(1) WebPilot: 透過自然語言操作瀏覽器, 自動完成中文網站的操作, (2) MRAG powered WebPilot: 透過資訊系統的使用手冊以及RAG的輔助, 自動完成 Web-Based 資訊系統的操作, (3)Interactive Voice RPA Agent: 透過語音互動釐清使用者的需求, 創建工作流程自動化RPA (Robotic Process Automation)，(4) CrossAPP PCPilot: 透過API串接及AutoHotkey 等腳本自動化電腦桌面端的操作, 自動完成跨應用程式的操作。對於上述每個AI代理人系統，我們將採兩階段模型來創建: 初期我們將以現有OpenAI、Anthropic等LLM來快速佈建Agentic AI系統, 第二階段則透過第一階段的測試資料, 訓練地端的模型, 確保資料的收集以及主權的AI. 我們也將訓練地端的模型, 確保資料的收集以及主權的AI. 我們希望透過這個計畫創造AI賦能的人機互動，提供更直覺、更人性化的使用者體驗，降低使用者操作、管理電腦的障礙, 提升台灣使用者在AI powered資訊發展的優勢。

This project investigates the application and future trends of AI in human-computer interfaces. Centered on large language models, it integrates visual models, voice input, and computer operations to automate complex tasks. From command-line interactions (Gorilla CLI) and browser agent operations (WebVoyager) to cross-application desktop operations (Claude Computer Use), recent studies presented at top conferences highlight the future trajectory of AI-driven PCs. Through this project, we aim to develop four Agentic AI Systems:

(1) WebPilot: Automates interactions with Chinese websites via natural language commands.

(2) MRAG-powered WebPilot: Uses information system manuals and Retrieval-Augmented Generation (RAG) to automate operations on web-based information systems.

(3) Interactive Voice RPA Agent: Leverages voice interaction technology to clarify user needs to create automated workflows through Robotic Process Automation (RPA).

(4) CrossAPP PCPilot: Automates cross-application operations on the desktop using APIs and tools like AutoHotkey based scripts.

For each of the above AI agent systems, we will adopt a two-phase model development approach: In the initial phase, we will rapidly deploy the Agentic AI systems using existing LLMs such as those provided by OpenAI and Anthropic. In the second phase, we will train on-premise models using test data collected during the first phase to ensure data sovereignty and ownership of the AI systems. Through this project, we hope to create AI-enabled human-computer interaction, provide a more intuitive and humane user experience, reduce barriers to user operation and computer management, and enhance the advantages of Taiwanese users in AI-powered information development.