AI-Avatar Agent

Architecture Overview

[User Mic (Push-To-Talk)] ➔ [OpenAI Whisper (STT)] ➔ [OpenAI GPT-4o (LLM)] ➔ [ElevenLabs (TTS)] ➔ [AudioSource + SALSA LipSync]

Phase 1: Pre-requisites & Unity Setup

1. Unity Version & Render Pipeline:

   • Install Unity 2022.3.62f1 via Unity Hub.

   • Create a new project using the HDRP (High Definition Render Pipeline) template (version 14.0.12).

2. VR Hardware Setup (Meta Quest Pro):

   • Import the Meta XR All-in-One SDK (or OpenXR Plugin) via the Unity Package Manager.

   • Configure project settings for OpenXR, target the Meta Quest Pro, and set up your VR Rig (Camera Offset, Controllers, and Input Actions).

3. API Keys Required:

   • OpenAI API Key (for Whisper and GPT-4o).

   • ElevenLabs API Key (for Text-to-Speech).

Phase 2: Offline Setup (Avatar & Voice Setup)

Step 1: Character Creation & Setup

You have two main paths for creating your 3D avatar:

Option A: Character Creator 5 (CC5 / Reallusion) — As shown in diagram

1. Author your character in Character Creator.

2. Export as FBX with standard facial blendshapes/visemes (e.g., ARKit blendshapes or CC3+ visemes).

3. Import into Unity and set the animation rig to Humanoid. Assign HDRP materials to shaders.

Option B: Custom Web Avatar via Avaturn ([https://avaturn.me/](https://avaturn.me/))

If you want to use your own customized avatar or create a digital twin:

1. Go to https://avaturn.me/ and create an avatar (you can upload a selfie to generate a custom 3D mesh).

2. Customize hair, clothing, and body features, then export the model as .glb or .fbx.

3. Ensure you select the option to export with Facial Blendshapes / Visemes (Avaturn exports include standard ARKit blendshapes for lip-syncing).

4. Import the .glb/.fbx file into Unity:

   • Set the Animation Type to Humanoid.

   • Convert textures/materials to HDRP Lit Shaders.

   • Place the avatar in your VR environment.

Step 2: Voice Cloning (ElevenLabs)

1. Log into your ElevenLabs account and go to Voice Lab.

2. Upload clear audio recordings of your target voice to create an Instant Voice Clone.

3. Save the generated Voice ID (you will need this string in your Unity script for API calls).

Step 3: SALSA LipSync Integration

1. Purchase and import SALSA LipSync v2 (or latest compatible version) from the Unity Asset Store.

2. Add the Salsa component to your Avatar's root GameObject.

3. Link the Avatar’s Head/Face mesh (containing the Blendshapes/Morph Targets) to SALSA.

4. Map Visemes: Map SALSA's mouth shapes (viseme_aa, viseme_E, viseme_I, viseme_O, viseme_U, etc.) to the corresponding blendshapes on your CC5 or Avaturn model mesh.

Phase 3: Runtime Loop Implementation

You can write a central orchestrator script (e.g., VRAIAgentManager.cs) attached to your VR scene to handle the continuous loop:

[1. Push-To-Talk] ➔ [2. Whisper STT] ➔ [3. GPT-4o LLM] ➔ [4. ElevenLabs TTS] ➔ [5. SALSA Playback]

1. User Input (Push-To-Talk Mic)

• Set up a VR controller input listener (e.g., holding down the primary trigger or 'A' button).

• When Pressed: Call Microphone.Start(null, false, lengthSec, sampleRate) to record audio from the Quest Pro microphone into an AudioClip.

• When Released: Call Microphone.End() and convert the recorded AudioClip into a WAV audio file byte array in memory.

2. Speech-to-Text (OpenAI Whisper API)

• Send an HTTP POST request (UnityWebRequest) containing the WAV audio byte payload to the OpenAI Whisper endpoint:

  • Endpoint: [https://api.openai.com/v1/audio/transcriptions](https://api.openai.com/v1/audio/transcriptions)

  • Model: whisper-1

• Response: Parse the JSON response to extract the output transcription text (e.g., "Hello, who are you?").

3. LLM Response Generation (OpenAI GPT-4o)

• Send an HTTP POST request with the transcribed string and system prompt to:

  • Endpoint: [https://api.openai.com/v1/chat/completions](https://api.openai.com/v1/chat/completions)

  • Model: gpt-4o

  • Payload: System instructions (persona definition) + User message history.

• Response: Extract the text reply string (e.g., "Hello! I am your VR AI assistant.").

4. Text-to-Speech (ElevenLabs API)

• Send an HTTP POST request with the GPT-4o text response to ElevenLabs:

  • Endpoint: [https://api.elevenlabs.io/v1/text-to-speech/](https://api.elevenlabs.io/v1/text-to-speech/){YOUR_VOICE_ID}

  • Header: xi-api-key: <YOUR_API_KEY>

  • Payload JSON: {"text": "Hello! I am your VR AI assistant.", "model_id": "eleven_monolingual_v1"}

• Receive the audio stream (MP3/WAV bytes) and convert it back into a Unity AudioClip at runtime.

5. Playback & Real-Time Lip-Sync

• Assign the generated AudioClip to the AudioSource component linked to your avatar's SALSA LipSync component.

• Call AudioSource.Play().

• SALSA LipSync automatically analyzes the audio frequency output in real-time and drives the avatar's facial blendshapes (whether CC5 or Avaturn) to match the speech seamlessly.

© You-Jin Kim
        Nebraska–Lincoln 🌽