Tutorial: Visualizing Word Embeddings in 3D Using Unity + Meta Quest

Overview

This tutorial walks through the complete pipeline for visualizing high-dimensional word embeddings as an interactive 3D point cloud in Unity, deployed to the Meta Quest 3 headset. The system renders 64 semantically categorized words as colored spheres in 3D space, with controller-based navigation and hover-to-inspect interaction.

The project compares three visualization conditions: a noisy 2D PCA projection, a clean 2D PCA projection, and a fully interactive Unity 3D environment — to evaluate how dimensionality and interactivity affect users' semantic understanding.

Apk: WordEmbeddings 

2D plots: 2D embeddings document 

Prerequisites

Unity 2022.3 LTS or newer with Android Build Support module installed

Meta XR SDK installed via Unity Package Manager (com.meta.xr.sdk.core)

Meta Quest 3 headset with Developer Mode enabled

Android SDK and ADB installed (via Android Studio or winget install Google.PlatformTools)

Python 3.9+ with numpy, scikit-learn, umap-learn, matplotlib installed

GloVe pretrained embeddings: glove.6B.300d.txt (download from nlp.stanford.edu/projects/glove)

Part 1: Python Pipeline

1.1 Dataset Selection

• The dataset consists of 64 words across four semantic categories: Emotions (16 words), Professions (16 words), Moral Concepts (16 words), and Nature (16 words). Words were chosen to create meaningful cluster separation while preserving interesting boundary cases.

• Example words per category:

  • Emotions: joy, sadness, fear, anger, love, hate, hope, grief, shame, envy, pride, guilt, happiness, disgust, anxiety, surprise

  • Professions: doctor, lawyer, teacher, engineer, nurse, artist, chef, soldier, banker, scientist, writer, pilot, judge, farmer, architect, accountant

  • Moral Concepts: justice, freedom, truth, loyalty, honor, courage, virtue, equality, mercy, duty, rights, fairness, conscience, authority, liberty, power

  • Nature: mountain, ocean, river, forest, desert, storm, fire, snow, sun, moon, thunder, wind, rain, earth, sky, valley

1.2 Generating Embeddings

• Load GloVe 300D vectors and extract embeddings for each word in the dataset: python scripts\generate_embeddings.py

• This script loads the GloVe file, looks up each word, and saves a 64x300 embedding matrix. Words not found in GloVe are skipped with a warning.

1.3 Dimensionality Reduction

• Three projections are generated from the 300D embeddings:

  • 2D Best (PCA dims 1 & 2): Maximally informative projection, captures highest variance. Used as the clean baseline condition.

  • 2D Worst (PCA dims 3 & 5): Skips the most informative dimensions, producing a visually scrambled layout. Used as the noisy baseline condition.

  • 3D UMAP: Three-dimensional layout preserving local neighborhood structure. Used in Unity.

  • 3D PCA: Alternative three-dimensional layout using PCA. Stored as fallback.

  • Run: python scripts\reduce_dimensions.py

  • Outputs saved to output/: X_2d.npy, X_3d_umap.npy, X_3d_pca.npy, words.npy, cats.npy

1.4 Generating embeddings.json

• Unity reads a single JSON file containing word, category, color, and all position arrays for each word: python scripts\export_json.py

• Each entry in the JSON has the following structure: { word, category, color: {r,g,b}, pos_2d, pos_3d_umap, pos_3d_pca }. All positions are normalized to [0, 1].

1.5 Generating 2D Plots

• Run the plot script to generate all four 2D plot images used in the study (labeled and unlabeled versions of both best and worst projections): python scripts\plot_2d.py

• Three mystery words (guilt, soldier, power) are rendered as gray dots labeled Dot A, Dot B, Dot C in all plots. These same words appear as gray spheres in Unity.

Part 2: Unity Setup

2.1 Project Configuration

• Create a new Unity project with the 3D template. Configure the following in Edit > Project Settings:

  • Platform: Android (File > Build Settings > Switch Platform)

  • Graphics APIs: Vulkan (primary) + OpenGLES3 (fallback) — remove any others

  • Minimum API Level: Android 12.0 (API 32) — required by Meta XR SDK

  • Scripting Backend: IL2CPP

  • Target Architectures: ARM64 only

  • Color Space: Linear

2.2 Installing Meta XR SDK

• In Window > Package Manager, add the following via Add package by name:

  • com.meta.xr.sdk.core — core XR support

  • com.unity.xr.openxr — OpenXR plugin

  • com.unity.inputsystem — new input system

• After installation, go to Edit > Project Settings > XR Plug-in Management > Android tab and enable OpenXR. Then under OpenXR > Features, enable Meta Quest feature group.

2.3 Scene Hierarchy

• The EmbeddingViz scene should contain the following objects at root level:

  • OVRCameraRig — replaces the default Main Camera. Handles all head tracking automatically. CloudNavigator script attached here.

  • EmbeddingCloud — empty GameObject with EmbeddingLoader script. Parent of all spawned word spheres.

  • InteractionManager — handles raycasting, hover detection, and info panel updates.

  • InfoCanvas — World Space canvas showing word info on hover. Contains InfoText (TextMeshProUGUI).

  • LaserPointer — LineRenderer component, start point at right controller anchor.

  • LegendAnchor — TextMeshPro 3D object with CategoryLegend script.

  • EventSystem — required for UI interaction.

  • Directional Light

2.4 Core Scripts

• EmbeddingLoader.cs 

  • Reads embeddings.json, spawns a WordPoint prefab for each word at its UMAP 3D position scaled to cloudSizeMeters (default 2.5m). Mystery words (guilt, soldier, power) are spawned gray with Dot A/B/C labels instead of their actual names.

• WordPoint.cs

  • Attached to each sphere. Manages color, scale, hover/selection state, and billboard label rotation. The Initialize method accepts an optional displayLabel parameter that overrides the word name — used for mystery dot relabeling.

• CloudNavigator.cs

  • Attached to OVRCameraRig. Right stick moves forward/back/strafe relative to the headset's look direction. Left stick X rotates the entire rig around the world Y axis (yaw only — pitch is handled by the headset). B button moves up, A button moves down.

• InteractionManager.cs

  • Casts a ray from the right controller anchor each frame. On hit, calls SetHovered on the WordPoint and updates the InfoCanvas with the word's display name and category. For mystery dots, the category shows as ??? instead of the actual category.

2.5 Building and Deploying

• Enable Developer Mode on the Quest via the Meta companion app. Connect via USB-C. Verify connection: adb devices

• In Unity: File > Build Settings > Build. Save as WordEmbeddingsAR.apk. Then install: adb install "C:\Users\amsye\Desktop\WordEmbeddingsAR.apk"

• For reinstalls (faster): adb install -r ... to skip the uninstall step. Find the app in the headset under Unknown Sources in the App Library.

Part 3: Known Issues and Limitations

• Label visibility requires pointing at a sphere — always-visible labels are technically possible but cause visual clutter with 64 words.

• Controller model rendering requires additional setup (OVRControllerPrefab) that was not included in this version.

• Passthrough AR was deferred — the current build runs in standard VR mode with a black background, not true passthrough.

• The info panel appears in world space and can end up behind the user. A HUD-anchored info panel would improve usability.

• The UMAP layout is computed offline and baked into embeddings.json — the layout cannot be regenerated at runtime.

Part 4: Software Stack Summary