Assignments

🤖 Assignment 1: Wide-area Reality Capture Digital Twin

Overview: Students will learn how to capture, clean, and ingest massive real-world spatial environments using photogrammetry to serve as the spatial substrate for location-based AR authoring. Initial point-cloud reconstruction and mesh remediation will be performed using Epic Games RealityScan and Unreal Engine, after which the shadowless asset will be exported (via FBX/OBJ/USD) to Unity for spatial alignment, stage hanging, UV texture auditing, and animated scene verification.

Technical Specifications: Epic Games RealityScan (or RealityCapture), Unreal Engine (Mesh Inspection & Material Cleanup), Unity (AR Foundation)

Core Tasks:

Site Scouting: When conducting site scouting for 3D photogrammetry, identify a suitable physical environment and capture over 1,000 sharp, high-resolution, heavily overlapping photographs under flat, overcast skies to eliminate harsh shadows for clean diffuse textures, making sure that when team members hand off the camera or split up the coverage, the next person begins their walking path at the exact same location where the previous person finished but moves along a different trajectory so the reconstruction software can seamlessly link the entire environment without breaking spatial alignment.
Cloud Reconstructions: Process the image sequence through RealityScan to generate a highly detailed 3D point cloud and high-density geometry.
Engine Optimization & Asset Export: Import the dense asset into Unreal Engine to inspect geometry and perform base color material remediation, then export optimized standardized 3D assets (FBX/OBJ/USD) for Unity ingestion.
Spatial Alignment, Stage Hanging & Lighting: Ingest the asset into Unity, calibrate real-world scale (1 Unity unit = 1 meter), inspect UV shader maps, hang virtual stage elements over the structure, and configure dynamic lighting with animated camera passes.
Submission Deliverables: The optimized Unity scene file, exported scan source files, raw photogrammetry source images, clean UV building texture maps, and an automated 10-second cinematic video showcasing stage hanging, UV texture fidelity, scale accuracy, and dynamic lighting sweeps across the environment. Unity Cinematic could be a great tool to capture this scene.

Grading Check Point and Rubric:

[ ] 1. Spatial Route Setup & Lighting Audit: Multi-waypoint navigation mesh established across a wide-area physical environment with heavy cloud cover or shade scouted to prevent baked-in sunlight.
[ ] 2. High-Density Image Capture: Capture a minimum dataset of 1,000+ distinct, sharp, overlapping photographs covering every surface angle of the designated location.
[ ] 3. RealityScan Cloud Processing: Upload and compute the image dataset using Epic Games RealityScan, monitoring the alignment point cloud for holes or coverage errors.
[ ] 4. Clean Unity Ingestion: Verify flawless asset import into Unity, ensuring hierarchy clean-up, correct material assignments, and zero broken mesh dependencies.
[ ] 5. Spatial Scale Calibration: Adjust bounding volume transforms within Unity to match real-world physical dimensions precisely (1 Unity unit = 1 meter).
[ ] 6. Mesh & UV Shader Mapping Audit: Inspect generated UV maps, building surface shaders, and texel density in Unity to ensure textures wrap accurately without stretching or seam distortion.
[ ] 7. Stage Hanging & Spatial Staging: Accurately place, anchor, and hang virtual stage elements and occlusion geometry onto the physical scanned building structure.
[ ] 8. Dynamic Lighting Calibration: Implement real-time lighting and light estimation inside Unity to demonstrate how the shadowless scan accepts dynamic directional light and scene shadows.
[ ] 9. Stage Animation Integration: Script or keyframe animated stage elements or camera motion sequences interacting with the scanned physical environment.
[ ] 10. Cinematic Video Deliverable: Render an automated 10-second video demonstrating correct scale, stage hanging alignment, UV texture fidelity, and dynamic lighting transitions using Unity Cinematic.

🧭 Assignment 2: Spatial AR Guidance & Non-Invasive Navigation Mechanics

Overview: Design and implement an intuitive wide-area AR navigation system that leads a user from Point A to Point B across a large physical site. The core challenge is creating natural, affordance-driven spatial cues that guide user movement without relying on invasive overlays or direct instructions.

Strict Guidance Rules:

❌ No Audio-Only Cues
(must provide visual/spatial presence)
❌ No Voice/Text Commands
(do NOT tell the user "Turn Left" or "Walk to the door")
❌ No Invasive Tunnel Effects
(no full-screen visual obstructions that block physical hazards)
✅ MUST Use Natural Diegetic Cues
(e.g., ambient fluid flows, subtle light paths, spatial affordances, organic particle trails, or peripheral visual cues that naturally elicit movement)

Technical Specifications:

Engine & AR System: Unity, AR Foundation, NavMesh / Spatial Pathfinding
Visual Systems: Shader Graph, VFX Graph, Light Probes / Dynamic Decals

Core Tasks:

Spatial Path Mapping: Define an intricate navigation route (minimum 30 meters) with multiple turn points across your scanned spatial map.
Diegetic Guidance Mechanic: Design a non-disruptive, visual guidance cue (e.g., dynamic light ripples on physical ground surfaces, directional particle breezes, floating spatial breadcrumbs, or ambient wall-following waves).
Safety & Pass-Through Optimization: Tune visual density so the user's view of real-world obstacles, stairs, and terrain remains clear at all times.
Adaptive Path Recalculation: Script C# logic using NavMeshPath to detect when a user veers off course and dynamically re-route the spatial cues toward the destination without jarring visual resets.

Submission Deliverables:

Unity project source code containing navigation C# scripts and VFX assets
System architecture diagram detailing route recalculation logic
60-second first-person video demonstrating an uninstructed user successfully navigating from Point A to Point B purely guided by your AR spatial cues

Grading Check Point and Rubric:

[ ] 1. Spatial Route Setup: Multi-waypoint navigation mesh established across a wide-area physical environment.
[ ] 2. Zero Voice/Text Compliance: System contains zero spoken instructions, text arrows, or UI pop-ups directing user actions.
[ ] 3. Anti-Tunneling Pass-Through: Guidance cues occupy $< 20\%$ of the screen viewport, maintaining physical safety awareness.
[ ] 4. Natural Affordance Design: Cue design uses intuitive visual physics (e.g., flow fields, light vectors) that prompt natural movement.
[ ] 5. Dynamic Re-routing Engine: Real-time distance tracking and C# path recalculation when the user strays from the primary path.
[ ] 6. Ground/Surface Conformance: Spatial path graphics conform dynamically to physical ground elevation, stairs, and uneven terrain.
[ ] 7. Arrival & Goal State Feedback: Clear, non-verbal spatial confirmation trigger when reaching Point B.
[ ] 8. Occlusion Integration: Guidance cues correctly pass behind real-world physical columns, walls, and structures.
[ ] 9. User Action Consistency: Validation that multi-user test subjects follow the intended path without hesitation or confusion.
[ ] 10. High-Resolution Nav Video: 60-second uncut video recording demonstrating smooth spatial guidance from Point A to Point B.

🤖 Assignment 3: Conversational AI Avatars & Crowd-Driven Spatial Direction

Overview: Transform your wide-area digital twin into an interactive narrative space by populating it with an interactive AI Avatar and a dynamic crowd system ("cloud of NPCs"). The AI Avatar must engage the user through dialogue and gestures, leveraging multi-agent crowd behavior to subtly nudge, funnel, or prompt the user to perform specific physical actions within the environment.

Technical Specifications:

Engine & Logic: Unity, C# State Machines / Behavior Trees, NavMesh AI Crowd / Flocking Systems
Dialogue & Avatar: Local or API-driven Conversational Agent (e.g., LLM wrapper or dynamic state-based dialogue engine), Unity Animator Controller
Spatial Systems: Spatialized Audio, NavMeshAgent, Trigger Volumes

Core Tasks:

Interactive AI Avatar Integration: Place a rigged humanoid AI Avatar in the AR scene that interacts with the user using synchronized conversational gestures, spatial look-at targets, and speech outputs.
Crowd Dynamics ("Cloud of People"): Deploy a flocking crowd system (15+ NPC agents) that moves dynamically through the wide-area environment. The crowd should act as physical/visual narrative funnels (e.g., gathering around an event, blocking paths, or walking in a directional stream).
Behavioral Nudge Pipeline: Program interaction logic where the Avatar's conversation—paired with crowd motion—naturally leads the user to take a specific action (e.g., inspect an object, cross a threshold, or open an AR interface).
Spatial Occlusion & Presence: Ensure all crowd members and the primary Avatar respect real-world depth maps and physical walls.

Submission Deliverables:

Complete integrated Unity simulation project
Flowchart mapping the AI Avatar dialogue tree, dynamic crowd state switches, and user action triggers
Video demonstration showing the Avatar speaking, the crowd adjusting around the user, and the user completing the intended spatial action

Grading Check Point and Rubric:

[ ] 1. AR Digital Twin Foundation: Scene successfully built on Assignment 1's wide-area scanned environment.
[ ] 2. Conversational AI Avatar: Avatar features dynamic dialogue execution, lip-sync/audio alignment, and spatial body gestures.
[ ] 3. Spatial Look-At & Awareness: Avatar turns head and body to maintain natural eye contact as the physical user moves around it.
[ ] 4. "Cloud of People" Flocking Logic: Crowd system demonstrates dynamic pathing, flocking density, and realistic agent avoidance.
[ ] 5. Spatial Funneling Mechanics: Crowd behavior actively influences physical locomotion (e.g., forming a corridor or surrounding a point of interest).
[ ] 6. Narrative Action Triggering: System tracks physical user positioning to detect successful completion of the prompted action.
[ ] 7. Multi-Agent Pathfinding Performance: Smooth execution of crowd AI state updates without dropping AR frame rates below target limits.
[ ] 8. Real-World Depth Occlusion: Avatar and crowd agents pass behind physical real-world pillars and structures correctly.
[ ] 9. Spatial Audio Attenuation: Avatar dialogue and crowd murmur feature realistic 3D distance attenuation and directionality.
[ ] 10. Video Walkthrough: Screen-captured demonstration highlighting conversational avatar engagement, crowd dynamics, and successful user directional nudging.

Page updated

Google Sites

Report abuse