Overview
In smart cities and urban planning, integrating real-time video with augmented reality (AR) has the potential to enhance situational awareness and enrich digital experiences. In this challenge, participants will work with drone-captured video footage of urban environments and integrate it into a Unity-based AR application. The goal is to trigger a virtual character (or object) at specific GPS coordinates within the video feed. When a user selects a particular GPS point, the AR character should be seamlessly overlaid on the video in real time, providing an interactive experience.
This challenge simulates practical applications such as interactive urban guides, emergency response augmentations, or gamified public experiences, where virtual elements dynamically interact with the real world based on precise geolocation data.
Challenge Tasks
Video Input Integration & Preprocessing
Video Footage Preparation:
Obtain drone-captured video footage of urban areas with embedded GPS metadata or overlay a separate GPS log file.
Preprocess the video to ensure uniform resolution and frame rate.
Note: in place of drone footage, you can use the live feed from your webcam/mobile device, with dummy Lat/lon values.
Unity Integration:
Import the video footage into Unity as a texture or video player component.
Ensure the video playback synchronizes with corresponding GPS data timestamps.
GPS Data Handling
Data Extraction & Mapping:
Extract GPS coordinates from the video metadata or an accompanying log.
Develop a mapping system to convert GPS coordinates into Unity’s world coordinates using appropriate scaling factors.
User Interface for GPS Selection:
Implement an in-application GUI that allows users to select or input a GPS coordinate (or a location marker) from the video’s geolocation data.
AR Character Implementation
Character Design & Integration:
Create or import a 3D character or object asset into Unity.
Define the AR behavior so that when a specified GPS point is selected, the character appears at that location.
Trigger Mechanism:
Develop a mechanism to continuously check if the current GPS location (or user’s selection) in the video frame is within a predefined threshold of the target GPS coordinate.
Once the condition is met, trigger the appearance of the AR character with a smooth transition (e.g., fade-in, scaling animation).
Real-Time Augmented Reality Overlay
Synchronization & Calibration:
Calibrate the alignment of the AR overlay with the video perspective to ensure the character appears anchored to real-world features.
Use camera calibration techniques or Unity’s AR Foundation (if applicable) to align the virtual camera with the physical perspective.
Interactive Experience:
Allow users to interact with the AR character (e.g., tap to trigger an animation or display information).
Provide visual cues (like an icon or marker) on the video to indicate where the AR character will appear.
Testing & Performance Evaluation
Robustness Testing:
Test the system with various drone video clips representing different urban settings and lighting conditions.
Validate the accuracy of the GPS-to-Unity coordinate mapping by comparing with real-world landmarks.
User Experience Evaluation:
Assess the fluidity of the AR overlay during video playback.
Ensure that the triggering of the character is intuitive and that the transition is seamless.
Dataset & Resources
Drone Video Footage:
A collection of publicly available drone videos of urban areas with geotagged metadata. For example, you might use resources from DroneDeploy or Airdata UAV.
GPS Log Data:
If not embedded in the video, a separate GPS log file (CSV or JSON) with timestamps and coordinates for the corresponding video footage.
3D Assets:
Pre-made 3D character models (available on Unity Asset Store or free sources like Kenney) or custom-created assets.
Unity Framework:
Use Unity 2021 or later, optionally incorporating AR Foundation if targeting mobile AR experiences.
Evaluation Criteria
Accuracy & Synchronization (30%)
How accurately the system maps the GPS coordinates to Unity’s world coordinates.
The synchronization between video playback, GPS data, and AR overlay.
Trigger Mechanism Effectiveness (25%)
The responsiveness and smoothness of the AR character appearance at the designated GPS point.
The reliability of the trigger under various test conditions.
User Interface & Interaction (20%)
Intuitiveness and ease of use of the GUI for selecting GPS points.
Quality of interactions provided with the AR character (animations, information display).
Technical Implementation & Code Quality (15%)
Modularity, readability, and documentation of the code.
Efficient use of Unity’s features to achieve real-time performance.
Innovation & Presentation (10%)
Creativity in design and additional features that enhance the augmented reality experience.
Quality of the final presentation/demo (video demonstration, slides, or live demo).
Tech Stack Recommendations
Unity Engine:
Unity 2021 or later with C# scripting.
Use Unity Video Player for handling drone footage.
AR & GPS Libraries:
Unity’s AR Foundation (if targeting mobile devices).
Plugins or custom scripts for handling GPS data (e.g., GPS Location Plugin).
3D Modeling Tools:
Blender, Maya, or pre-made assets from the Unity Asset Store.
Data Processing:
Python or similar tools can be used for preprocessing GPS logs or video metadata before import into Unity.
Deliverables
Source Code & Unity Project:
A complete Unity project folder with well-documented C# scripts and scene files.
Demonstration Video:
A short demo video showcasing the system in action: video playback with integrated GPS tracking and the AR character appearance.
Technical Report:
A document detailing the approach, system architecture, calibration process, and evaluation results. Include screenshots of key components (e.g., the GUI, coordinate mapping visualization).
User Guide:
Instructions on how to set up, run, and test the application locally, including any necessary configuration steps for GPS data and video import.