Embedded Files
Agenda
Forking Git Repo
Interface
Human-Computer Interaction
Gesture in XR
Forking Git Repo (Method for the assignment submission using the course material)
Fork is your own GitHub copy of someone else’s repository, while keeping a link to the original . Make sure to fork "for my own purposes"
Interface
The membrane of interaction between humans and technology
Human-computer interaction (HCI) is research in the design and the use of computer technology, which focuses on the interfaces between people (users) and computers. HCI researchers observe the ways humans interact with computers and design technologies that allow humans to interact with computers in novel ways. A device that allows interaction between human being and a computer is known as a "Human-computer Interface (HCI)".
Sub-specialties
speech recognition; voice control; gesture recognition (e.g., hand or eye tracking); behavior recognition; behavioral analytics; mood/emotion recognition; virtual assistants; visualization and display technology; tactile displays; haptics; biometric sensing; bioacoustic sensing; biosignal detection and processing.
Signal Processing
Signal processing is an electrical engineering subfield that focuses on analyzing, modifying and synthesizing signals, such as sound, images, potential fields, seismic signals, altimetry processing, and scientific measurements.
Research Groups
UMD HCI Lab
UMD HCI masters program
https://ischool.umd.edu/academics/masters-programs/master-of-science-in-human-computer-interaction/
MIT Media Lab
https://www.media.mit.edu/research/?filter=groups
Artful Design - Ge Wang, Stanford University
Academic Community
CHI Conference - Proceedings - ACM SIGCHI - YouTube
IEEE VR Conference - IEEE Virtual Reality Conference - YouTube
ISMAR International Symposium on Mixed and Augmented Reality (ISMAR) - YouTube
Gesture
: a movement usually of the body or limbs that expresses or emphasizes an idea, sentiment, or attitude
Gesture is the most direct, useful, and faithful manner to communicate data in real-time.
The human gesture is characterized by a smooth, continuously changing relationship of the limb to the body.
speech recognition
gesture recognition
behavior recognition
Brain interface
HCI in VR & AR
Virtual Reality involves providing sensory input to a user that replicates being present in a real or imagined environment. Most commonly, sensory input is limited to sight and sound, but it can also include other senses such as touch. Augmented Reality, on the other hand, involves a live direct or indirect experience of an environment, overlaid with computer-generated sensory input usually in the form of graphics, video and/or sound.
Applications of VR and AR include (but are not limited to): education – enhanced learning experiences; medical and healthcare – treatments for PTSD, phantom pain, anxieties and phobias, autism in children; support for complex tasks such as surgery, equipment assembly, or maintenance and repair by adding relevant information to the field of view of the user; training for medical personnel, law enforcement, military, and emergency responders; architectural design – experiencing a virtual building before it's built; engineering and design; telepresence – for meetings and remote workers; market research – experiencing a virtual product that doesn't yet exist; entertainment – cinema, music, and sports; tourism; product advertising and promotion; computer games.
(Eventually) Forget about the controllers. Our hands will be enough.
As XR glasses become smaller and more seamless, hand tracking is emerging as the primary interface.
It offers intuitive interaction, preserves social cues, and supports accessibility by design. Start with gestures, then add other input methods only when needed.
Mixed Reality Utility Kit
Mixed Reality Utility Kit
MRUK (Meta MR Utility Kit) is a toolkit that helps developers use the real-world environment in Mixed Reality applications on Meta Quest. (introduced in 2023)
It allows Unity applications to understand and interact with the physical space, including:
Detecting floors, walls, ceilings, tables, couches, and other surfaces
Placing virtual objects on real-world surfaces
Performing raycasts, collision, and occlusion with the physical environment
Using room geometry for navigation and spatial interaction
Combining with the Depth API to work with real-world geometry
The range of things we can do with it keeps expanding.
(2023) Quest 3 Depth API (Depth API Overview | Meta Horizon OS Developers) was introduced experimentally in September 2023 and became publicly available with Meta XR SDK v60 in December 2023.
(2025) Direct access to the raw RGB passthrough camera feed became available in 2025.
Early 2025, Meta XR / Horizon OS v74: the Passthrough Camera API was released as a public experimental feature. (Passthrough Camera API overview | Meta Horizon OS Developers)
Title: MR Simulated LiDAR Scanner / Artist: Richard Han (IMD)
<Video Demo:2025-12-10 18-47-37.mp4> (Scanner Sombre in real space)
This project is a mixed reality experience for the Meta quest 3 that allows the user to navigate and perceive the real world through a simulated LiDAR scanner. Instead of viewing the world through passthrough, the user has to scan the environment, revealing points in space that correspond to the detected real world surfaces. These points vary in color based on distance, providing special depth cues, and the scan angle can be adjusted to narrow or widen the FOV of the scanner.
Gesture Recognition example
MediaPipe
MediaPipe is a open-source cross-platform pipeline framework to build custom machine learning solutions for live and streaming media.
The framework was open-sourced by Google and is currently in the alpha stage.
Publications: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C21&q=mediapipe&btnG=&oq=+
*About License: https://choosealicense.com/
XR Hands Tracking
For hand-based XR interaction, Unity provides a fairly complete stack. XR Hands gives us access to tracked hand joints and poses, while the XR Interaction Toolkit builds higher-level interactions such as pinch, grab, poke, and direct manipulation on top of that tracking data. Unity also provides platform-specific examples, such as the PolySpatial visionOS samples, showing how similar hand interactions can be used in spatial computing environments.
So rather than building hand tracking from scratch, we can start with these existing components and focus on designing the interaction itself.
XR Hands | XR Hands | 1.10.0-pre.1
Hands Interaction Demo | XR Interaction Toolkit | 3.7.0-pre.1
How to communicate with external devices such as gyroscope, EEG, python, or other sensors?
About OSC (OSC spec 1_0 (stanford.edu))
Open Sound Control (OSC) is an open, transport-independent, message-based protocol developed for communication among computers, sound synthesizers, and other multimedia devices.
In unity:
example)
"Playful"
Play is an essential aspect of artful design, because it is an integral part of human life, having everything to do with purpose and a deep commitment to activities that have no extrinsic purpose.
Play is about engaging in an activity for its sheer intrinsic value -- its design entails the crafting of such internally meaningful experiences, it is psychology in motion -- how we set the conditions to motivate specific behaviors and induce certain mindsets in the player.
We design play into pure games and toys and can incorporate game-like elements into practical contexts.
"Aesthetics"
Aesthetics (also spelled esthetics) is the branch of philosophy concerned with the nature of beauty and the nature of taste and, in a broad sense, incorporates the philosophy of art.[1] Aesthetics examines the philosophy of aesthetic value, which is determined by critical judgments of artistic taste;[2] thus, the function of aesthetics is the "critical reflection on art, culture and nature".[3][4]
AESTHETICS IS NOT LUXURY
Consumerism has perverted the notion of the "designer object (e.g, "designer" handbag, "designer" watch) to signal you are paying for some extra, and often exterior, stylistic care to an otherwise purely utilitarian object. And while it at least recognizes that aesthetics is of emotional (and wallet-opening) appeal, it intentionally conveys an elitism around aesthetics, as if to say only those who can afford it deserve to have good, beautiful things. But design is about what is inside, and its aesthetics are derived from experiencing the totality of a thing. Truth and beauty can be imbued into all design, but they must be deeper than the surface polish our consumeristic society equates with design. It is time to take back design to mean something more authentic- through the products and experiences we design, but also through an awareness of beauty and truth when we see it, regardless of the price tag or marketing jargon. Elegance is not marked by lavishness, superficial sophistication, or opulence. Instead it is defined by authenticity and grace, plain and simple, deep and true.
Artful Design: Technology in Search of the Sublime, A MusiComic Manifesto (Ge Wang)
Page updated
Report abuse