An interactive experience combining movement and temperature.
This project was created as part of the Advanced Graphics and Interaction course, DH2413, at KTH, Sweden. This project was featured as an exhibition at Tekniska Museet, Stockholm, on October 31st 2025.
In this project, people were invited to interact with a particle system in Unity using their movements and body heat. A thermal camera captured body temperatures and used this data to control the colors of the particles in the particle system. Users could heat or cool their hands and then watch as the particle system responded to their changing temperature, along with creating beautiful and impressing shapes as it moved to follow the user's hand movement.
We wanted to create a fun and novel interaction based on body temperature. We wanted to combine thermal camera imaging with an interactive visualization, to allow people to see their own body heat as it changed. We were inspired by works such as the Fire & Ice interactive experience by Cinimod Studio & PRG in WestQuay Shopping Centre, Southampton, UK 2017, and Daisuke and Kosuke's paper Heat sensation in image creation with thermal vision from 2005, showcasing the "ThermoTablet". We liked the idea of visualizing temperatures for the user to play and interact with, and so envisioned creating such an experience.
Our goal was to allow users to view and control graphics through their movements and body temperature. We had a few different ideas throughout the developement of our project as to exactly how this would happen, and different constraints and challenges ended up shaping our project and ideas. We landed on creating a particle system for users to interact with on a large screen, with body temperature as an input for color and feedback provided to users through thermal and regular video.
Flir One Pro
We used a Flir One Pro thermal camera connected to a Raspberry Pi to capture the surface temperatures of our users. This was done using GTK+ application for FLIR ONE and modifying the code to extract and transmitt the relevant pixel data to our Unity project over Ethernet.
Unity Project
We developed a Unity WebGL application that accepts temperature data and hand movement data from Flir One Pro. By utilizing a Compute Shader-based GPU particle system and URP's indirect rendering in Unity, the application can efficiently map real-time data onto thousands of particles. We then map the temperature onto different colors and show it through a particle system. In this way, by changing the temperature of the user's hand, the user can wave their hand to interact with the particle system.
Python + OpenCV
We developed a Python client that processes the raw video feed from the Pi. Using Google MediaPipe for hand tracking and OpenCV for image analysis, we mask the thermal data to extract the specific temperature of the user's hand. This temperature value, along with the hand's position, is then sent to Unity via UDP to drive the interactive visualization in real-time.
We encountered several challenges along the way that ended up shaping our final project outcome. Our collective experience with the technologies we expected to work with was limited, so we had to adjust our expectations and plans throughout the process so as to keep progressing without getting stuck for too long.
We initially planned to identify relevant tempeature data from the thermal camera by analyzing the thermal images to identify approaching users' temperatures, but we later switched paths to use OpenCV's handtracking to only identify user's hands instead. We also encountered sensitivity issues with the thermal camera, which we attempted to circumvent throughout the project by imposing background restrictions for our physical setup, but unfortunately could not completely get rid of. As a result, our final project remained a bit finicky around extremer temperatures.
Latency issues in our Unity project were minimized visually by adjusting the behavior of our particle system, to better hide any delays from the user while improving the visual experience. We also decided to add visual feedback to users by showing the captured video being recorded by the thermal camera, in both thermal and regular video, reversed in order to better match users expectations.
Sofia Andersson
sofiaa5@kth.se
Data conversion, camera imaging
Jennifer Köhler
jenkoh@kth.se
Data conversion, camera imaging, physical interaction testing
Jingyi Yan
jingyiy@kth.se
Unity project, particle system, graphics
Yulu Zhang
yuluz@kth.se
Unity project, user testing, particle system, graphics, sound
Shangwen Wang
shangwen@kth.se
Data conversion, Unity project, particle system, hand tracking