Augmentation, Artificial Intelligence, and Robotics

These projects aim to improve human-machine interaction and to facilitate augmentation of the human mind.   We also wish to examine the complex relationships between and networks of existing humans and machines, and study the progressing convergence of computing and cognition.
 
2017   
 AR Thermal Camera Fusion

Orlosky, J., Kim, P., Kiyokawa, K., Mashita, T., Ratsamee, P., Uranishi, Y., & Takemura, H. (2017, October). VisMerge: Light Adaptive Vision Augmentation via Spectral and Temporal Fusion of Non-visible Light. In Mixed and Augmented Reality (ISMAR), 2017 IEEE International Symposium on (pp. 22-31). IEEE. Paper 
 Adaptive Drone Navigation

Thomasson, J., Photchara, R., Kriangkomol, P., Kiyokawa, K., Orlosky, J., Mashita, T., Uranishi, Y., Takemura, H. Adaptive View Management for Drone Teleoperation. To appear in the proceedings of Intelligent user interfaces 2017.  Paper 
2016   
 Low Latency VR Robot Embodiment


Theofilis, K., Orlosky, J., Nagai, Y., & Kiyokawa, K. (2016, November). Panoramic view reconstruction for stereoscopic teleoperation of a humanoid robot. In Humanoid Robots (Humanoids), 2016 IEEE-RAS 16th International Conference on (pp. 242-248). IEEE.

Project aimed at improving embodiment and bidirectional interaction in robots for telepresence applications.
Paper 
  Cognitive Monitoring in VR Using Integrated Eye-tracking 
Sonntag, D., Orlosky, J., Weber, M., Gu, Y., Sosnovsky, S., Toyama, T., & Toosi, E. N. (2015, June). Cognitive Monitoring via Eye Tracking in Virtual Reality Pedestrian Environments. In Proceedings of the 4th International Symposium on Pervasive Displays (pp. 269-270). ACM.

Virtual reality simulation of a pedestrian environment with integrated eye-tracking to monitor cognitive abilities. 
Demo Paper 
 Focus and Eye Tracking for Attentive Interfaces

Orlosky, J., Toyama, T., Sonntag, D., & Kiyokawa, K. The Role of Focus in Advanced Visual Interfaces. (2015, October) KI-Künstliche Intelligenz, 1-10.

This paper studies the role of focus (accommodation) in visual interfaces in combination with eye tracking.  This combination is used to make monoscopic wearable displays more usable and safer.


Paper 
  Pedestrian Simulator for Cognitive Monitoring

Orlosky, J., Weber, M., Gu, Y., Sonntag, D., & Sosnovsky, S. (2015, March). An Interactive Pedestrian Environment Simulator for Cognitive Monitoring and Evaluation. In Proceedings of the 20th International Conference on Intelligent User Interfaces Companion (pp. 57-60). ACM.

Simulator designed to provide training and cognitive monitoring for individuals that are at risk in pedestrian environments, including children and individuals with cognitive impairments.


Paper 
 Cognition Based Content Modulation and Notification

Toyama, T., Orlosky, J.Sonntag, D., & Kiyokawa, K. (2015, March). Attention Engagement and Cognitive State Analysis for Augmented Reality Text Display Functions. In Proceedings of the 20th International Conference on Intelligent User Interfaces (pp. 322-332). ACM.

Based on the gaze and cognitive state of the user, this method intelligently regulates properties of virtual content such as brightness, appearance, and notification timing

Paper 
  2014   
 AR Memory Augmentation

Orlosky, J., Toyama, T., Sonntag, D., Kiyokawa, K. Using Eye-gaze and Visualization To Augment Memory: A Framework for Improving Context Recognition and Recall. In the Proceedings of The 16th International Conference on Human-Computer Interaction (DAPI), LNCS 8530, pp. 282--291. Springer International Publishing Switzerland (2014).

Framework designed to augment human memory through AR.


Paper
 
  Matrix Mode ARrift
Experience the Matrix through Neo's eyes using the ARrift:
This was presented as a demonstration at Osaka University's Ichosai festival in May 2014 and as an informal demo during UIST 2014.  Check out the video of various people, including Mark Bolas, using matrix mode at UIST.
UIST 2014 Demo Video  
 
Combined Gaze Interaction and Language Model

Vero, A., B. Miksztai Rethey, B., Pinter, B., Voros, G., Orlosky, J., Toyoma, T., Sonntag, D., Lorincz. A. Gaze Tracking and Language Model for Flexible Augmentative and Alternative Communication in Practical Scenarios. To appear in the 16th Biennial Conference of the International Society for Augmentative and Alternative Communication (ISAAC), 2014.

Combination of gaze tracking and intelligent language models for facilitating alternative communication.

 
Poster
   
Mobile Alternative Communication
Voros, G., Miksztai Rethey, B., Vero, A., Orlosky, J., Toyoma, T., Sonntag, D., Lorincz. A., Mobile AAC Solutions using Gaze Tracking and Optical Character Recognition. To appear in the 16th Biennial Conference of the International Society for Augmentative and Alternative Communication (ISAAC), 2014.

Method used to Improve mobile communication using gaze and character recognition algorithms.
Presentation