03. Space: wormchat

// WORMCHAT: BI-DIRECTIONAL WORMHOLE COMMUNICATION SYSTEM //

The following is a development log of this project.

// BACKGROUND //

A wormhole is a speculative structure that connects two locations in our universe with a space-time bridge that resembles a tunnel between two black holes [1][2]. It is a popular phenomenon in science fiction because it allows for deep space travel within a human lifetime [3]. As wormholes only exist in theory, they have never been observed in the real world and their shape is only known through simulation (Figure I). Corvin Zahn created a visualization of a wormhole using a 4D ray tracer in 2008 (Figure II) [4]. The most important aspect of this structure is that it creates a shortcut between two distant points in a three-dimensional space. This sparked a string of ideas that lead me to experiment with projections in openFrameworks.

Figure I: Representation of a wormhole

Figure II: Still of Zahn's wormhole visualization [4].

// PROCESS //

I started experimenting with face recognition at first with the intention of building a competitive two-player game. Here, one player controls a virtual version of their face by looking directly at a webcam and moving their head in space. The other controls a small spaceship by means of hand gestures above a leap motion. The idea was to create a virtual interface where two people play together face-to-face, but are also separated by a translucent layer of asymmetric digital content. It was not my intention to completely realize this, but to create a rough proof of concept that could communicate the idea at the least. With this in mind, I downloaded the ofxCv and ofxFaceTracker addons as they include the full functionality to make face tracking possible.

I ran into problems trying to compile the ofxCv and ofxFaceTracker addons: the building process failed with linker errors related to these libraries. After consulting the web and trying to fix the problem myself for various hours I discovered that the problem could be solved by changing the Object File Name from $(IntDir) to $(IntDir)/%(RelativeDir) in the project properties of my app in Visual Studio.

After a while, I was able to remap a portion of the webcam texture onto a mesh, which would allow for creating a real-time face mesh that could be transformed in three-dimensional space. This functionality is demonstrated in Figure III where the mesh is positioned independently from the face.

Figure III: A face texture is extracted from the camera image and mapped to an independent face mesh.

This step in development got me thinking further about the possibilities of space and especially with relation to webcams. Video-communication systems allow for a window into each others current whereabouts without the need to be physically proximate to each other. Front-facing cameras have become a standard in the devices that we take with us wherever we go. The cameras provide a window for others to connect to one's current space and form a digital bridge between two locations in our universe. Therefore, webcams appear to share some interesting properties with wormholes.

I started reading about wormholes and found some interesting visualizations online. Looking at the work of Zahn, it seems as if two worlds morph into each other around a hole accompanied by heavy radial distortion. I asked myself what it would look like to look someone else in the eye from up close through a theoretical bridge between two distant points space. As a wormhole lies in three-dimensional space, to make a physical version of this I would have to create two spherical displays on which a full 360-degree view of another space can be viewed. An interesting idea, but unrealistic due to time and budget constraints. Instead, I decided to see how far I could get with the material I could get my hands on. Fortunately, I was able to borrow a Ricoh Theta SC 360-degree camera from Leiden University. Using this and my own laptop, I could build a single client for a bi-directional 360-degree communication system presented on a two-dimensional plane (my monitor).

Video produced by 360 degree cameras is mapped onto a rectangular image using equidistant cylindrical projection. In order to realize my idea in openFrameworks, I had to find a way to display such video on a sphere, so that it would resemble the radial distortion produced by wormholes. The most informative source that helped me learn more about this problem is Paul Bourke's website--a goldmine of great resources and contributions on everything that concerns linear algebra and projections and more 3D math. Especially his page on domes has been of great help to me. http://paulbourke.net/dome.

Eventually, I found a long blog post by Jason Fletcher with useful software and methods to deal with spherical projections (https://thefulldomeblog.com/2013/06/28/fisheye-lens-shader-options). This page lead me to an openFrameworks add-on called Pano2Dome (https://github.com/charlesveasey/Pano2Dome) that featured a shader that transforms equirectangular textures to an internal dome-master representation. The unfortunate limit of this implementation is that the field of view here is limited to about 250 degrees as it does not project onto a full sphere. For future reference, it would be interesting to study how the fish-eye renderer works in FishEyeQuake, an open-source mod of the game Quake by Wouter van Oortmerssen (http://strlen.com/gfxengine/fisheyequake/index.html).

After implementing the capability to display equirectangular video onto a sphere, I implemented real-time headtracking using the readily available webcam to transform the user's view based on their head position. Instead of using the ofxFacetracker add-on--saving some time--I decided to implement simple haarcascades for the detection of faces for better tracking performance. The transformation of the sphere creates an illusion of parallax for the observer, thus, introducing a fake third dimension into a two-dimensional plane. As the movement of the sphere was still a bit unstable I added a low-pass filter to smooth the tracked positions of the head using the ofxBiquadFilter add-on (https://github.com/dzlonline/ofxBiquadFilter).

Finally, I decided to add some themed graphics to the project to give the user the idea that they are video chatting using a futuristic wormhole communication system.

// RESULT //

The following video demonstrates the proof-of-concept version as presented at the NMNT mini-exhibition in class.

Video demonstration of proof-of-concept.

// CODE //

The source code of this project is available on GitHub under the MIT license: https://github.com/lshoek/wormchat

// REFERENCES //

[1] Perkowitz, Sidney. (2010). Wormhole. https://www.britannica.com/science/wormhole

[2] American Physics Society. (1998). Focus: Wormhole Construction: Proceed with Caution. https://physics.aps.org/story/v2/st7

[3] Grush, Loren. (2014). Will wormhole travel ever be possible?. https://www.popsci.com/article/technology/will-wormhole-travel-ever-be-possible

[4] Zahn, Corvin. (2008). Flight through a wormhole. https://www.spacetimetravel.org/wurmlochflug/wurmlochflug.html