🗓 3:00 P.M.-3:45 P.M. (Taiwan), 8:00 A.M.-8:45 A.M. (Norway/Germany/France)
The idea that an observer can be placed into an environment that consists partly of real and partly of virtual elements has been around for a few centuries at least. Although researchers have not achieved perfection in this interplay, we have learned a lot while in our attempts to create sufficiently immersive mixed environments.
This talk will give some insights into our experiences in three settings and the challenges that we are facing: The first setting merges a local real-world environment with a remote real-world environment, attempting to create a single, consistent experience. In the second setting, we explore to which degree we can move and act freely in the real-world to experience a virtual world through our avatar. In the third setting, we insert virtual elements into our view of the real world without visual inconsistencies.
We present these settings along with the data types, and our experiences with capture as well as display devices. The data types range from simple audio streams to point clouds, and each has its own potential and limitations. We are going to discuss our current understanding of the limits of human perception, the display devices that can use and that we are still missing, and the limits of current algorithms and processing power.
🗓 3:45 P.M.-4:30 P.M. (Taiwan), 8:45 A.M.-9:30 A.M. (Norway/Germany/France)
3D content is certainly nowadays a legitimate Multi-media content. (With curent regular lockdowns ;) demand on being able to visit, travel to, work, explore, or observe the world ‘online’ is more and more pregnant.
Similarly to computer graphics rendering involving object based and image based algorithms, the first part of this talk is on object based (3D), and the second part on image based (viewpoint dependent).
The first question to be answered is how to efficiently represent the virtual world, or online 3D scenes. Different models based on the context or application have been used like textured meshes in common place applications (e.g. games and e-commerce), parametric models in the CAD/CAM industry, implicit, volumetric models in some medical application. But the chosen representation may also be correlated with the evolution of capturing devices since they also biais the produced models : many capturing device produce (colored) point clouds, and such a representation can be kept and used directly. Image based reconstruction produces 3D point or textured meshes. We now imagine to provide this content online, by storing it on a server and distribute it to online clients.
Once the representation is chosen, it has to be transmitted; compressed models will be transmitted more rapidly, but progressive models maybe be streamed and rendered and observed progressively.
Moreover, different clients will need different resolutions. Thus the transmitted model needs to adapt to the client resources, like bandwidth, screen resolution.
In a second step, we focus on relative part of interest of the model by considering (literally) the client’s view point. The client views its object of interest from a given viewpoint, or navigate through the online environnement at a given position, the world is rendered to us through screens. This biased view of the online content is a chance to adapt the streaming of the model, but also the content representation. Giving more or less navigation freedom lowers or augments the possibility to target the online content needed by the client. Thus, also comes into play, the interaction of a client/user with the online content. We discuss how to define and benefit from interaction tools for the distribution of 3D content. For targeting the right content, and streaming policies, we also need qualitative and quantitative evaluation of the quality of the delivered model (QoS -quality of service) and QoE (quality of experience) of the user. We propose some approaches on 3D content distribution considering view dependent approaches.
🗓 4:30 P.M.-4:45 P.M. (Taiwan), 9:30 A.M.-9:45 A.M. (Norway/Germany/France)
In recent years, with the introduction of powerful HMDs such as Oculus Rift, HTC Vive Pro, the QoE that can be achieved with VR/360° videos has increased substantially. One component of QoE is the perceptual media quality of the viewing session, resulting from different technical characteristics of the end-to-end chain comprising the recording, post-processing, representation, coding, retrieval, transmission, playback and display of 360° videos. Furthermore, especially for 360° video, exploration behaviour, presence and simulator sickness also represent key aspects of QoE. Therefore, besides media quality, these three further constructs need to be assessed for a holistic understanding of QoE in case of omnidirectional videos. It should be noted that such findings are not only restricted to omnidirectional videos but also hold for other interactive video applications, for example, augmented reality, VR telepresence systems, interactive omnidirectional guides, etc. Since different constructs need to be assessed, the holistic and valid evaluation of QoE for 360° videos is a time-consuming task and requires a well-designed protocol. There are International Telecommunication Union (ITU) recommendations (ITUT Rec. P.910 and ITU-R Rec. BT.500-13) that provide guidelines for the assessment of video quality of 2D videos on 2D displays with respective test subjects. However, no such standard document exists for omnidirectional videos. Therefore, it is necessary to develop a set of guidelines to study visual quality assessment for omnidirectional videos. To this aim, this talk answers the following research questions: (1) What are the key aspects that need to be assessed besides media quality? (2) Which evaluation methodology and set of test protocols need to be applied for the assessment of perceived video quality of 360 videos? (3) What set of rating tasks/questionnaire types should be used to assess Simulator Sickness and Presence in the same test run together with quality?
🗓 4:45 P.M.-5:00 P.M. (Taiwan), 9:45 A.M.-10:00 A.M. (Norway/Germany/France)
As providing immersive user experience, Virtual Reality (VR) has become popular in recent years. It allows users to watch natural or virtual scenes with 3DoF (Degree-of-Freedom) interactions at arbitrary viewing orientations through commodity Head-Mounted Displays (HMDs). 3DoF interactions, unfortunately, fail to reflect the position changes of HMD users faithfully. In contrast, 6DoF interactions allow HMD users to freely move within the virtual world for a more immersive user experience. Due to future XR (Extended Reality) applications, e.g., online conferences, sports event broadcast, and remote surgeries, are inherently distributed, their success depends on the effective and efficient 6DoF immersive video streaming. We show challenges, opportunities, and research of 6DoF immersive video streaming in this talk.