About

Visiting Researcher at Microsoft Research, Interactive 3D Technologies group, August 2015 – October 2016.
Contractor / Research Intern at Microsoft Research, Mobility & Networking Research group, May 2014 – August 2015.
R&D Graphics Software Engineer at Transas, June 2008 – May 2014.
Lecturer at St. Petersburg Polytechnic University, September 2013 – August 2015.
Contractor / Research Intern at Microsoft Research, MIC group, June 2013 – August 2015.

M.S. and B.S. in Applied Mathematics and Computer Science from St. Petersburg Polytechnic UniversityApplied Mathematics Dept., 2007 – 2013. 


Contact email: degtyarev.y@gmail.com


Last update: October 2016

Research

Publications

1. Holoportation: Virtual 3D Teleportation in Real-time.
    S. Orts, C. Rhemann*, S. Fanello*, W. Chang*,  A. Kowdle*, Y. Degtyarev*D. Kim*,
 P. Davidson*, S. Khamis*, M. Dou*, V. Tankovich*, C. Loop*, Q. Cai*, P. Chou*,
 S. Mennicken, J. Valentin, P. Kohli, V. Pradeep, S. Wang, Y. Lutchyn, C. Keskin, S. Izadi*.
    ACM Symposium on User Interface Software and Technology (UIST), 2016.
    [PDF] [video[ACM DL]

2. Fusion4D: Real-time Performance Capture of Challenging Scenes.
    M. Dou, S. Khamis, Y. Degtyarev, P. Davidson*, S.R. Fanello*, A. Kowdle*, S. Orts Escolano*, C. Rhemann*,
 D. Kim, J. Taylor, P. Kohli, V. Tankovich, S. Izadi.
    ACM SIGGRAPH and Transactions on Graphics, 2016.
    [PDF] [video] [ACM DL]


3. Outatime: Using Speculation to Enable Low-Latency Continuous Interaction for Cloud Gaming.
    K. Lee, D. Chu, E. Cuervo, J. Kopf, Y. Degtyarev, S. Grizan, A. Wolman, J. Flinn.
    MobiSys, 2015. Best Paper Award
    [PDF] [video] [ACM DL]



4. Volumetric lighting of particle system using compute shaders.
    Yury Degtyarev, Sergey Zhukov. 
    
In Proceedings of the XLII Week of Science in SPbPU conference, 2014.
    Master's thesis. [rus], 2013
    [description] [
video1] [video2] [screenshots[slides: rus[week-of-science thesis preprint: rus]


5. Visualization of dynamic effects on the water surface.
    Yury Degtyarev, Sergey Zhukov. 
    In Proceedings of the XLI Week of Science in SPbPU conference, 2012.
    Bachelor's thesis.
 [rus], 2011
    [description] [video1] [video2] [screenshots] [abstract: eng] [slides: rus] [week-of-science thesis: rus]


6. Visualization of volumetric signs on 3D topographical maps.
    Yury Degtyarev, Peter Azmanov. 
    In Proceedings of the XLI Week of Science in SPbPU conference, 2012.
    Best Paper Award, in nationwide competition by Moscow Institute of Physics and Technology (in top 16 out of 2000 participants).
    [description] [video] [screenshots
] [week-of-science thesis: rus] [student's research report: rus]


7. Redistribution of projective grid's density.
    Yury Degtyarev
    Student's research report. St. Petersburg Polytechnic University, 2011.
    [screenshots
] [student's research report: rus]
  

Demos

Irides: Attaining Quality, Responsiveness and Mobility for Virtual Reality Head-mounted Displays.
Y. Degtyarev, E. Cuervo, D. Chu.
MobiSys, 2015.



Patents

1. Predictive server-side rendering of scenes. US patent, 2015
 
2. Reconstructing viewport upon user viewpoint misprediction. US patent, 2015

3. Synthesizing second eye viewport using interleaving. US patent, 2015


Conference talks

Visualization of dynamic effects on the water surface. 
Russian Game Developers Conference (KRI), Moscow, May 2013.
    [slides: rus]

Talks / Presentations

1. Untethered immersive stereo for cloud gaming.  Microsoft Research Vision & Graphics groups talk, August 2014.
 
2. Using CUDA for performance improvements in human body tracking algorithms.  Microsoft Research MIC group talk, September 2013.
 
3. Volumetric lighting of particle system using compute shaders.  Microsoft Research Vision & Graphics groups talk, August 2013.
    [slides: eng]

Selected Projects

Microsoft Research (Redmond, WA, USA)

holoportation: virtual 3D teleportation in real-time


Holoportation is a new type of 3D capture technology that allows high-quality 3D models of people to be reconstructed, compressed and transmitted anywhere in the world in real time. When combined with mixed reality displays such as HoloLens, this technology allows users to see, hear, and interact with remote participants in 3D as if they are actually present in the same physical space. Communicating and interacting with remote users becomes as natural as face-to-face communication.

Fusion4D: Real-time Performance Capture of Challenging Scenes


A key component of holoportation is its non-rigid surface reconstruction pipeline. We contribute a new pipeline for live multi-view performance capture, generating temporally coherent high-quality reconstructions in real-time. Our algorithm supports both incremental reconstruction, improving the surface estimation over time, as well as parameterizing the nonrigid scene motion. Our approach is highly robust to both large frame-to-frame motion and topology changes, allowing us to reconstruct extremely challenging scenes. We demonstrate advantages over related real-time techniques that either deform an online generated template or continually fuse depth data nonrigidly into a single reference model. Finally, we show geometric reconstruction results on par with offline methods which require orders of magnitude more processing time and many more RGBD cameras.

Untethered Immersive Stereo for Cloud Gaming

No foveation
Foveation to increase performance and save bandwidth
Click on screenshots for full-size images



I contributed to enabling untethered high-quality virtual reality with low perceived network latency through cloud assistance. The project explores collaborative rendering for mobile devices, speculative execution, perceptual likelihood-based foveation, and stereo graphics techniques (like reprojection, view interpolation); and it is built on top of the existing commercial games Doom 3 and Fable 3. Thin client requires only video decode and modest GPU capabilities. My work resulted in a stereo prototype (Irides), a few contributions to the existing mono prototype (DeLorean/Outatime), three patents, and accepted publications. 
Irides stereo demo was presented at Microsoft Research Devices and Networking Summit 2015. Also, both prototypes got coverage in news (e.g., Irides  NeowinMicrosoft News, VG247; Outatime/DeLorean – TechCrunch, Engadget, The Verge).

Human Body Tracking and Model Estimation


During my internship at Microsoft Research, I improved the performance of two tracking models: TenBo (CVPR’13) and SkinnedRig. I received an overall performance boost about 100x for SkinnedRig by implementing the given algorithm on the GPU using CUDA. In addition, several enhancements to the algorithm itself were made. I continued to work on SkinnedRig model as a research consultant and developed algorithms for learning model parameters (like skinning weights) from depth information on the GPU. Constraints and considerably bigger problem sizes made it especially challenging.


Transas (Saint Petersburg, Russia)


Volumetric lighting of particle system

Our method
Fourier Opacity Mapping method

I proposed and implemented a novel participating media lighting technique for real-time applications. Most modern approaches are based on approximation of transmittance functions with piecewise-linear curves, a combination of basis functions, or particles’ volume voxelization. Due to these approximations flickering or ringing artifacts can occur. I proposed a method of exact transmittance function computation for so-called “query points”. Given a grid of query points, we can perform transmittance function evaluation for them on the GPU and interpolate transmittance values for vertices of each particle (or for a grid per particle to increase sampling in the shadow plane). This algorithm has linear complexity, the same as existing approaches, but performs better on high-frequency, inhomogeneous volumetric data, which results in better overall quality. Future work includes exploring different parallel computation patterns, better cache utilization, and dealing with undersampling in shadow map plane. In this project, I've also implemented voxelization of particles density (as a prototype), radix sort on the GPU (for particles blending in correct order), particles shadow casting/receiving (with a new prototype for correct translucent shadows near depth discontinues), physical simulation using compute shaders, tessellation, LOD system, and efficient light sources binning using projective textures.


Physically based lighting


I implemented "physically based lighting" for our engine. Now the lighting is calculated in physical values like radiance and fully HDR rendering with eye adaptation is performed (tonemapping, bloom). For the sky model, I used Hosek's sky model, which I adapted for negative sun elevations. An illumination from the sky is included as an ambient component and the integration over hemisphere is performed on the GPU for irradiance map construction. 

The main problem was that the change of lighting system requires big amount of work in order to keep previous engine features. The consolidated commit in the version control system had modifications of over 200 source files. As a result, besides better image look, now it is easy to implement new rendering features "without hacks" and for artists it is more convenient to control visual parameters. In addition, our engine support day and night settings and requires no light sources' calibration due to this change.


Shadow system


I renewed the previous shadow system and added support of multiple spot shadow casting light sources. I worked on LiSPSM projection implementation and improvement (stability, optimality, and splits support), it was used for shadow warping transformation for big outdoor scenes. In order to support many light sources and shadow receivers we needed fast and efficient clipping system, which is described in the next section. I also implemented screen-space shadows for deferred renderer and worked on various filtration methods (screen-space bilateral, PCF, ESM). Now it is possible to have ~20 real-time shadow casting light sources in our engine with a decent framerate.


OBB hierarchical clipping system


I have built fast scene graph clipping system, nodes can be discarded by arbitrary set of convex bodies. This was needed to determine which visible geometry parts can be affected by light sources and shadows for ones. These operations can be expressed as intersections of convex bodies, and due to big amount of those, they should be fast. I have implemented convex body computational geometry, which outperforms Ogre3D-based implementation by 10x. Hierarchical information is also taken into account by using masking bits, fully-inside, fully-outside tests, and inter-frame coherency information. Overall performance gain with clipping was more than 50%.


Dynamic effects on the water surface


In this project, I proposed and implemented technique for local modification of water surface without increase of polygon budget by using parallax methods. Current approaches tend to represent water surface's vertex density uniformly or almost uniformly in screen-space. We found that these methods are unable to represent highly detailed and local effects even with significant increase of polygon budget, which results in poor performance. These effects are wakes after moving ships, splashes, hits of raindrops. Such dynamic effects require higher level of detail than the base water surface – ocean, sea, river or lake. Since we have uniformly detailed base water surface and highly detailed local dynamic effects, we decouple the resulting water surface representation. Dynamic effects' surface is represented as 2D heightmap with projective transform LiSPSM. We use Parallax Occlusion Mapping to determine final ray-surface intersection, a number of problems with it were solved: tangent space discontinuity on wave peaks, local approximation of base surface with tangent plane, limited tracing volume, edge-on parallax artifacts. We also utilize locality of dynamic effects: ray tracing occurs only for pixels, which are marked in special screen mask texture with rough representation of effects location on the screen.


Volumetric signs


For this project, I implemented volumetric signs rendering on 3D topographical maps for GIS applications. If we render those signs as polygonal meshes, several problems come up: we need to change signs geometry when moving or animation of the sign occurs, which requires tessellation or simplification; secondly, we need to query new terrain heights, and representation of those cannot be done with ideal precision. I proposed different approach by rendering signs as combination of caps and walls. For each part we can render the signs with pixel accuracy without querying terrain height or any geometrical manipulation: I combined ideas from shadow volumes (z-fail) and depth peeling algorithms in order to achieve this. As a result, we need only 2D contours of the signs to receive automatically their 3D image on the terrain.

Screenshots



Videos


Volumetric lighting ‎‎‎‎(day)‎‎‎



Volumetric lighting ‎‎(night)‎‎



Water FX



Water FX Day



Volumetric signs