Depth Estimation for Indoor Panoramas through Merging Multiple Perspective Monocular Depth Predictions

WACV Nov 2022  

We propose a method to estimate depths for indoor panoramas by merging multiple perspective depth maps produced by modern monocular depth estimation methods such as LeReS. The challenge is that the perspective depth maps, each covering a different subset of the panorama, tend to have different scales and shifts of the predicted depth values. Simply stitching them together led to inconsistent depth values for the whole panorama with visible seams. To address the challenge, we propose a novel approach to solve a single depth map for the whole panorama with information taken from each perspective depth estimations and a common panoramic depth map served as the reference for merging. 

Distortion Reduction for Off-Center Perspective Projection of Panoramas 

Best Short Paper Award    NICOGRAPH Nov 2021  

In this paper, we propose modifications to the equirectangular-to-perspective (E2P) projection that significantly reduce distortions when the camera position is away from the origin. This enables users to not only "look around" but also "walk around" virtually in a single panorama with more convincing renderings. We compare with other techniques that aim to augment panoramas with 3D information, including 1) panoramas with depth information and 2) panoramas augmented with room layouts, and show that our approach provides more visually convincing results.

AI Infra · Chapter 06.03: New Architectures

2023 Open source technical writing · Infrasys-AI · Apache-2.0

Section coverage:

- RNN → Transformer → SSM architectural evolution and efficiency/expressiveness trade-offs

- Structured State Space Models (S4): mathematical foundations, HiPPO matrix, and diagonal-plus-low-rank parameterization

- Mamba selective SSM: input-dependent parameters, hardware-aware parallel scan implementation, and comparison with Transformer/RNN

- MoE-Mamba: mixture-of-experts integration with SSMs; training/inference efficiency gains 

 NaniteTree Pipeline · Project Page

Invited Talk · Epic Games Shanghai · 2023 

An open-source automated pipeline that converts traditional non-Nanite vegetation models into Nanite-compatible geometry for Unreal Engine 5, bridging Houdini Digital Assets and a companion Unreal plugin for end-to-end asset-authoring workflows. 

 UnrealEngineGPT · Project Page

Honor Certificate · InternLM · Shanghai AI Lab · 2024 

An end-to-end open-source system combining a domain-specialized LLM for Unreal Engine (documentation Q&A, code generation) with an in-editor agentic plugin that drives Stable Diffusion + ControlNet workflows to generate 3D asset textures directly on selected actors.