Research Projects
Source-Free Domain Adaptation
Pre-trained Networks
Source-free domain adaptation (SFDA) aims to adapt a source model trained on a fully-labeled source domain to an unlabeled target domain. Large-data pre-trained networks are used to initialize source models during source training, and subsequently discarded. However, source training can cause the model to overfit to source data distribution and lose applicable target domain knowledge. We propose to integrate the pre-trained network into the target adaptation process as it has diversified features important for generalization and provides an alternate view of features and classification decisions different from the source model. We propose to distil useful target domain information through a co-learning strategy to improve target pseudolabel quality for finetuning the source model. Evaluation on 4 benchmark datasets show that our proposed strategy improves adaptation performance and can be successfully integrated with existing SFDA methods. Leveraging modern pre-trained networks that have stronger representation learning ability in the co-learning strategy further boosts performance.
Related Publications
“Rethinking the Role of Pre-Trained Networks in Source-Free Domain Adaptation”, [Paper][Poster][Slides]
Wenyu Zhang, Li Shen, Chuan-Sheng Foo,
in ICCV, 2023
@InProceedings{Zhang_2023_ICCV,
author = {Zhang, Wenyu and Shen, Li and Foo, Chuan-Sheng},
title = {Rethinking the Role of Pre-Trained Networks in Source-Free Domain Adaptation},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2023},
pages = {18841-18851}
}
Few-Shot UDA
Deep networks are prone to performance degradation when there is a domain shift between the source (training) data and target (test) data. Recent test-time adaptation methods update batch normalization layers of pre-trained source models deployed in new target environments with streaming data to mitigate such performance degradation. Although such methods can adapt on-the-fy without frst collecting a large target domain dataset, their performance is dependent on streaming conditions such as mini-batch size and class-distribution, which can be unpredictable in practice. In this work, we propose a framework for few-shot domain adaptation to address the practical challenges of data-effcient adaptation. Specifcally, we propose a constrained optimization of feature normalization statistics in pre-trained source models supervised by a small support set from the target domain. Our method is easy to implement and improves source model performance with as few as one sample per class for classifcation tasks. Extensive experiments on 5 cross-domain classifcation and 4 semantic segmentation datasets show that our method achieves more accurate and reliable performance than test-time adaptation, while not being constrained by streaming conditions.
Related Publications
“Few-Shot Adaptation of Pre-Trained Networks for Domain Shift”, [Paper][Code]
Wenyu Zhang, Li Shen, Wanyue Zhang, Chuan-Sheng Foo,
in IJCAI 2022
@inproceedings{ijcai2022p232,
title = {Few-Shot Adaptation of Pre-Trained Networks for Domain Shift},
author = {Zhang, Wenyu and Shen, Li and Zhang, Wanyue and Foo, Chuan-Sheng},
booktitle = {Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, {IJCAI-22}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Lud De Raedt},
pages = {1665--1671},
year = {2022},
month = {7},
url = {https://doi.org/10.24963/ijcai.2022/232},
}
Shadow Optimization from Structured Deep Edge Detection
Local structures of shadow boundaries as well as complex interactions of image regions remain largely unexploited by previous shadow detection approaches. In this paper, we present a novel learning-based framework for shadow region recovery from a single image. We exploit the local structures of shadow edges by using a structured CNN learning framework. We show that using the structured label information in the classification can improve the local consistency of the results and avoid spurious labelling. We further propose and formulate a shadow/bright measure to model the complex interactions among image regions. The shadow and bright measures of each patch are computed from the shadow edges detected in the image. Using the global interaction constraints on patches, we formulate a least-square optimization problem for shadow recovery that can be solved efficiently. Our shadow recovery method achieves state-of-the-art results on the major shadow benchmark databases collected under various conditions.
Related Publications
“Shadow Optimization from Structured Deep Edge Detection”, [arXiv][Poster]
Li Shen, Teck Wee Chua, and Karianto Leman,
in CVPR 2015
@InProceedings{Shen_2015_CVPR,
author = {Shen, Li and Chua, Teck Wee and Leman, Karianto},
title = {Shadow Optimization From Structured Deep Edge Detection},
journal = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2015}
}
BRDF Slices for Material Classification
The surface BRDF can be used to distinguish different materials. The BRDFs of many real materials are near isotropic and can be approximated well by a 2D function. When the camera principal axis is coincident with the surface normal of the material sample, the captured BRDF slice is nearly 1D, which suffers from significant information loss. Thus, improvement in classification performance can be achieved by simply setting the camera at a slanted view to capture a larger portion of the BRDF domain. We further use a handheld flashlight camera to capture a 1D BRDF slice for material classification. This 1D slice captures important reflectance properties such as specular reflection and retro-reflectance. We apply these results on ink classification, which can be used in forensics and analyzing historical manuscripts.
Related Publications
“New Perspective on Material Classification and Ink Identification”, [Pre-print] [Poster]
Rakesh Shiradkar, Li Shen, George Landon, Ping Tan and Ong Sim Heng,
in CVPR 2014
Download Ink Database
@InProceedings{Shiradkar_2014_CVPR,
author = {Shiradkar, Rakesh and Shen, Li and Landon, George and Tan, Ping and Heng, Ong Sim},
title = {New Perspective on Material Classification and Ink Identification},
journal = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2014}
}
Intrinsic Image Decomposition
Sparsity priors on reflectance components
Intrinsic image decomposition targets the recovery of shading and reflectance components from a single image.While this is an ill-posed problem on its own, we propose a novel approach for intrinsic image decomposition using reflectance sparsity priors that we have developed. Our sparse representation of reflectance is based on a simple observation: neighboring pixels with similar chromaticities usually have the same reflectance. We formalize and apply this sparsity constraint on local reflectance to construct a data-driven second-generation wavelet representation. We show that the reflectance component of natural images is sparse in this representation. We further propose and formulate a global sparse constraint on reflectance colors using the assumption that each natural image uses a small set of material colors. Using this sparse reflectance representation and the global constraint on a sparse set of reflectance colors, we formulate a constrained l1-norm minimization problem for intrinsic image decomposition that can be solved efficiently. Our algorithm can successfully extract intrinsic images from a single image, without using color models or any user interaction. Experimental results on a variety of images demonstrate the effectiveness of the proposed technique.
Related publications
“Intrinsic image decomposition using a sparse representation of reflectance”, [Pre-print]
Li Shen, Chuohao Yeo and Binh-Son Hua,
in IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Dec 2013
“Intrinsic Image Decomposition Using a Local and Global Sparse Representation of Reflectance”, [Pre-print] [Slides]
Li Shen and Chuohao Yeo,
in IEEE CVPR 2011. (Oral presentation)
@InProceedings{Shen_2013_PAMI,
author = {Shen, Li and Yeo, Chuohao and Hua, Binh-Son},
title = {Intrinsic Image Decomposition Using a Sparse Representation of Reflectance},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)},
month = {Dec},
year = {2013}
}
@InProceedings{Shen_2011_CVPR,
author = {Shen, Li and Yeo, Chuohao},
title = {Intrinsic Image Decomposition Using a Local and Global Sparse Representation of Reflectance},
journal = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2011}
}
Nonlocal texture constraints
Different from previous work, our method examines texture information to obtain constraints on reflectance among pixels that may be distant from one another in the image. We observe that distinct points with the same intensity-normalized texture configuration generally have the same reflectance value. We formulate the decomposition problem as the minimization of a quadratic function which incorporates both the Retinex constraint and our non-local texture constraint. The separation of shading and reflectance components should thus be performed in a manner that guarantees these non-local constraints. This optimization can be solved in closed form with the standard conjugate gradient algorithm
Related publications
“A Closed-Form Solution to Retinex with Nonlocal Texture Constraints”, [Pre-print]
Qi Zhao, Ping Tan, Qiang Dai, Li Shen, Enhua Wu, and Stephen Lin,
in IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), July 2012
“Intrinsic Image Decomposition with Non-Local Texture Cues”, [Pre-print]
Li Shen, Ping Tan and Stephen Lin,
in CVPR 2008
@InProceedings{Zhao_2012_PAMI,
author = {Zhao, Qi and Tan, Ping and Dai, Qiang and Shen, Li and Wu, Enhua and Lin, Stephen},
title = {A Closed-Form Solution to Retinex with Nonlocal Texture Constraints},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)},
month = {July},
year = {2012}
}
@InProceedings{Shen_2008_CVPR,
author = {Shen, Li and Tan, Ping and Lin, Stephen},
title = {Intrinsic Image Decomposition with Non-Local Texture Cues},
journal = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2008}
}
Outdoor Photometric Stereo and Weather Estimation
We extend photometric stereo to make it work with internet images, which are typically associated with different viewpoints and significant noise. For popular tourism sites, thousands of images can be obtained from internet search engines. With these images, our method computes the global illumination for each image and the surface orientation at some scene points. The illumination information can then be used to estimate the weather conditions (such as sunny or cloudy) for each image, since there is a strong correlation between weather and scene illumination. We demonstrate our method on several challenging examples.
Related Publications
“Photometric stereo and weather estimation using internet images”, [Pre-print]
Li Shen and Ping Tan
in CVPR 2009.
@InProceedings{Shen_2009_CVPR,
author = {Shen, Li and Tan, Ping},
title = {Photometric Stereo and Weather Estimation using Internet Images},
journal = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2009}
}
Interactive Design of All-frequency Lighting
We present an appearance-based user interface for artists to efficiently design customized image-based lighting environments. 1 Our approach avoids typical iterations of parameter editing, rendering, and confirmation by providing a set of intuitive user interfaces for directly specifying the desired appearance of the model in the scene. Then the system automatically creates the lighting environment by solving the inverse shading problem. To obtain a realistic image, all-frequency lighting is used with a spherical radial basis function (SRBF) representation. Rendering is performed using precomputed radiance transfer (PRT) to achieve a responsive speed.
Related Publications
“Illumination Brush: Interactive Design of All-frequency Lighting”, [Pre-print]
Makoto Okabe, Yasuyuki Matsushita, Li Shen and Takeo Igarashi
in PG 2007. (Oral presentation)
@InProceedings{Okabe_2007_CVPR,
author = {Okabe, Makoto and Matsushita, Yasuyuki and Shen, Li and Igarashi, Takeo},
title = {Illumination Brush: Interactive Design of All-Frequency Lighting},
journal = {Proceedings of the 15th Pacific Conference on Computer Graphics and Applications (PG)},
pages = {171-180},
year = {2007}
}
Recovery of Spatially-varying Materials under Complex Illumination
A major challenge in inverse reflectometry is the acquisition of spatially varying materials. We introduce a method to recover spatial reflectance from a sparse set of images under general illumination. Specifically, we first remove the high-frequency varying diffuse reflection term by using a low-order spherical harmonic approximation. This allows us to directly estimate the specular properties with a cluster fitting process, which simplifies the fitting processes and addresses the problem of data inadequacy for sparse images.
Related Publications
“Spatial Reflectance Recovery under Complex Illumination from Sparse Images”, [Pre-print]
Li Shen and Haruo Takemura,
in CVPR 2006
@InProceedings{Shen_2006_CVPR,
author = {Shen, Li and Takemura, Haruo},
title = {Spatial Reflectance Recovery under Complex Illumination from Sparse Images},
journal = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2006}
}
Efficient Photometric Stereo for 3D Objects with Unknown BRDF
Photometric Stereo technique primarily assumes the Lambertian reflection model only. For non-Lambertian objects, large number of images are required to recover 3D shapes and material reflectance. In this work, we propose a new Photometric Stereo technique to efficiently recover a full surface model from a small set of photographs. The proposed technique allows diffuse albedo to vary arbitrarily over surfaces while non-diffuse characteristics remain constant for a material. Specifically, the basic approach is to first recover the specular reflectance parameters of the surfaces by a novel optimization procedure. These parameters are then used to estimate the diffuse reflectance and surface normal for each point. As a result, a lighting-independent model of the geometry and reflectance properties of the surface is established using the proposed method, which can be used to re-render the images under novel lighting via traditional rendering methods.
Related Publications
“Efficient photometric stereo technique for three-dimensional surface”, [Pre-print]
Li Shen, Takeshi Machida and Haruo Takemura
in 3DIM 2005 (Oral penetration)
@InProceedings{Shen_2005_3DIM,
author = {Shen, Li and Machida, Takeshi and Takemura, Haruo},
title = {Efficient Photometric Stereo Technique for 3D Surface},
journal = {Fifth International Conference on 3D Digital Imaging and Modeling (3DIM) },
month = {June},
year = {2005}
}