Publications

Abstract

Convolutional neural networks (CNNs) learn filters in order to capture local correlation patterns in feature space. We propose to learn these filters as combinations of preset spectral filters defined by the Discrete Cosine Transform (DCT). Our proposed DCT-based harmonic blocks replace conventional convolutional layers to produce partially or fully harmonic versions of new or existing CNN architectures. Using DCT energy compaction properties, we demonstrate how the harmonic networks can be efficiently compressed by truncating high-frequency information in harmonic blocks thanks to the redundancies in the spectral domain. We report extensive experimental validation demonstrating benefits of the introduction of harmonic blocks into state-of-the-art CNN models in image classification, object detection and semantic segmentation applications. 

Bibtex

@article{ulicny2022harmonic,

  title={Harmonic convolutional networks based on discrete cosine transform},

  author={Ulicny, Matej and Krylov, Vladimir A and Dahyot, Rozenn},

  journal={Pattern Recognition},

  volume={129},

  pages={108707},

  year={2022},

  publisher={Elsevier} }

Abstract

Estimation of the Digital Surface Model (DSM) and building heights from single-view aerial imagery is a challenging inherently ill-posed problem that we address in this paper by resorting to machine learning. We propose an end-to-end trainable convolutional-deconvolutional deep neural network architecture that enables learning mapping from a single aerial imagery to a DSM for analysis of urban scenes. We perform multisensor fusion of aerial optical and aerial light detection and ranging (Lidar) data to prepare the training data for our pipeline. The dataset quality is key to successful estimation performance. Typically, a substantial amount of misregistration artifacts are present due to georeferencing/projection errors, sensor calibration inaccuracies, and scene changes between acquisitions. To overcome these issues, we propose a registration procedure to improve Lidar and optical data alignment that relies on Mutual Information, followed by Hough transform-based validation step to adjust misregistered image patches. We validate our building height estimation model on a high-resolution dataset captured over central Dublin, Ireland: Lidar point cloud of 2015 and optical aerial images from 2017. These data allow us to validate the proposed registration procedure and perform 3D model reconstruction from single-view aerial imagery. We also report state-of-the-art performance of our proposed architecture on several popular DSM estimation datasets.

Bibtex

@ARTICLE{Liu2020,

author = {Vladimir A. Krylov and Eamonn Kenny and Rozenn Dahyot},

title = {{IM2ELEVATION}: Building height estimation from single-view aerial imagery},

journal = {Remote Sens.},

year = 2020,

month = {Sept.},

volume = 12,

number = 17 } }

Abstract

Many applications, such as autonomous navigation, urban planning, and asset monitoring, rely on the availability of accurate information about objects and their geolocations. In this paper, we propose the automatic detection and computation of the coordinates of recurring stationary objects of interest using street view imagery. Our processing pipeline relies on two fully convolutional neural networks: the first segments objects in the images, while the second estimates their distance from the camera. To geolocate all the detected objects coherently we propose a novel custom Markov random field model to estimate the objects’ geolocation. The novelty of the resulting pipeline is the combined use of monocular depth estimation and triangulation to enable automatic mapping of complex scenes with the simultaneous presence of multiple, visually similar objects of interest. We validate experimentally the effectiveness of our approach on two object classes: traffic lights and telegraph poles. The experiments report high object recall rates and position precision of approximately 2 meters, which is approaching the precision of single-frequency GPS receivers.

Bibtex

@ARTICLE{Krylov2018a,

author = {Vladimir A. Krylov and Eamonn Kenny and Rozenn Dahyot},

title = {Automatic Discovery and Geotagging of Objects from Street View Imagery},

journal = {Remote Sens.},

year = 2018,

month = {May},

volume = 10,

number = 5} }

Video demo

Abstract

In this paper we address the problem of unsupervised change detection on two or more coregistered images of the same object or scene at several time instants. We propose a novel empirical-Bayesian approach that is based on a false discovery rate formulation for statistical inference on local patch-based samples. This alternative error metric allows to efficiently adjust the family-wise error rate in case of the considered large-scale testing problem. The designed change detector operates in an unsupervised manner under the assumption of the limited amount of changes in the analyzed imagery. The detection is based in the use of various statistical features, which enable the detector to address application-specific detection problems provided an appropriate ad hoc feature choice. In particular, we demonstrate the use of the rank-based statistics: Wilcoxon and Cramer-von Mises for image pairs, and multisample Levene statistic for short image sequences. The experiments with remotely sensed radar, dermatological, and still camera surveillance imagery demonstrate accurate performance and flexibility of the proposed method.

Bibtex

@ARTICLE{Krylov2016b,

author = {V. A. Krylov and G. Moser and S. B. Serpico and J. Zerubia},

title = {False Discovery Rate Approach to Unsupervised Image Change Detection},

journal = {IEEE Trans. Image Process.},

year = 2016,

month = {Oct.},

volume = 25,

number = 10,

pages = {4704--4718} }

Abstract

The accurate statistical modeling of synthetic aperture radar (SAR) images is a crucial problem in the context of effective SAR image processing, interpretation and application. In this paper a semi-parametric approach is designed within the framework of finite mixture models based on the generalized Gamma distribution (GΓD) in view of its flexibility and compact form. Specifically, we develop a generalized Gamma mixture model (GΓMM) to implement an effective statistical analysis of high-resolution SAR images and prove the identifiability of such mixtures. A low-complexity unsupervised estimation method is derived by combining the proposed histogram-based expectation-conditional maximization (ECM) algorithm and the Figueiredo-Jain algorithm. This results in a numerical maximum likelihood (ML) estimator that can simultaneously determine the ML estimates of component parameters and the optimal number of mixture components. Finally, the state-of-the-art performance of this proposed method is verified by experiments with a wide range of high-resolution SAR images.

Bibtex

@ARTICLE{Li2015,

author = {H.-C. Li and V. A. Krylov and P.-Z. Fan and J. Zerubia and W. J. Emery},

title = {Unsupervised Learning of Generalized Gamma Mixture Model with Application in Statistical Modeling of High-Resolution SAR Images},

journal = {IEEE Trans. Geosci. Remote Sens.},

year = 2015,

month = {Apr.},

volume = 54,

number = 4,

pages = {2153--2170} }

Abstract

The automatic extraction of elongated curvilinear structures (CLS) is an important task in various image processing applications, including numerous remote sensing, biometrical and medical problems. To address this task we develop a stochastic approach that relies on a fixed-grid, localized Radon transform for line segment extraction and a Conditional Random Field model to incorporate local interactions and refine the extracted curvilinear structures. We propose several different energy data terms, the appropriate choice of which allows us to process images with different noise and geometry properties. The contribution of this work is the design of a flexible and robust elongated CLS extraction framework that is comparatively fast due to the use of a fixed-grid configuration and a fast deterministic Radon-based line detector. We present several different applications of the developed approach, namely: CLS extraction in mammographic images, road networks extraction from optical remotely sensed images,  and line extraction from palmprint images. The experimental results demonstrate that the method is fairly robust to CLS curvature and can accurately extract blurred and low-contrast elongated CLS.

Bibtex

@ARTICLE{Krylov2014a,

author = {V. A. Krylov and J. D. B. Nelson},

title = {Stochastic extraction of elongated curvilinear structures with applications},

journal = {IEEE Trans. Image Process.},

year = 2014,

month = {Dec.},

volume = 23,

number = 12,

pages = {5360--5373} }

Abstract

Wavelet energy-based lacunarity features, which measure deviations from translational statistical invariance over multiple scales, were recently proposed for object detection and classification in sonar imagery. We here extend the idea to incorporate further robustness to background type whilst retaining sensitivity to local changes in texture caused by the presence of man-made objects. The resulting textural lacunarity features are constructed by estimating the joint distribution of local neighbourhoods with empirical distributions over an adaptive texton dictionary. Experiments on a synthetic aperture sonar imagery dataset suggest that the features offer significant improvements in the receiver operating curve.

Bibtex

@ARTICLE{Nelson2014,

author = {J.D.B. Nelson,  V.A. Krylov},

title = {Textural Lacunarity for semi-supervised Detection in Sonar Imagery},

journal = {IET Radar, Sonar & Navigation},

year = 2014,

month = {Jul.},

volume = 8,

number = 6,

pages = {616--621} }

Abstract

In this paper we develop a novel classification approach for multi-resolution, multi-sensor (optical and synthetic aperture radar, SAR) and/or multi-band images. This challenging image processing problem is of great importance for various remote sensing monitoring applications and has been scarcely addressed so far. To deal with this classification problem, we propose a two-step explicit statistical model. We first design a model for the multivariate joint class-conditional statistics of the co-registered input images at each resolution by resorting to multivariate copulas. Such copulas combine the class-conditional marginal probability density functions of each input channel that are estimated by finite mixtures of well-chosen parametric families. We consider different distribution families for the most common types of remote sensing imagery acquired by optical and SAR sensors. We then plug the estimated joint probability density functions into a hierarchical Markovian model based on a quad-tree structure, where each tree-scale corresponds to the different input image resolutions and to corresponding multi-scale decimated wavelet transforms, thus preventing a strong resampling of the initial images. To obtain the classification map, we resort to an exact estimator of the marginal posterior mode. We integrate a prior update in this model in order to improve the robustness of the developed classifier against noise and speckle. The resulting classification performance is illustrated on several remote sensing multi-resolution datasets including very high resolution and multi-sensor images acquired by COSMO-SkyMed and GeoEye-1.

Bibtex

@ARTICLE{Krylov2014a,

author = {A. Voisin and V. A. Krylov and G. Moser and S. B. Serpico and J. Zerubia},

title = {Supervised Classification of Multi-sensor and Multi-resolution Remote Sensing Images with a Hierarchical Copula-based Approach},

journal = {IEEE Trans. Geosci. Remote Sens.},

year = 2013,

month = {Jun.},

volume = 52,

number = 6,

pages = {3346--3358} }

Abstract

Parameter estimation of probability density functions is one of the major steps in the area of statistical image and signal processing. In this paper we explore several properties and limitations of the recently proposed method of logarithmic cumulants (MoLC) parameter estimation approach which is an alternative to the classical maximum likelihood (ML) and method of moments (MoM) approaches. We derive the general sufficient condition for a strong consistency of the MoLC estimates which represents an important asymptotic property of any statistical estimator. This result enables the demonstration of the strong consistency of MoLC estimates for a selection of widely used distribution families originating from (but not restricted to) synthetic aperture radar (SAR) image processing. We then derive the analytical conditions of applicability of MoLC to samples for the distribution families in our selection. Finally, we conduct various synthetic and real data experiments to assess the comparative properties, applicability and small sample performance of MoLC notably for the generalized gamma and K families of distributions. Supervised image classification experiments are considered for medical ultrasound and remote-sensing SAR imagery. The obtained results suggest that MoLC is a feasible and computationally fast yet not universally applicable alternative to MoM. MoLC becomes especially useful when the direct ML approach turns out to be unfeasible.

Bibtex

@ARTICLE{Krylov2013c,

author = {V. A. Krylov and G. Moser and S. B. Serpico and J. Zerubia},

title = {On the Method of Logarithmic Cumulants for Parametric Probability Density Function Estimation},

journal = {IEEE Trans. Image Process.},

year = 2013,

month = {Oct.},

volume = 22,

number = 10,

pages = {3791--3806} }

Abstract

This letter addresses the problem of classifying synthetic aperture radar (SAR) images of urban areas by using a supervised Bayesian classification method via a contextual hierarchical approach. We develop a bivariate copula-based statistical model that combines amplitude SAR data and textural information, which is then plugged into a hierarchical Markov random field model. The contribution of this letter is thus the development of a novel hierarchical classification approach that uses a quad-tree model based on wavelet decomposition and an innovative statistical model. The performance of the developed approach is illustrated on a high-resolution satellite SAR image of urban areas.

Bibtex

@ARTICLE{Krylov2013a,

author = {A. Voisin and V. A. Krylov and G. Moser and S. B. Serpico and J. Zerubia},

title = {Classification of Very High Resolution {SAR} Images of Urban Areas Using Copulas and Texture in a Hierarchical {M}arkov Random Field Model},

journal = {IEEE Geosci. Remote Sens. Lett.},

year = 2013,

month = {Jan.},

volume = 10,

number = 1,

pages = {96--100} }

Abstract

In this paper a novel supervised classification approach is proposed for high resolution dual polarization (dualpol) amplitude satellite synthetic aperture radar (SAR) images. A novel probability density function (pdf) model of the dual-pol SAR data is developed that combines finite mixture modeling for marginal probability density functions estimation and copulas for multivariate distribution modeling. The finite mixture modeling is performed via a recently proposed SAR-specific dictionarybased stochastic expectation maximization approach to SAR amplitude pdf estimation. For modeling the joint distribution of dual-pol data the statistical concept of copulas is employed, and a novel copula-selection dictionary-based method is proposed. In order to take into account the contextual information, the developed joint pdf model is combined with a Markov random field approach for Bayesian image classification. The accuracy of the developed dual-pol supervised classification approach is validated and compared with benchmark approaches on two high resolution dual-pol TerraSAR-X scenes, acquired during an epidemiological study. A corresponding single-channel version of the classification algorithm is also developed and validated on a single polarization COSMO-SkyMed scene.

Bibtex

@ARTICLE{Krylov2011b,

author = {V. A. Krylov and G. Moser and S. B. Serpico and J. Zerubia},

title = {Supervised High-Resolution Dual-Polarization {SAR} Image Classification by Finite Mixtures and Copulas},

journal = {IEEE J. Sel. Top. Signal Process.},

month = {Jun.},

year = 2011,

volume = 5,

number = 3,

pages = {554--566} }

Abstract

In this letter, we address the problem of estimating the amplitude probability density function (pdf) of single-channel synthetic aperture radar (SAR) images. A novel flexible method is developed to solve this problem, extending the recently proposed dictionary-based stochastic expectation maximization approach (developed for a medium-resolution SAR) to very high resolution (VHR) satellite imagery, and enhanced by introduction of a novel procedure for estimating the number of mixture components, that permits to reduce appreciably its computational complexity. The specific interest is the estimation of heterogeneous statistics, and the developed method is validated in the case of the VHR SAR imagery, acquired by the last-generation satellite SAR systems, TerraSAR-X and COSMO-SkyMed. This VHR imagery allows the appreciation of various ground materials resulting in highly mixed distributions, thus posing a difficult estimation problem that has not been addressed so far. We also conduct an experimental study of the extended dictionary of state-of-the-art SAR-specific pdf models and consider the dictionary refinements.

Bibtex

@ARTICLE{Krylov2011a,

author = {V. A. Krylov and G. Moser and S. B. Serpico and J. Zerubia},

title = {Enhanced Dictionary-Based {SAR} Amplitude Distribution Estimation and Its Validation With Very High-Resolution Data},

journal = {IEEE Geosci. Remote Sens. Lett.},

year = 2011,

month = {Jan.},

volume = 8,

number = 1,

pages = {148--152} }