Semi-Supervised Learning for Multi-Task Scene Understanding by Neural Graph Consensus

Accepted at the 35th AAAI Conference on Artificial Intelligence (AAAI 2021)

AAAI 2021 Paper (coming soon)

Abstract

We address the challenging problem of semi-supervised learning in the context of multiple visual interpretations of the world by finding consensus in a graph of neural networks. Each graph node is a scene interpretation layer, while each edge is a deep net that transforms one layer at one node into another from a different node. During the supervised phase edge networks are trained independently. During the next unsupervised stage edge nets are trained on the pseudo-ground truth provided by consensus among multiple paths that reach the nets' start and end nodes. These paths act as ensemble teachers for any given edge and strong consensus is used for high-confidence supervisory signal. The unsupervised learning process is repeated over several generations, in which each edge becomes a "student" and also part of different ensemble "teachers" for training other students. By optimizing such consensus between different paths, the graph reaches consistency and robustness over multiple interpretations and generations, in the face of unknown labels. We give theoretical justifications of the proposed idea and validate it on a large dataset. We show how prediction of different representations such as depth, semantic segmentation, surface normals and pose from RGB input could be effectively learned through self-supervised consensus in our graph. We also compare to state-of-the-art methods for multi-task and semi-supervised learning and show superior performance.

Idea

We propose the Neural Graph Consensus (NGC) model, a multi-class graph of deep neural networks, which approaches one of the most difficult problems in AI, that of unsupervised learning of multiple scene interpretations. We create a graph (or more generally, a hypergraph) of deep networks, such that each node in the graph represents a different interpretation layer of the world (e.g. semantic segmentation layer, depth layer, motion layer). The edges (or hyperedges) have dedicated deep networks, which predict the layer at one node from the layer, or layers, at one or several other nodes. We term the edges EdgeNets. The deep nets are trained in turn, using as supervision the consensual output of all paths reaching the same output node (when such consensus takes place). Since we do not have strict, well defined “worlds” as in reinforcement learning (RL), we let the natural structure in the data emerge, as in graph clustering. Similar to RL, in which agents in heavily constrained environments with well defined goals evolve by playing against each other, we put pressure on the deep nets by training them one versus the others, in complementary. Thus, each net brings something new, while also learning to agree with the other paths that reach the same output.

Dataset

To test the NGC approach in the case of many scene representations we capture a large dataset in a virtual environment (that we will publicly release), in which a drone flies above a city and learns to predict from a single image the scene depth, the 3D surface normals (both from the world and camera system of reference), its absolute location and orientation (6D pose), the scene wireframe (object boundaries) as well as the semantic segmentation of the scene in 12 classes: building, fence, pedestrian, pole, road line, road, sidewalk, vegetation, vehicle, wall, traffic sign, other (Figure below). The dataset is divided in four: supervised training set (subdivided in 8k images for training and 2k for validation), 2 test sets (10k images each, for unsupervised learning iterations 1 and 2) and a separate evaluation set (10k images, never seen during learning).

Quantitative results

Qualitative results

The plots show performance improvement (green) vs performance degradation (red) on five tasks. We compare the original predictions with two graph iterations and their distilled EdgeNets.

Useful Links

ArXiv Paper

Code

https://gitlab.com/mihaicristianpirvu/cycleconcepts/

https://gitlab.com/mihaicristianpirvu/neural-wrappers/

Dataset

Demo (video)

While our method is based on processing single frames (independently), we present the results in form of a continuous video in order to better assess the quality of the approach in the temporal domain, even though no temporal information was used by our specific NGC implementation - each frame is processed independently of the others.

The camera follows a part of the evaluation set trajectory -- approximately 20% of the total images. This continuous flight path was never seen during training or semi-supervised learning. We provide snippets of five different tasks: depth, semantic segmentation, surface normals with reference to the camera (C), surface normals with reference to the world (W) and wireframe.

Cite

If you intend to use our work please cite the following:

@article{leordeanu2020semi,

title={Semi-Supervised Learning for Multi-Task Scene Understanding by Neural Graph Consensus},

author={Leordeanu, Marius and Pirvu, Mihai and Costea, Dragos and Marcu, Alina and Slusanschi, Emil and Sukthankar, Rahul},

journal={arXiv preprint arXiv:2010.01086},

year={2020}

}

Acknowledgements

This work was supported by UEFISCDI, under Projects EEA-RO-2018-0496 and PN-III-P1-1.2-PCCDI-2017-0734.

We would like to express our gratitude to Aurelian Marcu and The Center for Advanced Laser Technologies (CETAL) for providing us GPU training resources.