Edge Domain Adaptation for Maritime Situational Awareness

People

Organisations

Maritime Situational Awareness

Australian maritime trade

Our simulation of Sydney Harbour

Civil maritime pressures

Definition of maritime situation awareness  


Project's goal   

To this end, we propose to employ domain adaptation as our primary strategy.


Research question

We recognise there are two fundamental research questions: "How to adapt?" and "When to adapt?".

For the question "How to adapt?", we propose "Human-in-the-loop test-time domain adaptation" method

Regarding the question "When to adapt?", we suggest employing "Assessing the domain gap" approach.

Both methodologies will be detailed in the following sections.

Human-in-the-loop test-time domain adaptation

Novelty

(a) Previous works have focused on developing fully autonomous solutions, primarily for self-driving vehicles. (b) Our approach, however, is proposed for visual surveillance, which are typically monitored by an operator. Therefore, our method will take advantage of the operator's involvement in the adaptation process. Also, In data-driven approaches, data can be viewed as a vital asset of businesses, thus our online adaptation will be source free to avoid data to be stolen. Furthermore, to significantly reduce labour cost of the human operator, our method only requires weak labels for online adaptation; see (c) for the definitions of full and weak labels in object detection

Methodology

Faster-RCNN is used as the object detector. In particular, given each image, the weak labels of human operator is used to compute the image-level loss and instance-level loss.  These losses are then used to update the batch normalisation layers of the network.

Results

Assessing domain gap for continual domain adaptation

Overview of our approach

RetinaNet is pre-trained on source images with full supervision                                                                                            

The outputs from different blocks C3 , C4 , and C5 are used to evaluate the domain gap between the source and target images

Correlation between the domain gap and object detection accuracy

We can see there is a strong correlation between the domain gap and detection accuracy. In particular, if the drop of detection accuracy is significant then the domain gap is large, and vice versa. 

Therefore, we apply "domain gap" as a criterion to decide when to adapt.

Application of domain gap evaluation in domain adaptation

For each target domain, the current RetinaNet is used to evaluate the domain gap between the current training data and target domain (without the need of labels). If the domain gap is found to be small, the target domain is discarded, and no adaptation is needed. However, if the domain gap is found to be large, target domain is added to the current training database to refine the current RetinaNet, resulting in a new RetinaNet model which replaces the current RetinaNet in the operating environment.

Benefits of domain gap evaluation (DGE) in domain adaptation (DA)

DGTA dataset: Target domain is sequentially changed 0h-1h ➜ 2h-3h ➜… ➜ 22h-23h

KITTI-Fog dataset: Target domain is sequentially changed fog-750m ➜ fog-375m ➜ fog-150m ➜ fog-75m ➜ fog-30m

BDD dataset: Target domain is sequentially changed clear-night ➜ cloudy-daytime ➜ overcast-daytime ➜ rainy-daytime ➜ rainy-night ➜ snowy-daytime ➜ snowy-night

Using domain gap as a criterion to decide when to adapt will sacrifices 1% accuracy but saves 50%-90% energy usage for continual domain adaptation

Relevant publications