Continual Learning and Optimizing DETR

In 2024, during my final year of undergraduate studies in Bangalore, India, I interned at Intel Corporation. My work focused on developing a method for continual learning in object detection, specifically for online, task-free streaming data scenarios. The goal was to enable object detection models to incrementally adapt to new data while retaining knowledge of previously learned classes, supporting lifelong adaptation in dynamic environments.

I explored dynamic architectures for adding new classes without catastrophic forgetting, along with knowledge preservation techniques such as knowledge distillation. I also explored strategies for efficient deployment on edge devices to enable real-time inference under resource constraints. One of my main areas of focus was on Detection Transformer models (DETR) and its variant, Deformable DETR. 

Limitation:

The Deformable DETR model was trained in a continual learning setting using PyTorch on four AWS A10G GPU instances. Due to computational constraints, training was expensive and each epoch took approximately one day.

Optimization:

To improve training efficiency, I explored a mimetic initialization strategy inspired by Trockman and Kolter (2021), originally proposed for Vision Transformers. This method initializes self-attention layers to reflect structural patterns observed in pretrained models, enabling faster convergence without additional pretraining.

Building on this idea, I analyzed learned weight patterns in DETR and designed a similar initialization approach. I observed consistent patterns in query-key and value-projection interactions, which I used to guide a more informed initialization of the attention layers.