Towards Inheritable Models for Open-Set Domain Adaptation
CVPR 2020 (Oral) Paper
Jogendra Nath Kundu* Naveen Venkat* Ambareesh Revanur Rahul M V R. Venkatesh Babu
Video Analytics Lab, Indian Institute of Science, Bangalore
Embedded Files
Citation
Citation
If you find our work helpful in your research, please cite our work:
@inproceedings{kundu_cvpr_2020,
title={Towards Inheritable Models for Open-Set Domain Adaptation},
author={Kundu, Jogendra Nath and Venkat, Naveen and Revanur, Ambareesh and M V, Rahul and Babu, R. Venkatesh},
inproceedings={The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month={June},
year={2020}
}
Licence
Licence
This project is licenced under an [MIT License].
Contact
Contact
If you have any queries, please get in touch via email : jogendranathkundu@gmail.com
Setup and Training Guide
Setup and Training Guide
1. Setup
1. Setup
This project is organized into 4 folders - code, data, summaries, weights.
Ensure that you have Python 2.7 and all the required dependencies installed.
pip install -r requirements.txt
Update the server_root_path variable (in code/inheritune_base/server_path.py file) to point to the folder where the project is downloaded.
Download and organize datasets inside the folder data/ (see data/readme.txt for more details).
1.1. Pre-processing datasets for experiments
1.1. Pre-processing datasets for experiments
To setup the dataset, we create the images along with their augmentations offline. Set dataset_dir and dataset_exp_names in the files code/inheritune_base/create_dataset.py and code/inheritune_base/augment_dataset.py. In create_dataset.py, set the datasets_sources and dataset_target variables to the names of the source and the target data folders. In augment_dataset.py, set dataset_name and experiments variables. An example is provided in the files.
Then execute the following in order:
python create_dataset.py
python augment_dataset.py
1.2 A note on the organization of datasets
1.2 A note on the organization of datasets
We use the following naming convention. The dataset_name corresponds to the folder inside data/ directory that contains the raw dataset. For Office-31, we have dataset_name = "office_31_dataset". The name of the processed dataset for Office-31 A->D task is of the form "inheritune_office_31_AtoD". Further note that names of the classes should not contain spaces (eg. rename the Office-Home class "Alarm Clock" to "Alarm_Clock").
The first num_C classes (class indices) are considered as shared classes. The next num_Ct_uk class indices in the target dataset are treated as target-unknown. This is indicated as follows:
Shared Classes: [0, 1, ..., num_C-1]
Target Private: [num_C, num_C+1, ..., num_Ct_uk]
1.3 Hardware Requirements
1.3 Hardware Requirements
This project requires an NVIDIA-GPU with CUDA 8.0 to run PyTorch with GPU computing capabilities. The GPU ID can be defined in settings['gpu'] in both code/inheritune_base/config_supervised.py and code/inheritune_base/config_adapt.py.
2. Vendor trains an Inheritable Model on the source domain
2. Vendor trains an Inheritable Model on the source domain
To train the vendor's source model, set the following parameters in code/inheritune_base/config_supervised.py:
settings['exp_name']: Name of the experiment to be performed (used for loading weights and continuing from checkpoints)
settings['dataset_exp_name']: Name of the dataset on which experiment should be performed. This is defined in code/inheritune_base/create_dataset.py, and used in code/inheritune_base/augment_dataset.py.
settings['max_iter']: Maximum number of iterations to train
settings['val_after']: Number of iterations after which to validate
settings['batch_size']: Number of positive (or negative) images per batch
settings['load_weights']: True / False (whether to load weights or not)
settings['load_exp_name']: If load_weights is True, what is the experiment name from which weights should be loaded. Modify this to the exp_name given to the pretraining step.
Note: Delete the files spliced_m_feats.npy and spliced_m_kmeans_labels.npy in the folder code/inheritune_base, between two sessions of inheritable model training.
Once the above settings are confirmed, execute:
python Train_supervised.py
The experiment statistics will be logged to tensorboard in the folder summaries/experiment_name, where experiment_name is defined in settings['exp_name']. These statistics can be viewed by running:
cd summaries/experiment_name
tensorboard --logdir .
The weights of the corresponding experiment are saved in weights/experiment_name.
2.1. Tip: Adding Custom Losses
2.1. Tip: Adding Custom Losses
This code base is extensible, in that one can include as many loss functions as required. For instance, to add a new loss function while training the inheritable model, one can create a dictionary key for settings['use_loss'], settings['optimizer'] in config_supervised.py and can include the loss definition in the function get_loss() defined in code/inheritune_base/Trainer_supervised.py. The code will take care of creating a new Adam optimizer for the loss and will follow an alternative minimization scheme. Here is an example of a custom loss:
Modify the dictionary items in code/inheritune_base/config_supervised.py:
settings['use_loss'] = {
'custom_loss': True # Set it to true, to enforce the loss.
}
settings['optimizer'] = {
'custom_loss': [<list of components to be trained using custom_loss>]
}
In the example above, a new optimizer will be created for the loss named 'custom_loss' which will take the parameters of the components provided in the list.
Add the definition of the loss in the function get_loss() defined in code/inheritune_base/Trainer_supervised.py:
def get_loss(self, which_loss):
...
elif which_loss == 'custom_loss': # Add a new condition elif condition
# Use activations stored in self.features
pass
...
self.summary_dict['loss/' + str(which_loss)] = loss.data.cpu().numpy()
return loss
Optionally, should you wish to enforce a set of losses before / after a few iterations (settings['val_after']), you may include the loss name in settings['losses_before_enough_iters'] / settings['losses_after_enough_iters'] in the file code/inheritune_base/config_supervised.py. For e.g.
settings['losses_after_enough_iters'] = ['custom_loss']
2.2. Tip: Adding new model components
2.2. Tip: Adding new model components
The code also supports the addition of new model components. To achieve this, add new components to modnet class in code/inheritune_base/net.py. For instance,
class modnet(nn.Module):
def __init__(self, num_C, num_Ct_uk, cnn=cnn_to_use, additional_components=[]):
super(modnet, self).__init__()
...
self.Custom_Component = nn.Linear(512, 512)
...
self.components = {
'M': self.M,
'Es': self.Es,
'Et': self.Et,
'Gs': self.Gs,
'Gn': self.Gn,
'CC': self.Custom_Component, # Add the dictionary item to make it visible to the rest of the code base
}
The forward-prop happens the function forward() in the file code/inheritune_base/Trainer_supervised.py. One can easily explore various strategies to train inheritable models by modifying these functions.
3. Client adapts to the target domain
3. Client adapts to the target domain
During adaptation, an inheritable model is given to the Client. Here, ensure that code/inheritune_base/config_adapt.py is set up before running the adaptation. Note that the performance on the target domain depends upon the model inheritability of the vendor model. The vendor model parameters are loaded from the experiment name mentioned in settings['load_exp_name'] defined in code/inheritune_base/config_adapt.py. For best results, modify the maximum iterations etc. according to the dataset and the number of training samples.
Ensure that the experiment name (settings['exp_name']) is appended with '_adapt'. This will be useful when continuing adaptation experiments from a previous checkpoint. In the current formulation, if settings['load_exp_name'] is appended with '_adapt', the model does not overwrite the weights of Et (which would otherwise happen while initializing Et from Es, during the adaptation phase).
Once the settings are confirmed, execute:
python Train_adapt.py
The experiment statistics will be logged to tensorboard at summaries/experiment_name, where experiment_name is defined in settings['exp_name']. These can be viewed by executing:
cd summaries/experiment_name
tensorboard --logdir .
The weights of the corresponding experiment are saved in weights/experiment_name.
3.1. Tip: Extensibility of code
3.1. Tip: Extensibility of code
Similar to inheritable model training, the sub-modules for adaptation (i.e. config_adapt.py and Trainer_adapt.py) support adding custom loss functions, modifying forward call etc (similar to the submodules for the vendor model training). Thus, the adaptation can be improved by exploring new training strategies.
Google Sites
Report abuse