Experiments

Result: Each operator in the designed PL-NL augmentation is effective in improving model robustness and when incorporating them, the robustness of pre-trained models is further enhanced.

Result. Through contrastive pre-training task to learn augmented variants constructed by a set of PL-NL operators, ContraBERT is able to group the semantically equivalent samples and push away the dissimilar samples, thus learn better vector representations.

Result. ContraBERT comprehensively improves the performance of original CodeBERT and GraphCodeBERT on four downstream tasks, we attribute the improvements to the enhanced robustness of the model has better performance on these tasks.

Result. Masked language modeling (MLM) and contrastive pre-training task play a different role for ContraBERT. When combining them together, the model achieves higher performance on different downstream tasks.