Embedded Files
SBAM: Salience-Based Adaptive Masking
We Introduce Saliency-Based Adaptive Masking (SBAM), a novel and cost-effective approach that significantly enhances the pre-training performance of Masked Image Modeling (MIM) approaches by prioritizing token salience. Our method provides robustness against variations in masking ratios, effectively mitigating the performance instability issues common in existing methods. This relaxes the sensitivity of MIM-based pre-training to masking ratios, which in turn allows us to propose an adaptive strategy for ‘tailored’ masking ratios for each data sample, which no existing method can provide. Toward this goal, we propose an Adaptive Masking Ratio (AMR) strategy that dynamically adjusts the proportion of masking for the unique content of each image based on token salience. We show that our method significantly improves over the state-of-the-art in mask-based pre-training on the ImageNet-1K dataset.
We Introduce Saliency-Based Adaptive Masking (SBAM), a novel and cost-effective approach that significantly enhances the pre-training performance of Masked Image Modeling (MIM) approaches by prioritizing token salience. Our method provides robustness against variations in masking ratios, effectively mitigating the performance instability issues common in existing methods. This relaxes the sensitivity of MIM-based pre-training to masking ratios, which in turn allows us to propose an adaptive strategy for ‘tailored’ masking ratios for each data sample, which no existing method can provide. Toward this goal, we propose an Adaptive Masking Ratio (AMR) strategy that dynamically adjusts the proportion of masking for the unique content of each image based on token salience. We show that our method significantly improves over the state-of-the-art in mask-based pre-training on the ImageNet-1K dataset.
Saliency-Based Adaptive Masking (SBAM)
SBAM introduces ‘token salience’ to prioritize and mask tokens with high significance. Hence, it is qualitatively confirmed that particularly important objects with high contribution within the image are selectively masked. Moreover, by integrating randomness with token salience, masks are probabilistically assigned to the background and less significant tokens, enriching the diversity of the token masking.
SBAM introduces ‘token salience’ to prioritize and mask tokens with high significance. Hence, it is qualitatively confirmed that particularly important objects with high contribution within the image are selectively masked. Moreover, by integrating randomness with token salience, masks are probabilistically assigned to the background and less significant tokens, enriching the diversity of the token masking.
The upper graphs display the performance of the methods at different masking ratios, while the lower graphs illustrate the Performance Improvement over Masking Ratio (PIMR) and Global PIMR.
SBAM significantly outperforms MAE in every measure, demonstrating its superior effectiveness in handling various masking ratios and enhanced pre-training performances.
The upper graphs display the performance of the methods at different masking ratios, while the lower graphs illustrate the Performance Improvement over Masking Ratio (PIMR) and Global PIMR.
SBAM significantly outperforms MAE in every measure, demonstrating its superior effectiveness in handling various masking ratios and enhanced pre-training performances.
The left graph displays fine-tuning accuracy, whereas the right graph illustrates linear probing accuracy, both over a range of pre-trained epochs.
The left graph displays fine-tuning accuracy, whereas the right graph illustrates linear probing accuracy, both over a range of pre-trained epochs.
The curves illustrate that SBAM surpasses MAE in pre-training effectiveness in every trained epoch, and also validates its quicker attainment of converged performance levels.
The curves illustrate that SBAM surpasses MAE in pre-training effectiveness in every trained epoch, and also validates its quicker attainment of converged performance levels.
Adaptive Masking Ratio (AMR)
Having masking ratios that adapt throughout training is highly effective, as it allows the masking process to be tailored to each sample in the dataset, accommodating the unique composition and object sizes within each image, as shown in the above qualitative samples.
Having masking ratios that adapt throughout training is highly effective, as it allows the masking process to be tailored to each sample in the dataset, accommodating the unique composition and object sizes within each image, as shown in the above qualitative samples.
The left graph illustrates classification accuracy across epochs, while the right graph shows the accuracy obtained through linear probing.
The left graph illustrates classification accuracy across epochs, while the right graph shows the accuracy obtained through linear probing.
Both results indicate a significant improvement in pre-training performance when AMR is applied, which not only achieves higher accuracy earlier in the training process but also maintains a lead at convergence.
Both results indicate a significant improvement in pre-training performance when AMR is applied, which not only achieves higher accuracy earlier in the training process but also maintains a lead at convergence.
Code and Model Parameters: SBAM
Page updated
Google Sites
Report abuse