Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training

Hyesong Choi; Hyejin Park; Kwang Moo Yi; Sungmin Cha; Dongbo Min

doi:10.1007/978-3-031-73229-4_20

Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training

Hyesong Choi
, Hyejin Park
, Kwang Moo Yi
, Sungmin Cha
, Dongbo Min

Department of Computer Science & Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

In this paper, we introduce Saliency-Based Adaptive Masking (SBAM), a novel and cost-effective approach that significantly enhances the pre-training performance of Masked Image Modeling (MIM) approaches by prioritizing token salience. Our method provides robustness against variations in masking ratios, effectively mitigating the performance instability issues common in existing methods. This relaxes the sensitivity of MIM-based pre-training to masking ratios, which in turn allows us to propose an adaptive strategy for ‘tailored’ masking ratios for each data sample, which no existing method can provide. Toward this goal, we propose an Adaptive Masking Ratio (AMR) strategy that dynamically adjusts the proportion of masking for the unique content of each image based on token salience. We show that our method significantly improves over the state-of-the-art in mask-based pre-training on the ImageNet-1K dataset. Code and model parameters are available at https://github.com/doihye/SBAM.

Original language	English
Title of host publication	Computer Vision – ECCV 2024 - 18th European Conference, Proceedings
Editors	Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, Gül Varol
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	343-359
Number of pages	17
ISBN (Print)	9783031732287
DOIs	https://doi.org/10.1007/978-3-031-73229-4_20
State	Published - 2025
Event	18th European Conference on Computer Vision, ECCV 2024 - Milan, Italy Duration: 29 Sep 2024 → 4 Oct 2024

Publication series

Name	Lecture Notes in Computer Science
Volume	15136 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	18th European Conference on Computer Vision, ECCV 2024
Country/Territory	Italy
City	Milan
Period	29/09/24 → 4/10/24

Bibliographical note

Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

Keywords

Masked autoencoder
Masked image modelling
Self-supervised learning

Access to Document

10.1007/978-3-031-73229-4_20

Cite this

Choi, H., Park, H., Yi, K. M., Cha, S., & Min, D. (2025). Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, & G. Varol (Eds.), Computer Vision – ECCV 2024 - 18th European Conference, Proceedings (pp. 343-359). (Lecture Notes in Computer Science; Vol. 15136 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-73229-4_20

Choi, Hyesong ; Park, Hyejin ; Yi, Kwang Moo et al. / Salience-Based Adaptive Masking : Revisiting Token Dynamics for Enhanced Pre-training. Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. editor / Aleš Leonardis ; Elisa Ricci ; Stefan Roth ; Olga Russakovsky ; Torsten Sattler ; Gül Varol. Springer Science and Business Media Deutschland GmbH, 2025. pp. 343-359 (Lecture Notes in Computer Science).

@inproceedings{735365e37e0b429297c80bf95730bc63,

title = "Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training",

abstract = "In this paper, we introduce Saliency-Based Adaptive Masking (SBAM), a novel and cost-effective approach that significantly enhances the pre-training performance of Masked Image Modeling (MIM) approaches by prioritizing token salience. Our method provides robustness against variations in masking ratios, effectively mitigating the performance instability issues common in existing methods. This relaxes the sensitivity of MIM-based pre-training to masking ratios, which in turn allows us to propose an adaptive strategy for {\textquoteleft}tailored{\textquoteright} masking ratios for each data sample, which no existing method can provide. Toward this goal, we propose an Adaptive Masking Ratio (AMR) strategy that dynamically adjusts the proportion of masking for the unique content of each image based on token salience. We show that our method significantly improves over the state-of-the-art in mask-based pre-training on the ImageNet-1K dataset. Code and model parameters are available at https://github.com/doihye/SBAM.",

keywords = "Masked autoencoder, Masked image modelling, Self-supervised learning",

author = "Hyesong Choi and Hyejin Park and Yi, \{Kwang Moo\} and Sungmin Cha and Dongbo Min",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.; 18th European Conference on Computer Vision, ECCV 2024 ; Conference date: 29-09-2024 Through 04-10-2024",

year = "2025",

doi = "10.1007/978-3-031-73229-4\_20",

language = "English",

isbn = "9783031732287",

series = "Lecture Notes in Computer Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "343--359",

editor = "Ale{\v s} Leonardis and Elisa Ricci and Stefan Roth and Olga Russakovsky and Torsten Sattler and G{\"u}l Varol",

booktitle = "Computer Vision – ECCV 2024 - 18th European Conference, Proceedings",

}

Choi, H, Park, H, Yi, KM, Cha, S & Min, D 2025, Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training. in A Leonardis, E Ricci, S Roth, O Russakovsky, T Sattler & G Varol (eds), Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. Lecture Notes in Computer Science, vol. 15136 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 343-359, 18th European Conference on Computer Vision, ECCV 2024, Milan, Italy, 29/09/24. https://doi.org/10.1007/978-3-031-73229-4_20

Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training. / Choi, Hyesong; Park, Hyejin; Yi, Kwang Moo et al.
Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. ed. / Aleš Leonardis; Elisa Ricci; Stefan Roth; Olga Russakovsky; Torsten Sattler; Gül Varol. Springer Science and Business Media Deutschland GmbH, 2025. p. 343-359 (Lecture Notes in Computer Science; Vol. 15136 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Salience-Based Adaptive Masking

T2 - 18th European Conference on Computer Vision, ECCV 2024

AU - Choi, Hyesong

AU - Park, Hyejin

AU - Yi, Kwang Moo

AU - Cha, Sungmin

AU - Min, Dongbo

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

PY - 2025

Y1 - 2025

N2 - In this paper, we introduce Saliency-Based Adaptive Masking (SBAM), a novel and cost-effective approach that significantly enhances the pre-training performance of Masked Image Modeling (MIM) approaches by prioritizing token salience. Our method provides robustness against variations in masking ratios, effectively mitigating the performance instability issues common in existing methods. This relaxes the sensitivity of MIM-based pre-training to masking ratios, which in turn allows us to propose an adaptive strategy for ‘tailored’ masking ratios for each data sample, which no existing method can provide. Toward this goal, we propose an Adaptive Masking Ratio (AMR) strategy that dynamically adjusts the proportion of masking for the unique content of each image based on token salience. We show that our method significantly improves over the state-of-the-art in mask-based pre-training on the ImageNet-1K dataset. Code and model parameters are available at https://github.com/doihye/SBAM.

AB - In this paper, we introduce Saliency-Based Adaptive Masking (SBAM), a novel and cost-effective approach that significantly enhances the pre-training performance of Masked Image Modeling (MIM) approaches by prioritizing token salience. Our method provides robustness against variations in masking ratios, effectively mitigating the performance instability issues common in existing methods. This relaxes the sensitivity of MIM-based pre-training to masking ratios, which in turn allows us to propose an adaptive strategy for ‘tailored’ masking ratios for each data sample, which no existing method can provide. Toward this goal, we propose an Adaptive Masking Ratio (AMR) strategy that dynamically adjusts the proportion of masking for the unique content of each image based on token salience. We show that our method significantly improves over the state-of-the-art in mask-based pre-training on the ImageNet-1K dataset. Code and model parameters are available at https://github.com/doihye/SBAM.

KW - Masked autoencoder

KW - Masked image modelling

KW - Self-supervised learning

UR - https://www.scopus.com/pages/publications/105018204520

U2 - 10.1007/978-3-031-73229-4_20

DO - 10.1007/978-3-031-73229-4_20

M3 - Conference contribution

AN - SCOPUS:105018204520

SN - 9783031732287

T3 - Lecture Notes in Computer Science

SP - 343

EP - 359

BT - Computer Vision – ECCV 2024 - 18th European Conference, Proceedings

A2 - Leonardis, Aleš

A2 - Ricci, Elisa

A2 - Roth, Stefan

A2 - Russakovsky, Olga

A2 - Sattler, Torsten

A2 - Varol, Gül

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 29 September 2024 through 4 October 2024

ER -

Choi H, Park H, Yi KM, Cha S, Min D. Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training. In Leonardis A, Ricci E, Roth S, Russakovsky O, Sattler T, Varol G, editors, Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. Springer Science and Business Media Deutschland GmbH. 2025. p. 343-359. (Lecture Notes in Computer Science). doi: 10.1007/978-3-031-73229-4_20

Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Fingerprint

Cite this