In multi-task learning (MTL) for visual scene understanding, it is crucial to transfer useful information between multiple tasks with minimal interferences. In this paper, we propose a novel architecture that effectively transfers informative features by applying the attention mechanism to the multi-scale features of the tasks. Since applying the attention module directly to all possible features in terms of scale and task requires a high complexity, we propose to apply the attention module sequentially for the task and scale. The cross-task attention module (CTAM) is first applied to facilitate the exchange of relevant information between the multiple task features of the same scale. The cross-scale attention module (CSAM) then aggregates useful information from feature maps at different resolutions in the same task. Also, we attempt to capture long range dependencies through the self-attention module in the feature extraction network. Extensive experiments demonstrate that our method achieves state-of-the-art performance on the NYUD-v2 and PASCAL-Context dataset. Our code is available at https://github.com/kimsunkyung/SCA-MTL.
|Title of host publication||2022 IEEE International Conference on Image Processing, ICIP 2022 - Proceedings|
|Publisher||IEEE Computer Society|
|Number of pages||5|
|State||Published - 2022|
|Event||29th IEEE International Conference on Image Processing, ICIP 2022 - Bordeaux, France|
Duration: 16 Oct 2022 → 19 Oct 2022
|Name||Proceedings - International Conference on Image Processing, ICIP|
|Conference||29th IEEE International Conference on Image Processing, ICIP 2022|
|Period||16/10/22 → 19/10/22|
Bibliographical noteFunding Information:
* Corresponding author This work was supported by the Mid-Career Researcher Program through the NRF of Korea (NRF-2021R1A2C2011624). Sunkyung Kim is grateful for financial support from Hyundai Motor Chung Mong-Koo Foundation.
© 2022 IEEE.
- cross attention
- monocular depth estimation
- Multi-task learning
- semantic segmentation