SAM2-UNet: segment anything 2 makes strong encoder for natural and medical image segmentation

Abstract Image segmentation plays an important role in vision understanding. Recently, the emerging vision foundation models continuously achieved superior performance on various tasks. Following such success, in this paper, we prove that the Segment Anything Model 2 (SAM2) can be a strong encoder for U-shaped segmentation models. We propose a simple but effective framework, termed SAM2-UNet, for versatile image segmentation. Specifically, SAM2-UNet adopts the Hiera backbone of SAM2 as the encoder, while the decoder uses the classic U-shaped design. Additionally, adapters are inserted into the encoder to enable parameter-efficient fine-tuning. Preliminary experiments on various downstream tasks, such as…

• NN
  National Natural Science Foundation of China

  Award: 62322608
• BA
  Basic and Applied Basic Research Foundation of Guangdong Province

  Award: 2024A1515010255