CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers

Scene understanding based on image segmentation is a crucial component of autonomous vehicles. Pixel-wise semantic segmentation of RGB images can be advanced by exploiting complementary features from the supplementary modality ( ${X}$ -modality). However, covering a wide variety of sensors with a modality-agnostic model remains an unresolved problem due to variations in sensor characteristics among different modalities. Unlike previous modality-specific methods, in this work, we propose a unified fusion framework, CMX, for RGB-X semantic segmentation. To generalize well across different modalities, that often include supplements as well as uncertainties, a unified cross-modal interaction is crucial for…

• NM
  Niedersächsisches Ministerium für Wissenschaft und Kultur

  Award: BW6-03
• BF
  Bundesministerium für Arbeit und Soziales

  Award: 01KM151112