Railroad is not a Train: Saliency as Pseudo-pxiel Supervision for Weakly Supervised Semantic Segmentation (CVPR 2021)

CVPR 2021 paper

Seungho Lee^1, , Minhyun Lee^1,, Jongwuk Lee², Hyunjung Shim¹

_{* indicates an equal contribution}

¹ _{School of Integrated Technology, Yonsei University}
² _{Department of Computer Science of Engineering, Sungkyunkwan University}

Introduction

Existing studies in weakly-supervised semantic segmentation (WSSS)
using image-level weak supervision have several limitations:
sparse object coverage, inaccurate object boundaries,
and co-occurring pixels from non-target objects.
To overcome these challenges, we propose a novel framework,
namely Explicit Pseudo-pixel Supervision (EPS),
which learns from pixel-level feedback by combining two weak supervisions;
the image-level label provides the object identity via the localization map
and the saliency map from the off-the-shelf saliency detection model
offers rich boundaries. We devise a joint training strategy to fully
utilize the complementary relationship between both information.
Our method can obtain accurate object boundaries and discard co-occurring pixels,
thereby significantly improving the quality of pseudo-masks.

Updates

12 Jul, 2021: Initial upload

19 Aug, 2021: Minor update on information about dCRF and the pre-trained model of the segmentation networks

• Please see the issuses: dCRF and pre-trained model

28 Aug, 2021: Major updates about MS-COCO 2014 dataset and minor updates (cleanup)

15 Apr, 2022: Minor update on information about the method setting up ‘cls_labels.npy’ the for ms-coco 17 dataset

• Please see the issue: coco17

22 Feb, 2023: Minor update on the download link for coco dataset (Masks, Saliency maps)

Installation

• Python 3.6
• Pytorch >= 1.0.0
• Torchvision >= 0.2.2
• MXNet
• Pillow
• opencv-python (opencv for Python)

Execution

Dataset & pretrained model

• PASCAL VOC 2012

  • Images
  • Saliency maps
    using PFAN

• MS-COCO 2014

  • Images
  • Saliency maps
    using PFAN
  • Segmentation masks

• Pretrained models

  • ImageNet-pretrained Model
    for ResNet38

• MS-COCO 2017

  • The way to setup “cls_labels.npy” for MS-COCO 2017 dataset. This work is done by JimmyMa99 and please see the codes.

Classification network

• Execute the bash file for training, inference and evaluation.

    # Please see these files for the detail of execution.
    # PASCAL VOC 2012 
    # Baseline
    bash script/vo12_cls.sh
    # EPS
    bash script/voc12_eps.sh
    # MS-COCO 2014
    # Baseline
    bash script/coco_cls.sh
    # EPS
    bash script/coco_eps.sh

• We provide checkpoints, training logs, and performances for each method and each dataset.

  Please see the details from the script files.

  | Dataset | METHOD | Train(mIoU) | Checkpoint | Training log |
  | ———————- | ——— | —————- | —————————————————————————————— | ——————————————————— |
  | PASCAL VOC 2012 | Base | 47.05 | Download | voc12_cls.log |
  | PASCAL VOC 2012 | EPS | 69.22 | Download | voc12_eps.log |
  | MS-COCO 2014 | Base | 31.23 | Download | coco_cls.log |
  | MS-COCO 2014 | EPS | 37.15 | Download | coco_eps.log |

• dCRF hyper-parameters
  • We did not use dCRF for our pseudo-masks, but only used for the comparision in the paper.
  • We chose the hyper-parameters for dCRF used in ResNet101-based DeepLabV2 among other candidates(OAA, and PSA)
  • Please see the official deeplab website for information

    CRF parameters: bi_w = 4, bi_xy_std = 67, bi_rgb_std = 3, pos_w = 3, pos_xy_std = 1.

    Segmentation network

• We utilize DeepLab-V2
  for the segmentation network.
• Please see deeplab-pytorch for the implementation in PyTorch.
• We used the pretrained model for VGG16 based network from DeepLab official and for ResNet101-based network from OAA official.

Results

Acknowledgement

This code is highly borrowed from PSA. Thanks to Jiwoon, Ahn.