CF计算编织>> segm_cfm>> 返回
项目作者： franckfotso

项目描述 ：
Implementation of Convolutional Feature Masking (CFM)
高级语言： Python
项目地址: git://github.com/franckfotso/segm_cfm.git
创建时间： 2017-08-27T15:16:34Z
项目社区：https://github.com/franckfotso/segm_cfm
开源协议：
下载


segm_cfm

Implementation of Convolutional Feature Masking (CFM)

Created by Franck FOTSO

Introduction:

CFM, which stands for Convolutional Feature Masking. This project is an attempt to implement the segmentation approach Convolutional Feature Masking for Joint Object and Stuff Segmentation). In the paper, they propose to use the shape information via masking convolutional features. The CNN features are masked out from the convolutional maps and used to train classifiers for recognition. We have setup an end-end framework suitable for data pre-processing, training, test, evalutation and visualization. There are several proposals involved in this project: Selective Search, Multiscale Combinatorial Group (MCG) and Convolutional Oriented Boundaries (COB).

example

Purposes:

Our main goals are:

  • Propose a framework to handle a custom dataset for a segmentation task
  • Propose a framework to handle a range of bounding boxes / segments proposals such as Selective search, MCG or COB.
  • Extract segments & rois from VOC Pascal Segmentation schema (if there is no SBD data).
  • Generate selective search proposals (segments & rois) from images set.
  • Implement a data generator and layers required to load segments/masks in Convolutional Network (AlexNet/ZF/VGG16).
  • Implement key layers involved in CFM approach: ROIPooling & MaskPooling.
  • Perform some post-processing (mask weighing & voting) required for the mask rendering.
  • Test the CFM approach over 4 types of network configs:
    • T1 (bbox cls + mask cls)
    • T2 (bbox cls + mask cls + bbox pred)
    • T3 (bbox cls + mask cls + bbox pred + mask pred) => our own layers
    • T4 (bbox cls + mask cls + bbox pred + mask pred) => author’s layers (CFM)

Datasets:

We have used the VOC2012 dataset, parsed as SBD (Semantic Boundaries Dataset). In addition, we also used the MCG and COB proposals related.
Required files are available here:

Those datasets were used to train model over CNN nets: AlexNet, ZF & VGG16. We have explored many CNN nets in the way to see how flexible this approach is. Futhermore, we have provided all prototxt required so that you can test this project in a wide range of Nvidia GPU card ( > 2 Gb)

Hardwares/Softwares:

  1. OS: Ubuntu 16.04 64 bit
  2. GPU: Nvidia GTX 950M 4G
  3. Cuda 8.0
  4. CuDNN 3.0.8
  5. Python 2.7.12
  6. OpenCV 3.1.0

Prerequisites:

  1. Caffe (rel. caffe-faster-rcnn & caffe-segm) prerequisites
  2. Python’s packages (requirements.txt)

Installation:

To install this project, please follow the steps below:

  1. Install OpenCV

  2. Download the repository:

    1.  $ cd /opt
    2.  $ sudo git clone --recursive https://github.com/franckfotso/segm_cfm.git
    3.  $ cd segm_cfm

  3. Install all externals repositories required:

    1.  $ cd /externals/caffe-segm

    Adjust Makefile.config, then

    1.  $ sudo mkdir build
    2.  $ cd build
    3.  $ cmake ..
    4.  $ make -j4
    5.  $ make install
    6.  $ cd ../..

    Install selective_search tool. Prequequisites: Boost.NumPy

    1.  $ cd /externals/selective_search_py
    2.  $ wget http://cs.brown.edu/~pff/segment/segment.zip; unzip segment.zip; rm segment.zip
    3.  $ mkdir build
    4.  $ cd build
    5.  $ cmake..
    6.  $ make -j 4
    7.  $ cd ../..

    Note: if you face any troubles, please have a look here selective_search for python 2.7

    Install project’s libs

    1.  $ cd libs
    2.  $ make
    3.  $ cd ..

  4. Install python’s packages required:

    1.  for req in $(cat requirements.txt); do pip install $req; done

Get the data and models required:

  1. Download the data and uncompress in ‘/opt/segm_cfm/data’

    • voc_2012: PASCAL -> data/voc_2012
    • bsd_voc2012: BSD -> data/bsd_voc2012
    • MCG proposals: MCG-RAW -> data/MCG-RAW.init
    • COB proposals: COB-RAW -> data/COB-RAW.init

  2. Download the models and uncompress in ‘/opt/segm_cfm/data/models’

    • bvlc_alexnet.caffemodel: Google Drive
    • VGG16.v2.caffemodel: Google Drive
    • AlexNet_segm_cfm_T3_iter_30000.caffemodel: coming soon
    • AlexNet_segm_cfm_T4_iter_30000.caffemodel: Google Drive
    • VGG16_segm_cfm_T3_iter_30000.caffemodel: coming soon
    • VGG16_segm_cfm_T4_iter_30000.caffemodel: Google Drive

Experiments:

WARNING: Be aware that some commands above are time consuming & high computing.

  1. Parse the groundtruth data (from pascal_voc segmentation format):

    1.  $ cd /opt/segm_cfm
    2.  $ python tools/parse_segm_data.py --dataset voc_2012 \ 
    3.    --gt_set trainval --in_seg_cls data/voc_2012/SegmentationClass \ 
    4.    --in_seg_inst data/voc_2012/SegmentationObject --num_proc 4 \ 
    5.    --out_seg_dir data/sbd_voc2012/SEGM-RAW

    For help use the command: python tools/parse_segm_data.py —help

  2. Parse the groundtruth data (from bsd segmentation format):

    1.  $ cd /opt/segm_cfm
    3.  $ python tools/parse_gt_data.py --dataset bsd_voc2012 \ 
    4.    --gt_set train --in_seg_cls data/bsd_voc2012/cls \ 
    5.    --in_seg_inst data/bsd_voc2012/inst --num_proc 4 \ 
    6.    --out_seg_dir data/bsd_voc2012/SEGM-RAW
    8.  $ python tools/parse_gt_data.py --dataset bsd_voc2012 \ 
    9.    --gt_set val --in_seg_cls data/bsd_voc2012/cls \ 
    10.    --in_seg_inst data/bsd_voc2012/inst --num_proc 4 \ 
    11.    --out_seg_dir data/bsd_voc2012/SEGM-RAW

    For help use the command: python tools/parse_gt_data.py —help

  3. Parse the proposals so that they can be easily handled by our project:

    1.  $ cd /opt/segm_cfm
    3.  $ python tools/parse_pr_data.py --dataset bsd_voc2012 \
    4.    --gt_set train --in_pr_dir data/bsd_voc2012/MCG-RAW.init \
    5.    --pr_meth mcg --top_k 1000 --num_proc 4 \
    6.    --out_pr_dir data/bsd_voc2012/MCG-RAW.parsed
    8.  Let's generate also MCG proposals for demo dataset
    10.  $ python tools/parse_pr_data.py --dataset demo \
    11.    --gt_set test --in_pr_dir data/bsd_voc2012/MCG-RAW.init \
    12.    --pr_meth mcg --top_k 1000 --num_proc 4 \
    13.    --out_pr_dir data/demo/MCG-RAW.parsed

    For help use the command: python tools/parse_pr_data.py —help

  4. (Optional) Generate ‘selective_search’ proposals:

    1.  $ cd /opt/segm_cfm
    3.  $ python tools/generate_data.py --dataset bsd_voc2012 \ 
    4.    --gt_set train --pr_meth selective_search --num_proc 4 \ 
    5.    --mode accurate --output_im data/bsd_voc2012/SS-IMG \ 
    6.    --data MASK --output_mat data/bsd_voc2012/SS-RAW
    8.  $ python tools/generate_data.py --dataset bsd_voc2012 \ 
    9.    --gt_set train --pr_meth selective_search --num_proc 4 \ 
    10.    --mode accurate --output_im data/bsd_voc2012/SS-IMG \ 
    11.    --data MAT --output_mat data/bsd_voc2012/SS-RAW
    13.  Let's do the same for the demo dataset
    15.  $ python tools/generate_data.py --dataset demo \ 
    16.    --gt_set test --pr_meth selective_search --num_proc 4 \ 
    17.    --mode accurate --output_im data/demo/SS-IMG \ 
    18.    --data MASK --output_mat data/demo/SS-RAW
    20.  $ python tools/generate_data.py --dataset demo \ 
    21.    --gt_set test --pr_meth selective_search --num_proc 4 \ 
    22.    --mode accurate --output_im data/demo/SS-IMG \ 
    23.    --data MAT --output_mat data/demo/SS-RAW

    For help use the command: python tools/generate_data.py —help

  5. Training: bsd_voc2012, AlexNet, selective_search | MCG

    1.  $ cd /opt/segm_cfm
    3.  Using selective_search proposals
    5.  $ python tools/train_net.py --gpu_id 0 --dataset bsd_voc2012 \
    6.    --gt_set train --task CFM --solver models/AlexNet/solver_CFM_T{1|2|3|4}.prototxt \
    7.    --weights pretrained/AlexNet/bvlc_alexnet.caffemodel \
    8.    --in_gt_dir data/bsd_voc2012/SEGM-RAW \
    9.    --in_pr_dir data/bsd_voc2012/SS-RAW.parsed \
    10.    --iters 30000 --cache_im_dir cache/bsd_voc2012_train_ss
    12.  Using MCG proposals
    14.  $ python tools/train_net.py --gpu_id 0 --dataset bsd_voc2012 \
    15.    --gt_set train --task CFM --solver models/AlexNet/solver_CFM_T{1|2|3|4}.prototxt \
    16.    --weights pretrained/AlexNet/bvlc_alexnet.caffemodel \
    17.    --in_gt_dir data/bsd_voc2012/SEGM-RAW \
    18.    --in_pr_dir data/bsd_voc2012/MCG-RAW.parsed \
    19.    --iters 30000 --cache_im_dir cache/bsd_voc2012_train_mcg

    For help use the command: python tools/train_net.py —help

  6. Training: bsd_voc2012, VGG16, selective_search | MCG

    1.  $ cd /opt/segm_cfm
    3.  Using selective_search proposals
    5.  $ python tools/train_net.py --gpu_id 0 --dataset bsd_voc2012 \
    6.    --gt_set train --task CFM --solver models/VGG16/solver_CFM_T{1|2|3|4}.prototxt \
    7.    --weights pretrained/VGG16/VGG16.v2.caffemodel \
    8.    --in_gt_dir data/bsd_voc2012/SEGM-RAW \
    9.    --in_pr_dir data/bsd_voc2012/MCG-RAW.parsed \
    10.    --iters 30000 --cache_im_dir cache/bsd_voc2012_train_ss
    12.  Using MCG proposals
    14.  $ python tools/train_net.py --gpu_id 0 --dataset bsd_voc2012 \
    15.    --gt_set train --task CFM --solver models/VGG16/solver_CFM_T{1|2|3|4}.prototxt \
    16.    --weights pretrained/VGG16/VGG16.v2.caffemodel \
    17.    --in_gt_dir data/bsd_voc2012/SEGM-RAW \
    18.    --in_pr_dir data/bsd_voc2012/MCG-RAW.parsed \
    19.    --iters 30000 --cache_im_dir cache/bsd_voc2012_train_mcg

    For help use the command: python tools/train_net.py —help

  7. Testing: demo, T4, AlexNet, selective_search | MCG

    1.  $ cd /opt/segm_cfm
    3.  Using selective_search proposals
    5.  $ python tools/demo.py --gpu_id 0 --net AlexNet \
    6.    --gt_set test --task CFM --dataset demo \
    7.    --proto models/AlexNet/AlexNet_CFM_T4_test.prototxt \
    8.    --weights outputs/bsd_voc2012/train_mcg/AlexNet_segm_cfm_T4_iter_30000.caffemodel \
    9.    --in_pr_dir data/demo/SS-RAW.parsed --kwargs 'cfm_t:T4'
    11.  Using MCG proposals
    13.  $ python tools/demo.py --gpu_id 0 --net AlexNet \
    14.    --gt_set test --task CFM --dataset demo \
    15.    --proto models/AlexNet/AlexNet_CFM_T4_test.prototxt \
    16.    --weights outputs/bsd_voc2012/train_mcg/AlexNet_segm_cfm_T4_iter_30000.caffemodel \
    17.    --in_pr_dir data/demo/MCG-RAW.parsed --kwargs 'cfm_t:T4'

    For help use the command: python tools/demo.py —help

  8. Testing: demo, VGG16, T4, selective_search | MCG

    1.  $ cd /opt/segm_cfm
    3.  Using selective_search proposals
    5.  $ python tools/demo.py --gpu_id 0 --net VGG16 \
    6.    --gt_set test --task CFM --dataset demo \
    7.    --proto models/VGG16/VGG16_CFM_T4_test.prototxt \
    8.    --weights outputs/bsd_voc2012/train_mcg/VGG16_segm_cfm_T4_iter_30000.caffemodel \
    9.    --in_pr_dir data/demo/SS-RAW.parsed --kwargs 'cfm_t:T4'
    11.  Using MCG proposals
    13.  $ python tools/demo.py --gpu_id 0 --net VGG16 \
    14.    --gt_set test --task CFM --dataset demo \
    15.    --proto models/VGG16/VGG16_CFM_T4_test.prototxt \
    16.    --weights outputs/bsd_voc2012/train_mcg/VGG16_segm_cfm_T4_iter_30000.caffemodel \
    17.    --in_pr_dir data/demo/MCG-RAW.parsed --kwargs 'cfm_t:T4'

    For help use the command: python tools/demo.py —help

Our results

All results related to demo dataset will be saved at ‘/opt/segm_cfm/data/demo/out’

example

Contact

Please feel free to leave suggestions or comments to Franck FOTSO (romyny9096@gmail.com)