Mononizing Binocular Videos

ACM Transactions on Graphics (SIGGRAPH Asia 2020 issue), Vol. 39, No. 6, December 2020, pp. 228:1—228:16.

[ Project Webpage ] [ arXiv ] [ Video ]

Mono3D is the implementation of mono-nizing binocular videos into a regular monocular video with the stereo information implicitly encoded, such that the original binocular videos can be restored with high quality.

Online demo

[ Mononized view ] [ Restored left view ] [ Restored right view ]

Environment

Please refer to env.yaml.

• Please carefully install following two packages with specific version, because the pretrained model is based on that version and newer versions are incompatible.
  ```
• mmcv==0.6.2
• mmdet==2.2.1 (build from source)
  ```

Dataset

We cannot release the whole 3D movie dataset due to copyright issues. But the binocular image dataset and part of the binocular video dataset used in the paper are publicly available: [ Flickr1024 ] and [ Inria ].

Prepare Flickr1024 for training the image version model

1. Download Flickr1024 from the website: https://yingqianwang.github.io/Flickr1024/
2. Download data list from https://drive.google.com/drive/folders/14oeXizbqTCxbmkZblt7YbWjaU2IIqNJf?usp=sharing

3. Organise the dataset as following (${DATASET is the root dir for maintaining our dataset}):

   ${DATASET}  
   |-- Flickr1024  
   |   |-- Train  
   |   |-- |-- 0001_L.png  
   |   |   |-- 0001_R.png
   |   |   |-- 0002_L.png  
   |   |   |-- 0002_R.png
   |   |   |-- ...
   |   |-- Validation  
   |   |-- |-- 0001_L.png  
   |   |   |-- 0001_R.png
   |   |   |-- 0002_L.png  
   |   |   |-- 0002_R.png
   |   |   |-- ...
   |   |-- Test 
   |   |-- |-- 0001_L.png  
   |   |   |-- 0001_R.png
   |   |   |-- 0002_L.png  
   |   |   |-- 0002_R.png
   |   |   |-- ...
   |   |-- list  
   |   |-- |-- train.txt
   |   |   |-- val.txt
   |   |   |-- test.txt

Demo

• Download pretrained model from https://drive.google.com/drive/folders/14oeXizbqTCxbmkZblt7YbWjaU2IIqNJf?usp=sharing, and put the ‘mono3d_img.pth.tar’ inside ‘Exp/model_zoo’.

• Demo on a single scene

$ PYTHONPATH=. python main/demo.py --left ./imgs/demo_L.png

Training

$ sh scripts/train.sh mono3d_img config/Flickr1024/mono3d_img.yaml

Evaluation

Evaluation on the testing set of Flickr1024

$ sh scripts/test.sh mono3d_img config/Flickr1024/mono3d_img.yaml

Copyright and License

You are granted with the LICENSE for both academic and commercial usages.

Acknowledgments

Thanks to Yingqian Wang for releasing the great dataset, Flickr1024.

Citation

@article{hu-2020-mononizing,
        author   = {Wenbo Hu and Menghan Xia and Chi-Wing Fu and Tien-Tsin Wong},
        title    = {Mononizing Binocular Videos},
        journal  = {ACM Transactions on Graphics (SIGGRAPH Asia 2020 issue)},
        month    = {December},
        year     = {2020},
        volume   = {39},
        number   = {6},
        pages    = {228:1-228:16}
    }