增强现实AR>> ViNet>> 返回
项目作者： samyak0210

项目描述 ：
ViNet Pushing the limits of Visual Modality for Audio Visual Saliency Prediction
高级语言： Python
项目地址: git://github.com/samyak0210/ViNet.git
创建时间： 2020-08-01T10:47:01Z
项目社区：https://github.com/samyak0210/ViNet
开源协议：MIT License
下载


ViNet: Pushing the limits of Visual Modality for Audio-Visual Saliency Prediction

This repository contains Pytorch Implementation of ViNet and AViNet.

PWC

Cite

Please cite with the following Bibtex code:

  1. @misc{jain2021vinet,
  2.       title={ViNet: Pushing the limits of Visual Modality for Audio-Visual Saliency Prediction}, 
  3.       author={Samyak Jain and Pradeep Yarlagadda and Shreyank Jyoti and Shyamgopal Karthik and Ramanathan Subramanian and Vineet Gandhi},
  4.       year={2021},
  5.       eprint={2012.06170},
  6.       archivePrefix={arXiv},
  7.       primaryClass={cs.CV}
  8. }

Abstract

We propose the ViNet architecture for audio-visual saliency prediction. ViNet is a fully convolutional encoder-decoder architecture. The encoder uses visual features from a network trained for action recognition, and the decoder infers a saliency map via trilinear interpolation and 3D convolutions, combining features from multiple hierarchies. The overall architecture of ViNet is conceptually simple; it is causal and runs in real-time (60 fps). ViNet does not use audio as input and still outperforms the state-of-the-art audio-visual saliency prediction models on nine different datasets (three visual-only and six audio-visual datasets). ViNet also surpasses human performance on the CC, SIM and AUC metrics for the AVE dataset, and to our knowledge, it is the first network to do so. We also explore a variation of ViNet architecture by augmenting audio features into the decoder. To our surprise, upon sufficient training, the network becomes agnostic to the input audio and provides the same output irrespective of the input. Interestingly, we also observe similar behaviour in the previous state-of-the-art models \cite{tsiami2020stavis} for audio-visual saliency prediction. Our findings contrast with previous works on deep learning-based audio-visual saliency prediction, suggesting a clear avenue for future explorations incorporating audio in a more effective manner.

Examples

We compare our model ViNet with UNISAL (previous state-of-the-art). Below are some examples of our model. The first section is original video, second is the ground-truth, third is our model’s prediction and last is the UNISAL’s prediction.

Architecture

Dataset

  • DHF1K and UCF Sports dataset can be downloaded from this link.
  • Hollywood-2 dataset can be downloaded from this link
  • The six audio-visual datasets - DIEM, AVAD, Coutrot-1&2, SumMe and ETMD can be downloaded from this link. You can also run the following command to fetch all the six dataset and its components - 
    1. $ bash fetch_data.sh

Testing

Clone this repository and download the pretrained weights of AViNet and ViNet on multiple datasets from this /g/personal/samyak_j_research_iiit_ac_in/EXYq5WiSbh9Kq9R_n-Gr3yABRyKPSkxM7ROLg-zPDXV_qA?e=5AL7UU">link.

  • ViNet

    Run the code using 
    1. $ python3 generate_result.py --path_indata path/to/test/frames --save_path path/to/results --file_weight path/to/saved/models
    This will generate saliency maps for all frames in the directory and dump these maps into results directory. The directory structure should be 
    1. └── Dataset  
    2.   ├── Video-Number  
    3.       ├── images

  • AViNet

    Run the code using 
    1. $ python3 generate_result_audio_visual.py --path_indata path/to/test/frames --save_path path/to/results --file_weight path/to/saved/models --use_sound True --split <split_number>
    2. <split_number>: {1,2,3}
    This will generate saliency maps for all frames in the directory and dump these maps into results directory. The directory structure should be 
    1. └── Dataset  
    2.   ├── video_frames  
    3.       ├── <dataset_name>
    4.           ├── Video-Name
    5.               ├── frames
    6.   ├── video_audio  
    7.       ├── <dataset_name>
    8.           ├── Video-Name
    9.               ├── audio  
    10.   ├── fold_lists
    11.       ├── <dataset_file>.txt
    Fold_lists consists of text file of video names and their corresponding fps in various splits. The directory structure is the same as the one generated by the fetch_data.sh file.

Training

For training the model from scratch, download the pretrained weights of S3D from /g/personal/samyak_j_research_iiit_ac_in/EYZ8Elhmc9tOmlVwnb41GEEBQNnmW31Q2mAwE8B9sFn7WA?e=co9Hvj">here and place these weights in the same directory. Run the following command to train 

  1. $ python3 train.py --train_path_data path/to/train/dataset --val_path_data  path/to/val/dataset --dataset <dataset_name> --use_sound <boolean_value>
  2. <dataset_name> : {"DHF1KDataset", "SoundDataset", "Hollywood", "UCF"}

In case of ViNet, the dataset directory structure should be

  1. └── Dataset  
  2.     ├── Video-Number  
  3.         ├── images  
  4.         |── maps
  5.         └── fixations

In case of AViNet, the dataset directory structure should be

  1. └── Dataset  
  2.     ├── video_frames  
  3.         ├── <dataset_name>
  4.             ├── Video-Name
  5.                 ├── frames
  6.     ├── video_audio  
  7.         ├── <dataset_name>
  8.             ├── Video-Name
  9.                 ├── audio
  10.     ├── annotations
  11.         ├── <dataset_name>
  12.             ├── Video-Name
  13.                 ├── <frame_id>.mat (fixations)
  14.                 ├── maps
  15.                     ├── <frame_id>.jpg (ground truth saliency map)
  16.     ├── fold_lists
  17.         ├── <dataset_file>.txt

For training the ViNet with Hollywood-2 or UCF-Sports dataset, first train the model with DHF1K dataset or you can directly use our model trained on DHF1K, and finetune the model weights on aforementioned datasets.

Similarly for training the AViNet with DIEM, AVAD, Coutrot-1&2, ETMD and SumMe dataset, first load model with DHF1K trained weights and finetune the model weights on aforementioned datasets.

Experiments

  • Audio

    For training the model, we provide argument to select the model between ViNet (Visual Net) and AViNet (Audio-Visual Net). Run the command - 
    1. $ python3 train.py --use_sound <boolean_value>

If you want to save the results of the generated map run the command - 

  1. $ python3 generate_result_audio_visual.py --use_sound <boolean_value> --file_weight <path/to/file> --path_indata <path/to/data>

  • Multiple Audio-Visual Fusion

    You can select the corresponding fusion technique’s model from the model.py file. By default it uses the model with Bilinear concatenation for fusing audio and visual cues. If you want to use Transformer-based fusion technique, call the model with name VideoAudioSaliencyTransformerFusionModel.

Quantitative Results

  • DHF1K

    The results of our models on DHF1K test dataset can be viewed here under the name ViNet. Comparison with other state-of-the-art saliency detection models

  • Hollywood2

  • UCF

  • DIEM

    We provide results on both our models - ViNet and AViNet on the DIEM dataset.

Contact

If any question, please contact samyak.j@research.iiit.ac.in, or pradeep.yarlagadda@students.iiit.ac.in , or use public issues section of this repository

License

This code is distributed under MIT LICENSE.