Disentangling Motion, Foreground and Background Features in Videos
This repo contains the source codes for our work as in title. Please refer to our project webpage or original paper for more details.
This project requires UCF-101 dataset and its localization annotations (bonding box for action region). Please note that the annotations only contain bounding boxes for 24 classes out of 101. We only use these 24 classes for further experiments.
UCF-101: http://crcv.ucf.edu/data/UCF101/UCF101.rarAnnotations (version 1): http://crcv.ucf.edu/ICCV13-Action-Workshop/index.files/UCF101_24Action_Detection_Annotations.zip
We split our dataset into training set, validation set and test set. Split lists of each set can be found under dataset folder.
As we are dealing with videos, using TF-records in TensorFlow can help to reduce I/O overheads. (Please refer to the official documentation if you’re not familiar with TF-records). Each SequenceExample in our TF-records includes 32 video frames, corresponding masks and so on.
A brief description of how we generate TF-records from videos and annotations: for each video, we split it into multiple chunks consisting of 32 frames and save each chunk as an example. The corresponding masks are generated through manipulations on localization annotations.
In order to generate TF-records used by this project, you need to modify certain paths in tools/generate_tfrecords.py. Including, videos_directory, annotation_directory and so on. As we use FFMPEG to decode videos, you may want to install it with the command below if you are using Anaconda:
conda install ffmpeg
After installation of FFMPEG, you need to specify the path to executable FFMPEG binary file in tools/ffmpeg_reader.py. (Usually it’s just ~/anaconda/bin/ffmpeg if you are using Anaconda). After specifying path to FFMPEG, you are good to go! Run the script as below to generate those TF-records:
python tools/generate_tfrecords.py
Our codes for training and testing are organized in the following fashion: we have scripts under models/ to construct TensorFlow graph for each model. And under top path, we have scripts named as `*_[train|val|test].py`**. These are the scripts accomplish external call and training/validation/test of each model.