项目作者: wanglimin

项目描述 :
Appearance-and-Relation Networks
高级语言: Python
项目地址: git://github.com/wanglimin/ARTNet.git
创建时间: 2017-11-23T16:09:00Z
项目社区:https://github.com/wanglimin/ARTNet

开源协议:

下载


Appearance-and-Relation Networks

We provide the code and models for the following report (arXiv Preprint):

  1. Appearance-and-Relation Networks for Video Classification
  2. Limin Wang, Wei Li, Wen Li, and Luc Van Gool
  3. in arXiv, 2017

Updates

  • November 23th, 2017
    • Initialize the repo.

Overview

ARTNet aims to learn spatiotemporal features from videos in an end-to-end manner. Its construction is based on a newly-designed module, termed as SMART block. ARTNet is a simple and general video architecture and all these relased models are trained from scratch on video dataset. Currently, for an engineering compromise between accuracy and efficiency, ARTNet is instantiated with the ResNet-18 architecture and trained on the input volume of 112*112*16.

Training on Kinetics

The training of ARTNet is based on our modified Caffe toolbox. Specical thanks to @zbwglory for modifying this code.

The training code is under folder of models/.

Performance on the validation set of Kinetics

Model Backbone architecture Spatial resolution Top-1 Accuracy Top-5 Accuracy
C2D ResNet18 112*112 61.2 82.6
C3D ResNet18 112*112 65.6 85.7
C3D ResNet34 112*112 67.1 86.9
ARTNet (s) ResNet18 112*112 67.7 87.1
ARTNet (d) ResNet18 112*112 69.2 88.3
ARTNet+TSN ResNet18 112*112 70.7 89.3

These models are trained on the Kinetics dataset from scratch and tested on the validation set. Our training is performed based on the input volume of 112*112*16. The test is performed by cropping 25 clips from the videos.

Fine tuning on HMDB51 and UCF101

The fine tuning process is conducted based on the TSN framework, where segment number is 2.

The fine tuning code is under folder of fine_tune/

Performance on the datasets of HMDB51 and UCF101

Model Backbone architecture Spatial resolution HMDB51 UCF101
C3D ResNet18 112*112 62.1 89.8
ARTNet (d) ResNet18 112*112 67.6 93.5
ARTNet+TSN ResNet18 112*112 70.9 94.3

These models learned on the Kinetics dataset are transferred to the HMDB51 and UCF101 datasets. The fine-tuning process is done with TSN framework where the segment number is 2. The performance is reported over three splits by using only RGB input.