Christoph Feichtenhofer, Axel Pinz and Andrew Zisserman

Overview

The objective of this work is object detection in video ‐ on this website we provide reference implementations (i.e. Code, Models and Data) for:

Please see our paper under Publications for more details.

Detect and Track (D&T) architecture

We propose a unified approach to tackle the problem of object detection in realistic video. Our objective is to directly infer a `tracklet' over multiple frames by simultaneously carrying out detection and tracking with a ConvNet. We train a fully convolutional architecture end-to-end using a detection and tracking based loss and term our approach D&T for joint Detection and Tracking. The input to the network consists of multiple frames which are first passed through a ConvNet trunk to produce convolutional features which are shared for the task of detection and tracking. We compute convolutional cross-correlation between the feature responses of adjacent frames to estimate the local displacement at different feature scales. On top of the features, we employ an RoI-pooling layer to classify and regress box proposals as well as an RoI-tracking layer that regresses box transformations (translation, scale, aspect ratio) across frames. Our architecture is fully convolutional up to RoI-pooling/tracking and can be trained end-to-end for object detection and tracking. Finally, to infer long-term tubes of objects across a video we link detections based on our tracklets.

Results

We evaluate our method on the ImageNet object detection from video (VID) dataset which contains 30 classes in 3862 training and 555 validation videos. The objects have ground truth annotations of their bounding box and track ID in a video. We measure performance as mean average precision (mAP) over the 30 classes on the validation set as is standard practice.

Qualitative results: (more videos here)

Quantitative results:

We show results for our models and the current state-of-the-art in the table below.

A quantitative evaluation on the large-scale ImageNet VID dataset shows that our approach, D&T (τ=1), is able to achieve better single-model performance than the winner of the last ILSVRC'16 challenge [5], despite being conceptually simple and much faster. Moreover, we show that including a tracking loss may improve feature learning for better static object detection D(&T loss), and we also present a very fast version of D&T (τ=10) that works on temporally-strided input frames. When employing a stronger base ConvNet (Inception-v4 instead of ResNet-101), we observe in the last row of the table above that the class-wise performance for D&T is greatly boosted for certain categories, e.g., dog (+5.7 AP), domestic cat (+9.4 AP) , lion (+11.4 AP), lizard (+4.5 AP), rabbit (+4.4 AP). An overall performance comparison with various base networks is shown in the Table below.


Code, Models & Data


Code Download

For installation instructions please follow the provided README.
Our implementation uses a modified version of the Caffe library which is included in the package.

Model Download

We release our models jointly trained on the ImageNet DET and VID datasets.

Data Download

Our models are trained using region proposals extracted using a Region Proposal Network. We provide the extracted proposals for training and testing on ImageNet DET and VID subsets.

Please cite our paper if you use the code or models.

Publications


C. Feichtenhofer, A. Pinz, A. Zisserman
[1] Detect to Track and Track to Detect  
IEEE International Conference on Computer Vision, 2017

References:

  • [2] K. Kang, H. Li, T. Xiao, W. Ouyang, J. Yan, X. Liu, and X. Wang. Object detection in videos with tubelet proposal networks. In Proc. CVPR, 2017.
  • [3] K. Kang, H. Li, J. Yan, X. Zeng, B. Yang, T. Xiao, C. Zhang, Z. Wang, R. Wang, X. Wang, and W. Ouyang. T-CNN: tubelets with convolutional neural networks for object detection from videos. arXiv preprint, 2016.
  • [4] K. Kang, W. Ouyang, H. Li, and X. Wang. Object detection from video tubelets with convolutional neural networks. In Proc. CVPR, 2016.
  • [5] J. Yang, H. Shuai, Z. Yu, R. Fan, Q. Ma, Q. Liu, and J. Deng. ILSVRC2016 object detection from video: Team NUIST. http://image-net.org/challenges/talks/2016/Imagenet%202016%20VID.pptx , 2016.

Acknowledgements

This work was supported by ERC grant VisRec no. 228180, EPSRC Programme Grant Seebibyte EP/M013774/1, and the Austrian Science Fund (FWF) under project P27076. Christoph Feichtenhofer is a recipient of a DOC Fellowship of the Austrian Academy of Sciences at the Institute of Electrical Measurement and Measurement Signal Processing, Graz University of Technology. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the GPUs used for this research.