Learning to Segment Instances in Videos with Spatial Propagation Network

This paper is available at the 2017 DAVIS Challenge website.

Check our results in this video.

Contact: Jingchun Cheng (chengjingchun at gmail dot com)

Cite the Paper

If you find that our method is useful in your research, please cite:

@article{DAVIS2017-6th,
  author = {J. Cheng and S. Liu and Y.-H. Tsai and W.-C. Hung and S. Gupta and J. Gu and J. Kautz and S. Wang and M.-H. Yang}, 
  title = {Learning to Segment Instances in Videos with Spatial Propagation Network}, 
  journal = {The 2017 DAVIS Challenge on Video Object Segmentation - CVPR Workshops}, 
  year = {2017}
}

About the Code

• The code released here mainly consistes of two parts in the paper: foreground segmentation and instance recognition.

• It contains the parent net for foreground segmentation and training codes for instance recognition networks.

• The matlab_code folder contains a simple version of our CRAF step for segmentation refinement.

Requirements

• Install caffe and pycaffe at http://caffe.berkeleyvision.org/.

• Download the DAVIS 2017 dataset and put it in the data folder.

• Download the pre-trained foreground/background model here and put it in the pretrained folder.

Training

• Train the per-object recognition model.
  cd training
python solve.py PATH_OF_MODEL PATH_OF_SOLVER
  Foe example, on the 'choreography' video for the 1st object, run:
  python solve.py ../pretrained/PN_ResNetF.caffemodel ../ResNetF/testnet_per_obj/choreography/solver_1.prototxt

Testing

• Test the general foreground/backgroung model.
  python infer_test_fgbg.py PATH_OF_MODEL PATH_OF_RESULT VIDEO_NAME
  Foe example, on the 'lions' video, run:
  python infer_test_fgbg.py pretrained/PN_ResNetF.caffemodel results/fgbg lions

• Test the object instance model.
  python infer_test_perobj.py MODEL_ITERATION VIDEO_NAME OBJECT_ID
  For example, on the 'lions' video for the 2nd object, run:
  python infer_test_perobj.py 3000 lions 2

• Run example_CRAF.m in the matlab_code folder for a demo on CRAF segmentation refinement.

Download Our Segmentation Results on 2017 DAVIS Challenge

• General foreground/background segmentation here
• Instance-level object segmentation without refinement here
• Final instance-level object segmentation with refinement here

Note

The model and code are available for non-commercial research purposes only.

• 09/2017: code and model released
• 03/2018: pre-trained model updated