MMSceneGraph

Introduction

MMSceneneGraph is an open source code hub for scene graph generation as well as supporting downstream tasks based on the scene graph on PyTorch. The frontend object detector is supported by open-mmlab/mmdetection.

Major features

• Modular design

  We decompose the framework into different components and one can easily construct a customized scene graph generation framework by combining different modules.

• Support of multiple frameworks out of box

  The toolbox directly supports popular and contemporary detection frameworks, e.g. Faster RCNN, Mask RCNN, etc.

• Visualization support

  The visualization of the groundtruth/predicted scene graph is integrated into the toolbox.

License

This project is released under the MIT license.

Changelog

Please refer to CHANGELOG.md for details.

Benchmark and model zoo

The original object detection results and models provided by mmdetection are available in the model zoo. The models for the scene graph generation are temporarily unavailable yet.

Supported methods and Datasets

Supported SGG (VRD) methods:

• Neural Motifs (CVPR'2018)
• VCTree (CVPR'2019)
• TDE (CVPR'2020)
• VTransE (CVPR'2017)
• IMP (CVPR'2017)
• KERN (CVPR'2019)
• GPSNet (CVPR'2020)
• HetH (ECCV'2020, ours)
• TopicSG (ICCV'2021, ours)

Supported saliency object detection methods:

• R3Net (IJCAI'2018)
• SCRN (ICCV'2019)

Supported image captioning methods:

• bottom-up (CVPR'2018)
• XLAN (CVPR'2020)

Supported datasets:

• Visual Genome: VG150 (CVPR'2017)
• VRD (ECCV'2016)
• Visual Genome: VG200/VG-KR (ours)
• MSCOCO (for object detection, image caption)
• RelCap (from VG and COCO, ours)

Installation

As our project is built on mmdetection 1.x (which is a bit different from their current master version 2.x), please refer to INSTALL.md. If you want to use mmdetection 2.x, please refer to mmdetection/get_start.md.

Getting Started

Please refer to GETTING_STARTED.md for using the projects. We will update it constantly.

Acknowledgement

We appreciate the contributors of the mmdetection project and Scene-Graph-Benchmark.pytorch which inspires our design.

Citation

If you find this code hub or our works useful in your research works, please consider citing:

@inproceedings{wang2021topic,
  title={Topic Scene Graph Generation by Attention Distillation from Caption},
  author={Wang, Wenbin and Wang, Ruiping and Chen, Xilin},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  pages={15900--15910},
  month = {October},
  year={2021}
}


@inproceedings{wang2020sketching,
  title={Sketching Image Gist: Human-Mimetic Hierarchical Scene Graph Generation},
  author={Wang, Wenbin and Wang, Ruiping and Shan, Shiguang and Chen, Xilin},
  booktitle={Proceedings of European Conference on Computer Vision (ECCV)},
  pages={222--239},
  year={2020},
  volume={12358},
  doi={10.1007/978-3-030-58601-0_14},
  publisher={Springer}
}

@InProceedings{Wang_2019_CVPR,
author = {Wang, Wenbin and Wang, Ruiping and Shan, Shiguang and Chen, Xilin},
title = {Exploring Context and Visual Pattern of Relationship for Scene Graph Generation},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
pages = {8188-8197},
month = {June},
address = {Long Beach, California, USA},
doi = {10.1109/CVPR.2019.00838},
year = {2019}
}