Visual Memorability for Robotic Interestingness via Unsupervised Online Learning (ECCV 2020 Oral and TRO)

Last update: Sep 08, 2022

Overview

Visual Interestingness

Refer to the project description for more details.
This code based on the following paper.

Chen Wang, Yuheng Qiu, Wenshan Wang, Yafei Hu, Seungchan Kim and Sebastian Scherer, Unsupervised Online Learning for Robotic Interestingness with Visual Memory, IEEE Transactions on Robotics (T-RO), 2021.
It is an extended version of the conference paper:

Chen Wang, Wenshan Wang, Yuheng Qiu, Yafei Hu, and Sebastian Scherer, Visual Memorability for Robotic Interestingness via Unsupervised Online Learning, European Conference on Computer Vision (ECCV), 2020.
If you want the original version, go to the ECCV Branch instead.
We also provide ROS wrapper for this project, you may go to interestingness_ros.

Install Dependencies

This version is tested in PyTorch 1.7

  pip3 install -r requirements.txt

Long-term Learning

You may skip this step, if you download the pre-trained vgg16.pt into folder "saves".

Download coco dataset into folder [data-root]:

bash download_coco.sh [data-root] # replace [data-root] by your desired location

The dataset will be look like:

data-root
├──coco
   ├── annotations
   │   ├── annotations_trainval2017
   │   └── image_info_test2017
   └── images
       ├── test2017
       ├── train2017
       └── val2017

Run

python3 longterm.py --data-root [data-root] --model-save saves/vgg16.pt

# This requires a long time for training on single GPU.
# Create a folder "saves" manually and a model named "ae.pt" will be saved.

Short-term Learning

Dowload the SubT front camera data (SubTF) and put into folder "data-root", so that it looks like:

data-root
├──SubTF
   ├── 0817-ugv0-tunnel0
   ├── 0817-ugv1-tunnel0
   ├── 0818-ugv0-tunnel1
   ├── 0818-ugv1-tunnel1
   ├── 0820-ugv0-tunnel1
   ├── 0821-ugv0-tunnel0
   ├── 0821-ugv1-tunnel0
   ├── ground-truth
   └── train

Run

python3 shortterm.py --data-root [data-root] --model-save saves/vgg16.pt --dataset SubTF --memory-size 100 --save-flag n100usage

# This will read the previous model "ae.pt".
# A new model "ae.pt.SubTF.n1000.mse" will be generated.

You may skip this step, if you download the pre-trained vgg16.pt.SubTF.n100usage.mse into folder "saves".

On-line Learning

Run

  python3 online.py --data-root [data-root] --model-save saves/vgg16.pt.SubTF.n100usage.mse --dataset SubTF --test-data 0 --save-flag n100usage

  # --test-data The sequence ID in the dataset SubTF, [0-6] is avaiable
  # This will read the trained model "vgg16.pt.SubTF.n100usage.mse" from short-term learning.

Alternatively, you may test all sequences by running
```
  bash test.sh
```
This will generate results files in folder "results".
You may skip this step, if you download our generated results.

Evaluation

We follow the SubT tutorial for evaluation, simply run

python performance.py --data-root [data-root] --save-flag n100usage --category normal --delta 1 2 3
# mean accuracy: [0.64455275 0.8368784  0.92165116 0.95906876]

python performance.py --data-root [data-root] --save-flag n100usage --category difficult --delta 1 2 4
# mean accuracy: [0.42088688 0.57836163 0.67878168 0.75491805]

This will generate performance figures and create data curves for two categories in folder "performance".

Citation

      @inproceedings{wang2020visual,
        title={Visual memorability for robotic interestingness via unsupervised online learning},
        author={Wang, Chen and Wang, Wenshan and Qiu, Yuheng and Hu, Yafei and Scherer, Sebastian},
        booktitle={European Conference on Computer Vision (ECCV)},
        year={2020},
        organization={Springer}
      }
      
      @article{wang2021unsupervised,
        title={Unsupervised Online Learning for Robotic Interestingness with Visual Memory},
        author={Wang, Chen and  Qiu, Yuheng and Wang, Wenshan and Hu, Yafei anad Kim, Seungchan and Scherer, Sebastian},
        journal={IEEE Transactions on Robotics (T-RO)},
        year={2021},
        publisher={IEEE}
      }

Download conferencec version paper.
Download journal version paper.

You may watch the following video to catch the idea of this work.

Code for the paper "Improving Vision-and-Language Navigation with Image-Text Pairs from the Web" (ECCV 2020)

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh

44 Dec 14, 2022

Code for ECCV 2020 paper "Contacts and Human Dynamics from Monocular Video".

Contact and Human Dynamics from Monocular Video This is the official implementation for the ECCV 2020 spotlight paper by Davis Rempe, Leonidas J. Guib

207 Jan 5, 2023

Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020)

Causality In Traffic Accident (Under Construction) Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020) Overview Data Prepa

21 Nov 20, 2022

Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

PWLQ Updates 2020/07/16 - We are working on getting permission from our institution to release our source code. We will release it once we are granted

54 Dec 15, 2022

dataset for ECCV 2020 "Motion Capture from Internet Videos"

Motion Capture from Internet Videos Motion Capture from Internet Videos Junting Dong*, Qing Shuai*, Yuanqing Zhang, Xian Liu, Xiaowei Zhou, Hujun Bao

98 Dec 7, 2022

Code for the paper: Adversarial Training Against Location-Optimized Adversarial Patches. ECCV-W 2020.

Adversarial Training Against Location-Optimized Adversarial Patches arXiv | Paper | Code | Video | Slides Code for the paper: Sukrut Rao, David Stutz,

32 Dec 13, 2022

SNE-RoadSeg in PyTorch, ECCV 2020

SNE-RoadSeg Introduction This is the official PyTorch implementation of SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentati

242 Dec 20, 2022

[ECCV 2020] Gradient-Induced Co-Saliency Detection

Gradient-Induced Co-Saliency Detection Zhao Zhang*, Wenda Jin*, Jun Xu, Ming-Ming Cheng ⭐ Project Home » The official repo of the ECCV 2020 paper Grad

35 Nov 25, 2022

Code for Towards Streaming Perception (ECCV 2020) :car:

sAP — Code for Towards Streaming Perception ECCV Best Paper Honorable Mention Award Feb 2021: Announcing the Streaming Perception Challenge (CVPR 2021

85 Dec 22, 2022

Comments

Variable

https://github.com/wang-chen/interestingness/blob/6994d50bd47d14b617f34f5c36c1beaba03acfdc/test_interest.py#L94

I think using Variable() will just return a tensor object in the new pytorch version.

opened by haleqiu 2

Visual Memorability for Robotic Interestingness via Unsupervised Online Learning (ECCV 2020 Oral and TRO)

Related tags

Overview

Visual Interestingness

Install Dependencies

Long-term Learning

Short-term Learning

On-line Learning

Evaluation

Citation

You might also like...

Code for the paper "Improving Vision-and-Language Navigation with Image-Text Pairs from the Web" (ECCV 2020)

Code for ECCV 2020 paper "Contacts and Human Dynamics from Monocular Video".

Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020)

Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

dataset for ECCV 2020 "Motion Capture from Internet Videos"

Code for the paper: Adversarial Training Against Location-Optimized Adversarial Patches. ECCV-W 2020.

SNE-RoadSeg in PyTorch, ECCV 2020

[ECCV 2020] Gradient-Induced Co-Saliency Detection

Code for Towards Streaming Perception (ECCV 2020) :car:

Comments

Variable

Releases(v2.0)

v2.0(Apr 12, 2021)

v1.0(Jun 19, 2020)

Owner

Chen Wang

Companion code for "Bayesian logistic regression for online recalibration and revision of risk prediction models with performance guarantees"

Marvis is Mastouri's Jarvis version of the AI-powered Python personal assistant.

Supplementary code for the experiments described in the 2021 ISMIR submission: Leveraging Hierarchical Structures for Few Shot Musical Instrument Recognition.

A list of all papers and resoureces on Semantic Segmentation

RANZCR-CLiP 7th Place Solution

Pytorch implementation of Generative Models as Distributions of Functions 🌿

This repository contains the code used in the paper "Prompt-Based Multi-Modal Image Segmentation".

This is the code of "Multi-view Contrastive Graph Clustering" in NeurlPS 2021.

A Pose Estimator for Dense Reconstruction with the Structured Light Illumination Sensor

OptNet: Differentiable Optimization as a Layer in Neural Networks

code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren*, Raymond A. Yeh*, Alexander G. Schwing.

FirmWire is a full-system baseband firmware emulation platform for fuzzing, debugging, and root-cause analysis of smartphone baseband firmwares

PyTorch implementation for the Neuro-Symbolic Sudoku Solver leveraging the power of Neural Logic Machines (NLM)

Flexible-Modal Face Anti-Spoofing: A Benchmark

Adversarially Learned Inference

Deep Learning with PyTorch made easy 🚀 !

Code and data accompanying our SVRHM'21 paper.

Revisiting Discriminator in GAN Compression: A Generator-discriminator Cooperative Compression Scheme (NeurIPS2021)

High-Resolution Image Synthesis with Latent Diffusion Models

PyTorch implementation for paper "Full-Body Visual Self-Modeling of Robot Morphologies".

code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren, Raymond A. Yeh, Alexander G. Schwing.