DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021)

Last update: Dec 26, 2022

Overview

Evaluation, Training, Demo, and Inference of DeFMO

DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021)

Denys Rozumnyi, Martin R. Oswald, Vittorio Ferrari, Jiri Matas, Marc Pollefeys

Qualitative results: https://www.youtube.com/watch?v=pmAynZvaaQ4

Pre-trained models

The pre-trained DeFMO model as reported in the paper is available here: https://polybox.ethz.ch/index.php/s/M06QR8jHog9GAcF. Put them into ./saved_models sub-folder.

Inference

For generating video temporal super-resolution:

python run.py --video example/falling_pen.avi

For generating temporal super-resolution of a single frame with the given background:

python run.py --im example/im.png --bgr example/bgr.png

Evaluation

After downloading the pre-trained models and downloading the evaluation datasets, you can run

python eval_dataset.py

Synthetic dataset generation

For the dataset generation, please download:

ShapeNetCore.v2 dataset: https://www.shapenet.org/.
Textures from the DTD dataset: https://www.robots.ox.ac.uk/~vgg/data/dtd/. The exact split used in DeFMO is from the "Neural Voxel Renderer: Learning an Accurate and Controllable Rendering Tool" model and can be downloaded here: https://polybox.ethz.ch/index.php/s/9Abv3QRm0ZgPzhK.
Backgrounds for the training dataset from the VOT dataset: https://www.votchallenge.net/vot2018/dataset.html.
Backgrounds for the testing dataset from the Sports1M dataset: https://cs.stanford.edu/people/karpathy/deepvideo/.
Blender 2.79b with Python enabled.

Then, insert your paths in renderer/settings.py file. To generate the dataset, run in renderer sub-folder:

python run_render.py

Note that the full training dataset with 50 object categories, 1000 objects per category, and 24 timestamps takes up to 1 TB of storage memory. Due to this and also the ShapeNet licence, we cannot make the pre-generated dataset public - please generate it by yourself using the steps above.

Training

Set up all paths in main_settings.py and run

python train.py

Evaluation on real-world datasets

All evaluation datasets can be found at http://cmp.felk.cvut.cz/fmo/. We provide a download_datasets.sh script to download the Falling Objects, the TbD-3D, and the TbD datasets.

Reference

If you use this repository, please cite the following publication ( https://arxiv.org/abs/2012.00595 ):

@inproceedings{defmo,
  author = {Denys Rozumnyi and Martin R. Oswald and Vittorio Ferrari and Jiri Matas and Marc Pollefeys},
  title = {DeFMO: Deblurring and Shape Recovery of Fast Moving Objects},
  booktitle = {CVPR},
  address = {Nashville, Tennessee, USA},
  month = jun,
  year = {2021}
}

Comments

Question about training set

Hi, thanks for your generous sharing.

I have a question about training set generating in your work. I generated a training set following your codes. Its size is about 100GB, far less than 1TB. Is there anything wrong?

Thanks.

opened by fan-hd 11
Apply your model on custom longer video clips

Hi thank you for releasing your code,

Can your model be applied on custom videos about high speed train crossing? Video clips last from 3 to 10 seconds, my idea was to preprocess them with your code in order to keep the same frame rate and have a better video quality for later object detection. This is an example frame from original video clip:

I tried to run your code on a video about 6 seconds and the result was a longer video (about 13min) with a lower level of detail, probably I'm doing something wrong. This is an example frame from output video clip:

How can I correctly reconstruct the quality of single frames usin all the information contained in the video?

opened by fabiozappo 4
Question about comparison with Jin et al.'s work (CVPR2018)

Hi, thank you for your interesting work! I have a question about the comparison of methods in your work. When making comparisons, did you retrain Jin et al.'s model ("Learning to Extract a Video Sequence from a Single Motion-Blurred Image" from CVPR 2018), or did you just use their pre-trained checkpoints? I couldn't find the training code on their github page.

opened by zzh-tech 2
Padding in Time-Consistency Loss

Hi,

Congratulations!

I found that "padding = tuple(side // 10 for side in sh[:2]) + (0,)" for normalized cross-correlation. Does it only implement padding to the height axis, since the padding tuple will be of size (4//10, H//10, 0)?

Thanks a lot.

opened by JLiu-Edinburgh 1
run on google colab!

I'm confused! and need to run the code on google colab or more explanation about how to implement that code in vscode or something else .if it know someone please help me

opened by ganikas 3

Releases(v1.0)

v1.0(Feb 7, 2021)

The first release of the method reported in the arxiv paper.
Source code(tar.gz)
Source code(zip)

Owner

Denys Rozumnyi

PhD student at ETH Zurich.

GitHub Repository https://github.com/rozumden/DeFMO

Code for a seq2seq architecture with Bahdanau attention designed to map stereotactic EEG data from human brains to spectrograms, using the PyTorch Lightning.

stereoEEG2speech We provide code for a seq2seq architecture with Bahdanau attention designed to map stereotactic EEG data from human brains to spectro

15 Nov 11, 2022

This repo contains the source code and a benchmark for predicting user's utilities with Machine Learning techniques for Computational Persuasion

Machine Learning for Argument-Based Computational Persuasion This repo contains the source code and a benchmark for predicting user's utilities with M

4 Nov 07, 2022

Pre-trained NFNets with 99% of the accuracy of the official paper

NFNet Pytorch Implementation This repo contains pretrained NFNet models F0-F6 with high ImageNet accuracy from the paper High-Performance Large-Scale

133 Dec 09, 2022

A PyTorch implementation of the baseline method in Panoptic Narrative Grounding (ICCV 2021 Oral)

52 Dec 19, 2022

This git repo contains the implementation of my ML project on Heart Disease Prediction

Introduction This git repo contains the implementation of my ML project on Heart Disease Prediction. This is a real-world machine learning model/proje

1 Feb 02, 2022

Official pytorch implementation of "Feature Stylization and Domain-aware Contrastive Loss for Domain Generalization" ACMMM 2021 (Oral)

Feature Stylization and Domain-aware Contrastive Loss for Domain Generalization This is an official implementation of "Feature Stylization and Domain-

22 Sep 22, 2022

METS/ALTO OCR enhancing tool by the National Library of Luxembourg (BnL)

Nautilus-OCR The National Library of Luxembourg (BnL) started its first initiative in digitizing newspapers, with layout recognition and OCR on articl

36 Dec 05, 2022

An open source library for face detection in images. The face detection speed can reach 1000FPS.

libfacedetection This is an open source library for CNN-based face detection in images. The CNN model has been converted to static variables in C sour

11.4k Dec 27, 2022

My solutions for Stanford University course CS224W: Machine Learning with Graphs Fall 2021 colabs (GNN, GAT, GraphSAGE, GCN)

machine-learning-with-graphs My solutions for Stanford University course CS224W: Machine Learning with Graphs Fall 2021 colabs Course materials can be

7 Dec 14, 2022

Code for the paper Progressive Pose Attention for Person Image Generation in CVPR19 (Oral).

Pose-Transfer Code for the paper Progressive Pose Attention for Person Image Generation in CVPR19(Oral). The paper is available here. Video generation

679 Jan 04, 2023

PyTorch implementation of SmoothGrad: removing noise by adding noise.

SmoothGrad implementation in PyTorch PyTorch implementation of SmoothGrad: removing noise by adding noise. Vanilla Gradients SmoothGrad Guided backpro

143 Jan 05, 2023

Python library for analysis of time series data including dimensionality reduction, clustering, and Markov model estimation

deeptime Releases: Installation via conda recommended. conda install -c conda-forge deeptime pip install deeptime Documentation: deeptime-ml.github.io

495 Dec 28, 2022

The 1st Place Solution of the Facebook AI Image Similarity Challenge (ISC21) : Descriptor Track.

ISC21-Descriptor-Track-1st The 1st Place Solution of the Facebook AI Image Similarity Challenge (ISC21) : Descriptor Track. You can check our solution

73 Dec 24, 2022

A PyTorch Implementation of "Neural Arithmetic Logic Units"

Neural Arithmetic Logic Units [WIP] This is a PyTorch implementation of Neural Arithmetic Logic Units by Andrew Trask, Felix Hill, Scott Reed, Jack Ra

181 Nov 18, 2022

Code repo for realtime multi-person pose estimation in CVPR'17 (Oral)

Realtime Multi-Person Pose Estimation By Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh. Introduction Code repo for winning 2016 MSCOCO Keypoints Cha

4.9k Dec 31, 2022

Code and dataset for ACL2018 paper "Exploiting Document Knowledge for Aspect-level Sentiment Classification"

Aspect-level Sentiment Classification Code and dataset for ACL2018 [paper] ‘‘Exploiting Document Knowledge for Aspect-level Sentiment Classification’’

146 Nov 29, 2022

A fast poisson image editing implementation that can utilize multi-core CPU or GPU to handle a high-resolution image input.

Poisson Image Editing - A Parallel Implementation Jiayi Weng (jiayiwen), Zixu Chen (zixuc) Poisson Image Editing is a technique that can fuse two imag

110 Dec 27, 2022

DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021)

Related tags

Overview

Evaluation, Training, Demo, and Inference of DeFMO

DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021)

Denys Rozumnyi, Martin R. Oswald, Vittorio Ferrari, Jiri Matas, Marc Pollefeys

Qualitative results: https://www.youtube.com/watch?v=pmAynZvaaQ4

Pre-trained models

Inference

Evaluation

Synthetic dataset generation

Training

Evaluation on real-world datasets

Reference

Comments

Question about training set

Apply your model on custom longer video clips

Question about comparison with Jin et al.'s work (CVPR2018)

Padding in Time-Consistency Loss

run on google colab!

Releases(v1.0)

v1.0(Feb 7, 2021)

Owner

Denys Rozumnyi

Code for a seq2seq architecture with Bahdanau attention designed to map stereotactic EEG data from human brains to spectrograms, using the PyTorch Lightning.

This repo contains the source code and a benchmark for predicting user's utilities with Machine Learning techniques for Computational Persuasion

Pre-trained NFNets with 99% of the accuracy of the official paper

A PyTorch implementation of the baseline method in Panoptic Narrative Grounding (ICCV 2021 Oral)

This git repo contains the implementation of my ML project on Heart Disease Prediction

Official pytorch implementation of "Feature Stylization and Domain-aware Contrastive Loss for Domain Generalization" ACMMM 2021 (Oral)

METS/ALTO OCR enhancing tool by the National Library of Luxembourg (BnL)

An open source library for face detection in images. The face detection speed can reach 1000FPS.

My solutions for Stanford University course CS224W: Machine Learning with Graphs Fall 2021 colabs (GNN, GAT, GraphSAGE, GCN)

Code for the paper Progressive Pose Attention for Person Image Generation in CVPR19 (Oral).

PyTorch implementation of SmoothGrad: removing noise by adding noise.

Python library for analysis of time series data including dimensionality reduction, clustering, and Markov model estimation

The 1st Place Solution of the Facebook AI Image Similarity Challenge (ISC21) : Descriptor Track.

A PyTorch Implementation of "Neural Arithmetic Logic Units"

Code repo for realtime multi-person pose estimation in CVPR'17 (Oral)

Code and dataset for ACL2018 paper "Exploiting Document Knowledge for Aspect-level Sentiment Classification"

A fast poisson image editing implementation that can utilize multi-core CPU or GPU to handle a high-resolution image input.

百度2021年语言与智能技术竞赛机器阅读理解Pytorch版baseline

GANSketchingJittor - Implementation of Sketch Your Own GAN in Jittor

Image classification for projects and researches