Localizing-Visual-Sounds-the-Hard-Way

Code and Dataset for "Localizing Visual Sounds the Hard Way".

The repo contains code and our pre-trained model.

Environment

Python 3.6.8
Pytorch 1.3.0

Flickr-SoundNet

We provide the pretrained model here.

To test the model, testing data and ground truth should be downloaded from learning to localize sound source.

Then run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --gt_path "path to ground truth" --testset "flickr"

VGG-Sound Source

We provide the pretrained model here.

To test the model, run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --testset "vggss"

(Note, some gt bounding boxes are updated recently, all results on VGG-SS cause a 2~3% difference on IoU.)

Both test data should be placed in the following structure.

data path
│
└───frames
│   │   image001.jpg
│   │   image002.jpg
│   │
└───audio
    │   audio011.wav
    │   audio012.wav

Citation

@InProceedings{Chen21,
              title        = "Localizing Visual Sounds the Hard Way",
              author       = "Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, Andrew Zisserman",
              booktitle    = "CVPR",
              year         = "2021"}

Localizing Visual Sounds the Hard Way

Related tags

Overview

Localizing-Visual-Sounds-the-Hard-Way

Environment

Flickr-SoundNet

VGG-Sound Source

Citation

Owner

Honglie Chen

Code for the paper "Balancing Training for Multilingual Neural Machine Translation, ACL 2020"

Deep ViT Features as Dense Visual Descriptors

Learning to Estimate Hidden Motions with Global Motion Aggregation

Simple STAC Catalogs discovery tool.

A toy project using OpenCV and PyMunk

Sync2Gen Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization

A tiny, friendly, strong baseline code for Person-reID (based on pytorch).

Predict stock movement with Machine Learning and Deep Learning algorithms

TorchX: A PyTorch Extension Library for More Efficient Deep Learning

Identify the emotion of multiple speakers in an Audio Segment

TensorFlow implementation of AlexNet and its training and testing on ImageNet ILSVRC 2012 dataset

Built a deep neural network (DNN) that functions as an end-to-end machine translation pipeline

Adjusting for Autocorrelated Errors in Neural Networks for Time Series

A dataset for online Arabic calligraphy

Vehicle detection using machine learning and computer vision techniques for Udacity's Self-Driving Car Engineer Nanodegree.

VOneNet: CNNs with a Primary Visual Cortex Front-End

CLASP - Contrastive Language-Aminoacid Sequence Pretraining

Code for the bachelors-thesis flaky fault localization

Forecasting directional movements of stock prices for intraday trading using LSTM and random forest

Implementation for "Manga Filling Style Conversion with Screentone Variational Autoencoder" (SIGGRAPH ASIA 2020 issue)