SAAVN - Sound Adversarial Audio-Visual Navigation,ICLR2022 (In PyTorch)

Last update: Aug 30, 2022

Related tags

Deep Learning SAAVN

Overview

SAAVN

SAAVN Code release for paper "Sound Adversarial Audio-Visual Navigation,ICLR2022" (In PyTorch)

These code are under cleaning! Some of bugs maybe happen, please tell me if you have any trouble.

Thanks

These codes are based on the SoundSpaces code base.

Usage

This repo supports AudioGoal Task on Replica and Matterport3D datasets.

Below we show the commands for training and evaluating AudioGoal with Depth sensor on Replica, but it applies to Matterport dataset as well.

Training

python main.py --default av_nav --run-type train --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Validation (evaluate each checkpoint and generate a validation curve)

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Test the best validation checkpoint based on validation curve

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Generate demo video with audio

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Note: [exp_config_file] is the main parameter configuration file of the experiment, while [tag_config_file] is special parameter configuration file for abalation experiments.

Citation

If you use this model in your research, please cite the following paper:

@inproceedings{YinfengICLR2022saavn,
	title = {Sound Adversarial Audio-Visual Navigation},
	author = {Yinfeng Yu, Wenbing Huang, Fuchun Sun, Changan Chen, Yikai Wang, Xiaohong Liu},
	year = {2022},
        booktitle={ICLR},
}

SAAVN - Sound Adversarial Audio-Visual Navigation,ICLR2022 (In PyTorch)

Related tags

Overview

SAAVN

SAAVN Code release for paper "Sound Adversarial Audio-Visual Navigation,ICLR2022" (In PyTorch)

These code are under cleaning! Some of bugs maybe happen, please tell me if you have any trouble.

Thanks

Usage

Citation

Owner

YinfengYu

Pseudo-rng-app - whos needs science to make a random number when you have pseudoscience?

一套完整的微博舆情分析流程代码，包括微博爬虫、LDA主题分析和情感分析。

Finding an Unsupervised Image Segmenter in each of your Deep Generative Models

Gans-in-action - Companion repository to GANs in Action: Deep learning with Generative Adversarial Networks

Pytorch implementation AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

Video Frame Interpolation with Transformer (CVPR2022)

Code for Mining the Benefits of Two-stage and One-stage HOI Detection

codes for IKM (arXiv2021, Submitted to IEEE Trans)

Official code repository for A Simple Long-Tailed Rocognition Baseline via Vision-Language Model.

VideoGPT: Video Generation using VQ-VAE and Transformers

A set of tools for converting a darknet dataset to COCO format working with YOLOX

PyTorch implementation of our paper How robust are discriminatively trained zero-shot learning models?

Official implementation of VaxNeRF (Voxel-Accelearated NeRF).

GND-Nets (Graph Neural Diffusion Networks) in TensorFlow.

PyTorch implementation of the paper:A Convolutional Approach to Melody Line Identification in Symbolic Scores.

Experiment about Deep Person Re-identification with EfficientNet-v2

This repository contains the source code for the paper First Order Motion Model for Image Animation

MPLP: Metapath-Based Label Propagation for Heterogenous Graphs

PyTorch implementation of MLP-Mixer

Official code for the CVPR 2021 paper "How Well Do Self-Supervised Models Transfer?"