AudioCLIP
Extending CLIP to Image, Text and Audio
This repository contains implementation of the models described in the paper arXiv:2106.13043. This work based on our previous works:
- ESResNe(X)t-fbsp: Learning Robust Time-Frequency Transformation of Audio (2021).
- ESResNet: Environmental Sound Classification Based on Visual Domain Models (2020).
Abstract
In the past, the rapidly evolving field of sound classification greatly benefited from the application of methods from other domains. Today, we observe the trend to fuse domain-specific tasks and approaches together, which provides the community with new outstanding models.
In this work, we present an extension of the CLIP model that handles audio in addition to text and images. Our proposed model incorporates the ESResNeXt audio-model into the CLIP framework using the AudioSet dataset. Such a combination enables the proposed model to perform bimodal and unimodal classification and querying, while keeping CLIP's ability to generalize to unseen datasets in a zero-shot inference fashion.
AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task, out-performing other approaches by reaching accuracies of 90.07% on the UrbanSound8K and 97.15% on the ESC-50 datasets. Further it sets new baselines in the zero-shot ESC-task on the same datasets (68.78% and 69.40%, respectively).
Finally, we also assess the cross-modal querying performance of the proposed model as well as the influence of full and partial training on the results. For the sake of reproducibility, our code is published.
How to Run the Model
The required Python version is >= 3.7.
AudioCLIP
On the ESC-50 dataset
python main.py --config protocols/audioclip-esc50.json --Dataset.args.root /path/to/ESC50
On the UrbanSound8K dataset
python main.py --config protocols/audioclip-us8k.json --Dataset.args.root /path/to/UrbanSound8K
Cite Us
@misc{guzhov2021audioclip,
title={AudioCLIP: Extending CLIP to Image, Text and Audio},
author={Andrey Guzhov and Federico Raue and Jörn Hees and Andreas Dengel},
year={2021},
eprint={2106.13043},
archivePrefix={arXiv},
primaryClass={cs.SD}
}
1 Aug 19, 2021
612 Jan 4, 2023
2 Mar 4, 2022
82 Dec 26, 2022
2.3k Dec 29, 2022
23 Nov 30, 2022
39 Oct 27, 2022
4.8k Dec 30, 2022
212 Dec 17, 2022
2 Aug 06, 2022
763 Dec 27, 2022
8 Mar 24, 2022
70 Dec 06, 2022
5 Sep 13, 2022
2 Feb 22, 2022
6 May 22, 2022
3k Jan 06, 2023
214 Nov 24, 2022
586 Dec 24, 2022
16 Mar 10, 2022
0 Mar 28, 2022
775 Jan 08, 2023
315 Jan 01, 2023
543 Jan 08, 2023
475 Jan 04, 2023
32 Nov 09, 2021
32 Dec 18, 2022
125 Dec 20, 2022