Code for paper 'Audio-Driven Emotional Video Portraits'.

Related tags

AudioEVP
Overview

Audio-Driven Emotional Video Portraits [CVPR2021]

Xinya Ji, Zhou Hang, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, Feng Xu

[Project] [Paper]

visualization

Given an audio clip and a target video, our Emotional Video Portraits (EVP) approach is capable of generating emotion-controllable talking portraits and change the emotion of them smoothly by interpolating at the latent space.

Installation

We train and test based on Python3.6 and Pytorch. To install the dependencies run:

pip install -r requirements.txt

Testing

  • Download the pre-trained models and data under the following link: google-drive (we release results of two target person: M003 and M030), unzip the test.zip and put the file in corresponding places.

  • Step1 : audio2landmark

    The emotion of predicted landmark motions can be manipulated by the emotion features (recommanded):

    python audio2lm/test.py --config config/target_test.yaml --audio path/to/audio --condition feature --emo_feature path/to/feature
    

    or by the emotional audio of the target person:

    python audio2lm/test.py --config config/target_test.yaml --audio path/to/audio --condition feature --emo_audio path/to/emo_audio
    

    The results will be stored in results/target.mov

  • Step2 : landmark2video

    A parametric 3D face model and the corresponding fitting algorithm should be used here to regress the geometry, expression and pose parameters of the predicted landmarks and the target video. Here we release some parameters of the testing results.

    lm2video/data/target/3DMM/3DMM: images and landmark positions of the video

    lm2video/data/target/3DMM/target_test: parameters of target's video

    lm2video/data/target/3DMM/target_test_pose: pose parameters of video

    lm2video/data/target/3DMM/test_results: parameters of predicted landmarks

    Here we use vid2vid to generate video from edgemaps:

    1. Generate the testing data by running:

      python lm2video/lm2map.py
      

      and copy the results in lm2video/results/ to vid2vid/datasets/face/.

    2. Replace the face_dataset.py and base_options.py in vid2vid to lm2video/face_dataset.py and lm2video/base_options.py, the 106 keypoint version.

    3. Copy lm2video/data/target/latest_net_G0.pth to vid2vid/checkpoints/target/ , lm2video/test_target.sh to vid2vid/scripts/face and run:

      bash ./scripts/face/test_target.sh
      

Training

  • Coming soon.

Citation

@article{ji2021audio,
  title={Audio-Driven Emotional Video Portraits},
  author={Ji, Xinya and Zhou, Hang and Wang, Kaisiyuan and Wu, Wayne and Loy, Chen Change and Cao, Xun and Xu, Feng},
  journal={arXiv preprint arXiv:2104.07452},
  year={2021}
}
โค๏ธ This Is The EzilaXMusicPlayer Advaced Repo ๐ŸŽต

Telegram EzilaXMusicPlayer Bot ๐ŸŽต A bot that can play music on telegram group's voice Chat โค๏ธ Requirements ๐Ÿ“ FFmpeg NodeJS nodesource.com Python 3.7+

Sadew Jayasekara 11 Nov 12, 2022
Audio processor to map oracle notes in the VoG raid in Destiny 2 to call outs.

vog_oracles Audio processor to map oracle notes in the VoG raid in Destiny 2 to call outs. Huge thanks to mzucker on GitHub for the note detection cod

19 Sep 29, 2022
A Python port and library-fication of the midicsv tool by John Walker.

A Python port and library-fication of the midicsv tool by John Walker. If you need to convert MIDI files to human-readable text files and back, this is the library for you.

Tim Wedde 52 Dec 29, 2022
Use python MIDI to write some simple music

Use Python MIDI to write songs

ๅฐๅฎ 1 Nov 19, 2021
GNOME powered sound conversion

SoundConverter A simple sound converter application for the GNOME environment. It reads anything the GStreamer library can read, and writes Ogg Vorbis

Gautier Portet 188 Dec 17, 2022
Datamoshing with FFmpeg

ffmosher Datamoshing with FFmpeg Drag and drop video onto mosh.bat to create a datamoshed video. To datamosh an image, please ensure the file is in a

18 Sep 11, 2022
A python program for visualizing MIDI files, and displaying them in a spiral layout

SpiralMusic_python A python program for visualizing MIDI files, and displaying them in a spiral layout For a hardware version using Teensy & LED displ

Gavin 6 Nov 23, 2022
Python library for audio and music analysis

librosa A python package for music and audio analysis. Documentation See https://librosa.org/doc/ for a complete reference manual and introductory tut

librosa 5.6k Jan 06, 2023
python wrapper for rubberband

pyrubberband A python wrapper for rubberband. For now, this just provides lightweight wrappers for pitch-shifting and time-stretching. All processing

Brian McFee 106 Nov 28, 2022
๐™ฐ ๐™ผ๐šž๐šœ๐š’๐šŒ ๐™ฑ๐š˜๐š ๐™ฒ๐š›๐šŽ๐šŠ๐š๐šŽ๐š ๐™ฑ๐šข ๐šƒ๐šŽ๐šŠ๐š–๐™ณ๐š•๐š ๐Ÿ’–

TeamDltmusic ๐™ฐ ๐™ผ๐šž๐šœ๐š’๐šŒ ๐™ฑ๐š˜๐š ๐™ฒ๐š›๐šŽ๐šŠ๐š๐šŽ๐š ๐™ฑ๐šข ๐šƒ๐šŽ๐šŠ๐š–๐™ณ๐š•๐š ๐Ÿ’– Deploy String Session String Click hear you can find string session OR join He

TeamDlt 5 Jan 18, 2022
Tune in is a Collaborative Music Playing Systems where multiple guests can join a room and enjoy the song being played

โœจA collaborative music playing systems๐ŸŽถ where multiple guests can join a room โžก๐Ÿšช and enjoy the song๐ŸŽง being played.

Vedansh Vijaywargiya 8 Nov 05, 2022
An app made in Python using the PyTube and Tkinter libraries to download videos and MP3 audio.

yt-dl (GUI Edition) An app made in Python using the PyTube and Tkinter libraries to download videos and MP3 audio. How do I download this? Windows: Fi

1 Oct 23, 2021
ianZiPu is a way to write notation for Guqin (ๅค็ด) music.

PyBetween Wrapper for Between - ๋น„ํŠธ์œˆ์„ ์œ„ํ•œ ํŒŒ์ด์ฌ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ Legal Disclaimer ์˜ค์ง ๊ต์œก์  ๋ชฉ์ ์œผ๋กœ๋งŒ ์‚ฌ์šฉํ• ์ˆ˜ ์žˆ์œผ๋ฉฐ, ๋น„ํŠธ์œˆ์€ VCNC์˜ ์ž์‚ฐ์ž…๋‹ˆ๋‹ค. ์•…์˜์  ๊ณต๊ฒฉ์— ์ด์šฉํ• ์‹œ ์ฒ˜๋ฒŒ ๋ฐ›์„์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์‚ฌ์šฉ์— ๋”ฐ๋ฅธ ์ฑ…์ž„์€ ์‚ฌ์šฉ์ž๊ฐ€

Nancy Yi Liang 8 Nov 25, 2022
Pyroomacoustics is a package for audio signal processing for indoor applications. It was developed as a fast prototyping platform for beamforming algorithms in indoor scenarios.

Summary Pyroomacoustics is a software package aimed at the rapid development and testing of audio array processing algorithms. The content of the pack

Audiovisual Communications Laboratory 1k Jan 09, 2023
Read music meta data and length of MP3, OGG, OPUS, MP4, M4A, FLAC, WMA and Wave files with python 2 or 3

tinytag tinytag is a library for reading music meta data of MP3, OGG, OPUS, MP4, M4A, FLAC, WMA and Wave files with python Install pip install tinytag

Tom Wallroth 577 Dec 26, 2022
๐ŸŽต A music bot for discord servers!

music bot A music bot for Discord Servers Features Play songs in your discord server Get the lyrics without going on a web explorer Commands Command P

1 Jul 25, 2022
Nayeli: cool telegram groups vc music project

Nayeli-music Nayeli ๐Ÿฅ€ is cool telegram ๐ŸŽ groups vc music project ๐ŸŽ‹ . Nayeli-music Nayeli Deployment ๐ŸŽ‹ ๐Ÿ“ฒ Esy deploy ๐Ÿพ๏ธ Source Owner โ™ฅ๏ธ โ„๏ธ He is s

Kasun bandara 2 Dec 20, 2021
Terminal-based audio-to-text converter

att Terminal-based audio-to-text converter Project description A terminal-based audio-to-text converter written in python, enabling you to convert .wa

Sven Eschlbeck 4 Dec 15, 2022
Dataset and baseline code for the VocalSound dataset (ICASSP2022).

VocalSound: A Dataset for Improving Human Vocal Sounds Recognition Introduction Citing Download VocalSound Dataset Details Baseline Experiment Contact

Yuan Gong 58 Jan 03, 2023
Hide Your Secret Message in any Wave Audio File.

HiddenWave Embedding secret messages in wave audio file What is HiddenWave Hiddenwave is a python based program for simple audio steganography. You ca

TechChip 99 Dec 28, 2022