Use VITS and Opencpop to develop singing voice synthesis; Maybe it will VISinger.

Overview

Init

Use VITS and Opencpop to develop singing voice synthesis; Maybe it will VISinger.

本项目基于

https://github.com/jaywalnut310/vits
https://github.com/SJTMusicTeam/Muskits/
https://wenet.org.cn/opencpop/ 歌声数据

使用muskit数据预处理,获得初步数据

cd egs/opencpop/svs1/
./local/data.sh

VISinger_data
--lable
--midi_dump
--wav_dump

采样率转换

python wave_16k.py
--wav_dump
--wav_dump_16k

使用muskit将数据处理成vits的格式

1, 将lable进行拆分
python muskit/data_label_single.py

label_dump,midi_dump,wav_dump:一个文件一个标注

注意:label和lable的混用(两个单词都是对的)

VISinger_data
--label_dump
--midi_dump
--wav_dump
--wav_dump_16k

2, 将label和midi处理为frame对应的发音单元和音符(基音)
python muskit/data_format_vits.py
VISinger_data
--label_vits
--label_dump
--midi_dump
--wav_dump
--wav_dump_16k

3, 生成VITS需要的files,并分割为train和dev,test不需要(可以手动设计)
python muskit/data_format_vits.py

vits_file.txt 中的内容格式:wave path|label path|pitch path;

cp vits_file.txt VISinger/filelists/
cd VISinger/

python preprocess.py 分割为train和dev

VITS训练

cd VISinger
CUDA_VISIBLE_DEVICES=0 python train.py -c configs/singing_base.json -m singing_base 2>exit_error.log;cat exit_error.log
python vsinging_infer.py

使用16K节约内存,方便模型修改

编辑midi,然后测试

cd ../;python muskit/infer_midi.py;cd -;python vsinging_edit.py

LOSS值 MEL谱

样例音频

vits_singing_样例.wav

You might also like...
In this project, we develop a face recognize platform based on MTCNN object-detection netcwork and FaceNet self-supervised network.
In this project, we develop a face recognize platform based on MTCNN object-detection netcwork and FaceNet self-supervised network.

模式识别大作业——人脸检测与识别平台 本项目是一个简易的人脸检测识别平台,提供了人脸信息录入和人脸识别的功能。前端采用 html+css+js,后端采用 pytorch,

Official codebase used to develop Vision Transformer, MLP-Mixer, LiT and more.

Big Vision This codebase is designed for training large-scale vision models on Cloud TPU VMs. It is based on Jax/Flax libraries, and uses tf.data and

Use MATLAB to simulate the signal and extract features. Use PyTorch to build and train deep network to do spectrum sensing.

Deep-Learning-based-Spectrum-Sensing Use MATLAB to simulate the signal and extract features. Use PyTorch to build and train deep network to do spectru

Transfer style api - An API to use with Tranfer Style App, where you can use two image and transfer the style

Transfer Style API It's an API to use with Tranfer Style App, where you can use

Voice of Pajlada with model and weights.

Pajlada TTS Stripped down version of ForwardTacotron (https://github.com/as-ideas/ForwardTacotron) with pretrained weights for Pajlada's (https://gith

A voice recognition assistant similar to amazon alexa, siri and google assistant.
A voice recognition assistant similar to amazon alexa, siri and google assistant.

kenyan-Siri Build an Artificial Assistant Full tutorial (video) To watch the tutorial, click on the image below Installation For windows users (run th

An implementation of
An implementation of "Optimal Textures: Fast and Robust Texture Synthesis and Style Transfer through Optimal Transport"

Optex An implementation of Optimal Textures: Fast and Robust Texture Synthesis and Style Transfer through Optimal Transport for TU Delft CS4240. You c

this is a lite easy to use virtual keyboard project for anyone to use
this is a lite easy to use virtual keyboard project for anyone to use

virtual_Keyboard this is a lite easy to use virtual keyboard project for anyone to use motivation I made this for this year's recruitment for RobEn AA

A collection of easy-to-use, ready-to-use, interesting deep neural network models
A collection of easy-to-use, ready-to-use, interesting deep neural network models

Interesting and reproducible research works should be conserved. This repository wraps a collection of deep neural network models into a simple and un

Comments
  • couple of questions

    couple of questions

    Hello how are you ! very cool stuff you have here ,I can clearly see you love singing voice synthesis (SVS) from your forks and repos !! i wanted to ask is that a fully working Visingerr or is it a try from you to make it to sing , like can it be tested on a custom English data and have like results the same as or near the demo in the paper. Also do you have like other samples i can hear , i know that you tested it on opencpop that has almost 5.2 hours of singing data , and also in the paper they trained Visingerr for 600k iterations right ? how many iterations did you achieve on the opencpop to get the result linked below (vits_singing_样例.wav). to be honest i thought vits is data hungry like tacotron2 or fastspeech (aka needs a lot of data to get great results) , that opencpop result of your is so impressive for 5.2 hours data , i also wonder if you lowered the sample rate of opencpop from 44.1 KHz to 22KHz as i heard 44.1 KHz takes alot of time to train x10 the time needed.

    迫不及待地想知道你的消息 :)

    opened by dutchsing009 5
  • 问题

    问题

    python prepare/data_vits.py 输出 1,../VISinger_data/label_vits/XXX._label.npy|XXX_score.npy|XXX_pitch.npy|XXX_slurs.npy 2,filelists/vits_file.txt 内容格式:wave path|label path|score path|pitch path|slurs path;

    请问1 2这两步是怎么操作?

    opened by baipeng0110 3
  • 训练结果

    训练结果

    目前模型缺乏时长预测模型和基音预测模型; 训练语料中的句子修改歌词的效果;

    原歌词:雨淋湿了天空灰得更讲究

    https://user-images.githubusercontent.com/16432329/164953151-4c2513cb-f336-416b-8f04-604f13e63368.MP4

    修改歌词:你闹够了没有让我更难受

    https://user-images.githubusercontent.com/16432329/164953155-16c72670-cc89-40bc-99fe-42781c9dcdc0.MP4

    help wanted 
    opened by MaxMax2016 0
  • About release models and VISinger

    About release models and VISinger

    Hi

    This is a fantastic project that I have ever seen.

    Could you please share the released model? As on the inference step, it is said that "using the released model"

    Also, is there any plan to implement the VISinger model?

    Thank you!

    opened by shiyanpei0826 1
Owner
AmorTX
Speech
AmorTX
Equipped customers with insights about their EVs Hourly energy consumption and helped predict future charging behavior using LSTM model

Equipped customers with insights about their EVs Hourly energy consumption and helped predict future charging behavior using LSTM model. Designed sample dashboard with insights and recommendation for

Yash 2 Apr 07, 2022
Implementation of Deep Deterministic Policy Gradiet Algorithm in Tensorflow

ddpg-aigym Deep Deterministic Policy Gradient Implementation of Deep Deterministic Policy Gradiet Algorithm (Lillicrap et al.arXiv:1509.02971.) in Ten

Steven Spielberg P 247 Dec 07, 2022
ShinRL: A Library for Evaluating RL Algorithms from Theoretical and Practical Perspectives

Status: Under development (expect bug fixes and huge updates) ShinRL: A Library for Evaluating RL Algorithms from Theoretical and Practical Perspectiv

37 Dec 28, 2022
Tensorflow Implementation of Pixel Transposed Convolutional Networks (PixelTCN and PixelTCL)

Pixel Transposed Convolutional Networks Created by Hongyang Gao, Hao Yuan, Zhengyang Wang and Shuiwang Ji at Texas A&M University. Introduction Pixel

Hongyang Gao 95 Jul 24, 2022
Catch-all collection of generative art made using processing

Generative art with Processing.py Some art I have created for fun. Dependencies Processing for Python, see how to download/use here Packages contained

2 Mar 12, 2022
Implementation of "GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddings" in PyTorch

PyGAS: Auto-Scaling GNNs in PyG PyGAS is the practical realization of our G NN A uto S cale (GAS) framework, which scales arbitrary message-passing GN

Matthias Fey 139 Dec 25, 2022
Keras Implementation of The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation by (Simon Jégou, Michal Drozdzal, David Vazquez, Adriana Romero, Yoshua Bengio)

The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation: Work In Progress, Results can't be replicated yet with the m

Yad Konrad 196 Aug 30, 2022
Human4D Dataset tools for processing and visualization

HUMAN4D: A Human-Centric Multimodal Dataset for Motions & Immersive Media HUMAN4D constitutes a large and multimodal 4D dataset that contains a variet

tofis 15 Nov 09, 2022
Supplementary code for the AISTATS 2021 paper "Matern Gaussian Processes on Graphs".

Matern Gaussian Processes on Graphs This repo provides an extension for gpflow with Matérn kernels, inducing variables and trainable models implemente

41 Dec 17, 2022
DA2Lite is an automated model compression toolkit for PyTorch.

DA2Lite (Deep Architecture to Lite) is a toolkit to compress and accelerate deep network models. ⭐ Star us on GitHub — it helps!! Frameworks & Librari

Sinhan Kang 7 Mar 22, 2022
Official repository of the paper Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision

Official repository of the paper Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision

Soubhik Sanyal 689 Dec 25, 2022
Pcos-prediction - Predicts the likelihood of Polycystic Ovary Syndrome based on patient attributes and symptoms

PCOS Prediction 🥼 Predicts the likelihood of Polycystic Ovary Syndrome based on

Samantha Van Seters 1 Jan 10, 2022
OpenMMLab Text Detection, Recognition and Understanding Toolbox

Introduction English | 简体中文 MMOCR is an open-source toolbox based on PyTorch and mmdetection for text detection, text recognition, and the correspondi

OpenMMLab 3k Jan 07, 2023
Supplemental learning materials for "Fourier Feature Networks and Neural Volume Rendering"

Fourier Feature Networks and Neural Volume Rendering This repository is a companion to a lecture given at the University of Cambridge Engineering Depa

Matthew A Johnson 133 Dec 26, 2022
Cleaned test data list of DukeMTMC-reID, ICCV2021

Cleaned DukeMTMC-reID Cleaned data list of DukeMTMC-reID released with our paper accepted by ICCV 2021: Learning Instance-level Spatial-Temporal Patte

14 Feb 19, 2022
PyTorch implementation for paper StARformer: Transformer with State-Action-Reward Representations.

StARformer This repository contains the PyTorch implementation for our paper titled StARformer: Transformer with State-Action-Reward Representations.

Jinghuan Shang 14 Dec 09, 2022
Full Stack Deep Learning Labs

Full Stack Deep Learning Labs Welcome! Project developed during lab sessions of the Full Stack Deep Learning Bootcamp. We will build a handwriting rec

Full Stack Deep Learning 1.2k Dec 31, 2022
Unity Propagation in Bayesian Networks Handling Inconsistency via Unity Smoothing

This repository contains the scripts needed to generate the results from the paper Unity Propagation in Bayesian Networks Handling Inconsistency via U

0 Jan 19, 2022
PyTorch implementation of 1712.06087 "Zero-Shot" Super-Resolution using Deep Internal Learning

Unofficial PyTorch implementation of "Zero-Shot" Super-Resolution using Deep Internal Learning Unofficial Implementation of 1712.06087 "Zero-Shot" Sup

Jacob Gildenblat 196 Nov 27, 2022
PyTorch implementation for "Sharpness-aware Quantization for Deep Neural Networks".

Sharpness-aware Quantization for Deep Neural Networks Recent Update 2021.11.23: We release the source code of SAQ. Setup the environments Clone the re

Zhuang AI Group 30 Dec 19, 2022