A complete speech segmentation system using Kaldi and x-vectors for voice activity detection (VAD) and speaker diarisation.

Last update: Oct 27, 2022

Overview

bbc-speech-segmenter: Voice Activity Detection & Speaker Diarization

A complete speech segmentation system using Kaldi and x-vectors for voice activity detection (VAD) and speaker diarisation.

The x-vector-vad system is described in the paper; Ogura, M. & Haynes, M. (2021) X-vector-vad for Multi-genre Broadcast Speech-to-text. The paper has been submitted to 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) and is currently under review as of June 2021.

ASRU 2021 website

Quickstart

$ docker pull bbcrd/bbc-speech-segmenter

# Test

$ docker run -w /wrk -v `pwd`:/wrk bbcrd/bbc-speech-segmenter ./test.sh

# Segmentation help

$ docker run bbcrd/bbc-speech-segmenter ./run-segmentation.sh --help
usage: run-segmentation.sh [options] input.wav input.stm output-dir

options:
  --nj NUM                 Maximum number of CPU cores to use
  --stage STAGE            Start from this stage
  --cluster-threshold THR  Cluster stopping criteria. Default: -0.3
  --vad-threshold THR      Xvector classifier threshold. Lower the number the
                           more speech segments shall be returned at the
                           expense of accuracy. Default: 0.2
  --vad-method             Filter segments on an individual or segment basis.
                           Default: individual
  --no-vad                 Skip xvector vad stages. Default: false
  --help                   Print this message

# Run segmentation (VAD + diarisation), results are in output-dir/diarize.stm

$ docker run -v `pwd`:/data bbcrd/bbc-speech-segmenter \
  ./run-segmentation.sh /data/audio.wav /data/audio.stm /data/output-dir

$ cat output-dir/diarize.stm
audio 0 audio_S00004 3.750 10.125 <speech>
audio 0 audio_S00003 10.125 13.687 <speech>
audio 0 audio_S00004 13.688 16.313 <speech>
...

# Train x-vector classifier

$ docker run -w /wrk/recipe -v `pwd`:/wrk bbcrd/bbc-speech-segmenter \
  local/xvector_utils.py data/bbc-vad-train/reference.stm            \
  data/bbc-vad-train/xvectors.ark new_model.pkl

# Evaluate x-vector classifier

$ docker run -w /wrk/recipe -v `pwd`:/wrk bbcrd/bbc-speech-segmenter \
  local/xvector_utils.py evaluate data/bbc-vad-eval/reference.stm    \
  data/bbc-vad-eval/xvectors.ark model/xvector-classifier.pkl

Audio & STM file format

In order to run the segmentation script you need your audio in 16Khz Mono WAV format. You also need an STM file describing the segments you want to apply voice activity detection and speaker diarization to.

For more information on the STM file format see XVECTOR_UTILS.md.

# Convert audio file to 16Khz mono wav

$ ffmpeg audio.mp3 -vn -ac 1 -ar 16000 audio.wav

# Create STM file for input

$ DURATION=$(ffprobe -i audio.wav -show_entries format=duration -v quiet -of csv="p=0")
$ DURATION=$(printf "%0.2f\n" $DURATION)

$ FILENAME=$(basename audio.wav)

$ echo "${FILENAME%.*} 0 ${FILENAME%.*} 0.00 $DURATION <label> _" > audio.stm

$ cat audio.stm
audio 0 audio 0.00 60.00 <label> _

Use Docker image to run code in local checkout

# Bulid Docker image

$ docker build -t bbc-speech-segmenter .

# Spin up a Docker container in an interactive mode

$ docker run -it -v `pwd`:/wrk bbc-speech-segmenter /bin/bash

# Inside a Docker container

$ cd /wrk/

# Run test

$ ./test.sh
All checks passed

Training and evaluation

X-vector utility

xvector_utils.py can be used to train and evaluate x-vector classifier, as well as o extract and visualize x-vectors. For more detailed information, see XVECTOR_UTILS.md.

The documentation also gives details on file formats such as ARK, SCP or STM, which are required to use this tool.

Run x-vector VAD training

Two files are required for x-vector-vad training:

Reference STM file
X-vectors ARK file

For example, from inside the Docker container:

$ cd /wrk/recipe

$ python3 local/xvector_utils.py train \
  data/bbc-vad-train/reference.stm     \
  data/bbc-vad-train/xvectors.ark      \
  new_model.pkl

The model will be saved as new_model.pkl.

Run x-vector VAD evaluation

Three files are needed in order to run VAD evaluation:

Reference STM file
X-vectors ARK file
x-vector-vad classifier model

For example, from inside the Docker container:

$ cd /wrk/recipe

$ python3 local/xvector_utils.py evaluate \
  data/bbc-vad-eval/reference.stm        \
  data/bbc-vad-eval/xvectors.ark         \
  model/xvector-classifier.pkl

WebRTC baseline

The code for the baseline WebRTC system referenced in the paper is available in the directory recipe/baselines/denoising_DIHARD18_webrtc.

Request access to `bbc-vad-train`

Due to size restriction, only bbc-vad-eval is included in the repository. If you'd like access to bbc-vad-train, please contact Matt Haynes.

Authors

Misa Ogura [email protected]
Matt Haynes [email protected]

A complete speech segmentation system using Kaldi and x-vectors for voice activity detection (VAD) and speaker diarisation.

Related tags

Overview

bbc-speech-segmenter: Voice Activity Detection & Speaker Diarization

Quickstart

Audio & STM file format

Use Docker image to run code in local checkout

Training and evaluation

X-vector utility

Run x-vector VAD training

Run x-vector VAD evaluation

WebRTC baseline

Request access to `bbc-vad-train`

Authors

Owner

BBC

MoveNet Single Pose on DepthAI

A stable algorithm for GAN training

Load What You Need: Smaller Multilingual Transformers for Pytorch and TensorFlow 2.0.

Modeling CNN layers activity with Gaussian mixture model

AWS provides a Python SDK, "Boto3" ,which can be used to access the AWS-account from the local.

bio_inspired_min_nets_improve_the_performance_and_robustness_of_deep_networks

A bunch of random PyTorch models using PyTorch's C++ frontend

An implementation of quantum convolutional neural network with MindQuantum. Huawei, classifying MNIST dataset

Hummingbird compiles trained ML models into tensor computation for faster inference.

Generative Adversarial Networks for High Energy Physics extended to a multi-layer calorimeter simulation

Facial Expression Detection In The Realtime

VID-Fusion: Robust Visual-Inertial-Dynamics Odometry for Accurate External Force Estimation

Code corresponding to The Introspective Agent: Interdependence of Strategy, Physiology, and Sensing for Embodied Agents

Repository for open research on optimizers.

Code implementation for the paper 'Conditional Gaussian PAC-Bayes'.

Learning where to learn - Gradient sparsity in meta and continual learning

Implementation of Nyström Self-attention, from the paper Nyströmformer

Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

Official repository for the paper "Going Beyond Linear Transformers with Recurrent Fast Weight Programmers"

PyTorch implementation of the ideas presented in the paper Interaction Grounded Learning (IGL)

A complete speech segmentation system using Kaldi and x-vectors for voice activity detection (VAD) and speaker diarisation.

Related tags

Overview

bbc-speech-segmenter: Voice Activity Detection & Speaker Diarization

Quickstart

Audio & STM file format

Use Docker image to run code in local checkout

Training and evaluation

X-vector utility

Run x-vector VAD training

Run x-vector VAD evaluation

WebRTC baseline

Request access to bbc-vad-train

Authors

Owner

BBC

MoveNet Single Pose on DepthAI

A stable algorithm for GAN training

Load What You Need: Smaller Multilingual Transformers for Pytorch and TensorFlow 2.0.

Modeling CNN layers activity with Gaussian mixture model

AWS provides a Python SDK, "Boto3" ,which can be used to access the AWS-account from the local.

bio_inspired_min_nets_improve_the_performance_and_robustness_of_deep_networks

A bunch of random PyTorch models using PyTorch's C++ frontend

An implementation of quantum convolutional neural network with MindQuantum. Huawei, classifying MNIST dataset

Hummingbird compiles trained ML models into tensor computation for faster inference.

Generative Adversarial Networks for High Energy Physics extended to a multi-layer calorimeter simulation

Facial Expression Detection In The Realtime

VID-Fusion: Robust Visual-Inertial-Dynamics Odometry for Accurate External Force Estimation

Code corresponding to The Introspective Agent: Interdependence of Strategy, Physiology, and Sensing for Embodied Agents

Repository for open research on optimizers.

Code implementation for the paper 'Conditional Gaussian PAC-Bayes'.

Learning where to learn - Gradient sparsity in meta and continual learning

Implementation of Nyström Self-attention, from the paper Nyströmformer

Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

Official repository for the paper "Going Beyond Linear Transformers with Recurrent Fast Weight Programmers"

PyTorch implementation of the ideas presented in the paper Interaction Grounded Learning (IGL)

Request access to `bbc-vad-train`