PyTorch Implementation for AAAI'21 "Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selection"

Last update: Nov 22, 2022

Related tags

Deep Learning UMS-ResSel

Overview

UMS for Multi-turn Response Selection

Implements the model described in the following paper Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selection.

@inproceedings{whang2021ums,
  title={Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selection},
  author={Whang, Taesun and Lee, Dongyub and Oh, Dongsuk and Lee, Chanhee and Han, Kijong and Lee, Dong-hun and Lee, Saebyeok},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2021}
}

This code is reimplemented as a fork of huggingface/transformers and taesunwhang/BERT-ResSel.

Setup and Dependencies

This code is implemented using PyTorch v1.6.0, and provides out of the box support with CUDA 10.1 and CuDNN 7.6.5.

Anaconda / Miniconda is the recommended to set up this codebase.

Anaconda or Miniconda

Clone this repository and create an environment:

git clone https://www.github.com/taesunwhang/UMS-ResSel
conda create -n ums_ressel python=3.7

# activate the environment and install all dependencies
conda activate ums_ressel
cd UMS-ResSel

# https://pytorch.org
pip install torch==1.6.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt

Preparing Data and Checkpoints

Pre- and Post-trained Checkpoints

We provide following pre- and post-trained checkpoints.

bert-base (english), bert-base-wwm (chinese)
bert-post (ubuntu, douban, e-commerce)
electra-base (english), electra-base (chinese)
electra-post (ubuntu, douban, e-commerce)

sh scripts/download_pretrained_checkpoints.sh

Data pkls for Fine-tuning (Response Selection)

Original version for each dataset is availble in Ubuntu Corpus V1, Douban Corpus, and E-Commerce Corpus, respectively.

sh scripts/download_datasets.sh

Domain-specific Post-Training

Post-training Creation

Data for post-training BERT

#Ubuntu Corpus V1
sh scripts/create_bert_post_data_creation_ubuntu.sh
#Douban Corpus
sh scripts/create_bert_post_data_creation_douban.sh
#E-commerce Corpus
sh scripts/create_bert_post_data_creation_e-commerce.sh

Data for post-training ELECTRA

sh scripts/download_electra_post_training_pkl.sh

Post-training Examples

BERT+ (e.g., Ubuntu Corpus V1)

python3 main.py --model bert_post_training --task_name ubuntu --data_dir data/ubuntu_corpus_v1 --bert_pretrained bert-base-uncased --bert_checkpoint_path bert-base-uncased-pytorch_model.bin --task_type response_selection --gpu_ids "0" --root_dir /path/to/root_dir --training_type post_training

ELECTRA+ (e.g., Douban Corpus)

python3 main.py --model electra_post_training --task_name douban --data_dir data/electra_post_training --bert_pretrained electra-base-chinese --bert_checkpoint_path electra-base-chinese-pytorch_model.bin --task_type response_selection --gpu_ids "0" --root_dir /path/to/root_dir --training_type post_training

Training Response Selection Models

Model Arguments

BERT-Base

task_name	data_dir	bert_pretrained	bert_checkpoint_path
ubuntu	data/ubuntu_corpus_v1	bert-base-uncased	bert-base-uncased-pytorch_model.bin
douban e-commerce	data/douban data/e-commerce	bert-base-wwm-chinese	bert-base-wwm-chinese_model.bin

BERT-Post

task_name	data_dir	bert_pretrained	bert_checkpoint_path
ubuntu	data/ubuntu_corpus_v1	bert-post-uncased	bert-post-uncased-pytorch_model.pth
douban	data/douban	bert-post-douban	bert-post-douban-pytorch_model.pth
e-commerce	data/e-commerce	bert-post-ecommerce	bert-post-ecommerce-pytorch_model.pth

ELECTRA-Base

task_name	data_dir	bert_pretrained	bert_checkpoint_path
ubuntu	data/ubuntu_corpus_v1	electra-base	electra-base-pytorch_model.bin
douban e-commerce	data/douban data/e-commerce	electra-base-chinese	electra-base-chinese-pytorch_model.bin

ELECTRA-Post

task_name	data_dir	bert_pretrained	bert_checkpoint_path
ubuntu	data/ubuntu_corpus_v1	electra-post	electra-post-pytorch_model.pth
douban	data/douban	electra-post-douban	electra-post-douban-pytorch_model.pth
e-commerce	data/e-commerce	electra-post-ecommerce	electra-post-ecommerce-pytorch_model.pth

Fine-tuning Examples

BERT+ (e.g., Ubuntu Corpus V1)

python3 main.py --model bert_post --task_name ubuntu --data_dir data/ubuntu_corpus_v1 --bert_pretrained bert-post-uncased --bert_checkpoint_path bert-post-uncased-pytorch_model.pth --task_type response_selection --gpu_ids "0" --root_dir /path/to/root_dir

UMS BERT+ (e.g., Douban Corpus)

python3 main.py --model bert_post --task_name douban --data_dir data/douban --bert_pretrained bert-post-douban --bert_checkpoint_path bert-post-douban-pytorch_model.pth --task_type response_selection --gpu_ids "0" --root_dir /path/to/root_dir --multi_task_type "ins,del,srch"

UMS ELECTRA (e.g., E-Commerce)

python3 main.py --model electra_base --task_name e-commerce --data_dir data/e-commerce --bert_pretrained electra-base-chinese --bert_checkpoint_path electra-base-chinese-pytorch_model.bin --task_type response_selection --gpu_ids "0" --root_dir /path/to/root_dir --multi_task_type "ins,del,srch"

Evaluation

To evaluate the model, set --evaluate to /path/to/checkpoints

UMS BERT+ (e.g., Ubuntu Corpus V1)

python3 main.py --model bert_post --task_name ubuntu --data_dir data/ubuntu_corpus_v1 --bert_pretrained bert-post-uncased --bert_checkpoint_path bert-post-uncased-pytorch_model.pth --task_type response_selection --gpu_ids "0" --root_dir /path/to/root_dir --evaluate /path/to/checkpoints --multi_task_type "ins,del,srch"

Performance

We provide model checkpoints of UMS-BERT+, which obtained new state-of-the-art, for each dataset.

Ubuntu	[email protected]	[email protected]	[email protected]
UMS-BERT+	0.875	0.942	0.988

Douban	MAP	MRR	[email protected]	[email protected]	[email protected]	[email protected]
UMS-BERT+	0.625	0.664	0.499	0.318	0.482	0.858

E-Commerce	[email protected]	[email protected]	[email protected]
UMS-BERT+	0.762	0.905	0.986

PyTorch Implementation for AAAI'21 "Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selection"

Related tags

Overview

UMS for Multi-turn Response Selection

Setup and Dependencies

Anaconda or Miniconda

Preparing Data and Checkpoints

Pre- and Post-trained Checkpoints

Data pkls for Fine-tuning (Response Selection)

Domain-specific Post-Training

Post-training Creation

Data for post-training BERT

Data for post-training ELECTRA

Post-training Examples

BERT+ (e.g., Ubuntu Corpus V1)

ELECTRA+ (e.g., Douban Corpus)

Training Response Selection Models

Model Arguments

BERT-Base

BERT-Post

ELECTRA-Base

ELECTRA-Post

Fine-tuning Examples

BERT+ (e.g., Ubuntu Corpus V1)

UMS BERT+ (e.g., Douban Corpus)

UMS ELECTRA (e.g., E-Commerce)

Evaluation

UMS BERT+ (e.g., Ubuntu Corpus V1)

Performance

Owner

Taesun Whang

Code release for our paper, "SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo"

Implementation of the paper Scalable Intervention Target Estimation in Linear Models (NeurIPS 2021), and the code to generate simulation results.

Detectron2-FC a fast construction platform of neural network algorithm based on detectron2

Code for "Unsupervised State Representation Learning in Atari"

Official Repository for Machine Learning class - Physics Without Frontiers 2021

A Web API for automatic background removal using Deep Learning. App is made using Flask and deployed on Heroku.

Repository for GNSS-based position estimation using a Deep Neural Network

An implementation of the 1. Parallel, 2. Streaming, 3. Randomized SVD using MPI4Py

Pytoydl: A toy deep learning framework built upon numpy.

Usable Implementation of "Bootstrap Your Own Latent" self-supervised learning, from Deepmind, in Pytorch

Graph Convolutional Networks for Temporal Action Localization (ICCV2019)

HeatNet is a python package that provides tools to build, train and evaluate neural networks designed to predict extreme heat wave events globally on daily to subseasonal timescales.

Collects many various multi-modal transformer architectures, including image transformer, video transformer, image-language transformer, video-language transformer and related datasets

Medical image analysis framework merging ANTsPy and deep learning

Experiments with Fourier layers on simulation data.

A simple python program that can be used to implement user authentication tokens into your program...

Improving Contrastive Learning by Visualizing Feature Transformation, ICCV 2021 Oral

[ICML 2021] "Graph Contrastive Learning Automated" by Yuning You, Tianlong Chen, Yang Shen, Zhangyang Wang

Nb workflows - A workflow platform which allows you to run parameterized notebooks programmatically

The code for the NSDI'21 paper "BMC: Accelerating Memcached using Safe In-kernel Caching and Pre-stack Processing".