BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

This is a Pytorch implementation of paper: DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

Multi-robot collaborative exploration and mapping through Voronoi partition and DRL in unknown environment

TensorFlow implementation of "Attention is all you need (Transformer)"

BabelCalib: A Universal Approach to Calibrating Central Cameras. In ICCV (2021)

Python package for covariance matrices manipulation and Biosignal classification with application in Brain Computer interface

Real-Time and Accurate Full-Body Multi-Person Pose Estimation&Tracking System

This is a TensorFlow implementation for C2-Rec

Deep learned, hardware-accelerated 3D object pose estimation

Few-shot NLP benchmark for unified, rigorous eval

Pytorch-diffusion - A basic PyTorch implementation of 'Denoising Diffusion Probabilistic Models'

Deepparse is a state-of-the-art library for parsing multinational street addresses using deep learning

Implementing yolov4 target detection and tracking based on nao robot

HDR Video Reconstruction: A Coarse-to-fine Network and A Real-world Benchmark Dataset (ICCV 2021)

Based on the paper "Geometry-aware Instance-reweighted Adversarial Training" ICLR 2021 oral

pq is a jq-like Pickle file viewer

Official code repository for the publication "Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons"

The first dataset on shadow generation for the foreground object in real-world scenes.

code for the ICLR'22 paper: On Robust Prefix-Tuning for Text Classification

Structure-Preserving Deraining with Residue Channel Prior Guidance (ICCV2021)

HODEmu, is both an executable and a python library that is based on Ragagnin 2021 in prep.