BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

A python3 tool to take a 360 degree survey of the RF spectrum (hamlib + rotctld + RTL-SDR/HackRF)

Multitask Learning Strengthens Adversarial Robustness

TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning

Graph Robustness Benchmark: A scalable, unified, modular, and reproducible benchmark for evaluating the adversarial robustness of Graph Machine Learning.

Malware Env for OpenAI Gym

Using Self-Supervised Pretext Tasks for Active Learning - Official Pytorch Implementation

Team Enigma at ArgMining 2021 Shared Task: Leveraging Pretrained Language Models for Key Point Matching

Pre-Trained Image Processing Transformer (IPT)

Multilingual Image Captioning

Learning Continuous Signed Distance Functions for Shape Representation

We present a regularized self-labeling approach to improve the generalization and robustness properties of fine-tuning.

object detection; robust detection; ACM MM21 grand challenge; Security AI Challenger Phase VII

Filtering variational quantum algorithms for combinatorial optimization

🚗 INGI Dakar 2K21 - Be the first one on the finish line ! 🚗

Erpnext app for make employee salary on payroll entry based on one or more project with percentage for all project equal 100 %

Code from PropMix, accepted at BMVC'21

Pretty Tensor - Fluent Neural Networks in TensorFlow

The official code for PRIMER: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization

A Game-Theoretic Perspective on Risk-Sensitive Reinforcement Learning

Open-Domain Question-Answering for COVID-19 and Other Emergent Domains