BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

Based on the given clinical dataset, Predict whether the patient having Heart Disease or Not having Heart Disease

A template repository for submitting a job to the Slurm Cluster installed at the DISI - University of Bologna

Inference code for "StylePeople: A Generative Model of Fullbody Human Avatars" paper. This code is for the part of the paper describing video-based avatars.

Implementation for "Exploiting Aliasing for Manga Restoration" (CVPR 2021)

The goal of the exercises below is to evaluate the candidate knowledge and problem solving expertise regarding the main development focuses for the iFood ML Platform team: MLOps and Feature Store development.

Source code for our CVPR 2019 paper - PPGNet: Learning Point-Pair Graph for Line Segment Detection

Implementing Vision Transformer (ViT) in PyTorch

Dieser Scanner findet Websites, die nicht direkt in Suchmaschinen auftauchen, aber trotzdem erreichbar sind.

source code of “Visual Saliency Transformer” (ICCV2021)

BrainGNN - A deep learning model for data-driven discovery of functional connectivity

Circuit Training: An open-source framework for generating chip floor plans with distributed deep reinforcement learning

pytorchのスライス代入操作をonnxに変換する際にScatterNDならないようにするサンプル

Implementation for Paper "Inverting Generative Adversarial Renderer for Face Reconstruction"

机器学习、深度学习、自然语言处理等人工智能基础知识总结。

repro_eval is a collection of measures to evaluate the reproducibility/replicability of system-oriented IR experiments

Explore extreme compression for pre-trained language models

A deep learning framework for historical document image analysis

StarGAN2 for practice

Semi-supervised Domain Adaptation via Minimax Entropy

PFENet: Prior Guided Feature Enrichment Network for Few-shot Segmentation (TPAMI).