Bert Axioms

This is the repository with the code for the Paper Diagnosing BERT with Retrieval Heuristics

Required Data

In order to run this code, you first need to download the dataset from the TREC 2019 Deep Learning Track Guidelines. The path for these should be specified in the config file

You also need a working installation of the Indri Toolkit for indexing and retrieval.

Parameters

There are a number of hyperparemeter that need to be set (like indri path, number of candidates to be retrieved, random seed etc). These can be set on a config YAML file at scripts/config-defaults.yaml. The parameters are handled by wandb, but can easily be addapted for any YAML reader (take a look at PyYAML.)

Observations

Note that, for LNC2, we use an external C++ code for dealing with Indri. This is so we can add the duplicated documents to the index without comprimissing scores. This code should be compiled with Indri's Makefile.app. This should be as easy as edditing Makefile.app from Indri and running make -f Makefile.app. (Check https://lemur.sourceforge.io/indri/ for more details).

The removal process of documents from the indri index does not guarantee that the index statistics will change immediately. This can cause slight differences than the more "correct" way to re-create the index from scratch for every duplicated document.

Expected Results

The results from this repository may not directly replicate the ones that appear on the paper. This is due to a few performance improvements made after the paper submission. These, however, do not change the final scores and conclusions. Mostly, you may see a increase on alpha-nDCG for all methods, and a increase on QL performance accross the board.

	`nDCG_cut`	`TFCI`	`TFCII`	`MTDC`	`LNC1`	`LNC2`	`TP`	`STMC1`	`STMC2`	`STMC3`
QL	0.3633	0.9936	0.7008	0.8759	0.5021	1.000	0.3852	0.4855	0.7047	0.7011
DistilBERT	0.4537	0.6109	0.3945	0.5130	0.5006	0.0003	0.4105	0.5040	0.5120	0.5099

Code for ECIR'20 paper Diagnosing BERT with Retrieval Heuristics

Related tags

Overview

Bert Axioms

Required Data

Parameters

Observations

Expected Results

Owner

Arthur Câmara

Cross-modal Retrieval using Transformer Encoder Reasoning Networks (TERN). With use of Metric Learning and FAISS for fast similarity search on GPU

Codes and Data Processing Files for our paper.

Aiming at the common training datsets split, spectrum preprocessing, wavelength select and calibration models algorithm involved in the spectral analysis process

Unrolled Generative Adversarial Networks

Federated Learning Based on Dynamic Regularization

The official re-implementation of the Neurips 2021 paper, "Targeted Neural Dynamical Modeling".

ImageNet Adversarial Image Evaluation

Privacy-Preserving Machine Learning (PPML) Tutorial Presented at PyConDE 2022

Unofficial PyTorch implementation of SimCLR by Google Brain

A PyTorch implementation of DenseNet.

Source code for paper: Knowledge Inheritance for Pre-trained Language Models

[NeurIPS 2019] Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss

PyTorch implementation of GLOM

The implemetation of Dynamic Nerual Garments proposed in Siggraph Asia 2021

PyTea: PyTorch Tensor shape error analyzer

Invert and perturb GAN images for test-time ensembling

Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners

An official reimplementation of the method described in the INTERSPEECH 2021 paper - Speech Resynthesis from Discrete Disentangled Self-Supervised Representations.

KGDet: Keypoint-Guided Fashion Detection (AAAI 2021)

A new framework, collaborative cascade prediction based on graph neural networks (CCasGNN) to jointly utilize the structural characteristics, sequence features, and user profiles.