Meta Language-Specific Layers in Multilingual Language Models

This repo contains the source codes for our paper

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov

EMNLP 2020

Introduction

This repo contains code to train multilingual language models (XLM) that (1) contain language-specific layers, and (2) meta-learn these layers through gradient of gradient.

Language-specific layers are served as meta parameters, optimized using an iterative procedure. The goal is to remedy negative transfer in multilingual models through a meta training objective. Please see our paper for details.

Dependencies

Python 3
XLM
NumPy
PyTorch

Usage

The code is based on the official implementation of XLM. This repo only contains files that we modified from the original codebase. To train a model, please merge code with the source code of XLM, and then follow the standard preprocessing and training instructions there.

Meta Language-Specific Layers in Multilingual Language Models

Related tags

Overview

Meta Language-Specific Layers in Multilingual Language Models

Introduction

Dependencies

Usage

Owner

Zirui Wang

A small demonstration of using WebDataset with ImageNet and PyTorch Lightning

Code for the ICML 2021 paper: "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision"

Implementation of H-UCRL Algorithm

Builds a LoRa radio frequency fingerprint identification (RFFI) system based on deep learning techiniques

[ ICCV 2021 Oral ] Our method can estimate camera poses and neural radiance fields jointly when the cameras are initialized at random poses in complex scenarios (outside-in scenes, even with less texture or intense noise )

Automatic number plate recognition using tech: Yolo, OCR, Scene text detection, scene text recognation, flask, torch

PRIN/SPRIN: On Extracting Point-wise Rotation Invariant Features

D-NeRF: Neural Radiance Fields for Dynamic Scenes

[cvpr22] Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation

Official implementation for the paper: Multi-label Classification with Partial Annotations using Class-aware Selective Loss

PyTorch code for: Learning to Generate Grounded Visual Captions without Localization Supervision

Code for the paper "Can Active Learning Preemptively Mitigate Fairness Issues?" presented at RAI 2021.

Unofficial PyTorch implementation of Guided Dropout

PyTorch deep learning projects made easy.

Pytorch library for end-to-end transformer models training and serving

This repository contains several image-to-image translation models, whcih were tested for RGB to NIR image generation. The models are Pix2Pix, Pix2PixHD, CycleGAN and PointWise.

PyTorch implementation of our ICCV 2021 paper Intrinsic-Extrinsic Preserved GANs for Unsupervised 3D Pose Transfer.

The Few-Shot Bot: Prompt-Based Learning for Dialogue Systems

A set of Deep Reinforcement Learning Agents implemented in Tensorflow.

This is the official implementation of 3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection, built on SECOND.