Meta Language-Specific Layers in Multilingual Language Models

This repo contains the source codes for our paper

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov

EMNLP 2020

Introduction

This repo contains code to train multilingual language models (XLM) that (1) contain language-specific layers, and (2) meta-learn these layers through gradient of gradient.

Language-specific layers are served as meta parameters, optimized using an iterative procedure. The goal is to remedy negative transfer in multilingual models through a meta training objective. Please see our paper for details.

Dependencies

Python 3
XLM
NumPy
PyTorch

Usage

The code is based on the official implementation of XLM. This repo only contains files that we modified from the original codebase. To train a model, please merge code with the source code of XLM, and then follow the standard preprocessing and training instructions there.

Meta Language-Specific Layers in Multilingual Language Models

Related tags

Overview

Meta Language-Specific Layers in Multilingual Language Models

Introduction

Dependencies

Usage

Owner

Zirui Wang

This project is the official implementation of our accepted ICLR 2021 paper BiPointNet: Binary Neural Network for Point Clouds.

Projects for AI/ML and IoT integration for games and other presented at re:Invent 2021.

Code for the paper "Balancing Training for Multilingual Neural Machine Translation, ACL 2020"

Image classification for projects and researches

StyleGAN2-ADA - Official PyTorch implementation

Tensorflow implementation of Human-Level Control through Deep Reinforcement Learning

PaddlePaddle GAN library, including lots of interesting applications like First-Order motion transfer, wav2lip, picture repair, image editing, photo2cartoon, image style transfer, and so on.

To propose and implement a multi-class classification approach to disaster assessment from the given data set of post-earthquake satellite imagery.

Development kit for MIT Scene Parsing Benchmark

A fast MoE impl for PyTorch

A Japanese Medical Information Extraction Toolkit

Official Implementation of "Transformers Can Do Bayesian Inference"

Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.

Implementation of the federated dual coordinate descent (FedDCD) method.

Data and code for the paper "Importance of Kernel Bandwidth in Quantum Machine Learning"

Video Background Music Generation with Controllable Music Transformer (ACM MM 2021 Oral)

Code accompanying the paper "Knowledge Base Completion Meets Transfer Learning"

LightningFSL: Pytorch-Lightning implementations of Few-Shot Learning models.

Control-Raspberry-Pi-Robot-using-Hand-Gestures - A 4WD Robot car based on Raspberry Pi that controlled by hand gestures(using openCV and mediapipe)

Differential fuzzing for the masses!