Meta Language-Specific Layers in Multilingual Language Models

This repo contains the source codes for our paper

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov

EMNLP 2020

Introduction

This repo contains code to train multilingual language models (XLM) that (1) contain language-specific layers, and (2) meta-learn these layers through gradient of gradient.

Language-specific layers are served as meta parameters, optimized using an iterative procedure. The goal is to remedy negative transfer in multilingual models through a meta training objective. Please see our paper for details.

Dependencies

Python 3
XLM
NumPy
PyTorch

Usage

The code is based on the official implementation of XLM. This repo only contains files that we modified from the original codebase. To train a model, please merge code with the source code of XLM, and then follow the standard preprocessing and training instructions there.

Meta Language-Specific Layers in Multilingual Language Models

Related tags

Overview

Meta Language-Specific Layers in Multilingual Language Models

Introduction

Dependencies

Usage

Owner

Zirui Wang

Computational Pathology Toolbox developed by TIA Centre, University of Warwick.

DeepI2I: Enabling Deep Hierarchical Image-to-Image Translation by Transferring from GANs

This repository contains code used to audit the stability of personality predictions made by two algorithmic hiring systems

CAMoE + Dual SoftMax Loss (DSL): Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss

This repository will be a summary and outlook on all our open, medical, AI advancements.

This is an official implementation for "ResT: An Efficient Transformer for Visual Recognition".

MVSDF - Learning Signed Distance Field for Multi-view Surface Reconstruction

A framework for multi-step probabilistic time-series/demand forecasting models

Implement Decoupled Neural Interfaces using Synthetic Gradients in Pytorch

MultiLexNorm 2021 competition system from ÚFAL

Load What You Need: Smaller Multilingual Transformers for Pytorch and TensorFlow 2.0.

Multi-task Learning of Order-Consistent Causal Graphs (NeuRIPs 2021)

The code for MM2021 paper "Multi-Level Counterfactual Contrast for Visual Commonsense Reasoning"

Pre-trained models for a Cascaded-FCN in caffe and tensorflow that segments

Use stochastic processes to generate samples and use them to train a fully-connected neural network based on Keras

Distributed DataLoader For Pytorch Based On Ray

PyTorch code for the paper "FIERY: Future Instance Segmentation in Bird's-Eye view from Surround Monocular Cameras"

Code for "Adversarial Training for a Hybrid Approach to Aspect-Based Sentiment Analysis

Official implementation of the paper WAV2CLIP: LEARNING ROBUST AUDIO REPRESENTATIONS FROM CLIP

Breaking the Dilemma of Medical Image-to-image Translation