Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Last update: Dec 29, 2022

Related tags

Deep Learning wechsel

Overview

WECHSEL

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

arXiv: https://arxiv.org/abs/2112.06598

Models from the paper are available on the HuggingFace Hub:

Installation

We distribute a Python Package via PyPI:

pip install wechsel

Alternatively, clone the repository, install requirements.txt and run the code in wechsel/.

Example usage

Transferring English roberta-base to Swahili:

import torch
from transformers import AutoModel, AutoTokenizer
from datasets import load_dataset
from wechsel import WECHSEL, load_embeddings

source_tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModel.from_pretrained("roberta-base")

target_tokenizer = source_tokenizer.train_new_from_iterator(
    load_dataset("oscar", "unshuffled_deduplicated_sw", split="train")["text"],
    vocab_size=len(source_tokenizer)
)

wechsel = WECHSEL(
    load_embeddings("en"),
    load_embeddings("sw"),
    bilingual_dictionary="swahili"
)

target_embeddings, info = wechsel.apply(
    source_tokenizer,
    target_tokenizer,
    model.get_input_embeddings().weight.detach().numpy(),
)

model.get_input_embeddings().weight.data = torch.from_numpy(target_embeddings)

# use `model` and `target_tokenizer` to continue training in Swahili!

Bilingual dictionaries

We distribute 3276 bilingual dictionaries from English to other languages for use with WECHSEL in dicts/.

Citation

Please cite WECHSEL as

@misc{minixhofer2021wechsel,
      title={WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models}, 
      author={Benjamin Minixhofer and Fabian Paischer and Navid Rekabsaz},
      year={2021},
      eprint={2112.06598},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Related tags

Overview

WECHSEL

Installation

Example usage

Bilingual dictionaries

Citation

Owner

Institute of Computational Perception

Official implementation of GraphMask as presented in our paper Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking.

Experimenting with computer vision techniques to generate annotated image datasets from gameplay recordings automatically.

A Java implementation of the experiments for the paper "k-Center Clustering with Outliers in Sliding Windows"

[ICML 2021] Break-It-Fix-It: Learning to Repair Programs from Unlabeled Data

UltraPose: Synthesizing Dense Pose with 1 Billion Points by Human-body Decoupling 3D Model

SIMULEVAL A General Evaluation Toolkit for Simultaneous Translation

DeepFaceLive - Live Deep Fake in python, Real-time face swap for PC streaming or video calls

This repository is dedicated to developing and maintaining code for experiments with wide neural networks.

Code for WSDM 2022 paper, Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation.

Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning

AI grand challenge 2020 Repo (Speech Recognition Track)

Cycle Consistent Adversarial Domain Adaptation (CyCADA)

Translation-equivariant Image Quantizer for Bi-directional Image-Text Generation

PyTorch implementation for our AAAI 2022 Paper "Graph-wise Common Latent Factor Extraction for Unsupervised Graph Representation Learning"

Creative Applications of Deep Learning w/ Tensorflow

Learning Synthetic Environments and Reward Networks for Reinforcement Learning

Constrained Language Models Yield Few-Shot Semantic Parsers

The implementation our EMNLP 2021 paper "Enhanced Language Representation with Label Knowledge for Span Extraction".

BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis

Implementation of algorithms for continuous control (DDPG and NAF).