Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Last update: Dec 29, 2022

Related tags

Deep Learning wechsel

Overview

WECHSEL

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

arXiv: https://arxiv.org/abs/2112.06598

Models from the paper are available on the HuggingFace Hub:

Installation

We distribute a Python Package via PyPI:

pip install wechsel

Alternatively, clone the repository, install requirements.txt and run the code in wechsel/.

Example usage

Transferring English roberta-base to Swahili:

import torch
from transformers import AutoModel, AutoTokenizer
from datasets import load_dataset
from wechsel import WECHSEL, load_embeddings

source_tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModel.from_pretrained("roberta-base")

target_tokenizer = source_tokenizer.train_new_from_iterator(
    load_dataset("oscar", "unshuffled_deduplicated_sw", split="train")["text"],
    vocab_size=len(source_tokenizer)
)

wechsel = WECHSEL(
    load_embeddings("en"),
    load_embeddings("sw"),
    bilingual_dictionary="swahili"
)

target_embeddings, info = wechsel.apply(
    source_tokenizer,
    target_tokenizer,
    model.get_input_embeddings().weight.detach().numpy(),
)

model.get_input_embeddings().weight.data = torch.from_numpy(target_embeddings)

# use `model` and `target_tokenizer` to continue training in Swahili!

Bilingual dictionaries

We distribute 3276 bilingual dictionaries from English to other languages for use with WECHSEL in dicts/.

Citation

Please cite WECHSEL as

@misc{minixhofer2021wechsel,
      title={WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models}, 
      author={Benjamin Minixhofer and Fabian Paischer and Navid Rekabsaz},
      year={2021},
      eprint={2112.06598},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Related tags

Overview

WECHSEL

Installation

Example usage

Bilingual dictionaries

Citation

Owner

Institute of Computational Perception

A solution to the 2D Ising model of ferromagnetism, implemented using the Metropolis algorithm

High accurate tool for automatic faces detection with landmarks

code for ICCV 2021 paper 'Generalized Source-free Domain Adaptation'

“英特尔创新大师杯”深度学习挑战赛赛道3：CCKS2021中文NLP地址相关性任务

Code reproduce for paper "Vehicle Re-identification with Viewpoint-aware Metric Learning"

Code Repo for the ACL21 paper "Common Sense Beyond English: Evaluating and Improving Multilingual LMs for Commonsense Reasoning"

Crowd-Kit is a powerful Python library that implements commonly-used aggregation methods for crowdsourced annotation and offers the relevant metrics and datasets

Applying PVT to Semantic Segmentation

Official pytorch implementation of Active Learning for deep object detection via probabilistic modeling (ICCV 2021)

PyTorch Code for "Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning"

Fast and robust certifiable relative pose estimation

Customer-Transaction-Analysis - This analysis is based on a synthesised transaction dataset containing 3 months worth of transactions for 100 hypothetical customers.

TransFGU: A Top-down Approach to Fine-Grained Unsupervised Semantic Segmentation

NIMA: Neural IMage Assessment

LegoDNN: a block-grained scaling tool for mobile vision systems

Torch-mutable-modules - Use in-place and assignment operations on PyTorch module parameters with support for autograd

Really awesome semantic segmentation

A simple and lightweight genetic algorithm for optimization of any machine learning model

A Kernel fuzzer focusing on race bugs

https://sites.google.com/cornell.edu/recsys2021tutorial

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Related tags

Overview

WECHSEL

Installation

Example usage

Bilingual dictionaries

Citation

Owner

Institute of Computational Perception

A solution to the 2D Ising model of ferromagnetism, implemented using the Metropolis algorithm

High accurate tool for automatic faces detection with landmarks

code for ICCV 2021 paper 'Generalized Source-free Domain Adaptation'

“英特尔创新大师杯”深度学习挑战赛 赛道3：CCKS2021中文NLP地址相关性任务

Code reproduce for paper "Vehicle Re-identification with Viewpoint-aware Metric Learning"

Code Repo for the ACL21 paper "Common Sense Beyond English: Evaluating and Improving Multilingual LMs for Commonsense Reasoning"

Crowd-Kit is a powerful Python library that implements commonly-used aggregation methods for crowdsourced annotation and offers the relevant metrics and datasets

Applying PVT to Semantic Segmentation

Official pytorch implementation of Active Learning for deep object detection via probabilistic modeling (ICCV 2021)

PyTorch Code for "Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning"

Fast and robust certifiable relative pose estimation

Customer-Transaction-Analysis - This analysis is based on a synthesised transaction dataset containing 3 months worth of transactions for 100 hypothetical customers.

TransFGU: A Top-down Approach to Fine-Grained Unsupervised Semantic Segmentation

NIMA: Neural IMage Assessment

LegoDNN: a block-grained scaling tool for mobile vision systems

Torch-mutable-modules - Use in-place and assignment operations on PyTorch module parameters with support for autograd

Really awesome semantic segmentation

A simple and lightweight genetic algorithm for optimization of any machine learning model

A Kernel fuzzer focusing on race bugs

https://sites.google.com/cornell.edu/recsys2021tutorial

“英特尔创新大师杯”深度学习挑战赛赛道3：CCKS2021中文NLP地址相关性任务