RBERT for Relation Extraction task for KLUE
Project Description
- Relation Extraction task is one of the task of Korean Language Understanding Evaluation(KLUE) Benchmark.
- Relation extraction can be defined as multiclass classification task for relationship between subject entity and object entity.
- Classes are such as
no_relation,per:employee_of,org:founded_by... totaling 30 labels. - This repo contains custom fine-tuning method utilizing monologg's R-BERT Implementation.
- Custom punctuations with Pororo NER has been added to the dataset prior to the model's training.
- If you want to refer to the experimentation note such as punctuation method of the entity, please refer to the blog post
Arguments Usage
| Argument | type | Default | Explanation |
|---|---|---|---|
| batch_size | int | 40 | batch size for training and inferece |
| num_folds | int | 5 | number of fold for Stratified KFold |
| num_train_epochs | int | 5 | number of epochs for training |
| loss | str | focalloss | loss function |
| gamma | float | 1.0 | focalloss's gamma value |
| optimizer | str | adamp | optimizer for training |
| scheduler | str | get_cosine_schedule_with_warmup | learning rate scheduler |
| learning_rate | float | 0.00005 | initial learning rate |
| weight_decay | float | 0.01 | Loss function's weight decay, preventing overfit |
| warmup_step | int | 500 | |
| debug | bool | false | debug with CPU device for better error representation |
| dropout_rate | float | 0.1 | |
| save_steps | int | 100 | number of steps for saving the model |
| evaluation_steps | int | 100 | number of step until the evaluation |
| metric_for_best_model | str | eval/loss | the metric for determining which is the best model |
| load_best_model_at_end | bool | True |
References
- monologg's R-BERT Implementation in Pytorch
- Enriching Pre-trained Language Model with Entity Information for Relation Classification
- KLUE: Korean Language Understanding Evaluation
- KLUE RE (Relation Extraction) Dataset / CC BY-SA
- Pororo's NER Documentation
Authorship
Hardware
GPU : Tesla V100 32GB
1 Jan 17, 2022
42 Dec 20, 2022
8 Apr 14, 2022
5 Oct 12, 2022
4 Nov 22, 2022
88 Dec 29, 2022
18 Jan 11, 2022
6 Feb 16, 2022
611 Dec 16, 2022
49 Aug 24, 2022
2 May 17, 2022
55 Nov 17, 2022
1 Nov 20, 2021
27 Dec 22, 2022
9 Jul 17, 2021
5 Dec 20, 2021
16 Dec 07, 2022
464 Jan 04, 2023
44 Dec 30, 2022
1.2k Dec 18, 2022