rapidnlp-datasets
Data pipelines for both TensorFlow and PyTorch !
If you want to load public datasets, try:
If you want to load local, personal dataset with minimized boilerplate, use rapidnlp-datasets!
installation
pip install -U rapidnlp-datasets
If you work with PyTorch, you should install PyTorch first.
If you work with TensorFlow, you should install TensorFlow first.
Usage
Here are few examples to show you how to use this library.
- QuickStart: Sequence Classification Task
- QuickStart: Question Answering Task
- QuickStart: Token Classification Task
- QuickStart: Masked Language Model Task
- QuickStart: SimCSE(Sentence Embedding)
sequence-classification-quickstart
In PyTorch,
>>> import torch
>>> from rapidnlp_datasets.pt import DatasetForSequenceClassification
>>> dataset = DatasetForSequenceClassification.from_jsonl_files(
input_files=["testdata/sequence_classification.jsonl"],
vocab_file="testdata/vocab.txt",
)
>>> dataloader = torch.utils.data.DataLoader(dataset, shuffle=True, batch_size=32, collate_fn=dataset.batch_padding_collate)
>>> for idx, batch in enumerate(dataloader):
... print("No.{} batch: \n{}".format(idx, batch))
...
In TensorFlow,
>>> from rapidnlp_datasets.tf import TFDatasetForSequenceClassifiation
>>> dataset, d = TFDatasetForSequenceClassifiation.from_jsonl_files(
input_files=["testdata/sequence_classification.jsonl"],
vocab_file="testdata/vocab.txt",
return_self=True,
)
>>> for idx, batch in enumerate(iter(dataset)):
... print("No.{} batch: \n{}".format(idx, batch))
...
Especially, you can save dataset to tfrecord format when working with TensorFlow, and then build dataset from tfrecord files directly!
>>> d.save_tfrecord("testdata/sequence_classification.tfrecord")
2021-12-08 14:52:41,295 INFO utils.py 128] Finished to write 2 examples to tfrecords.
>>> dataset = TFDatasetForSequenceClassifiation.from_tfrecord_files("testdata/sequence_classification.tfrecord")
>>> for idx, batch in enumerate(iter(dataset)):
... print("No.{} batch: \n{}".format(idx, batch))
...
question-answering-quickstart
In PyTorch:
>>> import torch
>>> from rapidnlp_datasets.pt import DatasetForQuestionAnswering
>>>
>>> dataset = DatasetForQuestionAnswering.from_jsonl_files(
input_files="testdata/qa.jsonl",
vocab_file="testdata/vocab.txt",
)
>>> dataloader = torch.utils.data.DataLoader(dataset, shuffle=True, batch_size=32, collate_fn=dataset.batch_padding_collate)
>>> for idx, batch in enumerate(dataloader):
... print("No.{} batch: \n{}".format(idx, batch))
...
In TensorFlow,
>>> from rapidnlp_datasets.tf import TFDatasetForQuestionAnswering
>>> dataset, d = TFDatasetForQuestionAnswering.from_jsonl_files(
input_files="testdata/qa.jsonl",
vocab_file="testdata/vocab.txt",
return_self=True,
)
2021-12-08 15:09:06,747 INFO question_answering_dataset.py 101] Read 3 examples in total.
>>> for idx, batch in enumerate(iter(dataset)):
print()
print("NO.{} batch: \n{}".format(idx, batch))
...
Especially, you can save dataset to tfrecord format when working with TensorFlow, and then build dataset from tfrecord files directly!
>>> d.save_tfrecord("testdata/qa.tfrecord")
2021-12-08 15:09:31,329 INFO utils.py 128] Finished to write 3 examples to tfrecords.
>>> dataset = TFDatasetForQuestionAnswering.from_tfrecord_files(
"testdata/qa.tfrecord",
batch_size=32,
padding="batch",
)
>>> for idx, batch in enumerate(iter(dataset)):
print()
print("NO.{} batch: \n{}".format(idx, batch))
...
3 Nov 23, 2022
8.9k Dec 30, 2022
30 Oct 9, 2022
96 Dec 23, 2022
68 Nov 24, 2022
704 Dec 27, 2022
341 Dec 29, 2022
1 Mar 5, 2022
53 Nov 22, 2022
76 Dec 14, 2022
35 Nov 24, 2022
106 Jan 06, 2023
1 Dec 23, 2021
1.5k Jan 08, 2023
3 Feb 15, 2022
131 Dec 30, 2022
31 Dec 13, 2022
956 Dec 19, 2022
6.6k Jan 08, 2023
2 Nov 08, 2021
36 Oct 12, 2022
0 Dec 23, 2021
5 Jan 01, 2022
9 Dec 29, 2022
5 Oct 09, 2022
28 Dec 12, 2022
1 Oct 26, 2021
124 Dec 30, 2022