While testing https://github.com/NatLibFi/Annif/pull/626, I noticed that Simplemma needs a lot of memory for Finnish language lemmatization and language detection. I did a little comparison to see how the choice of language affects memory consumption.

First the baseline, English:

#!/usr/bin/env python

from simplemma import text_lemmatizer

print(text_lemmatizer("she sells seashells on a seashore", lang='en'))

Results (partial) of running this with /usr/bin/time -v:

Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.14
Maximum resident set size (kbytes): 41652

So far so good. Then other languages I know.

Estonian:

print(text_lemmatizer("jääääre kuuuurija töööö", lang='et'))

Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.16
Maximum resident set size (kbytes): 41252

• practically same as English

Swedish:

print(text_lemmatizer("sju sköna sjuksköterskor skötte sju sjösjuka sjömän", lang='sv'))

Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.70
Maximum resident set size (kbytes): 190920

• somewhat slower, needs 150MB more memory

...

Finnish:

print(text_lemmatizer("mustan kissan paksut posket", lang='fi'))

Elapsed (wall clock) time (h:mm:ss or m:ss): 0:04.44
Maximum resident set size (kbytes): 1059328

• oops, now takes at least an order of magnitude longer and 1GB of memory! Why?

I checked the sizes of the data files for these four languages:

 14M fi.plzma
2.6M sv.plzma
604K et.plzma
479K en.plzma

There seems to be a correlation with memory usage here. The data file for Finnish is much bigger than the others, and Swedish is also big compared to English and Estonian. But I think something must be wrong if a data file that is ~10MB larger than the others leads to 1GB of extra memory usage.

I propose that Simplemma could support the Northern Sami language (ISO 639-1 code se). I understood from this discussion that adding a new language would require a corpus of word + lemma pairs. My colleague @nikopartanen found at least these two corpora that could perhaps be used as raw material:

SIKOR North Saami free corpus - this is a relatively large (9M tokens) corpus. From the description:

The corpus has been automatically processed and linguistically analyzed with the Giellatekno/Divvun tools. Therefore, it may contain wrong annotations.

Another obvious corpus is the Universal Dependency treebank for Northern Sami.

Thoughts on these two corpora? What needs to be done to make this happen?

This PR adds some generated suffix rules for Finnish language, as discussed in #19.

All these rules have an accuracy above 90% as evaluated on the Finnish language dictionary included with Simplemma. Collectively they cover around 6% of the dictionary entries.

This seems like a relatively simple library since I imagine most of the work is done on the data collection part. Would you consider porting it to C++ (and possibly other languages) so it can be more easily integrated to user applications?

Branch main refactored by Sourcery.

If you're happy with these changes, merge this Pull Request using the Squash and merge strategy.

See our documentation here.

git fetch origin sourcery/main
git merge --ff-only FETCH_HEAD
git reset HEAD^

Help us improve this pull request!

Hi @adbar,

thanks for providing this really cool (and capable) library.

Currently, all German definite articles ("der", "die", "das") are lemmatized to "der", which is wrong.

import simplemma
simplemma.lemmatize("Das", lang=('de',))
# Output: 'der'
simplemma.lemmatize("Die", lang=('de',))
# Output: 'der'
simplemma.lemmatize("Der", lang=('de',))
# Output: 'der'

Normally, I would not be too picky about wrong lemmatization (this happens all the time). But since these are among the most common German words, this should be correct. Their respective lemma should be the same word each: "Das" -> "Das", "Die" -> "Die", "Der" -> "Der". The lemma should be lowercase, if the original token is lowerecased, too.

Pull Request #23 refactored by Sourcery.

Since the original Pull Request was opened as a fork in a contributor's repository, we are unable to create a Pull Request branching from it.

To incorporate these changes, you can either:

1. Merge this Pull Request instead of the original, or

2. Ask your contributor to locally incorporate these commits and push them to the original Pull Request

   Incorporate changes via command line

   git fetch https://github.com/adbar/simplemma pull/23/head
   git merge --ff-only FETCH_HEAD
   git push

NOTE: As code is pushed to the original Pull Request, Sourcery will re-run and update (force-push) this Pull Request with new refactorings as necessary. If Sourcery finds no refactorings at any point, this Pull Request will be closed automatically.

See our documentation here.

Help us improve this pull request!

Pull Request #23 refactored by Sourcery.

Since the original Pull Request was opened as a fork in a contributor's repository, we are unable to create a Pull Request branching from it.

To incorporate these changes, you can either:

1. Merge this Pull Request instead of the original, or

2. Ask your contributor to locally incorporate these commits and push them to the original Pull Request

   Incorporate changes via command line

   git fetch https://github.com/adbar/simplemma pull/23/head
   git merge --ff-only FETCH_HEAD
   git push

NOTE: As code is pushed to the original Pull Request, Sourcery will re-run and update (force-push) this Pull Request with new refactorings as necessary. If Sourcery finds no refactorings at any point, this Pull Request will be closed automatically.

See our documentation here.

Help us improve this pull request!

Probably not intended, but this comes up with the current version 0.8.0

import simplemma simplemma.lemmatize("e", "en") 'spitshine'

BR, Bart Depoortere

First of all, thank you for creating this great lemmatizer!

I've used it for English and it's blazing fast (in my trials, it found the lemma in less than 10 ms). For other languages, such as Portuguese and Spanish it's still reasonably fast, with lemmatization working under 50ms.

For Polish, however, lemmatization is taking over 2 seconds. I know Polish is a more complicated language for lemmatizing because it's inflected, but is there a way I can speed it up? Ideally, I'd like to have it below 100ms, but even under 500ms I think it would be good enough.

In my trials, I'm passing a single word and doing it simply like this:

import  simplemma
import time

start = time.time()
langdata = simplemma.load_data('pl')
print(simplemma.lemmatize('robi', langdata))
end = time.time()
print(end-start)

Hi, the README says that this project uses data from Lemmatization lists by Michal Měchura, but the Asturian language is not supported in simplemma.

I'm wondering if there there are any plans to add the support for Asturian?

Hi @adbar ,

could you please in you documentation go into detail on what the greedy parameter actually does. In the documentation, it is mentioned as if it would be self-explanatory. However, I really cannot estimate its potential/dangers.

Thanks! :smiley:

Thanks for this great tool! I'm using it to align English lemmas with Czech lemmas for an MT use case. I'm finding that while the English lemmatizer is extremely fast (haven't benchmarked, but your ~250k/s estimate seems reasonably close), Czech is quite slow.

A tiny microbenchmark (admittedly misleading) shows disparity between the two:

In [1]: from simplemma import lemmatize

In [2]: %timeit -n1 lemmatize("fanoušků", lang="cs")
The slowest run took 768524.54 times longer than the fastest. This could mean that an intermediate result is being cached.
27.6 ms ± 67.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [3]: %timeit -n1 lemmatize("banks", lang="en")
The slowest run took 464802.09 times longer than the fastest. This could mean that an intermediate result is being cached.
16.9 ms ± 41.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

I see that lemmas are cached in the source code, with a single loop we see a near double penalty for Czech over English. I'm not exactly sure how to improve this immediately so I'm reporting it here for awareness and for others. 😄

I'm not sure if this is even worth reporting, but I was looking closer at how the language detection works and I realized that for languages that have rules (English and German currently), it can be quite easily tricked to accept nonsensical input and respond with high confidence that it is English (or probably German):

>>> from simplemma.langdetect import in_target_language, lang_detector
>>> in_target_language('grxlries cnwated pltdoms', lang='en')
1.0
>>> lang_detector('grxlries cnwated pltdoms', lang=('en','sv','de','fr','es'))
[('en', 1.0), ('unk', 0.0), ('sv', 0.0), ('de', 0.0), ('fr', 0.0), ('es', 0.0)]

This happens because the English language rules match the word suffixes (in this case, -ries, -ated and -doms) in the input words without looking too carefully at the preceding characters (the word length matters though).

This is unlikely to happen with real world text inputs, but it's possible for an attacker to deliberately trick the detector.

Hi, I'm the author of SSM which is a language learning utility for quickly making vocabulary flashcards. Thanks for this project! Without this it would have been difficult to provide multilingual lemmatization, which is an essential aspect of this tool.

However, I found that this is not particularly accurate for Russian. PyMorphy2 is a lemmatizer for Russian that I used in other projects. It's very fast and accurate in my experience, much more than spacy or anything else. Any chance you can include PyMorphy2's data in this library?

• [x] http://unimorph.ethz.ch/languages
• [x] https://github.com/lenakmeth/Wikinflection-Corpus
• [x] https://github.com/TALP-UPC/FreeLing/tree/master/data/
• [x] https://github.com/explosion/spacy-lookups-data/tree/master/spacy_lookups_data/data
• [x] https://github.com/tatuylonen/wiktextract
• [ ] http://pauillac.inria.fr/~sagot/index.html#udlexicons