Q: How to add custom words to Ruby lemmatizer?

Supply custom dict files with the format when initializing Lemmatizer.new, as shown in the README with sample.dict1.txt. This overrides built-in data and allows for domain-specific adjustments.

Question 1

How to add custom words to Ruby lemmatizer?

Accepted Answer

Supply custom dict files with the format

Question 2

Does lemmatizer gem support part-of-speech tagging?

Accepted Answer

Yes, it supports lemmatization for nouns, verbs, adjectives, and adverbs, and you can specify POS symbols like :noun or :verb. If not specified, it tries each POS in order, as described in the usage section.

Question 3

How accurate is lemmatizer compared to Python's NLTK?

Accepted Answer

Lemmatizer is inspired by NLTK's morphy and uses WordNet data, so accuracy is similar for English. However, it's Ruby-specific and may require custom dictionaries for edge cases that NLTK handles differently with more advanced rules.

Question 4

Can lemmatizer handle abbreviations in text?

Accepted Answer

Yes, by using 'abbr' tags in custom dictionary files, you can resolve abbreviations like 'utexas' to 'University of Texas', as demonstrated in the README with sample.dict2.txt. This is useful for preprocessing technical documents.

Question 5

What happens if a word isn't in the dictionary?

Accepted Answer

Lemmatizer leaves unknown words intact to preserve text integrity, such as proper names like 'James' or 'MacBooks'. You can add custom entries to handle these cases via user dictionaries.

Question 6

How to fix lemmatizer giving unexpected results for words like 'higher'?

Accepted Answer

This occurs because 'higher' is listed as a lemma in the dictionary. To fix it, modify the original dict files or supply custom dicts with corrected entries, as recommended in the limitations section of the README.

lemmatizer

What is lemmatizer?

Overview

Use Cases

Best For

Found a gem we're missing?

Not Ideal For

Pros & Cons

Pros

Cons

Frequently Asked Questions