Lemmatization

Lemmatization is an NLP technique that reduces words to their dictionary base form (lemma) — converting "running", "ran", "runs" to "run" using linguistic rules, improving search, text analysis, and vocabulary reduction.

What Is Lemmatization?

• Definition: Reduce words to dictionary form (lemma).
• Examples: "better" → "good", "was" → "be", "mice" → "mouse".
• Method: Uses vocabulary, morphology, and part-of-speech.
• Tools: spaCy, NLTK WordNet, Stanford CoreNLP.
• vs Stemming: Lemmatization produces valid words, stemming may not.

Why Lemmatization Matters

• Search: Match "running" query to "run" documents.
• Vocabulary Reduction: Fewer unique tokens to process.
• Text Analysis: Group word variants for frequency counts.
• Feature Engineering: Better features for ML models.
• Normalization: Standardize text for comparison.

Lemmatization vs Stemming

spaCy Example

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("The mice were running quickly")
lemmas = [token.lemma_ for token in doc]
# ["the", "mouse", "be", "run", "quickly"]

Lemmatization produces linguistically correct base forms — more accurate than stemming for NLP.