Entity Extraction

Named Entity Recognition (NER) is the NLP task that identifies and classifies specific named entities — people, organizations, locations, dates, and domain-specific concepts — within unstructured text — forming the foundation of knowledge extraction pipelines, financial intelligence systems, clinical data processing, and document understanding applications.

What Is Named Entity Recognition?

• Definition: Given an input text, identify spans of text that refer to named entities and classify each span into predefined categories (PER, ORG, LOC, DATE, etc.).
• Output Format: Tagged sequence or span list — e.g., "Apple [ORG] announced the iPhone [PRODUCT] in San Francisco [LOC] on January 9, 2007 [DATE]."
• Task Formulation: Token classification problem — assign an entity tag (BIO or BIOES scheme) to each token in the input sequence.
• Evaluation: F1-score at entity span level (exact match of span boundaries and entity type required).

Why NER Matters

• Knowledge Base Construction: Automatically extract entities from millions of documents to populate databases, knowledge graphs, and structured catalogs.
• Financial Intelligence: Identify company names, executive mentions, financial figures, and events in news streams for automated trading signals and research.
• Clinical Data Extraction: Extract diagnoses, medications, dosages, and procedures from unstructured clinical notes for EHR structuring and clinical trial matching.
• Legal Document Analysis: Identify parties, dates, jurisdictions, and monetary amounts in contracts and legal filings for review automation.
• Search Enhancement: Entity-aware search systems understand "Apple" as a company in a technology query context versus a fruit in a recipe context.

Standard Entity Categories

Coarse-Grained (Universal):
- PER (Person): Albert Einstein, Elon Musk, Dr. Sarah Chen.
- ORG (Organization): TSMC, FDA, Stanford University, NATO.
- LOC (Location): Taiwan, Silicon Valley, Pacific Ocean.
- DATE / TIME: Q3 2024, January 9, 2007, 3:45 PM.
- MISC (Miscellaneous): Languages, nationalities, events (Olympic Games).

Fine-Grained / Domain-Specific:
- Biomedical: Disease (Alzheimer's), Gene (BRCA1), Drug (metformin), Protein (p53).
- Financial: Ticker (TSMC), Currency amount ($4.2B), Financial instrument (10-year Treasury).
- Legal: Case citation, Statute reference, Party name, Jurisdiction.

NER Architectures — Evolution

Rule-Based Systems (1990s–2000s):
- Hand-crafted regex patterns and gazetteers (entity dictionaries).
- High precision on known entities; brittle for novel entities and domains.
- Still used for specialized domains with well-defined entity formats (e.g., IBAN numbers, PO numbers).

Statistical CRF Models (2000s–2010s):
- Conditional Random Field (CRF) sequence labeling with hand-engineered features (capitalization, POS tags, word shape, gazetteer lookup).
- Standard production approach pre-deep learning; SpaCy's original models.

BiLSTM-CRF (2015–2018):
- Bidirectional LSTM encodes context; CRF decodes globally consistent label sequence.
- Major accuracy jump over feature-engineered approaches; became the DL baseline.

BERT-Based Token Classification (2019–present):
- Fine-tune BERT/RoBERTa on entity-labeled data with a linear classification head over token representations.
- State-of-the-art on all standard benchmarks; particularly strong on contextual disambiguation.
- Example: "Apple" classified as ORG in "Apple acquired the startup" vs. not-entity in "I ate an apple."

Generative NER (2023–present):
- Prompt LLMs (GPT-4, Claude) to extract entities in structured JSON format.
- Excellent zero-shot and few-shot performance; no labeled data needed for new entity types.
- Higher latency and cost; strong for prototype systems and rare entity categories.

Popular NER Tools & Models

BIO Tagging Scheme

• B-XXX: Beginning of entity of type XXX.
• I-XXX: Inside (continuation) of entity of type XXX.
• O: Outside any entity.

Example: "TSMC [B-ORG] Taiwan [B-LOC] semiconductor [O] plant [O]"

NER is the first extraction layer that transforms raw text into structured, queryable knowledge — as transformer models achieve near-human accuracy on standard categories and LLM-based zero-shot approaches handle novel entity types without labeled data, NER is becoming an automated utility embedded in every document intelligence pipeline.