Narrativeqa Long

NarrativeQA (Long) is the full-document variant of the NarrativeQA benchmark — requiring models to read entire movie scripts or Gutenberg novels averaging 50,000-80,000 words to answer free-form questions, representing the frontier challenge of long-document comprehension where the answer may be embedded anywhere in a text far exceeding the context window of standard models.

What Is NarrativeQA?

• Origin: Kočiský et al. (2018) from DeepMind.
• Scale: 1,567 stories (783 books + 789 movie scripts) with 46,765 question-answer pairs.
• Format: Each story has ~30 questions; answers are free-form text (averaging ~4 words), not multiple-choice.
• Answer Source: Questions were written by human annotators who read only the plot summary — ensuring questions probe deep story understanding, not surface pattern matching.
• Two Evaluation Variants: Context = summary (~700 words) OR context = full text (~50,000-80,000 words).

Why the Long Version Is Hard

The "Long" setting — using the full book or script rather than a summary — exposes three fundamental challenges:

Challenge 1 — Context Window Overflow:
- Most transformer models cap at 4k-8k tokens (~3k-6k words). A 60,000-word novel = ~80,000 tokens.
- Solutions: RAG (retrieve relevant passages), sliding window attention, hierarchical summarization, or very long context models (Claude 100k, Gemini 1M).

Challenge 2 — Holistic Understanding:
- Some questions require synthesizing character development from chapter 1 and chapter 30: "How did [character] change throughout the story?"
- RAG retrieval of top-3 passages cannot answer these — the entire arc is needed.

Challenge 3 — Needle in a Haystack:
- Specific factual questions ("What was the name of the detective's partner's dog?") require finding a single sentence in 80,000 words.
- Retrieval can find this efficiently, but with ~5% retrieval failure rate, 5% of answers become impossible.

Performance Results

Evaluation Metrics

NarrativeQA uses three complementary metrics because answers are free-form and often have multiple valid phrasings:
- BLEU: N-gram precision between generated answer and reference answers.
- ROUGE-L: Longest common subsequence recall.
- METEOR: Unigram recall with stemming and synonym matching.

Why NarrativeQA (Long) Matters

• Ultimate Long-Context Test: No benchmark better distinguishes models with 8k vs. 100k context windows than NarrativeQA long — the performance gap is stark and meaningful.
• Literary Understanding: Books contain subtle character psychology, narrative irony, and thematic arcs that require understanding the whole text — a genuine test of deep reading comprehension.
• Application Relevance: AI research assistants, legal discovery (reading full case files), and educational summarization all require NarrativeQA-style full-document comprehension.
• RAG Architecture Driver: NarrativeQA long motivated significant research into passage retrieval optimization, dense passage indexing, and hierarchical document representation.
• Context Utilization Research: NarrativeQA long is used to study "lost in the middle" — the finding that models best use information at the beginning and end of context, missing information in the middle of long documents.

Famous "Needle in a Haystack" Test Connection

The NarrativeQA long setting directly inspired the "Needle in a Haystack" evaluation (Kamradt, 2023) — placing a specific fact anywhere in a 100k-token document and testing whether the model can retrieve it. NarrativeQA long is the naturalistic version of this synthetic test.

NarrativeQA (Long) is consuming the novel — the frontier benchmark of truly long-form document comprehension, where genuine understanding requires reading and integrating an entire book rather than finding and extracting a relevant passage.