Executable Semantic Parsing

Executable semantic parsing is the NLP task of converting natural language utterances into executable formal representations — such as SQL queries, API calls, Python code, or logical forms — that can be directly run against a database, knowledge base, or programming environment to produce concrete answers or actions.

Why Executable Parsing?

• Traditional NLP often produces text answers — which may be vague, incomplete, or hallucinated.
• Executable parsing produces structured, runnable code — the answer is computed by executing the generated program, ensuring precision and grounding in actual data.
• The output is verifiable: you can check whether the generated code does what the user asked, and the execution result is deterministic.

Executable Parsing Pipeline

1. Natural Language Input: User asks a question or gives a command in plain language.
2. Semantic Parsing: The model (LLM or specialized parser) converts the utterance into an executable representation.
3. Execution: The generated code or query is executed against the target system (database, API, interpreter).
4. Result: The execution output is returned to the user as the answer.

Target Representations

• SQL: For database queries — "How many customers are in New York?" → `SELECT COUNT(*) FROM customers WHERE state = 'NY'`
• SPARQL: For knowledge graph queries — "Who directed Inception?" → `SELECT ?d WHERE { :Inception :director ?d }`
• Python/Code: For calculations and data processing — "Plot sales by month" → Python code using pandas and matplotlib.
• API Calls: For interacting with services — "Book a flight from NYC to London tomorrow" → structured API request.
• Lambda Calculus: For compositional semantic representations — formal logical forms that can be evaluated.
• Robot Commands: For embodied AI — "Pick up the red block" → structured action sequence.

Semantic Parsing with LLMs

• Modern LLMs have made executable semantic parsing much more accessible — they can generate SQL, Python, and API calls from natural language with high accuracy.
• In-context learning: Few-shot examples of (question, code) pairs enable LLMs to parse new questions without fine-tuning.
• Schema/API awareness: Providing the database schema or API documentation in the prompt helps the LLM generate syntactically and semantically correct code.

Challenges

• Schema Grounding: The parser must correctly map natural language terms to database columns, table names, and relationships.
• Compositional Generalization: Handling complex, nested queries that combine multiple clauses — "Show customers who bought more than the average."
• Ambiguity: Natural language is ambiguous — "top customers" could mean highest spending, most frequent, or most recent.
• Safety: Executing generated code poses security risks — SQL injection, destructive operations, unauthorized access.

Evaluation

• Execution Accuracy: Does the generated code produce the correct answer when executed? (Preferred over exact match because multiple queries can produce the same result.)
• Benchmarks: Spider (SQL), WikiTableQuestions, MTOP (API calls), GeoQuery.

Executable semantic parsing is the bridge between natural language and computation — it transforms human intent into precise, executable actions, making databases, APIs, and code accessible to non-programmers.