LLM Code Generation

LLM Code Generation: From Codex to DeepSeek-Coder — transformer models for code completion and synthesis

Code generation via large language models (LLMs) has transformed developer productivity. Codex (GPT-3 fine-tuned on GitHub code) pioneered GitHub Copilot; successor models (GPT-4, DeepSeek-Coder, StarCoder) achieve higher accuracy and context understanding.

Codex and Semantic Understanding

Codex (OpenAI, released 2021) is GPT-3 (175B parameters) fine-tuned on 159 GB high-quality GitHub code. Language semantics learned from code enable understanding variable names, API conventions, library dependencies. Evaluated on HumanEval benchmark: 28.8% pass@1 (single attempt succeeds, verified via execution). pass@k metric tries k generations, measuring probability of correct solution within k attempts. pass@100: 80%+ for Codex, capturing capability within multiple candidates.

GitHub Copilot and Integration

GitHub Copilot (commercial) integrates Codex into VS Code, Vim, Neovim, JetBrains IDEs. Real-time completion (50-100 ms latency required) leverages cache optimization and batching. Copilot X adds multi-line suggestions, chat interface (explanation, code fixes), documentation generation. GPT-4-based Copilot (2023) improves accuracy further.

DeepSeek-Coder and Specialized Models

DeepSeek-Coder (DeepSeek, 2024) achieves 88.3% HumanEval pass@1, outperforming GPT-3.5 and matching GPT-4. Training on 87B tokens code + 13B tokens diverse data balances code-specific and general knowledge. StarCoder (BigCode) trained on 783B Python/JavaScript tokens via BigCode dataset (permissive licenses); 15.3B parameter variant achieves competitive HumanEval performance.

Fill-in-the-Middle Objective

Fill-in-the-middle (FIM) training enables code infilling: given prefix and suffix, predict middle code. Codex uses FIM via probabilistic prefix/suffix masking during training. FIM improves code completion accuracy—context from both directions significantly reduces ambiguity.

Repository-Level and Multi-File Context

Modern code generation incorporates repository context: related files, function definitions, import statements. RAG-augmented generation retrieves relevant code snippets; in-context learning adds examples to prompt. Multi-file context (up to 4K-8K tokens) enables coherent APIs and cross-file consistency.

Evaluation and Unit Tests

HumanEval evaluates 164 Python coding problems (LeetCode difficulty). Test generation and execution (sandbox) verify correctness. Real-world evaluation remains open: does generated code pass production tests? Newer benchmarks (MBPP—Mostly Basic Python Programming, SWE-Bench for software engineering) address diverse coding tasks and problem sizes.