Capability Testing

Capability Testing is a model evaluation approach that tests specific, named capabilities rather than overall accuracy — evaluating whether the model can handle negation, handle rare classes, maintain consistency, or perform other specific skills required for its application.

Capability Categories

• Core Skills: Basic classification/regression accuracy on standard inputs.
• Edge Cases: Performance on boundary cases, rare events, and extreme values.
• Compositional: Ability to handle combinations of features or conditions not seen during training.
• Temporal: Stability over time as data distribution drifts.

Why It Matters

• Targeted Improvement: When a capability test fails, you know exactly what to improve.
• Requirements Alignment: Map model capabilities directly to application requirements.
• Regression Detection: Track capabilities across model versions to catch regressions in specific skills.

Capability Testing is testing skills, not scores — evaluating whether the model possesses specific abilities required for its real-world role.