Neural Chat

Neural Chat is a 7B parameter language model developed by Intel as a fine-tune of Mistral-7B, aligned using Direct Preference Optimization (DPO) and optimized to showcase high-performance LLM inference on Intel hardware — demonstrating that competitive language models can run efficiently on Intel Gaudi2 accelerators and Intel Xeon CPUs without requiring NVIDIA GPUs, using the Intel Extension for Transformers (ITREX) for advanced INT8/INT4 quantization.

What Is Neural Chat?

• Definition: A fine-tuned language model from Intel Labs — starting from Mistral-7B base, further trained with supervised fine-tuning on high-quality instruction data (OpenOrca), then aligned using DPO (Direct Preference Optimization) to improve response quality and helpfulness.
• Intel Hardware Showcase: Neural Chat is designed to demonstrate that high-quality LLM inference doesn't require NVIDIA GPUs — Intel optimized the model to run efficiently on Intel Gaudi2 AI accelerators, Intel Xeon Scalable processors, and Intel Arc GPUs.
• Leaderboard Achievement: At release, Neural Chat V3.1 topped the Hugging Face Open LLM Leaderboard for the 7B parameter category — beating the base Mistral-7B model and demonstrating the value of DPO alignment.
• ITREX Optimization: The Intel Extension for Transformers provides advanced quantization (INT8, INT4, mixed precision) and kernel optimizations specifically for Intel hardware — enabling Neural Chat to run at competitive speeds on CPUs that are typically considered too slow for LLM inference.

Key Features

• DPO Alignment: Uses Direct Preference Optimization rather than RLHF — a simpler alignment method that directly optimizes the model from preference pairs without training a separate reward model.
• CPU-Optimized Inference: Intel's optimizations make Neural Chat one of the fastest models to run on x86 CPUs — important for enterprise deployments where GPU availability is limited.
• INT4 Quantization: ITREX provides INT4 quantization with minimal accuracy loss — reducing memory requirements by 8× and enabling inference on standard server CPUs.
• OpenVINO Integration: Neural Chat can be exported to OpenVINO format for optimized inference on Intel hardware — including Intel integrated GPUs and Intel Neural Processing Units (NPUs) in laptops.

Neural Chat is Intel's demonstration that competitive LLM performance doesn't require NVIDIA hardware — by fine-tuning Mistral-7B with DPO alignment and optimizing inference with ITREX quantization, Intel proved that high-quality language models can run efficiently on Xeon CPUs and Gaudi accelerators, expanding the hardware options for enterprise AI deployment.