BigDL
An LLM acceleration library for Intel XPU (GPU, NPU, CPU) to speed up local inference and finetuning of popular models.
What is BigDL?
IPEX-LLM is an LLM acceleration library developed by Intel to optimize the performance of large language models on Intel XPU hardware, including GPUs, NPUs, and CPUs. It solves the problem of slow and resource-intensive local LLM inference by providing low-bit quantization, hardware-specific optimizations, and seamless integration with popular AI frameworks. This enables users to run models like LLaMA and Mistral efficiently on consumer Intel devices.
Developers and researchers working with local LLM deployment on Intel hardware, including those using integrated GPUs (e.g., Intel Core Ultra), discrete Arc GPUs, or NPUs for inference and finetuning tasks.
Developers choose IPEX-LLM for its deep optimization of LLMs specifically for Intel architectures, offering superior performance and lower memory usage compared to generic solutions. Its unique selling point is the ability to run large models like DeepSeek V3 671B on just 1-2 Intel Arc GPUs through techniques like FlashMoE, which is not commonly available in other acceleration libraries.
Overview
Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.
Use Cases
Best For
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
