Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss. Use when deploying large models (7B-70B) on limited GPU memory, when you need faster inference than GPTQ with better accuracy preservation, or for instruction-tuned and multimodal models. MLSys 2024 Best Paper Award winner.
The awq-quantization skill enables 4-bit weight quantization that preserves important weights based on activation patterns, offering an effective solution for large language models (LLMs). It solves the challenge of deploying large models, such as those in the 7B to 70B range, on limited GPU memory without sacrificing significant accuracy. This technique provides up to a 3x speedup in inference times, making it a highly efficient approach for fast and memory-optimized deployment of models, particularly when using GPUs with Ampere architecture or later (A100, H100, RTX 40xx). The skill is designed to minimize accuracy loss, ensuring that models still perform well even after quantization, unlike traditional methods that might significantly compromise model performance for the sake of speed or memory efficiency.
You can install the skill using the following command: pip install autoawq. For optimized CUDA kernels, use pip install autoawq[kernels], and for Intel CPU/XPU optimization, use pip install autoawq[cpu].
AWQ requires GPUs with Ampere architecture or later (e.g., A100, H100, or RTX 40xx) to fully leverage Marlin kernel support for enhanced performance.
Quantizing a 7B model typically takes around 10-15 minutes, while quantizing a 70B model can take up to 1 hour.
While AWQ works well with instruction-tuned models, it can also be applied to other types of models with relatively low accuracy loss. However, it generalizes best for instruction-tuned and chat models.
AWQ is not optimized for on-the-fly quantization. If you require zero calibration overhead for on-the-fly quantization, you should consider using bitsandbytes instead.
Quick Setup:
.claude/skills/Repository
zechenzhangAGI/AI-research-SKILLs