Model Optimizer (GitHub Repo)
NVIDIA open-sourced its Model Optimizer library, providing standard tools like quantization and pruning to accelerate model inference for deployment.
Deep dive
- Supports various optimization techniques: Post-Training Quantization (PTQ), Quantization-Aware Training (QAT), pruning, and speculative decoding.
- Offers specialized FP4/FP8 quantization workflows for NVIDIA GPU architectures.
- Provides Python APIs for integrating into existing PyTorch-based training pipelines.
- Features a pre-configured agent skill for Claude Code.
Decoder
- Quantization: The process of reducing the precision of the numbers used to represent a model's parameters (e.g., from 16-bit to 4-bit), which reduces memory usage and speeds up inference.
- Pruning: The process of removing unnecessary or redundant weights from a neural network to reduce model size without significantly impacting accuracy.
- Distillation: Training a smaller model to produce outputs consistent with a larger, more complex model.
Original article
NVIDIA Model Optimizer
NVIDIA Model Optimizer (referred to as Model Optimizer, or ModelOpt) is a library comprising state-of-the-art model optimization techniques including quantization, pruning, Neural Architecture Search (NAS), distillation, speculative decoding and sparsity to accelerate models.
[Input] Model Optimizer currently supports inputs of a Hugging Face, PyTorch or ONNX model.
[Optimize] Model Optimizer provides Python APIs for users to easily compose the above model optimization techniques and export an optimized quantized checkpoint. Model Optimizer is also integrated with NVIDIA Megatron-Bridge, Megatron-LM and Hugging Face Accelerate for training required inference optimization techniques.
[Export for deployment] Seamlessly integrated within the NVIDIA AI software ecosystem, the quantized checkpoint generated from Model Optimizer is ready for deployment in downstream inference frameworks like SGLang, TensorRT-LLM, TensorRT, or vLLM. The unified Hugging Face export API now supports both transformers and diffusers models.
Latest News
- [2026/09/16] End-to-end W4A4 NVFP4 + QAD tutorial for Qwen3.6-35B-A3B: NVFP4 W4A4 PTQ plus quantization-aware distillation, reaching up to 1.30x vLLM throughput over BF16 and 3.1x smaller checkpoints while recovering the accuracy W4A4 costs.
- [2026/09/09] BLOG: Improving NVFP4 Accuracy with Local-Hessian Weight Scales
- [2026/08/24] BLOG: AutoQuantize: A Fast Automatic Mixed-Precision Assignment
- [2026/08/17] BLOG: Developing Nemotron 3.5 Lightning NVFP4 with QAD Using NVIDIA Model Optimizer: Learn how quantization-aware distillation recovers accuracy from aggressive NVFP4 quantization while reducing model size and increasing throughput.
- [2026/06/26] BLOG: Creating the NVIDIA Nemotron 3 Ultra NVFP4 Checkpoint with NVIDIA Model Optimizer: How we quantized Nemotron 3 Ultra (550B) to NVFP4 with Model Optimizer — up to 5.9× higher decode-heavy inference throughput than GLM-5.1 754B FP4 while matching BF16 accuracy. NVFP4 Checkpoint on Hugging Face.
- [2026/05/27] End-to-end Optimization tutorial for Nemotron-3-Nano-30B-A3B: Pruning + two-phase distillation + FP8 quantization achieving 2.6× vLLM throughput and 2.6× memory reduction.
- [2026/05/13] Puzzletron: A new algorithm for heterogeneous pruning & NAS of LLM and VLM models.
- [2026/04/15] Customer story: Domyn compresses Colosseum-355B → 260B using ModelOpt's Minitron pruning + distillation
- [2026/03/17] Customer story: Bielik.AI builds Bielik Minitron 7B (33% smaller, 50% faster, 90% quality retained) using ModelOpt's Minitron pruning + distillation
- [2026/03/11] Model Optimizer quantized Nemotron-3-Super checkpoints are available on Hugging Face for download: FP8, NVFP4.
- [2025/12/11] BLOG: Top 5 AI Model Optimization Techniques for Faster, Smarter Inference
Install
To install stable release packages for Model Optimizer with pip from PyPI:
pip install -U nvidia-modelopt[all]
To install from source in editable mode with all development dependencies or to use the latest features, run:
git clone git@github.com:NVIDIA/Model-Optimizer.git cd Model-Optimizer pip install -e .[dev]
Techniques
| Technique | Description | Docs |
|---|---|---|
| Post Training Quantization | Compress model size by 2x-4x, speeding up inference while preserving model quality! | docs |
| Quantization Aware Training / Distillation | Refine accuracy of quantized models even further with a few training steps! | docs |
| Pruning | Reduce your model parameters or memory footprint and accelerate inference by removing unnecessary weights! | |
| Distillation | Reduce deployment model size by teaching small models to behave like larger models! | docs |
| Speculative Decoding | Train draft modules to predict extra tokens during inference! | docs |
| Sparsity | Efficiently compress your model by storing only its non-zero parameter values and their locations | docs |
Deprecation Policy
Model Optimizer follows a structured approach to managing deprecated features:
- Communication: Deprecation notices are documented in the Changelog.
- Migration Period: Since Model Optimizer is still pre-1.0, we provide a 1-release (~1-month) migration period after deprecation.
- Removal: Following the migration period, deprecated elements are removed in alignment with semantic versioning standards.
Citation
@misc{nvidia-modelopt,
author = {{NVIDIA Corporation}},
title = {{NVIDIA Model Optimizer}},
howpublished = {\url{https://github.com/NVIDIA/Model-Optimizer}},
year = {2024--2026},
note = {GitHub repository}
}
AI Agents
ModelOpt's agent skills can be installed from this repository and used in any workspace.
Claude Code
claude plugin marketplace add https://github.com/NVIDIA/Model-Optimizer.git claude plugin install modelopt@modelopt
Codex
codex plugin marketplace add https://github.com/NVIDIA/Model-Optimizer.git
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi