Unsloth Dynamic v3.0

Unsloth Dynamic v3.0 is the next iteration of our Dynamic quantization and a major improvement over Dynamic v2.0.

Today, we’re releasing Qwen3.8-27B Dynamic v3.0 quants that deliver >10% top-1% better accuracy at the same size compared to every other provider. This is an update of our first shared early preview version of Dynamic v3.0. The new 3.0 GGUFs work with most inference engines including llama.cpp and Unsloth Desktop.

Dynamic v3.0 overall preserves more model quality while keeping the same size, with stronger results across metrics like Divergence-300 @32 and KL Divergence.

Also a huge thanks to all your support! We saw over 5.1 million Unsloth Qwen3.8 downloads in just 5 days!

Our new methodology composes of many new features and improvements. We now use a much higher-quality imatrix calibration dataset from diverse sources. The dataset is refined for agentic coding, chat, and multilingual performance. We also improved layer selection and introduced many more quantization techniques to preserve as much model quality as possible.

We do not train on the imatrix calibration dataset, and we do NOT use QAT or QAD. Everything is done through post-training quantization. Our imatrix file used is available for the community to test, evaluate, and use. We encourage researchers and developers to create variations and fine-tunes of Qwen3.8 using our Unsloth quants/imatrix.

• We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed
• We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller.
• UD-Q2_K_XL is around +8% more accurate on top-1% than the next best and it's 9.83GB and managed to create a working HTML program with 1 small JS bug - previously it would break.

🔀 Divergence-300 @32

We generally report top-1% accuracy like how for Kimi-K3 "Dynamic 1-bit reaches ~78.9% top-1 accuracy while being 62% smaller." However top-1% is an argmax on 1 prediction, so it's not really effective on gauging actual inference.

We created a dataset of 300 held out examples (NOT in calibration dataset) from Terminal-Bench 2.1 + DeepSWE + Harbor + MathArena 2025-26 + non-Latin/long-doc prompts and we did greedy argmax decoding for 32 tokens for BF16 vs all quants and providers.

This allows us to gauge if there is overfitting and if quant outputs are similar to BF16's trajectories over multiple tokens. This is a better metric than top-1% accuracy since we extend KLD top-1% to more like KLD top-1% at 32 tokens.

❓1-bit should not be used for agentic use-cases

As seen in Divergence-300 @32, there is a sharp drop off from UD-Q2_K_XL to UD-IQ2_S for 32 token prediction from around 25% accuracy to under 8-10%. This sharp drop off means tool calling and non thinking modes break down. Some issues and mitigations if using the 1-bit:

1. Excessive looping You will see a lot of looping when using quants below UD-Q2_K_XL - use presence_penalty = 1.5 in all cases (or higher)
2. Empty responses Always enable thinking at least on low reasoning for 1-bit quants - non reasoning modes cause the model to not even output anyway
3. Agentic use cases and tool calling Do not use the model for tool calling - only general knowledge is retained when quantized heavily, and the model will either fail to call tools, keep calling tools or not even call them.
4. General Knowledge works A Top-1% recovery of 77% is not a replacement for Divergence-300 @32 at 8% which is a better indicator for actual inference workloads - you can use the model for very short general knowledge fact questions, but best to use UD-Q2_K_XL.

🔀 KL Divergence Benchmarks

We ran KLD benchmarks for all providers as well and report Top-1% and KLD mean. At all levels especially on the smaller quant sizes, Unsloth UD-3 quants get up to +10% extra top-1% accuracy at the same disk space!

All plots remove the MTP head from the x axis when calculating disk space to provide a fair comparison to everyone.

🕊️Not Overfitting

When comparing to our older UD-2 on unseen Wikitext and Code, we show great improvement on KLD - the bigger ones not so much, so we still use our old UD-2 for the larger quants - we plan to experiment and improve them as well!

We also control for overfitting by using totally different datasets for calibration and remove all leakages as much as possible. We test KLD on these unseen datasets, and also we do NOT do QAD / QAT, just pure PTQ so overfitting is less of a concern vs other QAD / QAT approaches.

Similarly Divergence-300 @32 uses an unseen dataset of 300 prompts from DeepSWE, Terminal Bench and others, and acts as another dataset to gauge overfitting - and shows our new UD-3 methods do not overfit.

Dynamic v2.0 (Old)

We're introducing Unsloth Dynamic v2.0 quantization - a major upgrade to our previous quants. This new method outperforms leading quantization methods and sets new benchmarks for Aider Polyglot, 5-shot MMLU and KL Divergence.

This means you can now run + fine-tune quantized LLMs while preserving as much accuracy as possible! You can run the 2.0 GGUFs on most inference engines like llama.cpp, Unsloth Studio etc.

The key advantage of using the Unsloth package and quants is our active role in fixing bugs in major models. We've collaborated directly with teams behind Qwen3, Meta (Llama 4), Mistral (Devstral), Google (Gemma 1–3) and Microsoft (Phi-3/4), contributing fixes that increase accuracy.

💡 What's New in Dynamic v2.0?

• Revamped Layer Selection for GGUFs + safetensors: Unsloth Dynamic 2.0 now selectively quantizes layers much more intelligently and extensively. Rather than modifying only select layers, we now dynamically adjust the quantization type of every possible layer, and the combinations will differ for each layer and model.
• Current selected and all future GGUF uploads will utilize Dynamic 2.0 and our new calibration dataset. The dataset contains more than >1.5M tokens (depending on model) and comprise of high-quality, hand-curated and cleaned data - to greatly enhance conversational chat performance.
• Previously, our Dynamic quantization (DeepSeek-R1 1.58-bit GGUF) was effective only for MoE architectures. Dynamic 2.0 quantization now works on all models (including MOEs & non-MoEs).
• Model-Specific Quants: Each model now uses a custom-tailored quantization scheme. E.g. the layers quantized in Gemma 3 differ significantly from those in Llama 4.
• To maximize efficiency, especially on Apple Silicon and ARM devices, we now also add Q4_NL, Q5.1, Q5.0, Q4.1, and Q4.0 formats.

📊 Why KL Divergence?

Accuracy is Not All You Need showcases how pruning layers, even by selecting unnecessary ones still yields vast differences in terms of "flips". A "flip" is defined as answers changing from incorrect to correct or vice versa. The paper shows how MMLU might not decrease as we prune layers or do quantization,but that's because some incorrect answers might have "flipped" to become correct. Our goal is to match the original model, so measuring "flips" is a good metric.

KL Divergence should be one of the gold standards for reporting quantization errors as per the research paper "Accuracy is Not All You Need". Using perplexity is incorrect since output token values can cancel out, so we must use KLD or harder benchmarks like Aider.

⚖️ Calibration Dataset Overfitting

Most frameworks report perplexity and KL Divergence using a test set of Wikipedia articles. However, we noticed using the calibration dataset which is also Wikipedia related causes quants to overfit, and attain lower perplexity scores. We utilize Calibration_v3 and Calibration_v5 datasets for fair testing which includes some wikitext data amongst other data. Also instruct models have unique chat templates, and using text only calibration datasets is not effective for instruct models (base models yes). In fact most imatrix GGUFs are typically calibrated with these issues.

🔢 MMLU Replication Adventure

• Replicating MMLU 5 shot was nightmarish. We could not replicate MMLU results for many models including Llama 3.1 (8B) Instruct, Gemma 3 (12B) and others due to subtle implementation issues.
• Llama 3.1 (8B) Instruct has a MMLU 5 shot accuracy of 67.8% using a naive MMLU implementation. We find however Llama tokenizes "A" and "_A" (A with a space in front) as different token ids. If we consider both spaced and non spaced tokens, we get 68.2% (+0.4%)
• There are many other subtle issues, and so to benchmark everything in a controlled environment, we designed our own MMLU implementation from scratch by investigating github.com/hendrycks/test directly, and verified our results across multiple models and comparing to reported numbers.

✨ Gemma 3 QAT Replication, Benchmarks

We benchmarked all Q4_0 GGUF versions, and did extensive experiments on the 12B model. We see the 12B Q4_0 QAT model gets 67.07% whilst the full bfloat16 12B version gets 67.15% on 5 shot MMLU. That's very impressive! The 27B model is mostly nearly there!

🦙 Llama 4 Bug Fixes + Run

We also helped and fixed a few Llama 4 bugs:

• Llama 4 Scout changed the RoPE Scaling configuration in their official repo. We helped resolve issues in llama.cpp to enable this change.
• Llama 4's QK Norm's epsilon for both Scout and Maverick should be from the config file - this means using 1e-05 and not 1e-06. We helped resolve these in llama.cpp and transformers.
• The Llama 4 team and vLLM also independently fixed an issue with QK Norm being shared across all heads (should not be so). MMLU Pro increased from 68.58% to 71.53% accuracy.