Repository navigation
[Desktop] weights failed to load: Agents-A1-NVFP4 #684
Description
Before you start
- I have checked the supported models and this checkpoint is not listed there. A listed checkpoint that fails is a bug; use a Bug report instead.
- I have read the Roadmap and this model is not already planned there.
- I have read the FAQ.
- I have searched existing issues and found no duplicate.
- This is not a GGUF checkpoint. GGUF support is on the Roadmap; please do not open an issue for it.
Hugging Face link
https://huggingface.co/r0b0tlab/Agents-A1-NVFP4
Is the model architecture already supported
Yes, but this checkpoint or quantization does not load
Is the quantization already supported
Yes, but this checkpoint's weight format does not load
What happens when you load it
>>>>>>>> PASTE THE FULL SERVER LOG HERE <<<<<<<<
(copied with the "Copy server log" button in the failure dialog, or Logs -> Server status -> Copy)
WeightLoadError: ValueError: model.layers.0.mlp.gate.weight is torch.bfloat16 but the checkpoint's quant config declares model.layers.0.mlp.gate QuantScheme(nvfp4, WeightDesc(elem='e2m1', group=(1, 16
Anything else
Desktop 0.2.0-beta.23; engine 0.1.3+gfa0e9537d; NVIDIA GeForce RTX 4060, 8.00 GiB, driver 616.92; AMD Ryzen 5 5500, 31.9 GiB
Activity
silvertakana commentedon Oct 14, 2026
I cannot paste the log in, github clears my paste, so here it is.
log.txt
silvertakana commentedon Oct 14, 2026
Update after inspecting the checkpoint
I checked the safetensors headers and confirmed that all 40 model.language_model.layers..mlp.gate.weight router tensors are BF16. The quantization metadata declares NVFP4 globally but does not exclude these routers. The expert projections are packed NVFP4, so the checkpoint is mixed precision; the router tensors are where the metadata and stored dtypes disagree.
FreeToken’s loader follows the metadata and raises when it encounters a BF16 router where NVFP4 was declared. I reproduced the error with the installed loader. Adding model.language_model.layers.*.mlp.gate to quantization.exclude_modules made that tensor pass the loader’s check while keeping the expert projections quantized. I have not yet verified a full model load.
The local shard hashes match the Hugging Face download cache records, so this doesn’t appear to be download corruption. The mismatch looks to be in the checkpoint metadata or export. Could the repository metadata be updated to exclude the BF16 routers? FreeToken might also handle this mixed-precision case more gracefully, but it is currently reporting the metadata/tensor mismatch accurately.
Model card: https://huggingface.co/r0b0tlab/Agents-A1-NVFP4
NVIDIA MLP-only recipe: https://github.com/NVIDIA/Model-Optimizer/blob/main/modelopt_recipes/configs/ptq/presets/model/nvfp4_mlp_only.yaml