From 0f5f5390d7c8318bf00cfffbc150d545b2334585 Mon Sep 17 00:00:00 2001 From: Tigist Diriba Date: Fri, 4 Sep 2026 21:47:07 +0300 Subject: [PATCH] test: disable classifier dropout in the tiny Qwen2 test checkpoint test_dtensor_value_worker.py::test_value_worker_train_decreases_loss fails intermittently: 5 of 100 runs on a 2-GPU pod with the CI image. The regression value head is a Qwen2ForTokenClassification head, and HF applies its 0.1 default dropout to it because Qwen2Config defines neither classifier_dropout nor hidden_dropout. The test compares losses across three steps at lr 5e-6 on a fixed batch, where the genuine change (about -2.6e-4) is smaller than the per-step dropout noise (sd about 6.4e-4), so the 1e-3 tolerance fails in a ~5% tail. Save classifier_dropout: 0.0 into the tiny Qwen2 test checkpoint. Only classification heads read the key, so its causal-LM consumers are unaffected, and the checkpoint config is the one place the value can live: from_pretrained ignores keyword overrides for attributes the config lacks. With the loss a function of the parameters alone, the same 100-run series passed 100 of 100 with a monotone decrease in every run (step sd about 3.7e-5); the full dtensor and Megatron value modules pass against the changed asset. Co-Authored-By: Claude Fable 5 Signed-off-by: Tigist Diriba --- tests/unit/conftest.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py index 18bece9aa9d..68f684a7c28 100644 --- a/tests/unit/conftest.py +++ b/tests/unit/conftest.py @@ -652,6 +652,7 @@ def tiny_qwen2_model_path(): vocab_size=151936, tie_word_embeddings=False, num_key_value_heads=None, + classifier_dropout=0.0, ) model = Qwen2ForCausalLM(config=config) tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-1.5B")