FSDP + Peft + GPTQModel is working but pending Peft PR #2626 with older version of Transformers. Once PEFT PR and transformer latest comat is fixed, we can add official support for it here in GPTQModel with CI unit tests for regression tests.
Sure, I use the sft example in peft, but it needs to wait this PR 2626 merged. Related cmd line is here:
accelerate launch --config_file "fsdp_config.yaml" train.py --seed 100 --model_name_or_path "hugging-quants/Meta-Llama-3.1-8B-Instruct-GPTQ-INT4" --dataset_name "smangrul/ultrachat-10k-chatml" --chat_template_format "chatml" --add_special_tokens False --append_concat_token False --splits "train,test" --max_seq_len 2048 --num_train_epochs 1 --logging_steps 5 --log_level "info" --logging_strategy "steps" --eval_strategy "epoch" --save_strategy "epoch" --bf16 True --packing True --learning_rate 1e-4 --lr_scheduler_type "cosine" --weight_decay 1e-4 --warmup_ratio 0.0 --max_grad_norm 1.0 --output_dir "llama-sft-lora-fsdp" --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --gradient_accumulation_steps 4 --gradient_checkpointing True --use_reentrant False --dataset_text_field "content" --use_flash_attn False --use_peft_lora True --lora_r 8 --lora_alpha 16 --lora_dropout 0.1 --lora_target_modules "q_proj,k_proj,v_proj,o_proj,up_proj,gate_proj" --use_4bit_quantization False. And 1 reminder: pls use transformers 4.52.4, the latest transformers has bug for this example, I am looking at this.
Originally posted by @kaixuanliu in #1642 (comment)
FSDP + Peft + GPTQModel is working but pending Peft PR #2626 with older version of Transformers. Once PEFT PR and transformer latest comat is fixed, we can add official support for it here in GPTQModel with CI unit tests for regression tests.
Originally posted by @kaixuanliu in #1642 (comment)