Correct FTPO slowness diagnosis: fixed-length padding, not compute-bound
Measured actual FTPO training context lengths (real tokenizer, all 12,000 examples): mean 530 tokens, p99 1080, max 1126 -- against a configured finetune_max_seq_length of 4000. ftpo_trainer.py's collator pads every batch to that fixed length rather than to the longest sequence in the batch, so every forward pass was processing ~4000 tokens of mostly padding (~13% utilization on average). This also explains why batch_size 1->4 had no effect: total padded-token compute is invariant to the batch/accum split. Lowered finetune_max_seq_length to 1280 (covers p99 with headroom, nothing in the dataset gets truncated) -- should cut per-step compute roughly 3x. Updated DGX_SPARK_SETUP.md §7 with the measurement and corrected takeaway. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -210,7 +210,7 @@ finetune_mode: "ftpo" # ftpo / dpo / dpo_final_token
|
||||
finetune_ftpo_dataset: "" # you can specify an existing ftpo dataset, or leave unset to let the
|
||||
# pipeline use the one produced in the generation step
|
||||
finetune_base_model_id: null # Base model for DPO (if unset, uses model_id)
|
||||
finetune_max_seq_length: 4000 # this may truncate some outputs
|
||||
finetune_max_seq_length: 1280 # measured p99 context length is 1080 tokens (max observed: 1126) -- 4000 was mostly wasted padding (~13% utilization), ~3x more compute per step than needed on this dataset
|
||||
finetune_load_in_4bit: true # qlora
|
||||
|
||||
# --- Early Stopping ---
|
||||
|
||||
Reference in New Issue
Block a user