Nibmeister/auto-antislop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix chosen-token regularization

Browse Source
This commit is contained in:
sam-paech
2026-07-23 15:42:54 -07:00
parent 6299030455
commit bc9e75fdec
10 changed files with 273 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
configs/Mistral-Small-3.2-24B-Instruct-2506.yaml
View File

@@ -265,7 +265,8 @@ finetune_shuffle_seed: 666
# 0 = off; 0.9 strongly downsamples overrepresented rule violations
# (this is useful because the raw generated dataset is typically very skewed)
ftpo_sample_rejected_regularisation_strength: 0.7
ftpo_sample_chosen_regularisation_strength: 0.2
# 0 = off; positive values trim globally overrepresented chosen-token slots
ftpo_sample_chosen_regularisation_strength: 0.0
ftpo_sample_min_chosen_tokens: 3 # filter out ftpo samples that have fewer than this number in the chosen tokens list
@@ -283,4 +284,4 @@ ftpo_tau_mse_target: 0.5 # Grace bandwidth (logits) before the above MSE l
# MSE loss term 2: stronger mse term applied to remaining (non-target) vocab
ftpo_lambda_mse: 0.4
ftpo_clip_epsilon_logits: 2 # For a chosen token: "after winning vs rejected token by this margin, preference loss turns off"
ftpo_clip_epsilon_logits: 2 # For a chosen token: "after winning vs rejected token by this margin, preference loss turns off"