Add DGX Spark (aarch64/Blackwell) setup fixes and documentation

- utils/vllm_manager.py: drop --disable-log-requests, removed in vLLM 0.26.0
- core/ftpo_trainer.py: pass token_type_ids to the 3 model forward calls in
  compute_loss -- transformers 5.5.0's Gemma3 requires it during training
  for causal-mask construction (Gemma3 is natively multimodal)
- configs/gemma-3-4b-it.yaml: lower vllm_gpu_memory_utilization 0.85->0.5,
  since the DGX Spark's 121GB is unified CPU/GPU memory and the default
  starved the OS, causing swap thrashing
- DGX_SPARK_SETUP.md: full writeup of the above plus the parts that don't
  live in this repo (two-conda-env split to resolve a vllm/unsloth
  transformers version conflict, flash-attn source build flags, torch/CUDA
  version matching, ~/.triton/cache permissions)

The antislop-vllm submodule also needed a one-line fix (removing an invalid
reference_compile kwarg in utils/refusal_detector.py that was silently
disabling refusal filtering) -- documented in DGX_SPARK_SETUP.md rather than
committed as a submodule pointer change, since we don't have push access to
upstream's antislop-vllm repo.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-08 07:39:37 -05:00
parent da2231574f
commit 4edb2135be
5 changed files with 227 additions and 3 deletions

View File

@@ -22,7 +22,7 @@ vllm_model_id: null # Model served by vLLM (if unset, will use model_id)
vllm_port: 8000
vllm_hf_token: null # Optional: Your Hugging Face token if model is gated
vllm_cuda_visible_devices: "0" # set to e.g. "0,1,2,3" for multiple gpus
vllm_gpu_memory_utilization: 0.85 # leave some room for the refusal classifier if you are using it (about 3gb)
vllm_gpu_memory_utilization: 0.5 # DGX Spark has unified CPU/GPU memory (121GB shared) -- 0.85 starved the OS and caused swap thrashing. 0.5 (~60GB) is generous for a 4B model + KV cache.
vllm_max_model_len: 4500
vllm_dtype: "bfloat16"
# Additional raw CLI arguments for vLLM server, e.g., ["--tensor-parallel-size", "4"] for multiple gpus