38 lines
678 B
Plaintext
38 lines
678 B
Plaintext
# Core dependencies
|
|
pyyaml>=6.0
|
|
pandas>=1.5
|
|
numpy>=1.20
|
|
nltk>=3.6
|
|
requests>=2.25
|
|
tqdm>=4.60
|
|
tiktoken
|
|
wordfreq>=3.0
|
|
regex
|
|
flash-attn
|
|
dotenv
|
|
scipy
|
|
|
|
# For vLLM (if managed by this script)
|
|
# vllm # User should install this separately if managing vLLM manually or if specific version needed
|
|
# torch # Or newer, compatible with vLLM and Unsloth
|
|
|
|
|
|
|
|
# For DPO Finetuning (Unsloth and its dependencies)
|
|
unsloth
|
|
bitsandbytes
|
|
accelerate
|
|
peft
|
|
trl
|
|
transformers
|
|
datasets
|
|
sentencepiece
|
|
protobuf
|
|
hf_transfer
|
|
tensorboard
|
|
|
|
|
|
|
|
|
|
# Note: Users will need to ensure compatible versions of torch, CUDA, and vLLM/Unsloth
|
|
# are installed for their specific hardware if using vLLM management or finetuning. |