TRL

論文索引

此部分正在建設中。歡迎貢獻！

組序列策略最佳化 (Group Sequence Policy Optimization)

📜 論文: https://huggingface.co/papers/2507.18071

GSPO 是 GRPO 的一種變體，它在序列級別而不是每個標記級別計算重要性取樣權重。要復現論文中的設定，請使用此配置

from trl import GRPOConfig

training_args = GRPOConfig(
    importance_sampling_level="sequence",
    loss_type="grpo",
    beta=0.0,  # GSPO set kl regularization to zero: https://github.com/volcengine/verl/pull/2775#issuecomment-3131807306 
    epsilon=3e-4,  # GSPO paper (v2), section 5.1
    epsilon_high=4e-4,  # GSPO paper (v2), section 5.1
    gradient_accumulation_steps=1,
    steps_per_generation=4,  # partition rollout batch into 4 mini-batches. GSPO paper (v2), section 5.1. Must be 4 times gradient_accumulation_steps
)

< > 在 GitHub 上更新