Skip to content

rec_gsm8k

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

rec_gsm8k

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.md
README.md
 
 
gsm8k.yaml
gsm8k.yaml
 
 

README.md

Example: group-relative REINFORCE variants on GSM8k dataset

This example shows the usage of group-relative REINFORCE variants on the GSM8k dataset.

For more details about algorithm design, please refer to our paper.

The config file is located in gsm8k.yaml.

Group-relative REINFORCE variants

This folder provides example configurations for running different group-relative REINFORCE variants within Trinity-RFT. It includes three major families:

  • REC family (regularization by clipping)
  • REP family (regularization by an additive loss term)
  • RED family (actively shaping data distribution)

These include baseline algorithms like vanilla REINFORCE and GRPO as special cases.

All algorithms are instantiated through modular YAML configs for easy reproduction and extension.

Summary Table 📝

Family Variants Key Idea
Baselines REINFORCE, GRPO Standard references
REC OneSide/TwoSide/Ring-IS/NoIS Clipping as regularization, with or without importance sampling
REP AsymRE, OPMD Regularization by an additive loss term
RED Drop, Weight Actively shaping data distribution

Instantiations

Baselines

Vanilla REINFORCE with group mean as baseline:

algorithm:
  algorithm_type: rec
  policy_loss_fn_args:
    epsilon_low: 0.2
    epsilon_high: 0.2
    clip_mode: "none" # no clipping
    weight: "none" # uniform weighting for samples
    temp: 1.0
    regularizer: "none" # no regularizer
    regularizer_coef: 0.0
  advantage_fn_args:
    std_normalize: false

GRPO with KL regularization (enabled via kl_loss_fn and kl_loss_fn_args):

algorithm:
  algorithm_type: rec
  policy_loss_fn_args:
    epsilon_low: 0.2
    epsilon_high: 0.2
    clip_mode: "one-side"
    weight: "importance_sampling"
    temp: 1.0
    regularizer: "none"
    regularizer_coef: 0.0
  advantage_fn_args:
    std_normalize: true
  kl_loss_fn: 'k2'
  kl_loss_fn_args:
    kl_coef:  0.0

REC family

REC-OneSide-NoIS:

algorithm:
  algorithm_type: rec
  policy_loss_fn_args:
    epsilon_low: 0.2
    epsilon_high: 0.2
    clip_mode: "one-side"
    weight: "none"
  advantage_fn_args:
    std_normalize: false
  kl_loss_fn_args:
    kl_coef: 0.0

REC-OneSide-IS:

algorithm:
  algorithm_type: rec
  policy_loss_fn_args:
    epsilon_low: 0.2
    epsilon_high: 0.2
    clip_mode: "one-side"
    weight: "importance_sampling"
  advantage_fn_args:
    std_normalize: false
  kl_loss_fn_args:
    kl_coef: 0.0

REC-GSPO-NoIS:

algorithm:
  algorithm_type: rec
  policy_loss_fn_args:
    epsilon_low: 3e-4
    epsilon_high: 4e-4
    clip_mode: "gspo-one-side"
    weight: "none"
  advantage_fn_args:
    std_normalize: false
  kl_loss_fn_args:
    kl_coef: 0.0

REC-GSPO-IS:

algorithm:
  algorithm_type: rec
  policy_loss_fn_args:
    epsilon_low: 3e-4
    epsilon_high: 4e-4
    clip_mode: "gspo-one-side"
    weight: "gspo_importance_sampling"
  advantage_fn_args:
    std_normalize: false
  kl_loss_fn_args:
    kl_coef: 0.0

REC-TwoSide-IS:

algorithm:
  algorithm_type: rec
  policy_loss_fn_args:
    epsilon_low: 0.2
    epsilon_high: 0.2
    clip_mode: "two-side"
    weight: "importance_sampling"
    temp: 1.0
    regularizer: "none"
    regularizer_coef: 0.0
  advantage_fn_args:
    std_normalize: false
  kl_loss_fn_args:
    kl_coef: 0.0

REC-Ring-NoIS:

algorithm:
  algorithm_type: rec
  policy_loss_fn_args:
    epsilon_low: 0.2
    epsilon_high: 0.2
    epsilon_low_prime: 0.6
    epsilon_high_prime: 2.0
    clip_mode: "ring"
    weight: "none"
    temp: 1.0
    regularizer: "none"
    regularizer_coef: 0.0
  advantage_fn_args:
    std_normalize: false
  kl_loss_fn_args:
    kl_coef: 0.0

REP family

Meta's AsymRE:

algorithm:
  algorithm_type: rec
  policy_loss_fn_args:
    clip_mode: "none"
    weight: "none"
    temp: 1.0
    regularizer: "forward-kl"
    regularizer_coef: 0.1
  advantage_fn_args:
    std_normalize: false
  kl_loss_fn_args:
    kl_coef: 0.0

Kimi's OPMD:

algorithm:
  algorithm_type: rec
  policy_loss_fn_args:
    clip_mode: "none"
    weight: "none"
    regularizer: "k2"
    regularizer_coef: 0.1
  advantage_fn_args:
    std_normalize: false
  kl_loss_fn_args:
    kl_coef: 0.0

RED family

RED-Drop:

algorithm:
  algorithm_type: rec
  policy_loss_fn_args:
    clip_mode: "none"
    weight: "none"
    regularizer: "none"
  advantage_fn_args:
    std_normalize: false
    drop: "balance"
  kl_loss_fn_args:
    kl_coef: 0.0

RED-Weight:

algorithm:
  algorithm_type: rec
  policy_loss_fn_args:
    clip_mode: "none"
    weight: "advantage"
    regularizer: "none"
    temp: 1.0
  advantage_fn_args:
    std_normalize: false
  kl_loss_fn_args:
    kl_coef: 0.0

Citation

@misc{yao2025grouprelativereinforcesecretlyoffpolicy,
      title={Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends},
      author={Chaorui Yao and Yanxi Chen and Yuchang Sun and Yushuo Chen and Wenhao Zhang and Xuchen Pan and Yaliang Li and Bolin Ding},
      year={2025},
      eprint={2509.24203},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2509.24203},
}