Reward gating is killing your gradients — here's how to fix it

tl;dr

• Don’t gate your rewards — decay them.
• Adaptive reward weighting can turn sparse signals into denser ones.
• GDPO + ARW is a stronger pairing for multi-reward RL than GRPO + gating.

---

Multi-reward RL, when done naively, often leads to undesirable outcomes — models optimize for whichever reward is easiest to exploit. For example, training a model to be both correct and concise often collapses to “be concise” because conciseness is easier to satisfy than correctness, and the model converges to a degenerate solution: short, wrong answers.

The standard practice for fixing this is reward gating: auxiliary rewards are gated behind one or more primary rewards. The tradeoff is significantly less reward hacking, at the cost of gradient signal — the auxiliary rewards only fire when the primary reward triggers, which sparsifies the learning signal.

Motivation

A while back I started playing around with Prime Intellect’s Verifiers repository (their nanoGPT-like repo for RL). As I was launching my first training runs, I got curious about auxiliary rewards — rewarding the model for more than just getting the answer right. That’s when I learned how necessary robust reward gating is for preventing models from settling into degenerate solutions that just clear auxiliary thresholds. Around the same time, I stumbled across a very well-timed paper from NVIDIA introducing GDPO — Group reward-Decoupled Normalization Policy Optimization.

The key issue GDPO points at is that GRPO is insufficient for multi-reward RL because of how it confounds reward signals during advantage computation.

Take two rollouts, A and B, each with two reward components $[r_1, r_2]$:

# --- CASE 1 ---
rollout_A = [0, 0]
rollout_B = [1, 1]

# Sum r1 and r2
A_sum = 0
B_sum = 2

# Normalize
mean = 1
std  = 1
A_advantage = (0 - 1) / 1  # = -1
B_advantage = (2 - 1) / 1  # = +1

# --- CASE 2 ---
rollout_A = [0, 0]
rollout_B = [1, 0]

# Sum r1 and r2
A_sum = 0
B_sum = 1

# Normalize
mean = 0.5
std  = 0.5
A_adv = (0 - 0.5) / 0.5  # = -1
B_adv = (1 - 0.5) / 0.5  # = +1

Even with a totally different reward distribution, the advantages come out identical under GRPO. By aggregating rewards before normalizing, we wash out each individual component’s signal. Different reward decompositions can collapse to the same normalized summed advantage, so the policy gradient has no information about which objective improved — only that the aggregate did.

GDPO offers a clean fix. Where GRPO aggregates and then normalizes, GDPO normalizes and then aggregates.

This decouples each reward’s contribution to the advantage. The same example, recomputed under GDPO:

# --- CASE 1 ---
rollout_A = [0, 0]
rollout_B = [1, 1]

# Normalize r1 separately
r1_vals = [0, 1]
r1_mean, r1_std = 0.5, 0.5
A_r1_norm = -1
B_r1_norm = +1

# Normalize r2 separately
r2_vals = [0, 1]
r2_mean, r2_std = 0.5, 0.5
A_r2_norm = -1
B_r2_norm = +1

# Sum the normalized values
A_adv = -1 + -1  # = -2
B_adv = +1 + +1  # = +2

# --- CASE 2 ---
rollout_A = [0, 0]
rollout_B = [1, 0]

# Normalize r1 separately
r1_vals = [0, 1]
r1_mean, r1_std = 0.5, 0.5
A_r1_norm = -1
B_r1_norm = +1

# Normalize r2 separately
r2_vals = [0, 0]
r2_mean, r2_std = 0, 0
A_r2_norm = 0
B_r2_norm = 0

# Sum the normalized values
A_adv = -1 + 0  # = -1
B_adv = +1 + 0  # = +1

Now the two cases yield clearly different signals. The model gets fine-grained feedback about which reward shifted. Simple fix, powerful result.

The paper notes that reward gating is still recommended on top of GDPO, to prevent reward hacking. As I implemented GDPO in my fork of Prime-RL, that recommendation kept bothering me — it feels counterintuitive to completely shut down the gradient on auxiliary rewards until a primary one fires. I tried building a hierarchical rubric that let some gradient flow through regardless, but it didn’t resolve the issue: it still silenced large chunks of the signal. The fundamental tradeoff of reward gating is preventing degenerate behavior at the cost of zero gradient signal unless the threshold is met.

Note. Reward gating is also commonly used to route between reward functions depending on which outcome was reached — partial rewards in environments like Wordle, for example. My qualms are with reward gating as a reward-hacking band-aid, not as a process-supervision router.

Adaptive Reward Weighting

If I’m training for conciseness + correctness, and a model picks up conciseness immediately, why not reward it for that — and gradually shift its focus toward the rewards that are still struggling? Tell it: “hey, you’ve got that one locked in, let’s focus on the harder ones.”

That’s the core insight of Adaptive Reward Weighting (ARW): if a reward is saturated (the model consistently achieves it), decay its weight toward a minimum floor.

This gives us the best of both worlds. Early in training, every reward contributes signal. As training progresses, saturated rewards decay in weight and harder rewards take over. Throughout training, every reward contributes some signal — just with shifting emphasis.

# Track an EMA of each reward's mean across batches
ema[r] = alpha * batch_mean[r] + (1 - alpha) * ema[r]

# Map EMA to weight via decay curve. Assumes rewards are in [0, 1].
normalized   = clamp(ema[r] / saturation_threshold, 0, 1)
decay_factor = 1 - normalized ** decay_exponent

# decay_exponent controls how aggressive the decay is
weight[r] = base_weight[r] * max(decay_factor, min_weight[r])

Crucially we enforce a minimum weight floor (e.g. 0.1) so reward signals never disappear entirely. Otherwise the model can saturate a reward, watch its weight go to zero, and then forget how to satisfy it.

A naive implementation of ARW is prone to reward oscillation. Imagine a model optimizes for reward A early, the EMA brings A’s weight to the floor, and the model starts optimizing other rewards. As it focuses elsewhere, it may begin “forgetting” A (e.g. responses get longer again). A’s reward drops, its weight spikes back up, the model re-optimizes A, and we cycle.

To prevent this we use a ratcheting mechanism with a slow leak — an asymmetric response that creates hysteresis:

# Apply hysteresis per reward
if weight[r] < ratchet_floor[r]:
    # decay immediately
    ratchet_floor[r] = weight[r]
else:
    # recover slowly
    ratchet_floor[r] += recovery_rate * (weight[r] - ratchet_floor[r])

weight[r] = ratchet_floor[r]

# base_weight[r]:          starting weight for each reward (e.g. [1.0, 1.0])
# min_weight[r]:           floor per reward as a fraction of base weight
# saturation_threshold:    EMA value at which decay kicks in hard
# ratchet_floor[r]:        tracked per reward, prevents oscillation

Decay is allowed to happen immediately on saturation, but when a reward starts dropping again we bring its weight back up slowly. No thrashing.

Results

Experiments were run on top of the Prime-RL repository, using Llama-3.2-1B-Instruct and the GSM8K train split, for 500 steps. The reward setup: a binary correctness reward (verifiable answer match) plus a binary length reward (response under 384 tokens). For ARW runs, decay was enabled on all rewards, with the following parameters:

ema_alpha            = 0.1
saturation_threshold = 0.95
decay_exponent       = 2.0
min_weights          = [0.1, 0.1]
recovery_rate        = 0.01

Four variants were compared: GRPO + gating, GRPO + ARW, GDPO + gating, GDPO + ARW.

In both GRPO and GDPO, ARW yields a higher training reward than the gating baseline. Base GRPO + gating ends at 0.806; GRPO + ARW reaches 0.818; GDPO + ARW reaches 0.863. All four variants comfortably satisfy the length reward, but under gating most of that conciseness signal is silenced — only ARW lets it keep contributing once it’s saturated.

Evaluation was done on both ARW checkpoints using Prime Intellect’s math-python environment against openai/gsm8k (eval split):

GDPO + ARW (61% pass@1) beats GRPO + ARW (57%) — and uses over 5× fewer tokens on average (201 vs 1140). Combining ARW’s signal density with GDPO’s decoupled advantages lets the training process tune itself to the model’s actual learning needs at any given moment.

Another observation worth flagging: ARW seems noticeably more stable to reward collapse than gating. Confirming that properly will need more ablations than I had compute for.

Conclusion

Reward gating is useful for preventing reward hacking, and remains very powerful as a reward router. But the lack of gradient signal it introduces is genuinely counterintuitive for multi-objective optimization. Why silence the signal completely when you can just turn down the noise?

Adaptive reward weighting is a simple alternative: decay saturated rewards instead of gating them. The model still learns from every objective throughout training — just with shifting emphasis.

ARW isn’t a special contribution. If anything, the fact that such a small change buys this much extra signal density is a sign of how much we still have left to figure out about multi-reward RL.

— sp (@blackplasma22)