Rollout Entropy Dynamics

06 Jul, 2026

Introduction

What is being asked in these experiments is if some future information, in the form of a rollout being successful or unsuccessful, is "leaked" (in the form of entropy over time) somewhere early enough in a completion that by intervening mid-generation or otherwise you can induce likely-to-fail rollouts into becoming successful.

Background

The experiment was done using TRL's AsyncGRPO library because of its slim design, ease of setup, and great "hackability" (and it works well on single node setups). All the code can be found here. The dataset used was trl-lib/DeepMath-103K and model was Qwen/Qwen3-4B.

Initially, I was wondering about the token-entropy dynamics over a completion and how you might use that information to intervene on a rollout mid-completion to potentially "course-correct" a rollout so you wouldn't need to waste compute on a rollout you knew with some certainty would fail. For instance, a model being successful when k=16 but not k=8 implies the same model can solve the problem. The idea is that by intervening on a completion based on its entropy-dynamics could be one way to lower some model's solve-at-k threshold.

With a baseline group-solve rate (group size of 8) that sits fairly comfortably above 50% in the beginning of training, if you could induce a single completion to be successful, that's training signal you might otherwise not get. This thought led me to this paper, titled "EDIS: Diagnosing LLM Reasoning via Entropy Dynamics" and their conclusion lined up with my own intuition on what entropy looked like over a rollout. They say "Correct responses maintain stable, low entropy, while incorrect responses exhibit distinctive instability patterns: peak-valley spikes (entropy drops then rebounds) and burst spikes (progressive entropy rise)". They score completed trajectories post-hoc using hard-coded entropy thresholds, then use those scores to either select the most stable completions at inference time (best-of-N) or reweight GRPO advantages during training. Interestingly, this paper by Meta just came across my feed a couple weeks ago. Rather than intervening mid-generation, they use the model itself to identify the first poor reasoning step in a known failed trajectory then resample multiple suffixes from that point.

I like EDIS for its visualizations in showing that entropy-dynamics can almost be windowed over a span of tokens, but I didn't want to hardcode some threshold then need offline recomputation with every experiment I might run. I imagine the entropy dynamics across different models, tasks, and training-steps to be quite different.

Wanting a metric that speaks to local volatility and how it changes over tokens led me to wanting to use some sort of VIX-like measure. The measure is based off computing the top-k (say, k=3) entropy at each token position, then treating this entropy-sequence as a time series and applying rolling-window statistics borrowed from financial volatility analysis. We create our own little "VIX" measurement based on the RMS of token-to-token entropy changes, drift that captures directional trend, and a way to capture how far entropy has rebounded from its local minimum. (Shoutout Euan Sinclair and his great books on volatility).

For the first test, the idea is that during training, as (un)successful completions finish, we have a separate thread computing several VIX metrics over windows of completed rollouts, then based on a hand-crafted trigger rule (i.e. firing when vix exceeds a threshold) and our reward labels, we search for VIX threshold values that best separate failed from successful completions (i.e. maximize the failure trigger rate while constraining the success trigger rate) and pass those thresholds to the server, which applies the same trigger rule during generation to decide when to fork. This thread keeps some number of lookback (un)successful completions in their own fifo buffers, and re-tunes the thresholds as n new completions arrive.

Speaking more to this trigger rule. On the vLLM side, we create a new completions endpoint that our workers will point to. What's happening on the inference server itself is that instead of calling vllm_engine.generate with max_tokens set to the maximum number of output tokens that the entire completion is allowed to have, we set it to something like window_size * windows_per_completion. Then within this multi-window amount of tokens, use the current VIX-like-threshold values, transform the logprobs of the current chunk of output tokens, then search for occurrences where some window of tokens causes the trigger-rule to fire. If the rule did trigger, we then re-sample n=num_samples times from beginning of the window and select the partial-rollout with the lowest vix-score. To the worker this is all opaque; they just get back a completion and score the group as normal.

V1

Speaking to the results of the first experiment, the hand-crafted classifier, whose job it is to answer the question of "is this completion heading toward failure?" works weakly. We use Youden's J to capture how well the classifier separates our two groups.

The 3x selectivity ratio is to say that the classifier fires on failing completions three times more often than successful ones. It can use entropy patterns to predict failed completions, but only weakly.

Within our groups, every other completion is eligible for intervention. The success rates of each show that intervention produces effectively zero measurable improvement.

When the classifier fires and the system forks, 82% of the time the replacement branch has a lower VIX score than the original trajectory. The fork is successfully selecting "calmer" continuations. But calmer doesn't seem to imply "more likely to be correct". The intervened completions succeed only 24.9% of the time, versus 30.3% for non-intervened completions (−5.5%). This isn't necessarily evidence that forking hurts because there is selection bias. Completions that trigger the classifier are already likely to fail.

I think that forking 43% into the completion is notable. By that point the model has probably committed to a reasoning approach. So, perhaps resampling tokens near the entropy spike might change the tokens but wouldn't change the high-level strategy of the completion.

I think this is probably the core learning from V1. That detection and intervention are very separable problems. The classifier can solve detection well enough but the intervention mechanism (i.e. when to branch and how to rank) are what's broken. A VIX-like measure might be a good means by which you can make a prediction about a rollout's terminal reward, but it's at best unclear whether its appropriate to use it as a proxy for when to intervene.

V2

On the detection side, V2 replaces V1's hand-crafted VIX threshold rule with a 3-layer MLP while keeping the same fork-and-resample intervention method. The hypothesis being that better detection might translate into better interventions.

The classifier is trained online during the RL run where the dataset is past completions and their reward labels. Every m new completions, the MLP is retrained on a rolling buffer of successful and unsuccessful completion windows. Each window is a chunk of chunk_size tokens characterized by entropy-window features (the raw top-k entropy trajectory over the chunk) concatenated with VIX summary features (volatility, drift, etc). The classifier outputs a failure probability for each window, and the system searches a Pareto frontier of operating points (i.e. thresholds at different caps on the success trigger rate (10%, 15%, 20%, 25%)) to find the best separation. Once trained, the classifier parameters are pushed to the vLLM server via HTTP, where they're loaded into an identical MLP for inference-time scoring.

On the intervention side, whereas V1 forks at the beginning of whichever chunk triggered the rule, V2 finds the window with the highest classifier failure probability, then within that window computes per-token VIX values and forks just before the token with the highest VIX scores. This allows us to fork closer to the actual entropy spike. How the "best" branch is chosen amongst the num_samples branches generated is by looking at each potential branch's MLP score and choosing the lowest failure probability.

The MLP classifier works substantially better than V1's hand-crafted thresholds, which is no surprise. At the operating point used in the experiment (cap=0.10), the classifier achieves 4.5x selectivity and Youden's J of 0.31, compared to V1's 3.0x and 0.19.

In terms of intervention-effectiveness, cumulative_treatment_minus_control measures (cumulative_treatment_successes / cumulative_treatment_total) - (cumulative_control_successes / cumulative_control_total), which is to say, the differences in success rates between completions that take the intervention path (but may not necessarily get intervened upon) and those that don't take it.

What's interesting to me is that early in the training, there appears to be some semblance of signal that dissipates over the course of training. Putting sense to these graphs, early in training when zero_solve_group is high (60-75%), most groups have no successful completions at all. So in that regime, even a marginal intervention that flips a single completion from failure to success creates training signal. This coincides with cumulative_treatment_minus_control being highest (i.e. > 1.5%). But as training progresses and the model improves, zero_solve_group drops to around 40%, meaning the model can solve most problems on its own, making the intervention path redundant. The group already has successful completions for the optimizer to learn from, so flipping one failure to success doesn't change the training dynamics much, so you see cumulative lift decay toward zero.

Looking at different intervention improvements between V1 and V2:

The fork is clearly broken. The branch selection is worse than random (30% improvement rate, where 50% would be chance), hardly anyone gets forked (81% noop), and the ones that do get forked almost never succeed (3.3%). By every measure of the fork itself, the intervention is broken. And yet the cumulative metrics seem to tell a different story.

I think the resolution is that the fork-level metrics and the cumulative A/B test are actually telling the same story. The classifier fires on hard problems, resampling from the same point produces branches that face the same difficulty, and branch selection is worse than chance. The early cumulative lift (around 1.5%) coincides with very small sample counts and decays steadily as more data arrives, which is probably indicative of noise rather than a real effect.

The classifier was trained with completion-level labels applied to every window, and so it learned which entropy patterns are statistically more common in failed completions, not which continuations will causally succeed. This is coupled with intervention in V2 occurring at the point within the "most-failure-like" window at the point of maximum volatility. The core problem remains what it was after V1. When the classifier detects a failing trajectory and forks, it's asking the same model to resample from roughly the same point. The model already committed to a reasoning strategy before the entropy spike appeared. Resampling tokens at the spike changes the surface-level tokens but not the underlying approach. It seems better detection can tell you that the model is failing and when the failure signal appears, but it can't tell the model what to do instead.

V3

From V2, one conclusion as it relates to intervention is that the model has committed itself to a poor reasoning strategy before the entropy-classified window. Intervening by sampling num_samples from the same prefix doesn't appear to lead to more successful outcomes, especially not at the point of maximum volatility. So, taking ideas from educational psychology and off-policy learning literature, the next idea was to inject a hint (like "Wait, let me reconsider.") a window's worth of tokens before the classified failure window, then continue generation from there. We also trained the classifier on only the first third of each completion's entropy trajectory, reasoning that early failure signals (before the model has committed to a reasoning approach) are the ones worth learning.

The first V3 run, which injected the hint and continued with a single path (no branching), was actively destructive (cumulative_treatment_minus_control stabilized around -1.6%). A second run added back branching (num_samples=3) with VIX-based branch selection and restricted intervention to the first 300 tokens. The early-window classifier training did work as intended and intervened completions succeeded at 9.5% versus V2's 3.3%, but the cumulative lift was still negative at -1.3%.

The hint strings are semantically meaningful to a human, but they are very off-policy tokens. Unlike tool use or multi-turn chat (where models are trained to expect off-policy tokens at structural boundaries) mid-reasoning injection (at least on a 4b model) actively hurts training. Given the generality of the hint, the model doesn't know what to reconsider or why, and can't recover coherently.

V4

V1 through V3 all shared the same premise, that of wanting to detect then correct a failing completion mid-generation. V1 and V2 tried resampling from the failure point. V3 tried injecting a hint to redirect reasoning. None of that worked so well. By the time entropy signals appear, the model has already committed to a reasoning strategy, and no amount of local resampling or prompting can change it.

So V4 abandons correction entirely, opting instead to use the classifier for selection before generation begins. The mechanism is that for every treatment completion, generate num_samples=4 candidate prefixes of 256 tokens each (a single prefill, branched into four parallel decodes), score each prefix with the online classifier, select the one with the lowest maximum failure probability across its entropy windows, and then complete only that prefix to full length. Control completions generate normally. The classifier is the same 3-layer MLP architecture as V2, trained on the first 256 tokens of control-arm completions only, so it never sees the selection-biased treatment data.

V4 basically reframes the question from "can we rescue a failing completion" to "can we avoid starting one".

The classifier is weaker than V2's, which makes sense in that it's trained on truncated early windows (256 tokens) rather than full completions, giving it less signal to learn from.

At the operating point used (cap=0.20), Youden's J is 0.23 and selectivity is 2.2x, compared to V2's 0.37 and 2.9x at the same cap. Whereas V2 needed to make a binary threshold decision (i.e. "is this window failing badly enough to trigger a fork?"), V4 only needs to rank four candidates. Comparing V2 and V4, V2 outperforms V4, though they both seem to converge to the same value towards the end.

Notably, V4 does log a score_gap metric, which is the difference between the highest and lowest classifier scores among the 4 candidates for each prompt.

The classifier scores all four candidates within ~1.7 percentage points of each other. Meanwhile, scores vary by ~11 percentage points across different prompts. The within-prompt variation is just 15% of the cross-prompt variation.

The selected_first_rate of 0.231 (almost exactly 25%) confirms that the classifier is picking uniformly at random among the candidates. The classifier achieves 2-4x selectivity and real Youden's J scores because it does genuinely learn that certain entropy patterns correlate with failure. But what it's really learning seems closer to problem difficulty as opposed to completion quality. Hard problems produce volatile, high-entropy trajectories regardless of which of the four prefixes you look at and easy problems produce calm trajectories. So the classifier discriminates well between a hard prompt and an easy prompt, but when given four prefixes from the same prompt, they all look identical.

V2's data corroborates this from a different angle. V2's branch improvement rate of 0.30 (worse than coin-flip) shows that when the classifier is forced to rank completions within a prompt, it actually anti-selects. It picks worse branches more often than better ones. Raw entropy features do vary between completions of the same prompt (V2's treatment and control VIX means have near-zero correlation despite seeing the same prompts), but the classifier has learned that this within-prompt variation is noise. And when it leaks through into branch scoring, it's misleading rather than helpful.

This also explains why no version of intervention worked. V1 and V2 forked and resampled, but the new branches faced the same underlying difficulty. V3 injected hints, but the difficulty didn't change. V4 selected among candidates, but there was nothing meaningful to select between. V2's early cumulative lift (~1.5% when zero_solve_group was high) likely came from the extra resampling giving the model additional shots at a different reasoning path and not from the classifier making smart selections, which is why it decayed as the model improved and groups no longer needed the help.

V5

V4 really is trying to ask whether or not early entropy carries any intra-prompt signal, or is the classifier purely determining problem difficulty. The numbers are from a classifier trained using standard binary cross-entropy, which has every incentive to not favor the (clearly more subtle) within-prompt signal. So, V5 aims to train a classifier to distinguish completions within the same prompt.

V5 does everything the same as the baseline, but only adds a sort of "side-channel" observation. The training setup consists of a contrastive classifier using a Bradley-Terry loss on within-group pairs. For every group, where at least one completion succeeded and at least one failed, we pair entropy windows from the successful and unsuccessful completions at matching token positions (i.e. each pair sees the same prompt and same position in the completion, but one is from a successful and the other from an unsuccessful completion). The classifier finetunes every m new groups after an initial 32 classifiable groups have finished.

The results are pretty unambiguous as to the effect of entropy on completion result. The contrastive accuracy, which looks at how often the classifier correctly ranks score(failure) > score(success) starts around 51%, gets up to 53% before trending back down to 50% across the training run. The classifier's Bradley-Terry loss hovers a little below ln(2) = .693 for the entire run, which effectively says the scoring is random.

V5 answers the cleanest version of the question we could ask. Given two entropy trajectories from the same prompt, at the same token position, one from a completion that will succeed and one from a completion that will fail, can you tell which is which. The answer (at least in this setup) is no. Perhaps a bigger classifier, trained longer, etc might alter that answer.

V6

V5 showed that the Bradley-Terry classifier cannot distinguish completions within a prompt, but the original classifier is genuinely good at estimating problem difficulty. So in V6, rather than intervening during generation, we use the classifier to modulate the group-level GRPO advantages after scoring. Specifically, we compute a group-level difficulty weight, derived from the scores of each completion in the group, and multiply all advantages by that weight. In this way, harder prompts push more gradient through the loss. V6 is basically asking whether we should learn more aggressively from harder problems. The result is a reward curve that's indistinguishable from the baseline.

Conclusion

The entropy signal is real. Across all six versions, classifiers trained on windowed entropy features can reliably tell you whether a completion is more likely to fail. But the signal is closer to recognizing hard problems, not bad completions. What I find interesting is how consistent the negative result is across very different mechanisms. Fork and resample, inject hints, select prefixes, train contrastively, reweight advantages, etc, all grounded in the same observation that entropy trajectories look different for successes and failures, and none of them produced measurable improvement. Entropy tells you the problem is hard, and hard problems are hard for every completion the model generates.

If there is a usable within-prompt signal in entropy, it's below the noise floor of what a small MLP can extract from 256-token windows during online RL. A larger model, longer windows, or a fundamentally different featurization might change that, but the simpler interpretation is that entropy is the manifestation of a model solving problems that are currently hard for it.

Follow up

In wondering about follow-up experiments, given these results, I thought it might be interesting to put these entropy-dynamics to use in creating a sort of verifier-free curriculum learning guide. That thought led me to this paper, which shows that ordering data from low to high entropy is a strong lever for improving model performance. Similar to our classifier, their entropy metric measures problem difficulty. The difference is they use it to decide when to train on a problem, not whether to intervene on a completion.