Start from outcomes
Use outcome-level behavioral signals, such as task correctness, rather than relying only on surface text patterns.
The question
A correct answer is not an explanation.
Which neurons or sparse features contribute to successful multi-step reasoning—and how do their contributions accumulate across generation?
Trace across steps
Weight token log-probability gradients with outcome-derived advantages to attribute contributions across the generated trajectory.
Identify and intervene
Integrate along activation paths, aggregate across questions, and modulate the selected internal components at inference time.
An outcome you can see
Same question. Different reasoning.
Carlos invests $90 in a lemon tree. Each year it produces 7 lemons at $1.50 each, with $3 in upkeep. When does he start earning a positive profit?
$90 initial cost$10.50 yearly revenue$3 yearly upkeep$7.50 yearly net income
Enhanced reasoning
7.50n − 90 > 0
n > 12 → the first whole-number year is 13.
The strict positive-profit constraint is respected.
Model answer13years
Actual profit at year 13: +$7.50
Selected excerpts from the provided case study. Suppression double-counts upkeep; the original response uses a non-strict constraint; enhancement changes the intermediate reasoning and returns 13 years.
Abstract
From final outcomes to internal representations.
Large Language Models (LLMs) demonstrate remarkable reasoning capabilities, yet the internal mechanisms driving these multi-step processes remain opaque. Existing mechanistic interpretability approaches often rely on superficial text-pattern co-occurrences or capture only short-term effects, struggling to trace the long-horizon, outcome-driven influence of internal components such as neurons or sparse features in multi-step reasoning. We introduce Integrated Policy Gradient (IPG), a training-free framework that brings policy-based attribution into mechanistic interpretability for LLM reasoning. IPG uses outcome-derived advantages to attribute behavioral signals across inference trajectories and integrates gradients along baseline-to-activation paths. Aggregated attributions identify outcome-relevant components for interpretation and intervention. Empirical evaluations on representative open-source LLMs demonstrate localization of sparse subsets of components, complementary spatial, semantic, and temporal insights, and improvements in reasoning performance through inference-time modulation.
Methodology
Trace. Integrate. Aggregate.
One frozen model. Outcome signals at the end. Component-level attribution across the trajectory.
QuestionReadPlanCalculateCheckOutcome
← Outcome-weighted credit across generation steps
Internal neurons
or SAE features
or SAE features
Baseline → activation
path integration
path integration
Aggregate over questions
rank components
rank components
Sequential attribution
Outcome-based credit
Outcome-derived advantages weight token log-probability gradients. Parameters remain frozen: this is attribution, not model training.
Integrated attribution
Baseline-to-activation paths
Accumulate gradients as component activations move from a baseline to their original values, then combine contributions across generation steps.
Inference-time control
Selected-component scaling
Aggregate IPG scores across questions and select top-ranked components. Scale their activations by γ to suppress or enhance their influence.
Technical interpretation
IPG is an outcome-oriented, sequential attribution framework motivated by a linear approximation of mediation effects. Its gradient-based scores are evaluated through component interventions and alignment with a group-based zero-ablation mediation proxy; they are not exact counterfactual mediation estimates.
Experimental evidence
Training-free interventions improve reasoning performance.
Accuracy (%) on Qwen2.5-Math-1.5B-Instruct under greedy decoding. Values after ± are half-widths of 95% confidence intervals, in percentage points.
| Method | GSM8K | MATH-500 | AIME 2024 | GPQA-D |
|---|---|---|---|---|
| Original | 82.41 ± 0.65 | 63.00 ± 0.95 | 10.00 ± 3.17 | 24.75 ± 1.92 |
| IPG | 84.61 ± 0.40 | 65.80 ± 0.88 | 20.00 ± 3.17 | 33.33 ± 1.68 |
| IPG-SAE | 84.76 ± 0.50 | 66.00 ± 0.76 | 20.00 ± 4.75 | 35.35 ± 1.20 |
Selected methods from the manuscript. See the paper for the full baseline comparison, Llama3.1-8B results, and attribution-baseline experiments.
Ablation · Table 1
Intervention tests the selected components.
63.028.8
MATH-500 accuracy drops when IPG-selected features are zero-ablated. Random ablation yields 59.8%; the unmodified model yields 63.0%.
Mediation proxy · Table 3
Alignment with an intervention-based reference.
0.305± 0.113
Mean Overlap@k ± standard deviation over five random seeds on Qwen. IPG reaches 0.246 ± 0.030 on DeepSeek-R1-Distill-Qwen-1.5B.
Where
Layer-wise localization shows where outcome-relevant components concentrate.
What
High-activation contexts reveal associations with planning, calculation, and summarization.
When
Activation dynamics provide a temporal view of component activity during generation.
A closer look
From an example to experimental evidence.
A second visual introduction combines the case study, component attribution, and intervention results.
48 seconds · Download video
Citation
If you find IPG useful, please cite our paper.
BibTeX
@inproceedings{li2026ipg,
title = {From Outcome to Representation: Tracing Reasoning
Mechanisms through Integrated Policy Gradient},
author = {Li, Changming and Zhang, Kaixing and Shi, Yingdong
and Xu, Haoyun and Zhang, Zheng and Song, Kaitao
and Ren, Kan},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026}
}