This entry posts the active full outline from info-geo/full_paper_outline.md as part of the running work trail.

Source file: {{ page.source_path }}

Paper Outline v2: Proxy Gap, Transport Geometry, and Falsifiable Mechanism Tests

Target Venues

  1. ICML 2026
  2. NeurIPS 2026
  3. ICLR 2027

Working Title Options

  1. When Reconstruction Looks Solved but Behavior Disagrees
  2. Proxy Failure in SAE Evaluation Across Scale
  3. Transport-Informed Diagnostics for Sparse Autoencoders

Submission-Safe Abstract (Template)

Sparse autoencoders (SAEs) are commonly evaluated with reconstruction metrics such as $R^2$, yet behavior-preservation metrics can disagree sharply in practice. We document a robust disagreement regime in mid-layer residual streams where model scaling improves reconstruction while worsening cross-entropy recovery at fixed sparsity budget. We propose a falsifiable mechanism class: Euclidean reconstruction error underweights behavior-sensitive directions induced by downstream geometry. Motivated by recent work on entropic-transport interpretations of attention, information geometry of softmax, and task-relevant dimensionality, we test whether sensitivity-/geometry-aware diagnostics outperform standard reconstruction proxies after controlling for optimization budget. We provide a pre-registered experimental ladder that separates optimization artifacts from intrinsic residual effects, and report which claims are supported, contradicted, or unresolved.

Claim Ladder (Core)

C1 (Observed, current evidence)

Proxy mismatch exists in budgeted low-$k$ mid-layer settings:

  1. $\Delta R^2 > 0$ while $\Delta CE_{rec} < 0$ across scale in key cells.

C2 (Must test next)

Optimization-vs-intrinsic discrimination:

  1. Does low-$k$ CE gap vanish with higher SAE token budget?

C3 (Mechanism test)

Geometry-aware proxies (SWD / pullback approximations) outperform $R^2$ in predicting behavior.

C4 (Optional extension)

Task-relevant dimensional mismatch contributes to where proxy failure is strongest.

Section-by-Section Outline (v2)

1. Introduction (1.5-2 pages)

  1. Why SAE evaluation quality matters.
  2. Empirical anomaly at scale.
  3. Three-gap decomposition: - metric-space gap, - optimization gap, - dimensionality gap.
  4. Contributions: empirical regime map + falsification framework.

2. Experimental Setup and Data (1-1.5 pages)

  1. Models/layers/SAE configs.
  2. Metrics definitions (R^2, CE_rec, rate, usage proxies).
  3. Current artifact provenance and caveats (single-seed, 10M budget).

3. Theory as Hypothesis Generator (1.5-2 pages)

  1. Attention-EOT interpretation (bounded claim).
  2. Output-space information geometry and pullback concept.
  3. MI task-relevant dimensionality framing.
  4. Clear non-claims.

4. Discriminating Experiments (1.5-2 pages)

  1. Exp-A: low-$k$ token sweep for H0 vs H1.
  2. Exp-B: SWD vs reconstruction proxies.
  3. Exp-C: anisotropy controls.
  4. Exp-D/E optional second wave.

5. Results (2-3 pages)

Structure this section as:

  1. Observed now (proxy mismatch).
  2. After new runs (H0/H1 decision).
  3. Mechanistic lift (proxy leaderboard with uncertainty).

6. Discussion (1 page)

  1. What is now supported.
  2. What remains unresolved.
  3. Practical implications for SAE evaluation/steering.

7. Reproducibility and Limits (0.5-1 page)

  1. Seed vs bootstrap uncertainty.
  2. Compute constraints and confounds.
  3. Open-source artifact checklist.

Pre-Registered Decision Rules

D1: Optimization gate

Fit: $$ \Delta CE(T,k)=\Delta_\infty(k)+A_kT^{-\beta_k}. $$

Interpretation:

  1. $\Delta_\infty(k=8)$ near 0 with tight CI -> optimization-dominant.
  2. $\Delta_\infty(k=8)$ negative with CI excluding 0 -> residual intrinsic component.

D2: Mechanism gate

Support for geometry-aware mechanism requires:

  1. SWD/pullback proxy out-of-sample predictive lift over $R^2$.
  2. Lift stable under resampling/seed variation.

D3: Dimensionality gate (optional)

Support requires:

  1. MI-derived dimensional indicators explain additional CE variance after controlling for basic capacity proxies.

Statistical Plan (Minimum)

  1. Distinguish eval bootstrap uncertainty from training-seed uncertainty.
  2. Mark single-seed findings as descriptive.
  3. Use held-out cell prediction for proxy comparisons.
  4. Avoid fixed rho-threshold claims without uncertainty context.

Figure Plan (v2)

  1. Fig 1: Core mismatch panel (R^2 vs CE_rec by scale at fixed $k$).
  2. Fig 2: Token budget vs Delta CE (low-$k$ asymptote fit).
  3. Fig 3: Proxy leaderboard (R^2, cosine, SWD, optional pullback metric).
  4. Fig 4: Layer anisotropy summaries vs mismatch magnitude.
  5. Fig 5 (optional): MI dimension indicators vs proxy-failure regimes.

Coauthor Work Breakdown (Execution-Ready)

Workstream A: Runs

  1. Launch low-$k$ token sweep (70M/410M, mid, $k=8,16$, 10M/50M/100M).
  2. Add 2-3 seeds on anchor cells.

Workstream B: Analysis

  1. Implement SWD extraction.
  2. Build standardized table generator with uncertainty fields.

Workstream C: Writing

  1. Maintain two claim tables: - supported now, - contingent on pending experiments.
  2. Keep abstract/conclusion conditional until D1 and D2 are settled.

Timeline (ASAP realistic)

  1. Week 1: Exp-A minimal + Exp-B first pass + draft figures 1-3.
  2. Week 2: seed repeats + anisotropy controls + claims update.
  3. Week 3: optional pullback/MI second wave if D1 supports intrinsic residual.
  4. Week 4: full manuscript freeze for internal review.

“Do Not Overclaim” Checklist (Pre-Submission)

  1. No “intrinsic geometry proven” language without D1 support.
  2. No theorem labels for unproved approximations in this setting.
  3. No direct “replace PR with MI k*” claims.
  4. Every major mechanism claim tied to one explicit falsification result.

Target Length

  1. Main paper: 9-10 pages (conference format).
  2. Appendix: 6-10 pages (methods, extra tables, ablations).