Stats Check

Core Rule

Every statistical claim must report effect size + uncertainty (a CI), N, the named test, and whether its assumptions hold. "Significant" means statistically significant — never "large" or "important". Association is not causation: correlational/observational designs license "associated with / predicts", never "causes / drives / improves". And no p-hacking / HARKing / selective reporting — the analysis you report is the analysis you pre-specified, with exploratory results labeled exploratory. A bare p-value, a missing N, a causal verb on an ablation, or a number that disagrees between the abstract and Table 2 are the failures this skill exists to catch.

This is the deterministic, numbers-and-design half of verification, operationalizing agent_docs/statistics.md. It flags and proposes fixes — it does not change a reported quantity (a Protected Claim per CLAUDE.md); recomputing a number to reconcile text and table is fine, altering what was measured or which test was run is not.

When to Use

Invoke with /stats-check when:

• A Results section with statistics is "done" and you want it Reviewer-2-proof.
• A revision added or changed a number, test, or comparison.
• Before submission, on Results + the abstract + every table — the consistency check needs all three open at once.
• The prompt-router flagged the task [Statistics] or [Causation].

Scope it: /stats-check sections/results.tex. Run the whole Results section with its tables and the abstract in view — half the findings are cross-location number mismatches.

Process

Phase 1: Load the Checklist and the Numbers

1. Read agent_docs/statistics.md — the reporting checklist this skill runs. It is the source of truth; this skill is its executor.
2. Read the target Results .tex, plus every table/figure it summarizes and the abstract. Numbers must agree across all three (statistics.md → text ↔ tables ↔ abstract); you cannot check consistency with only one open.
3. Read MANUSCRIPT_MAP.md — the Data & reproducibility line (was the design an RCT, an ablation, an observational comparison? — this decides what causal language is licensed) and Claims that need extra care.
4. For ML/agents work, read agent_docs/field/ai-ml.md — variance over seeds/rollouts, matched-budget comparison, and partial-vs-full success are field-specific statistics expectations.

If a number's test or assumptions are not stated and you cannot tell which was run, that is a finding — do not infer the test. Which test was run is a Protected Claim; ask the author.

Phase 2: Find Every Statistical Claim

Walk the section and extract each quantitative comparison or inferential statistic: every p-value, CI, effect size, mean difference, correlation, regression coefficient, percentage, ratio, and "significant"/"more"/"better" comparison. Record for each: the point estimate, the uncertainty (CI/SE/SD, or none), the N, the named test (or none), the verb (causes / is associated with / predicts), and the scope (which harness, which task set, all vs subset).

Phase 3: Check Each Claim Against the Checklist

Run every claim through agent_docs/statistics.md:

1. Effect size + uncertainty, not just p. A bare "p < 0.05" with no magnitude and no CI is incomplete — the reportable triplet is point estimate · interval · test (with N).
2. "Significant" used correctly. Flag "significant" deployed as a synonym for large / important / meaningful. Magnitude needs a number, stated separately.
3. N, test, assumptions stated. N for that comparison (not the study total); the test named; assumptions addressed (normality/variance for a t-test; linearity for Pearson r) and what was done if violated.
4. Multiple comparisons. A family of tests needs a correction (Bonferroni/Holm/BH-FDR) or a justification; all tests run must be disclosed, not just the winners. A subgroup "significant" after many slices is a hypothesis, not a finding.
5. Causation vs association. Check the verb against the design (Phase 1). A causal verb ("causes / drives / improves / the effect of") on a correlational comparison or an uncontrolled ablation is OVERCLAIM. RCT licenses causation within its population; a natural experiment does conditionally with the identifying assumption stated.
6. p-hacking / HARKing signals. Outcomes in Methods that vanish in Results (or appear in Results unannounced); p-values clustered just under 0.05; "trending toward significance"; post-hoc results framed as a-priori hypotheses; undisclosed exclusions or optional stopping.
7. Significant digits & units. False precision ("94.732%" off a ±1% estimate); the last digit should be the first uncertain one; units present and consistent; percentage points ≠ percent (a 70→88 rise is 18 pp, ~26% relative).
8. Cross-location consistency. Recompute: every number matches across text, tables, figures, and abstract; N is the same everywhere; totals add up; derived numbers (a difference, ratio, percent change) recompute from the reported inputs.

Phase 4: Report Findings and Fixes

Produce the findings table (claim → issue → fix), severity-ordered. The fix says what the author must do. Do not edit reported quantities here — flag the mismatch, propose the calibration. Reconciling a text number to a table (a clerical correction) can be proposed; changing what was measured or which test was run is a Protected Claim needing sign-off.

Output Format

# Stats Check — sections/results.tex

## Summary
- Statistical claims examined: 12
- Clean: 6
- Issues: 6  (2 causal overclaim · 1 bare-p · 1 missing-N · 1 mult-comparison · 1 cross-location mismatch)

## Findings (claim → issue → fix)
| # | Locator | Claim (quoted) | Issue | Fix (author action) |
|---|---|---|---|---|
| 1 | res ¶2 | "the gate caused an 18-pp gain in tool-call accuracy" | Causal verb on an uncontrolled ablation (MANUSCRIPT_MAP: not an RCT) | Soften to "was associated with an 18-pp gain"; or justify a causal design |
| 2 | res ¶3 | "tool-call accuracy was significantly higher (p = 0.03)" | Bare p; no effect size, no CI | Add point estimate + 95% CI + named test + N: "X pp (95% CI a–b; two-sample t-test, n = …)" |
| 3 | res ¶3 | "the gate reduced hallucinated tool calls (p = 0.02)" | N not stated for this comparison | State N for the comparison |
| 4 | res ¶5 | "the gate helped on 3 of 8 task subsets" | 8 subgroup tests, no correction, winners only | Report all 8; apply BH-FDR or justify; label exploratory if post-hoc |
| 5 | res ¶2 vs Table 2 | "21% → 6%" in text; Table 2 shows "21% → 8%" | Cross-location number mismatch | Reconcile to the real value; fix everywhere at once (Protected — confirm which is correct) |
| 6 | res ¶4 | "a significant improvement" | "significant" as a synonym for large | Give the magnitude with a number; reserve "significant" for the test result |

## Causation flags (design vs verb)
- Design per MANUSCRIPT_MAP: ablation comparison (not randomized at the unit of inference).
- Licensed: "associated with", "predicts", "co-occurs with". NOT: "causes", "drives", "the effect of".
- Finding 1 violates this — highest priority.

## Cross-location consistency
- Hallucinated-call rate: text 6% vs Table 2 8% — MISMATCH (finding 5).
- N: text "512 tasks" vs Methods "512 held-out" — consistent.
- 18-pp gain recomputes from Table 2 (39% → 57%) — OK.

## Notes for the author
- Findings 1, 5 are Protected (causal claim / changed number) — need your sign-off, not a silent edit.
- Confirm which test produced each p-value; I did not infer any (that would change what was reported).

End with the tally (clean / issues) and a one-line worst-risk. Never report a Results section "clean" while a bare p-value, a causal overclaim, or a cross-location mismatch stands.

Pairs With

• agent_docs/statistics.md — the checklist this skill executes; read it first, it is the authority for every rule above.
• integrity-reviewer agent — escalate when the signals look like patterned selective-reporting/HARKing across the whole manuscript, not isolated reporting gaps; it scans breadth, /stats-check verifies each number.
• /claim-check — run alongside: claim-check verifies verbs against cited sources, stats-check verifies numbers against the data and the design. Together they cover claims + quantities.
• citation-gate.sh (PostToolUse) — orthogonal (it checks \cite resolution), but run it so the Results section is structurally clean before the numerical pass.

Common Rationalizations

Notes

• This skill never invents a number, a test, or an assumption. A missing value is [VALUE — verify]; an unknown test is a question for the author — per the cardinal rule in CLAUDE.md.
• Causation-vs-association and selective-reporting judgments are Reasoner-tier (CLAUDE.md → Model Selection); the cross-location number-matching is deterministic — recompute, do not estimate (CLAUDE.md → Model vs Code).
• A recurring statistics slip (the author keeps correcting "significant", or bare p-values) is a rule — log under tasks/reviews/, applies_to: [statistics], promote to ## Top Rules if it recurs (CLAUDE.md → Self-Improvement Loop).