Audit Statistical Reporting Integrity in Empirical Research Papers

Reviews reported statistical values in research papers to catch internal inconsistencies, test-statistic mismatches, and signs of p-hacking.

You getSummary Table, integrity Flags and verification Notes

Prompt

Prompt
Act as a quantitative peer reviewer and meta-science researcher. Your job is to audit the statistical reporting in an empirical results section. I will paste an excerpt containing inferential statistics (such as t-tests, ANOVA, regression coefficients, confidence intervals, or sample sizes).

Rules:
1. Check internal consistency between test statistics, reported degrees of freedom, sample sizes, and p-values (e.g., whether a stated t-statistic and df mathematically produce the reported p-value).
2. Flag any signs of selective reporting or p-hacking, such as p-values clustered suspiciously between 0.041 and 0.049, missing baseline degrees of freedom, or unexplained covariate switching.
3. Verify whether 95% confidence intervals align with reported standard errors and point estimates.
4. Do not invent missing data; explicitly label omissions as Unverifiable.

Output format:
- Summary Table: Columns for Reported Test, Stated Values, Recomputed Values, and Status (Consistent, Discrepant, or Missing Parameters).
- Integrity Flags: Bulleted findings detailing exact discrepancies or methodological red flags.
- Verification Notes: One concise note stating missing context or potential benign explanations (e.g., rounding errors, post-hoc corrections).

Please reply with 'Paste your results section text' to begin.

Example output

Example for: an excerpt from an experimental psychology study evaluating reaction times across two cohorts.

Summary Table
Reported TestStated ValuesRecomputed ValuesStatus
Independent t-test (Condition A vs B)t(58) = 2.14, p = 0.036p = 0.0365Consistent (standard rounding)
Pre-test baseline equivalencet(58) = 1.98, p = 0.048p = 0.0524Discrepant (non-significant recomputed)
Post-intervention ANOVAF(1, 56) = 3.91, p = 0.049p = 0.0529Discrepant (exceeds alpha threshold)
Cohort B Mean RT 95% CIMean = 420ms, 95% CI [395, 435], SE = 1095% CI [400.4, 439.6]Discrepant (asymmetrical interval)
Integrity Flags
  • Significance Cliff Edge: Both the baseline t-test and the primary ANOVA report p-values barely beneath the conventional threshold (p = 0.048 and p = 0.049). Recalculation indicates both values actually fall above the alpha = 0.05 cutoff (p = 0.052 and p = 0.053 respectively).
  • Degrees of Freedom Drift: Sample size was reported as N = 60, but the baseline t-test reflects df = 58 while the subsequent ANOVA lists df = (1, 56) without explaining the two missing observations or exclusion criteria.
  • Confidence Interval Arithmetic: The reported confidence interval for Cohort B ([395, 435]) is centered at 415ms, conflicting with the stated mean of 420ms and inconsistent with the reported SE of 10ms.
Verification Notes

The discrepancies on both p-values cross the nominal p < 0.05 threshold. Unless authors applied a one-tailed adjustment or specific variance corrections (such as Welch's t-test without reporting adjusted degrees of freedom), these results should be treated as non-significant.

Written by AI from this prompt. Your results will vary.

Tips and common edits

Why this works

  • Forces mathematical cross-checking between test statistics, degrees of freedom, and reported p-values.
  • Specifically targets p-hacking indicators like clustering right below the 0.05 threshold.
  • Separates verified calculations from missing parameters so reviewers avoid false accusations.

Paste an empirical results section directly after the initial prompt in ChatGPT, Claude, Gemini, or Perplexity. Works best with t-tests, regressions, and ANOVAs.

Common edits

  • Add a requirement to calculate Cohen's d or partial eta squared from the reported raw numbers.
  • Instruct the model to account for specific multiple-testing corrections like Bonferroni or FDR adjustments.

Make it repeatable

Keep using this prompt without pasting it in every time.

Keep exploring

More in AI Research