Red-Team a Research Proposal for Confounders and Failure Modes

Get a severity-ranked validity audit table that exposes confounding variables, sampling biases, and failure modes in your research proposal with concrete fixes.

You getExecutive Verdict, validity Audit Table and top 2 Dealbreakers

Prompt

Prompt
Act as a senior research methodologist and adversarial peer reviewer. I will paste my research proposal or study protocol right after this prompt. Your job is to red-team the design for confounding variables, validity threats, and operational failure modes before data collection begins.

Follow these rules:
- Identify at least 6 distinct threats covering both internal validity (confounders, selection bias, measurement drift) and external validity (generalizability, ecological limits).
- Rank each threat as Critical, High, or Medium based on its likelihood of invalidating the primary hypothesis.
- Pair every threat with one concrete fix: a design control (blocking, blinding, placebo) or a specific statistical sensitivity check.
- Reference the exact part of my proposal that creates each flaw. Do not invent citations or statistics.

Output format:
1. Executive Verdict: Two sentences stating whether the study is ready to run or needs major revision.
2. Validity Audit Table: Markdown table with columns: Threat, Validity Type, Severity, Proposal Trigger, and Concrete Fix.
3. Top 2 Dealbreakers: One short paragraph naming the two flaws to fix first and why.

End with one line listing missing protocol details, such as power calculations or attrition plans, that limit this audit.

Example output

At a glanceValidity threats by severity count6 threats found: 1 Critical, 2 High, 3 Medium
  • Critical: 1
  • High: 2
  • Medium: 3

Example for: a 12-week quasi-experimental study testing whether a 4-day workweek improves code review quality across two engineering offices (Austin and Berlin) at a 300-person software firm.

Executive Verdict

This study requires major revision before launch. Comparing two different offices without randomization conflates the 4-day workweek with site-specific culture, management, and timezone differences, making causal claims about code quality indefensible.

Validity Audit Table
ThreatValidity TypeSeverityProposal TriggerConcrete Fix
Site-level confoundingInternalCriticalAssigning Austin to treatment and Berlin to controlUse a stepped-wedge cluster randomized design across teams within both offices.
Hawthorne effectInternalHighAnnouncing code review quality as the primary KPITrack baseline PR defect rates for 8 weeks prior and include unobtrusive production incident metrics.
Measurement driftInternalHighRelying on senior engineers to subjectively score PRsBlind reviewers to the author's schedule and run an inter-rater reliability check (Krippendorff's alpha).
Differential attritionInternalMediumAllowing engineers to opt out of the 4-day scheduleConduct an intention-to-treat analysis and run a Lee bounds sensitivity check for dropouts.
Novelty and fatigueExternalMediumRunning the trial for only 12 weeksExtend observation to 24 weeks or segment results into weeks 1-6 versus weeks 7-12 to test decay.
Firm-size limitsExternalMediumSampling only one 300-person SaaS engineering orgRestrict claims to mid-sized product teams and report team-level workload variance.
Top 2 Dealbreakers

Fix site-level confounding and measurement drift first. If Austin and Berlin differ in legacy codebase complexity or manager strictness, no statistical adjustment can separate office effects from the 4-day schedule. Similarly, unblinded subjective PR scoring invites confirmation bias from managers who want the pilot to succeed.

Missing protocol details: The proposal lacks a statistical power calculation for team-level clustering, a definition of "code review quality," and a plan for handling public holidays that differ between Texas and Germany.

Written by AI from this prompt. Your results will vary.

Tips and common edits

Why this works

  • Forcing both internal and external validity threats prevents the model from only flagging obvious sample-size issues.
  • Pairing every flaw with a concrete control or sensitivity check turns vague criticism into an actionable revision plan.
  • Requiring exact references to the pasted proposal grounds the audit in your actual design rather than generic textbook advice.

Paste your research proposal, study design, or grant abstract right after this prompt in Claude, Gemini, ChatGPT, or Perplexity before peer review or IRB submission.

Common edits

  • Add your target journal or funding body (such as NIH or NeurIPS) to tailor the severity ranking to their review criteria.
  • Ask the model to include a column estimating the cost or timeline impact of each proposed fix if your budget is locked.

Make it repeatable

Keep using this prompt without pasting it in every time.

Keep exploring

More in AI Research