Audit Industry Benchmark Reports for Bias and Missing Data

This prompt analyzes industry reports to identify cherry-picked metrics, hidden survivorship bias, and missing control group data that undermine statistical validity.

Prompt

Prompt
Act as a data auditor and research analyst. I will provide an industry benchmark report or a summary of its findings. Your goal is to critically evaluate the claims for statistical rigor. Follow these rules: 1. Identify all metrics presented and cross-reference them against implicit missing denominators or baseline controls. 2. Flag potential survivorship bias where only successful entities are included. 3. Highlight timeline gaps or non-standard reporting periods that skew results. 4. State if the report lacks a clear definition of the sample size and selection criteria. Output format: 1. Executive Summary of Findings. 2. Table with columns: Claimed Metric, Potential Omission, Risk Level. 3. List of 3 questions to ask the report author to clarify validity. This is not professional financial or legal advice; consult a qualified professional for investment or business decisions.

Example output

At a glanceAudit: claimed metrics vs potential omissions2024 Digital Marketing ROI Benchmark report
ItemPotential OmissionRisk Level
40 percent conversion liftBaseline data of non-usersHigh
95 percent retention rateInclusion criteria for active accountsMedium
Average lead cost of 5 dollarsGeographic distribution of samplesHigh

Example for: an audit of a 2024 Digital Marketing ROI Benchmark report. 1. Executive Summary of Findings: The report claims a 40 percent increase in lead conversion for firms using proprietary automation tools. However, the analysis ignores the baseline conversion rates of non-users during the same period and fails to disclose if the study excluded firms that went bankrupt or discontinued the software, suggesting significant survivorship bias. 2. Audit Table: | Claimed Metric | Potential Omission | Risk Level |
| --- | --- | --- |
| 40 percent conversion lift | Baseline data of non-users | High |
| 95 percent retention rate | Inclusion criteria for active accounts | Medium |
| Average lead cost of 5 dollars | Geographic distribution of samples | High | 3. Questions for the author: 1. What was the exact attrition rate of the firms included in the study over the 12-month period? 2. How did the conversion rate of the control group compare to the treatment group during the same fiscal quarters? 3. Were firms that ceased operations during the study period included in the final retention averages?

Written by AI from this prompt. Your results will vary.

Tips and common edits

Why this works

  • Forces identification of missing denominators which are often hidden in marketing reports.
  • Uses a structured table to force a clear comparison between claims and potential risks.
  • Requires explicit questioning of the author to close information gaps.

Use this with Perplexity or ChatGPT when you need to verify marketing-heavy industry reports or white papers. Paste the text of the report or upload the PDF directly after entering the prompt.

Common edits

  • Add a specific instruction to check if the report's source data is proprietary or publicly verifiable.
  • Adjust the risk level assessment criteria to align with your specific industry standards.

Make it repeatable

Keep using this prompt without pasting it in every time.

Keep exploring

More in AI Research