Evidence Types Catalog

Different claims require different types of evidence. This catalog describes evidence types, their strengths and limitations, and when each is most appropriate.

---

Overview: Matching Evidence to Claims

---

Evidence Types

1. Statistical Evidence

What it is: Numerical data from systematic measurement or research.

Subtypes:

• Descriptive statistics (averages, percentages, distributions)
• Inferential statistics (correlations, significance tests)
• Population data (census, surveys)
• Longitudinal data (trends over time)

Strengths:

• Quantifiable and precise
• Can show patterns across large populations
• Harder to dismiss than anecdote
• Can demonstrate magnitude and scale

Limitations:

• Numbers can be misleading without context
• Correlation ≠ causation
• Depends entirely on data quality
• Can be manipulated through selection

Best for:

• Establishing prevalence or frequency
• Showing trends and patterns
• Demonstrating magnitude of a problem
• Comparing groups or conditions

Quality markers:

• Sample size adequate
• Methodology transparent
• Confidence intervals provided
• Limitations acknowledged

---

2. Case Study Evidence

What it is: Detailed examination of specific real-world examples.

Subtypes:

• Organizational case studies (companies, institutions)
• Individual case studies (people, projects)
• Historical case studies (events, decisions)
• Comparative case studies (multiple examples examined together)

Strengths:

• Rich, detailed, and engaging
• Shows how things work in practice
• Memorable and persuasive
• Can illustrate complex interactions

Limitations:

• May not generalize
• Selection bias (why this case?)
• Often lacks counterfactual
• Details can be wrong or idealized

Best for:

• Illustrating principles in action
• Making abstract concepts concrete
• Providing memorable examples
• Showing practical application

Quality markers:

• Named and verifiable entities
• Specific details and outcomes
• Sources for key facts
• Acknowledgment of limitations

---

3. Expert Testimony

What it is: Statements from recognized authorities in the relevant field.

Subtypes:

• Direct quotes from experts
• Expert opinion in publications
• Expert consensus statements
• Expert disagreement/debate

Strengths:

• Borrows credibility from the expert
• Can provide synthesis and interpretation
• Signals that smart people agree
• Adds authority to claims

Limitations:

• Expert opinion ≠ empirical evidence
• Experts can be wrong
• Appeal to authority fallacy risk
• Experts may have conflicts of interest

Best for:

• Establishing current thinking in a field
• Interpreting complex evidence
• Adding credibility to conclusions
• Identifying open questions

Quality markers:

• Expert has relevant credentials
• Quote is in context
• Multiple experts if citing consensus
• Dissenting views acknowledged

---

4. Historical Evidence

What it is: Information about past events, developments, and contexts.

Subtypes:

• Primary historical documents
• Historical analysis by scholars
• Chronological development
• Historical analogies

Strengths:

• Establishes origins and development
• Provides perspective and context
• Shows what has been tried before
• Can reveal patterns over time

Limitations:

• Historical conditions may differ from present
• Records may be incomplete or biased
• Hindsight bias can distort interpretation
• Historical analogies can mislead

Best for:

• Establishing context and background
• Showing development over time
• Learning from past successes/failures
• Grounding current discussions

Quality markers:

• Primary sources used when available
• Context acknowledged
• Multiple perspectives considered
• Limitations of historical analogy noted

---

5. Experimental Evidence

What it is: Results from controlled experiments designed to test specific hypotheses.

Subtypes:

• Randomized controlled trials (gold standard)
• Laboratory experiments
• Field experiments
• Natural experiments

Strengths:

• Can establish causation (not just correlation)
• Controls for confounding variables
• Reproducible in principle
• Considered strongest evidence type

Limitations:

• Often narrow scope
• Lab conditions may not reflect real world
• Expensive and time-consuming
• Ethical limits on what can be tested

Best for:

• Establishing causal relationships
• Testing specific hypotheses
• Comparing interventions
• Settling empirical debates

Quality markers:

• Random assignment
• Adequate sample size
• Pre-registration of hypotheses
• Replication by other researchers

---

6. Survey/Poll Evidence

What it is: Systematic collection of opinions, behaviors, or characteristics from a sample.

Subtypes:

• Public opinion polls
• Academic surveys
• Industry surveys
• Internal organizational surveys

Strengths:

• Can capture attitudes and beliefs
• Can measure self-reported behavior
• Relatively fast and scalable
• Can track changes over time

Limitations:

• Self-report bias
• Question wording effects
• Sampling challenges
• Response rates matter

Best for:

• Understanding attitudes and beliefs
• Measuring prevalence of behaviors
• Capturing self-reported experiences
• Identifying trends

Quality markers:

• Methodology disclosed
• Sample is representative
• Response rate adequate
• Question wording neutral

---

7. Anecdotal Evidence

What it is: Individual stories, personal experiences, or illustrative examples.

Subtypes:

• Personal stories
• Illustrative examples
• Testimonials
• Observational accounts

Strengths:

• Highly engaging and memorable
• Makes abstract concepts vivid
• Connects emotionally
• Can illustrate what statistics can't capture

Limitations:

• Doesn't generalize
• Subject to selection bias
• Can be atypical or misleading
• Easy to find anecdotes for any position

Best for:

• Opening chapters or sections
• Illustrating points made with other evidence
• Humanizing data
• Engaging reader emotionally

Quality markers:

• Presented as illustration, not proof
• Verifiable if possible
• Representative of broader patterns
• Not cherry-picked outlier

---

8. Observational Evidence

What it is: Systematic observation without experimental intervention.

Subtypes:

• Ethnographic observation
• Naturalistic studies
• Cohort studies
• Cross-sectional studies

Strengths:

• Studies real-world conditions
• Can capture complex phenomena
• Doesn't require intervention
• Can study rare events or long timeframes

Limitations:

• Cannot establish causation
• Observer may influence observed
• Confounding variables uncontrolled
• Selection bias possible

Best for:

• Understanding phenomena in context
• Generating hypotheses
• Studying things that can't be experimentally manipulated
• Documenting patterns

Quality markers:

• Systematic methodology
• Multiple observers when possible
• Limitations acknowledged
• Triangulation with other evidence types

---

9. Counterargument Evidence

What it is: Evidence that challenges or complicates the main thesis.

Subtypes:

• Direct refutation
• Competing theories
• Edge cases and exceptions
• Methodological critiques

Strengths:

• Demonstrates intellectual honesty
• Strengthens credibility
• Prepares reader for objections
• Shows depth of engagement

Limitations:

• Can undermine argument if not handled well
• May confuse readers if overemphasized
• Requires skill to integrate

Best for:

• Building trust with skeptical readers
• Addressing obvious objections preemptively
• Showing nuance
• Demonstrating balanced perspective

Quality markers:

• Strongest version presented (steelman)
• Response provided
• Not straw-manned
• Acknowledges valid points

---

10. Comparative Evidence

What it is: Evidence from comparing different conditions, approaches, or groups.

Subtypes:

• Cross-cultural comparisons
• Before/after comparisons
• Treatment vs. control comparisons
• Benchmarking

Strengths:

• Shows what's possible under different conditions
• Provides concrete alternatives
• Can control for some variables
• Often very persuasive

Limitations:

• Comparisons may not be apples-to-apples
• Confounding differences between groups
• Success may not transfer across contexts

Best for:

• Showing alternatives work
• Demonstrating impact of interventions
• Identifying best practices
• Making the case for change

Quality markers:

• Comparisons are genuinely comparable
• Key differences acknowledged
• Not cherry-picked comparisons
• Mechanism for difference explained

---

Evidence Combinations

Strong arguments typically combine multiple evidence types:

For Persuasion

1. Hook: Anecdotal (engagement)
2. Scale: Statistical (magnitude)
3. Mechanism: Experimental or Expert (explanation)
4. Application: Case Study (practical)
5. Credibility: Address Counterarguments (trust)

For Establishing New Ideas

1. Foundation: Historical (context)
2. Theory: Expert (framework)
3. Test: Experimental (validation)
4. Application: Case Study (real-world)

For Challenging Conventional Wisdom

1. Contrast: Counterargument (what people think)
2. Evidence: Statistical or Experimental (what's actually true)
3. Explanation: Expert (why the misconception exists)
4. Proof: Case Study (examples of the truth)

---

Matching Evidence to Proof Burden

---

Use this catalog when specifying evidence types in research prompts and when evaluating whether returned research meets the chapter's needs.