---
title: "Common Methodological and Statistical Issues in Scientific Manuscripts"
description: "This document catalogs frequent issues encountered during peer review, organized by category. Use this as a reference to identify potential problems and provide constructive feedback."
type: skill
canonical_url: https://claudary.paisolsolutions.com/skills/common-issues-2-2
source: "Claudary"
difficulty: intermediate
author: "Claude Code Knowledge Pack"
date: 2026-07-10T11:18:34.856Z
license: CC-BY-4.0
attribution: "Common Methodological and Statistical Issues in Scientific Manuscripts — Claudary (https://claudary.paisolsolutions.com/skills/common-issues-2-2)"
---

# Common Methodological and Statistical Issues in Scientific Manuscripts
This document catalogs frequent issues encountered during peer review, organized by category. Use this as a reference to identify potential problems and provide constructive feedback.

## Overview

# Common Methodological and Statistical Issues in Scientific Manuscripts

This document catalogs frequent issues encountered during peer review, organized by category. Use this as a reference to identify potential problems and provide constructive feedback.

## Statistical Issues

### 1. P-Value Misuse and Misinterpretation

**Common Problems:**
- P-hacking (selective reporting of significant results)
- Multiple testing without correction (familywise error rate inflation)
- Interpreting non-significance as proof of no effect
- Focusing exclusively on p-values without effect sizes
- Dichotomizing continuous p-values at arbitrary thresholds (p=0.049 vs p=0.051)
- Confusing statistical significance with biological/clinical significance

**How to Identify:**
- Suspiciously high proportion of p-values just below 0.05
- Many tests performed but no correction mentioned
- Statements like "no difference was found" from non-significant results
- No effect sizes or confidence intervals reported
- Language suggesting p-values indicate strength of effect

**What to Recommend:**
- Report effect sizes with confidence intervals
- Apply appropriate multiple testing corrections (Bonferroni, FDR, Holm-Bonferroni)
- Interpret non-significance cautiously (lack of evidence ≠ evidence of lack)
- Pre-register analyses to avoid p-hacking
- Consider equivalence testing for "no difference" claims

### 2. Inappropriate Statistical Tests

**Common Problems:**
- Using parametric tests when assumptions are violated (non-normal data, unequal variances)
- Analyzing paired data with unpaired tests
- Using t-tests for multiple groups instead of ANOVA with post-hoc tests
- Treating ordinal data as continuous
- Ignoring repeated measures structure
- Using correlation when regression is more appropriate

**How to Identify:**
- No mention of assumption checking
- Small sample sizes with parametric tests
- Multiple pairwise t-tests instead of ANOVA
- Likert scales analyzed with t-tests
- Time-series data analyzed without accounting for repeated measures

**What to Recommend:**
- Check assumptions explicitly (normality tests, Q-Q plots)
- Use non-parametric alternatives when appropriate
- Apply proper corrections for multiple comparisons after ANOVA
- Use mixed-effects models for repeated measures
- Consider ordinal regression for ordinal outcomes

### 3. Sample Size and Power Issues

**Common Problems:**
- No sample size justification or power calculation
- Underpowered studies claiming "no effect"
- Post-hoc power calculations (which are uninformative)
- Stopping rules not pre-specified
- Unequal group sizes without justification

**How to Identify:**
- Small sample sizes (n<30 per group for typical designs)
- No mention of power analysis in methods
- Statements about post-hoc power
- Wide confidence intervals suggesting imprecision
- Claims of "no effect" with large p-values and small n

**What to Recommend:**
- Conduct a priori power analysis based on expected effect size
- Report achieved power or precision (confidence interval width)
- Acknowledge when studies are underpowered
- Consider effect sizes and confidence intervals for interpretation
- Pre-register sample size and stopping rules

### 4. Missing Data Problems

**Common Problems:**
- Complete case analysis without justification (listwise deletion)
- Not reporting extent or pattern of missingness
- Assuming data are missing completely at random (MCAR) without testing
- Inappropriate imputation methods
- Not performing sensitivity analyses

**How to Identify:**
- Different n values across analyses without explanation
- No discussion of missing data
- Participants "excluded from analysis"
- Simple mean imputation used
- No sensitivity analyses comparing complete vs. imputed data

**What to Recommend:**
- Report extent and patterns of missingness
- Test MCAR assumption (Little's test)
- Use appropriate methods (multiple imputation, maximum likelihood)
- Perform sensitivity analyses
- Consider intention-to-treat analysis for trials

### 5. Circular Analysis and Double-Dipping

**Common Problems:**
- Using the same data for selection and inference
- Defining ROIs based on contrast then testing that contrast in same ROI
- Selecting outliers then testing for differences
- Post-hoc subgroup analyses presented as planned
- HARKing (Hypothesizing After Results are Known)

**How to Identify:**
- ROIs or features selected based on results
- Unexpected subgroup analyses
- Post-hoc analyses not clearly labeled as exploratory
- No data-independent validation
- Introduction that perfectly predicts findings

**What to Recommend:**
- Use independent datasets for selection and testing
- Pre-register analyses and hypotheses
- Clearly distinguish confirmatory vs. exploratory analyses
- Use cross-validation or hold-out datasets
- Correct for selection bias

### 6. Pseudoreplication

**Common Problems:**
- Technical replicates treated as biological replicates
- Multiple measurements from same subject treated as independent
- Clustered data analyzed without accounting for clustering
- Non-independence in spatial or temporal data

**How to Identify:**
- n defined as number of measurements rather than biological units
- Multiple cells from same animal counted as independent
- Repeated measures not acknowledged
- No mention of random effects or clustering

**What to Recommend:**
- Define n as biological replicates (animals, patients, independent samples)
- Use mixed-effects models for nested or clustered data
- Account for repeated measures explicitly
- Average technical replicates before analysis
- Report both technical and biological replication

## Experimental Design Issues

### 7. Lack of Appropriate Controls

**Common Problems:**
- Missing negative controls
- Missing positive controls for validation
- No vehicle controls for drug studies
- No time-matched controls for longitudinal studies
- No batch controls

**How to Identify:**
- Methods section lists only experimental groups
- No mention of controls in figures
- Unclear baseline or reference condition
- Cross-batch comparisons without controls

**What to Recommend:**
- Include negative controls to assess specificity
- Include positive controls to validate methods
- Use vehicle controls matched to experimental treatment
- Include sham surgery controls for surgical interventions
- Include batch controls for cross-batch comparisons

### 8. Confounding Variables

**Common Problems:**
- Systematic differences between groups besides intervention
- Batch effects not controlled or corrected
- Order effects in sequential experiments
- Time-of-day effects not controlled
- Experimenter effects not blinded

**How to Identify:**
- Groups differ in multiple characteristics
- Samples processed in different batches by group
- No randomization of sample order
- No mention of blinding
- Baseline characteristics differ between groups

**What to Recommend:**
- Randomize experimental units to conditions
- Block on known confounders
- Randomize sample processing order
- Use blinding to minimize bias
- Perform batch correction if needed
- Report and adjust for baseline differences

### 9. Insufficient Replication

**Common Problems:**
- Single experiment without replication
- Technical replicates mistaken for biological replication
- Small n justified by "typical for the field"
- No independent validation of key findings
- Cherry-picking representative examples

**How to Identify:**
- Methods state "experiment performed once"
- n=3 with no justification
- "Representative image shown"
- Key claims based on single experiment
- No validation in independent dataset

**What to Recommend:**
- Perform independent biological replicates (typically ≥3)
- Validate key findings in independent cohorts
- Report all replicates, not just representative examples
- Conduct power analysis to justify sample size
- Show individual data points, not just summary statistics

## Reproducibility Issues

### 10. Insufficient Methodological Detail

**Common Problems:**
- Methods not described in sufficient detail for replication
- Key reagents not specified (vendor, catalog number)
- Software versions and parameters not reported
- Antibodies not validated
- Cell line authentication not verified

**How to Identify:**
- Vague descriptions ("standard protocols were used")
- No information on reagent sources
- Generic software mentioned without versions
- No antibody validation information
- Cell lines not authenticated

**What to Recommend:**
- Provide detailed protocols or cite specific protocols
- Include reagent vendors, catalog numbers, lot numbers
- Report software versions and all parameters
- Include antibody validation (Western blot, specificity tests)
- Report cell line authentication method (STR profiling)
- Make protocols available (protocols.io, supplementary materials)

### 11. Data and Code Availability

**Common Problems:**
- No data availability statement
- "Data available upon request" (often unfulfilled)
- No code provided for computational analyses
- Custom software not made available
- No clear documentation

**How to Identify:**
- Missing data availability statement
- No repository accession numbers
- Computational methods with no code
- Custom pipelines without access
- No README or documentation

**What to Recommend:**
- Deposit raw data in appropriate repositories (GEO, SRA, Dryad, Zenodo)
- Share analysis code on GitHub or similar
- Provide clear documentation and README files
- Include requirements.txt or environment files
- Make custom software available with installation instructions
- Use DOIs for permanent data citation

### 12. Lack of Method Validation

**Common Problems:**
- New methods not compared to gold standard
- Assays not validated for specificity, sensitivity, linearity
- No spike-in controls
- Cross-reactivity not tested
- Detection limits not established

**How to Identify:**
- Novel assays presented without validation
- No comparison to existing methods
- No positive/negative controls shown
- Claims of specificity without evidence
- No standard curves or controls

**What to Recommend:**
- Validate new methods against established approaches
- Show specificity (knockdown/knockout controls)
- Demonstrate linearity and dynamic range
- Include positive and negative controls
- Report limits of detection and quantification
- Show reproducibility across replicates and operators

## Interpretation Issues

### 13. Overstatement of Results

**Common Problems:**
- Causal language for correlational data
- Mechanistic claims without mechanistic evidence
- Extrapolating beyond data (species, conditions, populations)
- Claiming "first to show" without thorough literature review
- Overgeneralizing from limited samples

**How to Identify:**
- "X causes Y" from observational data
- Mechanism proposed without direct testing
- Mouse data presented as relevant to humans without caveats
- Claims of novelty with missing citations
- Broad claims from narrow samples

**What to Recommend:**
- Use appropriate language ("associated with" vs. "caused by")
- Distinguish correlation from causation
- Acknowledge limitations of model systems
- Provide thorough literature context
- Be specific about generalizability
- Propose mechanisms as hypotheses, not conclusions

### 14. Cherry-Picking and Selective Reporting

**Common Problems:**
- Reporting only significant results
- Showing "representative" images that may not be typical
- Excluding outliers without justification
- Not reporting negative or contradictory findings
- Switching between different statistical approaches

**How to Identify:**
- All reported results are significant
- "Representative of 3 experiments" with no quantification
- Data exclusions mentioned in results but not methods
- Supplementary data contradicts main findings
- Multiple analysis approaches with only one reported

**What to Recommend:**
- Report all planned analyses regardless of outcome
- Quantify and show variability across replicates
- Pre-specify outlier exclusion criteria
- Include negative results
- Pre-register analysis plan
- Report effect sizes and confidence intervals for all comparisons

### 15. Ignoring Alternative Explanations

**Common Problems:**
- Preferred explanation presented without considering alternatives
- Contradictory evidence dismissed without discussion
- Off-target effects not considered
- Confounding variables not acknowledged
- Limitations section minimal or absent

**How to Identify:**
- Single interpretation presented as fact
- Prior contradictory findings not cited or discussed
- No consideration of alternative mechanisms
- No discussion of limitations
- Specificity assumed without controls

**What to Recommend:**
- Discuss alternative explanations
- Address contradictory findings from literature
- Include appropriate specificity controls
- Acknowledge and discuss limitations thoroughly
- Consider and test alternative hypotheses

## Figure and Data Presentation Issues

### 16. Inappropriate Data Visualization

**Common Problems:**
- Bar graphs for continuous data (hiding distributions)
- No error bars or error bars not defined
- Truncated y-axes exaggerating differences
- Dual y-axes creating misleading comparisons
- Too many significant figures
- Colors not colorblind-friendly

**How to Identify:**
- Bar graphs with few data points
- Unclear what error bars represent (SD, SEM, CI?)
- Y-axis doesn't start at zero for ratio/percentage data
- Left and right y-axes with different scales
- Values reported to excessive precision (p=0.04562)
- Red-green color schemes

**What to Recommend:**
- Show individual data points with scatter/box/violin plots
- Always define error bars (SD, SEM, 95% CI)
- Start y-axis at zero or indicate breaks clearly
- Avoid dual y-axes; use separate panels instead
- Report appropriate significant figures
- Use colorblind-friendly palettes (viridis, colorbrewer)
- Include sample sizes in figure legends

### 17. Image Manipulation Concerns

**Common Problems:**
- Excessive contrast/brightness adjustment
- Spliced gels or images without indication
- Duplicated images or panels
- Uneven background in Western blots
- Selective cropping
- Over-processed microscopy images

**How to Identify:**
- Suspicious patterns or discontinuities
- Very high contrast with no background
- Similar features in different panels
- Straight lines suggesting splicing
- Inconsistent backgrounds
- Loss of detail suggesting over-processing

**What to Recommend:**
- Apply adjustments uniformly across images
- Indicate spliced gels with dividing lines
- Show full, uncropped images in supplementary materials
- Provide original images if requested
- Follow journal image integrity policies
- Use appropriate image analysis tools

## Study Design Issues

### 18. Poorly Defined Hypotheses and Outcomes

**Common Problems:**
- No clear hypothesis stated
- Primary outcome not specified
- Multiple outcomes without correction
- Outcomes changed after data col

---

Source: [Claudary](https://claudary.paisolsolutions.com/skills/common-issues-2-2) · https://claudary.paisolsolutions.com