Evidence Synthesis and Guideline Integration Guide

Overview

Evidence synthesis involves systematically reviewing, analyzing, and integrating research findings to inform clinical recommendations. This guide covers guideline sources, evidence hierarchies, systematic reviews, meta-analyses, and integration of multiple evidence streams for clinical decision support.

Major Clinical Practice Guidelines

Oncology Guidelines

NCCN (National Comprehensive Cancer Network)

• Scope: 60+ cancer types, supportive care guidelines
• Update Frequency: Continuous (online), 1-3 updates per year per guideline
• Evidence Categories:
  • Category 1: High-level evidence, uniform NCCN consensus
  • Category 2A: Lower-level evidence, uniform consensus (appropriate)
  • Category 2B: Lower-level evidence, non-uniform consensus (appropriate)
  • Category 3: Major disagreement or insufficient evidence
• Access: Free for patients, subscription for providers (institutional access common)
• Application: US-focused, most widely used in clinical practice

ASCO (American Society of Clinical Oncology)

• Scope: Evidence-based clinical practice guidelines
• Methodology: Systematic review, GRADE-style evidence tables
• Endorsements: Often endorses NCCN, ESMO, or other guidelines
• Focused Topics: Specific clinical questions (e.g., biomarker testing, supportive care)
• Guideline Products: Full guidelines, rapid recommendations, endorsements
• Quality: Rigorous methodology, peer-reviewed publication

ESMO (European Society for Medical Oncology)

• Scope: European guidelines for cancer management
• Evidence Levels:
  • I: Evidence from at least one large RCT or meta-analysis
  • II: Evidence from at least one well-designed non-randomized trial, cohort study
  • III: Evidence from well-designed non-experimental study
  • IV: Evidence from expert committee reports or opinions
  • V: Evidence from case series, case reports
• Recommendation Grades:
  • A: Strong evidence for efficacy, substantial clinical benefit (strongly recommended)
  • B: Strong or moderate evidence, limited clinical benefit (generally recommended)
  • C: Insufficient evidence, benefit not sufficiently well established
  • D: Moderate evidence against efficacy or for adverse effects (not recommended)
  • E: Strong evidence against efficacy (never recommended)
• ESMO-MCBS: Magnitude of Clinical Benefit Scale (grades 1-5 for meaningful benefit)

Cardiovascular Guidelines

AHA/ACC (American Heart Association / American College of Cardiology)

• Scope: Cardiovascular disease prevention, diagnosis, management
• Class of Recommendation (COR):
  • Class I: Strong recommendation - should be performed/administered
  • Class IIa: Moderate recommendation - is reasonable
  • Class IIb: Weak recommendation - may be considered
  • Class III - No Benefit: Not recommended
  • Class III - Harm: Potentially harmful
• Level of Evidence (LOE):
  • A: High-quality evidence from >1 RCT, meta-analyses
  • B-R: Moderate-quality evidence from ≥1 RCT
  • B-NR: Moderate-quality evidence from non-randomized studies
  • C-LD: Limited data from observational studies, registries
  • C-EO: Expert opinion based on clinical experience
• Example: "Statin therapy is recommended for adults with LDL-C ≥190 mg/dL (Class I, LOE A)"

ESC (European Society of Cardiology)

• Scope: European cardiovascular guidelines
• Class of Recommendation:
  • I: Recommended or indicated
  • II: Should be considered
  • III: Not recommended
• Level of Evidence: A (RCTs), B (single RCT or observational), C (expert opinion)

Other Specialties

IDSA (Infectious Diseases Society of America)

• Antimicrobial guidelines, infection management
• GRADE methodology
• Strong vs weak recommendations

ATS/ERS (American Thoracic Society / European Respiratory Society)

• Respiratory disease management
• GRADE methodology

ACR (American College of Rheumatology)

• Rheumatic disease guidelines
• Conditionally recommended vs strongly recommended

KDIGO (Kidney Disease: Improving Global Outcomes)

• Chronic kidney disease, dialysis, transplant
• GRADE-based recommendations

GRADE Methodology

Assessing Quality of Evidence

Initial Quality Assignment

Randomized Controlled Trials: Start at HIGH quality (⊕⊕⊕⊕)

Observational Studies: Start at LOW quality (⊕⊕○○)

Factors Decreasing Quality (Downgrade)

Risk of Bias (-1 or -2 levels)

• Lack of allocation concealment
• Lack of blinding
• Incomplete outcome data
• Selective outcome reporting
• Other sources of bias

Inconsistency (-1 or -2 levels)

• Unexplained heterogeneity in results across studies
• Wide variation in effect estimates
• Non-overlapping confidence intervals
• High I² statistic in meta-analysis (>50-75%)

Indirectness (-1 or -2 levels)

• Different population than target (younger patients in trials, applying to elderly)
• Different intervention (higher dose in trial than used in practice)
• Different comparator (placebo in trial, comparing to active treatment)
• Surrogate outcomes (PFS) when interested in survival (OS)

Imprecision (-1 or -2 levels)

• Wide confidence intervals crossing threshold of benefit/harm
• Small sample size, few events
• Optimal information size (OIS) not met
• Rule of thumb: <300 events for continuous outcomes, <200 events for dichotomous

Publication Bias (-1 level)

• Funnel plot asymmetry (if ≥10 studies)
• Known unpublished studies with negative results
• Selective outcome reporting
• Industry-sponsored studies only

Factors Increasing Quality (Upgrade - Observational Only)

Large Magnitude of Effect (+1 or +2 levels)

• +1: RR >2 or <0.5 (moderate effect)
• +2: RR >5 or <0.2 (large effect)
• No plausible confounders would reduce effect

Dose-Response Gradient (+1 level)

• Clear dose-response or duration-response relationship
• Strengthens causal inference

All Plausible Confounders Would Reduce Effect (+1 level)

• Observed effect despite confounders biasing toward null
• Rare, requires careful justification

Final Quality Rating

After adjustments, assign final quality:

• High (⊕⊕⊕⊕): Very confident in effect estimate
• Moderate (⊕⊕⊕○): Moderately confident; true effect likely close to estimate
• Low (⊕⊕○○): Limited confidence; true effect may be substantially different
• Very Low (⊕○○○): Very little confidence; true effect likely substantially different

Systematic Reviews and Meta-Analyses

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)

Search Strategy

• Databases: PubMed/MEDLINE, Embase, Cochrane Library, Web of Science
• Search Terms: PICO (Population, Intervention, Comparator, Outcome)
• Date Range: Typically last 10-20 years or comprehensive
• Language: English only or all languages with translation
• Grey Literature: Conference abstracts, trial registries, unpublished data

Study Selection

PRISMA Flow Diagram:

Records identified through database searching (n=2,450)
Additional records through other sources (n=15)
                ↓
Records after duplicates removed (n=1,823)
                ↓
Records screened (title/abstract) (n=1,823)  → Excluded (n=1,652)
                ↓                                 - Not relevant topic (n=1,120)
Full-text articles assessed (n=171)              - Animal studies (n=332)
                ↓                                 - Reviews (n=200)
Studies included in qualitative synthesis (n=38) → Excluded (n=133)
                ↓                                 - Wrong population (n=42)
Studies included in meta-analysis (n=24)          - Wrong intervention (n=35)
                                                  - No outcomes reported (n=28)
                                                  - Duplicate data (n=18)
                                                  - Poor quality (n=10)

Data Extraction

• Study characteristics: Design, sample size, population, intervention
• Results: Outcomes, effect sizes, confidence intervals, p-values
• Quality assessment: Risk of bias tool (Cochrane RoB 2.0 for RCTs)
• Dual extraction: Two reviewers independently, resolve disagreements

Meta-Analysis Methods

Fixed-Effect Model

• Assumption: Single true effect size shared by all studies
• Weighting: By inverse variance (larger studies have more weight)
• Application: When heterogeneity is low (I² <25%)
• Interpretation: Estimate of common effect across studies

Random-Effects Model

• Assumption: True effect varies across studies (distribution of effects)
• Weighting: By inverse variance + between-study variance
• Application: When heterogeneity moderate to high (I² ≥25%)
• Interpretation: Estimate of average effect (center of distribution)
• Wider CI: Accounts for heterogeneity, more conservative

Heterogeneity Assessment

I² Statistic

• Percentage of variability due to heterogeneity rather than chance
• I² = 0-25%: Low heterogeneity
• I² = 25-50%: Moderate heterogeneity
• I² = 50-75%: Substantial heterogeneity
• I² = 75-100%: Considerable heterogeneity

Q Test (Cochran's Q)

• Test for heterogeneity
• p<0.10 suggests significant heterogeneity (liberal threshold)
• Low power when few studies, use I² as primary measure

Tau² (τ²)

• Estimate of between-study variance
• Used in random-effects weighting

Subgroup Analysis

• Explore sources of heterogeneity
• Pre-specified subgroups: Disease stage, biomarker status, treatment regimen
• Test for interaction between subgroups

Forest Plot Interpretation

Study               n     HR (95% CI)          Weight
─────────────────────────────────────────────────────────────
Trial A 2018        450   0.62 (0.45-0.85)     ●───┤      28%
Trial B 2019        320   0.71 (0.49-1.02)      ●────┤     22%
Trial C 2020        580   0.55 (0.41-0.74)    ●──┤       32%
Trial D 2021        210   0.88 (0.56-1.38)        ●──────┤  18%

Overall (RE model)  1560  0.65 (0.53-0.80)      ◆──┤
Heterogeneity: I²=42%, p=0.16

                          0.25  0.5  1.0  2.0  4.0
                                Favors Treatment  Favors Control

Guideline Integration

Concordance Checking

Multi-Guideline Comparison

Recommendation: First-line treatment for advanced NSCLC, PD-L1 ≥50%

Guideline    Version   Recommendation                               Strength
─────────────────────────────────────────────────────────────────────────────
NCCN         v4.2024   Pembrolizumab monotherapy (preferred)       Category 1
ESMO         2023      Pembrolizumab monotherapy (preferred)       I, A
ASCO         2022      Endorses NCCN guidelines                    Strong
NICE (UK)    2023      Pembrolizumab approved                      Recommended

Synthesis: Strong consensus across guidelines for pembrolizumab monotherapy.
Alternative: Pembrolizumab + chemotherapy also Category 1/I-A recommended.

Discordance Resolution

• Identify differences and reasons (geography, cost, access, evidence interpretation)
• Note date of each guideline (newer may incorporate recent trials)
• Consider regional applicability
• Favor guidelines with most rigorous methodology (GRADE-based)

Regulatory Approval Landscape

FDA Approvals

• Track indication-specific approvals
• Accelerated approval vs full approval
• Post-marketing requirements
• Contraindications and warnings

EMA (European Medicines Agency)

• May differ from FDA in approved indications
• Conditional marketing authorization
• Additional monitoring (black triangle)

Regional Variations

• Health Technology Assessment (HTA) agencies
• NICE (UK): Cost-effectiveness analysis, QALY thresholds
• CADTH (Canada): Therapeutic review and recommendations
• PBAC (Australia): Reimbursement decisions

Real-World Evidence (RWE)

Sources of RWE

Electronic Health Records (EHR)

• Clinical data from routine practice
• Large patient numbers
• Heterogeneous populations (more generalizable than RCTs)
• Limitations: Missing data, inconsistent documentation, selection bias

Claims Databases

• Administrative claims for billing/reimbursement
• Large scale (millions of patients)
• Outcomes: Mortality, hospitalizations, procedures
• Limitations: Lack clinical detail (labs, imaging, biomarkers)

Cancer Registries

• SEER (Surveillance, Epidemiology, and End Results): US cancer registry
• NCDB (National Cancer Database): Hospital registry data
• Population-level survival, treatment patterns
• Limited treatment detail, no toxicity data

Prospective Cohorts

• Framingham Heart Study, Nurses' Health Study
• Long-term follow-up, rich covariate data
• Expensive, time-consuming

RWE Applications

Comparative Effectiveness

• Compare treatments in real-world settings (less strict eligibility than RCTs)
• Complement RCT data with broader populations
• Example: Effectiveness of immunotherapy in elderly, poor PS patients excluded from trials

Safety Signal Detection

• Rare adverse events not detected in trials
• Long-term toxicities
• Drug-drug interactions in polypharmacy
• Postmarketing surveillance

Treatment Patterns and Access

• Guideline adherence in community practice
• Time to treatment initiation
• Disparities in care delivery
• Off-label use prevalence

Limitations of RWE

• Confounding by indication: Sicker patients receive more aggressive treatment
• Immortal time bias: Time between events affecting survival estimates
• Missing data: Incomplete or inconsistent data collection
• Causality: Association does not prove causation without randomization

Strengthening RWE

• Propensity score matching: Balance baseline characteristics between groups
• Multivariable adjustment: Adjust for measured confounders in Cox model
• Sensitivity analyses: Test robustness to unmeasured confounding
• Instrumental variables: Use natural experiments to approximate randomization

Meta-Analysis Techniques

Binary Outcomes (Response Rate, Event Rate)

Effect Measures

• Risk Ratio (RR): Ratio of event probabilities
• Odds Ratio (OR): Ratio of odds (less intuitive)
• Risk Difference (RD): Absolute difference in event rates

Example Calculation

Study 1:
- Treatment A: 30/100 responded (30%)
- Treatment B: 15/100 responded (15%)
- RR = 0.30/0.15 = 2.0 (95% CI 1.15-3.48)
- RD = 0.30 - 0.15 = 0.15 or 15% (95% CI 4.2%-25.8%)
- NNT = 1/RD = 1/0.15 = 6.7 (treat 7 patients to get 1 additional response)

Pooling Methods

• Mantel-Haenszel: Common fixed-effect method
• DerSimonian-Laird: Random-effects method
• Peto: For rare events (event rate <1%)

Time-to-Event Outcomes (Survival, PFS)

Hazard Ratio Pooling

• Extract HR and 95%