Imaging Data Commons

Overview

Use the idc-index Python package to query and download public cancer imaging data from the National Cancer Institute Imaging Data Commons (IDC). No authentication required for data access.

Current IDC Data Version: v23 (always verify with IDCClient().get_idc_version())

Primary tool: idc-index (GitHub)

CRITICAL - Check package version and upgrade if needed (run this FIRST):

REQUIRED_VERSION = "0.11.14"  # Must match metadata.idc-index in this file
installed = idc_index.__version__

if installed < REQUIRED_VERSION:
    print(f"Upgrading idc-index from {installed} to {REQUIRED_VERSION}...")

    subprocess.run(["pip3", "install", "--upgrade", "--break-system-packages", "idc-index"], check=True)
    print("Upgrade complete. Restart Python to use new version.")
else:
    print(f"idc-index {installed} meets requirement ({REQUIRED_VERSION})")

Verify IDC data version and check current data scale:

from idc_index import IDCClient
client = IDCClient()

# Verify IDC data version (should be "v23")
print(f"IDC data version: {client.get_idc_version()}")

# Get collection count and total series
stats = client.sql_query("""
    SELECT
        COUNT(DISTINCT collection_id) as collections,
        COUNT(DISTINCT analysis_result_id) as analysis_results,
        COUNT(DISTINCT PatientID) as patients,
        COUNT(DISTINCT StudyInstanceUID) as studies,
        COUNT(DISTINCT SeriesInstanceUID) as series,
        SUM(instanceCount) as instances,
        SUM(series_size_MB)/1000000 as size_TB
    FROM index
""")
print(stats)

Core workflow:

1. Query metadata → client.sql_query()
2. Download DICOM files → client.download_from_selection()
3. Visualize in browser → client.get_viewer_URL(seriesInstanceUID=...)

When to Use This Skill

• Finding publicly available radiology (CT, MR, PET) or pathology (slide microscopy) images
• Selecting image subsets by cancer type, modality, anatomical site, or other metadata
• Downloading DICOM data from IDC
• Checking data licenses before use in research or commercial applications
• Visualizing medical images in a browser without local DICOM viewer software

Quick Navigation

Core Sections (inline):

• IDC Data Model - Collection and analysis result hierarchy
• Index Tables - Available tables and joining patterns
• Installation - Package setup and version verification
• Core Capabilities - Essential API patterns (query, download, visualize, license, citations, batch)
• Best Practices - Usage guidelines
• Troubleshooting - Common issues and solutions

Reference Guides (load on demand):

IDC Data Model

IDC adds two grouping levels above the standard DICOM hierarchy (Patient → Study → Series → Instance):

• collection_id: Groups patients by disease, modality, or research focus (e.g., tcga_luad, nlst). A patient belongs to exactly one collection.
• analysis_result_id: Identifies derived objects (segmentations, annotations, radiomics features) across one or more original collections.

Use collection_id to find original imaging data, may include annotations deposited along with the images; use analysis_result_id to find AI-generated or expert annotations.

Key identifiers for queries:

Index Tables

The idc-index package provides multiple metadata index tables, accessible via SQL or as pandas DataFrames.

Complete index table documentation: Use https://idc-index.readthedocs.io/en/latest/indices_reference.html for quick check of available tables and columns without executing any code.

Important: Use client.indices_overview to get current table descriptions and column schemas. This is the authoritative source for available columns and their types — always query it when writing SQL or exploring data structure.

Available Tables

Auto = loaded automatically when IDCClient() is instantiated fetch_index() = requires client.fetch_index("table_name") to load

Joining Tables

Key columns are not explicitly labeled, the following is a subset that can be used in joins.

Note: Subjects, Updated, and Description appear in multiple tables but have different meanings (counts vs identifiers, different update contexts).

For detailed join examples, schema discovery patterns, key columns reference, and DataFrame access, see references/index_tables_guide.md.

Clinical Data Access

# Fetch clinical index (also downloads clinical data tables)
client.fetch_index("clinical_index")

# Query clinical index to find available tables and their columns
tables = client.sql_query("SELECT DISTINCT table_name, column_label FROM clinical_index")

# Load a specific clinical table as DataFrame
clinical_df = client.get_clinical_table("table_name")

See references/clinical_data_guide.md for detailed workflows including value mapping patterns and joining clinical data with imaging.

Data Access Options

Cloud storage organization

IDC maintains all DICOM files in public cloud storage buckets mirrored between AWS S3 and Google Cloud Storage. Files are organized by CRDC UUIDs (not DICOM UIDs) to support versioning.

Files are stored as <crdc_series_uuid>/<crdc_instance_uuid>.dcm. Access is free (no egress fees) via AWS CLI, gsutil, or s5cmd with anonymous access. Use series_aws_url column from the index for S3 URLs; GCS uses the same path structure.

See references/cloud_storage_guide.md for bucket details, access commands, UUID mapping, and versioning.

DICOMweb access

IDC data is available via DICOMweb interface (Google Cloud Healthcare API implementation) for integration with PACS systems and DICOMweb-compatible tools.

See references/dicomweb_guide.md for endpoint URLs, code examples, supported operations, and implementation details.

Direct Parquet access

All idc-index metadata tables are published as Parquet files to a public GCS bucket (idc-index-data-artifacts) with unrestricted CORS. This enables DuckDB or pandas queries without installing idc-index, including cross-table joins and queries against volume_geometry_index and rtstruct_index.

See references/parquet_access_guide.md for URL patterns, available files, and DuckDB query examples.

Installation and Setup

Required (for basic access):

pip install --upgrade idc-index

Important: New IDC data release will always trigger a new version of idc-index. Always use --upgrade flag while installing, unless an older version is needed for reproducibility.

IMPORTANT: IDC data version v23 is current. Always verify your version:

print(client.get_idc_version())  # Should return "v23"

If you see an older version, upgrade with: pip install --upgrade idc-index

Tested with: idc-index 0.11.14 (IDC data version v23)

Optional (for data analysis):

pip install pandas numpy pydicom

Core Capabilities

1. Data Discovery and Exploration

Discover what imaging collections and data are available in IDC:

from idc_index import IDCClient

client = IDCClient()

# Get summary statistics from primary index
query = """
SELECT
  collection_id,
  COUNT(DISTINCT PatientID) as patients,
  COUNT(DISTINCT SeriesInstanceUID) as series,
  SUM(series_size_MB) as size_mb
FROM index
GROUP BY collection_id
ORDER BY patients DESC
"""
collections_summary = client.sql_query(query)

# For richer collection metadata, use collections_index
client.fetch_index("collections_index")
collections_info = client.sql_query("""
    SELECT collection_id, CancerTypes, TumorLocations, Species, Subjects, SupportingData
    FROM collections_index
""")

# For analysis results (annotations, segmentations), use analysis_results_index
client.fetch_index("analysis_results_index")
analysis_info = client.sql_query("""
    SELECT analysis_result_id, analysis_result_title, Subjects, Collections, Modalities
    FRO