For the complete documentation index, see llms.txt. This page is also available as Markdown.

Data model

IDC relies on the DICOM data model for organizing images and image-derived data. At the same time, IDC includes certain attributes and data types that are outside of the DICOM data model. The Entity-Relationship (E-R) diagram and examples below summarize a simplified view of the IDC data model (you will find the explanation of how to interpret the notation used in this E-R diagram in this page from Mermaid documentation).

IDC content is organized in Collections: groups of DICOM files that were collected through certain research activity. We sometimes refer to these as Original Collections to distinguish them from Analysis Results collections described below.

Collections are organized into Programs, which group related collections, or those collections that were contributed under the same funding initiative or a consortium. Example: TCGA program contains TCGA-GBM, TCGA-BRCA and other collections. You will see Collections nested under Programs in the upper left section of the IDC Portal. You will also see the list of collections that meet the filter criteria in the top table on the right-hand side of the portal interface.

Individual DICOM files included in the collection contain attributes that organize content according to the DICOM data model.

Each collection will contain data for one or more cases, or patients. Data for the individual patient is organized in DICOM studies, which group images and derived objects collected in the context of a single imaging exam or encounter. Studies are composed of DICOM series, which in turn consist of DICOM instances. Each DICOM instance corresponds to a single file on disk. As an example, in radiology imaging, individual instances will most often correspond to image slices in multi-slice acquisitions (although a single instance of an enhanced multi-frame object, or of a Segmentation, can hold an entire volume), and in digital pathology you will see a separate file/instance for each resolution layer of the image pyramid, plus any label and overview images. Instance is not a level you browse in the IDC Portal - you will encounter individual instances once you download data to your computer.

Analysis results

The Analysis Results collection is a very important concept in IDC, and the peer of the Original Collection introduced above. An analysis result - we use the shorter form from here on - is the DICOM encoded result of some analysis performed on data from one or more original collections. Such analysis results are often contributed by investigators unrelated to those that submitted the analyzed images, and may span images across multiple collections.

An analysis result is a collection in its own right. It has its own identifier, title, DOI, license, description and citation, recorded in analysis_results_index just as original collections are recorded in collections_index. That grouping is what establishes the provenance of the derived content and gives credit to the people who produced it - they are cited for their contribution, and their terms of reuse travel with it.

IDC's metadata treats the two as peers explicitly. A collection is composed of one or more contributing sources, listed in the sources field of collections_index. Each entry carries its own source_doi, license, citation and source_title, and is tagged with a source_type of either original_data or analysis_result. For an analysis result, the source_id in that list is its analysis_result_id.

What is different is where the content sits. An analysis result brings no images, patients or studies of its own; it enriches data that is already in IDC. So each derived series belongs to two collections at the same time, and IDC records both:

  • collection_id - the original collection that supplied the analyzed images. A derived series carries this exactly like the images it describes, which is what places it in the right patient and study.

  • analysis_result_id - the analysis result that contributed it. Set for derived series only; NULL for originally submitted images.

The two are orthogonal grouping axes, not a hierarchy: an analysis result is not nested under one original collection - tcga_sbu_til_maps spans 23 of them - which is why the IDC Portal offers analysis results as a search scope of their own, alongside programs and collections. Filter on whichever axis you actually mean.

Two properties of derived content follow from all this:

  • Analysis results do not introduce new patients, and almost always attach to a study that already exists, alongside the images they describe - which is why you see them overlaid when you open the study in the viewer. As of v24, every patient with derived series also has original imaging, and only 260 of the ~92,000 studies containing derived series consist of derived series alone.

  • Provenance and licensing follow the contributing source, not the collection: series within a single study can carry different source_DOI and license_short_name values, which is why the diagram above places both attributes on SERIES. Derived content is not the only reason - a collection can have several original sources too, as NLST does with its CT images (CC BY 4.0) and its DICOM-converted slide microscopy (CC BY 3.0). As of v24, 37 of the 176 collections have more than one original source. License information is available programmatically at series granularity - see Licensing and attribution.

The model on a concrete example

The specific counts and version numbers below are as of IDC data release v24, and will change as new data is added.

Radiology: PROSTATEx

Consider patient ProstateX-0217 from the PROSTATEx collection (open in the IDC viewer):

What this single patient illustrates:

  • A derived series belongs to two collections at once. All four carry collection_id = 'prostatex', just like the MR images, and the analysis_result_id of the analysis result that contributed them - which is where their DOI and license come from.

  • They attach to the existing study. No new patient, no new study - the SEG and SR objects land in the same StudyInstanceUID as the images they describe.

  • One study can carry several licenses and DOIs. Here, three DOIs under CC BY 3.0 and two under CC BY 4.0. Licensing and provenance attach at the series level.

  • A study accretes content over releases. The images arrived in IDC v2, the BAMF segmentation in v19, the lesion annotations in v23.

  • SeriesDescription is not an identifier. 35 series here share the description tfl_3d dynamisch fast; only SeriesInstanceUID distinguishes them.

  • A case maps to many clinical records, spread across several tables and joined on dicom_patient_id - and one of those tables was contributed by an analysis result rather than by the original submitters.

Digital pathology: TCGA-LUAD

The same model applies to slide microscopy, where the INSTANCE level looks quite different. Patient TCGA-80-5608 from the TCGA-LUAD collection (open in the SLIM viewer):

Two additional points this example makes:

  • In pathology, one instance is one pyramid layer, not one slice. The whole slide image above is a single series of 4 files, ranging from a 128 MB base layer at 0.5 Β΅m/px down to a thumbnail. (A small number of series in IDC split a single layer across several instances.)

  • A single analysis result can span many collections. tcga_sbu_til_maps covers 23 TCGA collections and pan_cancer_nuclei_seg_dicom covers 14 - this is the many-to-many relationship between ANALYSIS_RESULT and COLLECTION in the diagram above. A single analysis result can also contribute more than one type of object, as pan_cancer_nuclei_seg_dicom does with ANN and SEG series.

Where each entity lives in the metadata

The table below maps each entity to the identifier you would use in idc-index. The same DICOM identifiers are used in the BigQuery dicom_all table, which additionally exposes every other DICOM attribute - see Files and metadata.

Entity
Identifier
Where to find it

PROGRAM

program_id

collections_index

COLLECTION

collection_id

index, collections_index

CASE

PatientID

index

STUDY

StudyInstanceUID

index

SERIES

SeriesInstanceUID

index

INSTANCE

SOPInstanceUID

BigQuery dicom_all; sm_instance_index for slide microscopy

ANALYSIS_RESULT

analysis_result_id

index, analysis_results_index

CLINICAL_DATA

dicom_patient_id

per-collection clinical tables, see clinical.md

Note that the identifiers above are the ones you search with. The files themselves are named using IDC-assigned UUIDs (crdc_series_uuid and crdc_instance_uuid) so that IDC can support versioning - see UUIDs and GUIDs.

Last updated

Was this helpful?