> For the complete documentation index, see [llms.txt](https://learn.canceridc.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learn.canceridc.dev/data/data-model.md). # Data model IDC relies on the DICOM data model for organizing images and image-derived data. At the same time, IDC includes certain attributes and data types that are outside of the DICOM data model. The *Entity-Relationship (E-R) diagram* and examples below summarize a simplified view of the IDC data model (you will find the explanation of how to interpret the notation used in this E-R diagram in [this page](https://mermaid.js.org/syntax/entityRelationshipDiagram.html) from Mermaid documentation). ```mermaid erDiagram PROGRAM ||--o{ COLLECTION : contains COLLECTION ||--o{ CASE : contains CASE ||--o{ STUDY : contains STUDY ||--o{ SERIES : contains SERIES ||--o{ INSTANCE : contains ANALYSIS_RESULT |o--o{ SERIES : contributes ANALYSIS_RESULT }o--o{ COLLECTION : spans CASE |o--o{ CLINICAL_DATA : "may have" PROGRAM { string program_id PK } COLLECTION { string collection_id PK string collection_name string program_id FK record sources "one per contributing source" } CASE { string PatientID PK } STUDY { string StudyInstanceUID PK } SERIES { string SeriesInstanceUID PK string collection_id FK string analysis_result_id FK string source_DOI string license_short_name } INSTANCE { string SOPInstanceUID PK } ANALYSIS_RESULT { string analysis_result_id PK string analysis_result_title string source_DOI string license_short_name string citation } CLINICAL_DATA { string dicom_patient_id FK } ``` IDC content is organized in **Collections**: groups of DICOM files that were collected through certain research activity. We sometimes refer to these as **Original Collections** to distinguish them from Analysis Results collections described below. Collections are organized into **Programs**, which group related collections, or those collections that were contributed under the same funding initiative or a consortium. Example: TCGA program contains TCGA-GBM, TCGA-BRCA and other collections. You will see Collections nested under Programs in the upper left section of the [IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/). You will also see the list of collections that meet the filter criteria in the top table on the right-hand side of the portal interface. Individual DICOM files included in the collection contain attributes that organize content according to the [DICOM data model](/dicom/data-model.md). Each collection will contain data for one or more cases, or **patients**. Data for the individual patient is organized in DICOM **studies**, which group images and derived objects collected in the context of a single imaging exam or encounter. Studies are composed of DICOM **series**, which in turn consist of DICOM **instances**. Each DICOM instance corresponds to a single file on disk. As an example, in radiology imaging, individual instances will most often correspond to image slices in multi-slice acquisitions (although a single instance of an enhanced multi-frame object, or of a Segmentation, can hold an entire volume), and in digital pathology you will see a separate file/instance for each resolution layer of the image pyramid, plus any label and overview images. Instance is not a level you browse in the IDC Portal - you will encounter individual instances once you download data to your computer. {% hint style="warning" %} `PatientID` is unique *within* a collection, but is not guaranteed to be unique across IDC. If you join or group data across collections, always use `PatientID` together with `collection_id`. {% endhint %} ## Analysis results The **Analysis Results collection** is a very important concept in IDC, and the peer of the Original Collection introduced above. An analysis result - we use the shorter form from here on - is the DICOM encoded result of some analysis performed on data from one or more original collections. Such analysis results are often contributed by investigators unrelated to those that submitted the analyzed images, and may span images across multiple collections. An analysis result is a collection in its own right. It has its own identifier, title, DOI, license, description and citation, recorded in `analysis_results_index` just as original collections are recorded in `collections_index`. That grouping is what establishes the provenance of the derived content and gives credit to the people who produced it - they are cited for their contribution, and their terms of reuse travel with it. IDC's metadata treats the two as peers explicitly. A collection is composed of one or more **contributing sources**, listed in the `sources` field of `collections_index`. Each entry carries its own `source_doi`, `license`, `citation` and `source_title`, and is tagged with a `source_type` of either `original_data` or `analysis_result`. For an analysis result, the `source_id` in that list is its `analysis_result_id`. What is different is *where the content sits*. An analysis result brings no images, patients or studies of its own; it enriches data that is already in IDC. So each derived series belongs to two collections at the same time, and IDC records both: * `collection_id` - the original collection that supplied the analyzed images. A derived series carries this exactly like the images it describes, which is what places it in the right patient and study. * `analysis_result_id` - the analysis result that contributed it. Set for derived series only; NULL for originally submitted images. The two are **orthogonal grouping axes, not a hierarchy**: an analysis result is not nested under one original collection - `tcga_sbu_til_maps` spans 23 of them - which is why the [IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/) offers analysis results as a search scope of their own, alongside programs and collections. Filter on whichever axis you actually mean. {% hint style="warning" %} **A `collection_id` filter returns both axes at once.** Derived series carry the `collection_id` of the images they analyze, so filtering on a collection returns the analysis results contributed to it alongside its original images - and there can be far more of those than you expect. `collection_id = 'nlst'` matches 590,572 series, but only 204,346 of them are NLST's own images; the other 386,226 were contributed by analysis results. 55 of the 176 collections contain derived series. * **Original images only**: add `analysis_result_id IS NULL`. That is SQL, so it applies wherever you query `index` with SQL - `idc-index`, `POST /v3/sql`, the MCP `run_sql` tool, or BigQuery. The column is genuinely NULL for original data, not an empty string, so `= ''` matches nothing. There is no cohort-filter equivalent: `analysis_result_id` is a filterable term, but its only values are the 24 analysis result ids, so no term means "none". * **One analysis result**: filter on its `analysis_result_id` - which does work on every surface - rather than on the collections it spans. `tcga_sbu_til_maps` contributed 21,030 series across 23 collections; selecting those 23 collections instead returns 66,878 series, most of them original imaging you did not ask for. {% endhint %} Two properties of derived content follow from all this: * **Analysis results do not introduce new patients**, and almost always attach to a study that already exists, alongside the images they describe - which is why you see them overlaid when you open the study in the viewer. As of v24, every patient with derived series also has original imaging, and only 260 of the \~92,000 studies containing derived series consist of derived series alone. * **Provenance and licensing follow the contributing source, not the collection**: series within a single study can carry different `source_DOI` and `license_short_name` values, which is why the diagram above places both attributes on SERIES. Derived content is not the only reason - a collection can have several original sources too, as NLST does with its CT images (CC BY 4.0) and its DICOM-converted slide microscopy (CC BY 3.0). As of v24, 37 of the 176 collections have more than one original source. License information is available programmatically at series granularity - see [Licensing and attribution](/data/licensing.md). ## The model on a concrete example {% hint style="info" %} The specific counts and version numbers below are as of IDC data release v24, and will change as new data is added. {% endhint %} ### Radiology: PROSTATEx Consider patient `ProstateX-0217` from the [PROSTATEx](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=prostatex) collection ([open in the IDC viewer](https://viewer.imaging.datacommons.cancer.gov/v3/viewer/?StudyInstanceUIDs=1.3.6.1.4.1.14519.5.2.1.7310.5101.239746591836843122771107560214)): ``` PROGRAM community └── COLLECTION prostatex (PROSTATEx) └── CASE ProstateX-0217 └── STUDY 1.3.6.1.4.1.14519.5.2.1.7310.5101.239746591836843122771107560214 │ ├── 43 original MR series DOI 10.7937/k9tcia.2017.murs5cl CC BY 3.0 since v2 │ ├── t2_tse_tra 23 INSTANCEs (one file per slice) │ ├── t2_tse_sag 19 INSTANCEs │ ├── diffusie-3Scan-4bval_fs 60 INSTANCEs │ ├── 35 × "tfl_3d dynamisch fast" 12 INSTANCEs each (DCE time points) │ └── … │ └── 4 series contributed by ANALYSIS_RESULTs, added to this same study: ├── SEG prostatex_seg_zones 10.7937/tcia.nbb4-4655 CC BY 3.0 since v2 ├── SEG prostatex_seg_hires 10.7937/tcia.2019.deg7zg1u CC BY 3.0 since v2 ├── SEG bamf_aimi_annotations 10.5281/zenodo.8345959 CC BY 4.0 since v19 └── SR prostatex_targets 10.5281/zenodo.15643312 CC BY 4.0 since v23 CLINICAL_DATA for ProstateX-0217, across 4 of the 6 clinical tables for this collection: prostatex_images 8 rows │ prostatex_findings 1 │ prostatex_ktrans 1 │ bamf…qa_results 1 ``` What this single patient illustrates: * **A derived series belongs to two collections at once.** All four carry `collection_id = 'prostatex'`, just like the MR images, *and* the `analysis_result_id` of the analysis result that contributed them - which is where their DOI and license come from. * **They attach to the existing study.** No new patient, no new study - the SEG and SR objects land in the same `StudyInstanceUID` as the images they describe. * **One study can carry several licenses and DOIs.** Here, three DOIs under CC BY 3.0 and two under CC BY 4.0. Licensing and provenance attach at the series level. * **A study accretes content over releases.** The images arrived in IDC v2, the BAMF segmentation in v19, the lesion annotations in v23. * **`SeriesDescription` is not an identifier.** 35 series here share the description `tfl_3d dynamisch fast`; only `SeriesInstanceUID` distinguishes them. * **A case maps to many clinical records**, spread across several tables and joined on `dicom_patient_id` - and one of those tables was contributed by an analysis result rather than by the original submitters. ### Digital pathology: TCGA-LUAD The same model applies to slide microscopy, where the INSTANCE level looks quite different. Patient `TCGA-80-5608` from the [TCGA-LUAD](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=tcga_luad) collection ([open in the SLIM viewer](https://viewer.imaging.datacommons.cancer.gov/slim/studies/2.25.67565461533469433863078107064394326180)): ``` PROGRAM tcga └── COLLECTION tcga_luad (TCGA-LUAD) └── CASE TCGA-80-5608 └── STUDY 2.25.67565461533469433863078107064394326180 │ ├── SM series "FFPE HE TP DX1" DOI 10.5281/zenodo.12689915 CC BY 3.0 since v8 │ └── 4 INSTANCEs, one per pyramid level: │ 21987 × 17849 (0.50 µm/px, base layer, 128 MB) │ 5496 × 4462 (2.00 µm/px) │ 2748 × 2231 (4.00 µm/px) │ 946 × 768 (11.6 µm/px, thumbnail) │ ├── ANALYSIS_RESULT pan_cancer_nuclei_seg_dicom CC BY 4.0 │ ├── ANN series 1 INSTANCE (nuclei as bulk annotations) since v19 │ └── SEG series 4 INSTANCEs (nuclei as segmentations) since v20 │ └── ANALYSIS_RESULT tcga_sbu_til_maps CC BY 4.0 since v23 ├── SEG "Stony Brook CNN-generated TIL Map" ├── SEG "Stony Brook Inception-V4 Binary TIL Map" └── SEG "Stony Brook Inception-V4 Fractional TIL Map" ``` Two additional points this example makes: * **In pathology, one instance is one pyramid layer**, not one slice. The whole slide image above is a single series of 4 files, ranging from a 128 MB base layer at 0.5 µm/px down to a thumbnail. (A small number of series in IDC split a single layer across several instances.) * **A single analysis result can span many collections.** `tcga_sbu_til_maps` covers 23 TCGA collections and `pan_cancer_nuclei_seg_dicom` covers 14 - this is the many-to-many relationship between ANALYSIS\_RESULT and COLLECTION in the diagram above. A single analysis result can also contribute more than one type of object, as `pan_cancer_nuclei_seg_dicom` does with ANN and SEG series. ## Where each entity lives in the metadata The table below maps each entity to the identifier you would use in [`idc-index`](https://github.com/ImagingDataCommons/idc-index). The same DICOM identifiers are used in the BigQuery `dicom_all` table, which additionally exposes every other DICOM attribute - see [Files and metadata](/data/organization-of-data/files-and-metadata.md).
EntityIdentifierWhere to find it
PROGRAMprogram_idcollections_index
COLLECTIONcollection_idindex, collections_index
CASEPatientIDindex
STUDYStudyInstanceUIDindex
SERIESSeriesInstanceUIDindex
INSTANCESOPInstanceUIDBigQuery dicom_all; sm_instance_index for slide microscopy
ANALYSIS_RESULTanalysis_result_idindex, analysis_results_index
CLINICAL_DATAdicom_patient_idper-collection clinical tables, see clinical.md
Note that the identifiers above are the ones you search with. The files themselves are named using IDC-assigned UUIDs (`crdc_series_uuid` and `crdc_instance_uuid`) so that IDC can support versioning - see [UUIDs and GUIDs](/data/organization-of-data/guids-and-uuids.md).