Highlights

• >85 TB of data: IDC contains radiology, brightfield (H&E) and fluorescence slide microscopy images, along with image-derived data (annotations, segmentations, quantitative measurements) and accompanying clinical data

• free: all of the data in IDC is publicly available: no registration, no access requests

• commercial-friendly: >95% of the data in IDC is covered by the permissive CC-BY license, which allows commercial reuse (small subset of data is covered by the CC-NC license); each file in IDC is tagged with the license to make it easier for you to understand and follow the rules

• cloud-based: all of the data in IDC is available from both Google and AWS public buckets: fast and free to download, no out-of-cloud egress fees

• harmonized: all of the images and image-derived data in IDC is harmonized into standard DICOM representation

Functionality

IDC is as much about data as it is about what you can do with the data! We maintain and actively develop a variety of tools that are designed to help you efficiently navigate, access and analyze IDC data:

• visualization: examine images and image-derived annotations and analysis results from the convenience of your browser using integrated OHIF, VolView and Slim open source viewers

• cohort building: use rich and extensive metadata to build subsets of data programmatically using idc-index or BigQuery SQL

• download: use your favorite S3 API client or idc-index to efficiently fetch any of the IDC files from our public buckets

Easy and efficient access to public cancer imaging data

We ingest and distribute datasets from variety of sources and contributors, primarily focusing on large data collection initiatives sponsored by US National Cancer Institute.

On ingestion, we harmonize images and image-derived data into DICOM format for interoperability, whenever data is represented in a non-DICOM format.

Upon conversion, the data undergoes Extract-Transform-Load (ETL), which extracts DICOM metadata to make the data searchable, ingests the DICOM files into public S3 storage buckets and a DICOMweb store. Once the data is released, we provide various interfaces to access data and metadata.

Tools to simplify the use of the data

We are actively developing a variety of capabilities to make it easier for the users to work with the data in IDC. Some of the examples of those tools include

Support of continuous enrichment of data

We welcome you to apply to contribute analysis results and annotations of the images available in IDC! These can be expert manual annotations, analysis results generated using AI tools, segmentations, contours, metadata attributes describing the data (e.g., annotation of the scan type), expert evaluation of the quality of existing AI-generated annotations in IDC.

If your contribution is accepted by the IDC stakeholders:

• we will work with you to choose the appropriate DICOM object type for your data and convert it into DICOM representation

• once published in IDC

  • your data will become searchable and viewable in IDC Portal, so it is easier for the users of your data to discover and work with your data

  • files can be downloaded very efficiently using S3 interface and idc-index

Integration of cancer imaging data with other components of CRDC

Discounted use and training materials for NIH-funded investigators

Publications by the IDC team

Publications referencing IDC (a subset)

We want Imaging Data Commons to be your companion in your cancer imaging research activities - from discovering relevant data to sharing your analysis results and showcasing the tools you developed!

Explore the data available

IDC Portal is integrated with powerful visualization tools: just with your web browser you will be able to see IDC images and annotations using OHIF Viewer, Slim viewer and VolView!

Subset the content you need

We have many tools to help you search data in IDC, so that you download only what you need!

Download the data you liked

• once you have idc-index python package installed, download from the command line is as easy as running idc download <manifest_file>, or idc download <collection_id>.

Experiment with analysis tools

We want to make it easier to understand performance of the latest advances in AI on real-world cancer imaging data!

Scale the analysis to thousands of cloud VMs

With the cloud, you can do things that are simply impossible to do with your local resources.

Share analysis results or annotations

If you have an algorithm, that you evaluated/published, that can enrich data in IDC with analysis results and you want to contribute those, or if you are a domain expert and would like to publish results of manual annotations you prepared - we want to hear from you!

• through a dedicated Zenodo record you will have a citation and DOI to get credit for your work; your data is ingested from Zenodo into IDC, and citation will be generated for the users of your data in IDC

Questions?

Imaging Data Commons is being developed by a team of engineers and imaging scientists with decades of experience in cancer imaging informatics, cloud computing, imaging standards, security, open source tool development and data sharing.

Our team includes the following sites and project leads:

• Brigham and Women's Hospital, Boston, MA, USA (BWH)

  • Andrey Fedorov, PhD, and Ron Kikinis, MD - Co-PIs of the project

  • Hugo Aerts, PhD

  • Cosmin Ciausu, MS

  • Deepa Krishnaswamy, PhD

  • Katie Mastrogiacomo

  • Maria Loy

• Institute for Systems Biology, Seattle, WA, USA (ISB)

  • David Gibbs, PhD - site PI

  • William Longabaugh, MS

  • William Clifford, MS

  • Suzanne Paquette, MS

  • George White

  • Ilya Shmulevich, PhD

• General Dynamics Information Technology, Bethesda, MD, USA (GDIT)

  • David Pot, PhD - site PI

  • Poojitha Gundluru

  • Fabian Seidl

  • Prema Venkatesun

  • Anthony Le

• Fraunhofer MEVIS, Bremen, Germany (Fraunhofer MEVIS)

  • André Homeyer, PhD - site PI

  • Daniela Schacherer, MS

  • Henning Höfener, PhD

• Massachusetts General Hospital, Boston, MA, USA (MGH)

  • Chris Bridge, DPhil - site PI

  • Chris Gorman, PhD

• Radical Imaging LLC, Boston, MA, USA (Radical Imaging)

  • Rob Lewis, PhD - site PI

  • Igor Octaviano

  • Pedro Kohler

• PixelMed Publishing, Bangor, PA, USA (PixelMed)

  • David Clunie, MB, BS - site PI

• Isomics Inc, Cambridge, MA, USA (Isomics)

  • Steve Pieper, PhD - site PI

Oversight:

• Leidos Biomedical Research

  • Ulrike Wagner - project manager

  • Todd Pihl - project manager

• National Cancer Institute

  • Erika Kim - federal lead

  • Granger Sutton - federal lead

IDC Alumni

We are grateful to the following individuals who contributed to IDC in the past, but are no longer directly involved in the development of IDC.

• Keyvan Farahani (NCI)

• Markus Herrmann (MGH)

• Davide Punzo (Radical Imaging)

• James Petts (Radical Imaging)

• Erik Ziegler (Radical Imaging)

• Gitanjali Chhetri (Radical Imaging)

• Rodrigo Basilio (Radical Imaging)

• Jose Ulloa (Radical Imaging)

• Madelyn Reyes (GDIT)

• Derrick Moore (GDIT)

• Mark Backus (GDIT)

• Rachana Manandhar (BWH)

• Rasmus Kiehl (Fraunhofer MEVIS)

• Chad Osborne (GDIT)

• Afshin Akbarzadeh (BWH)

• Dennis Bontempi (BWH)

• Vamsi Thiriveedhi (BWH)

• Jessica Cienda (GDIT)

• Bernard Larbi (GDIT)

• Mi Tian (ISB)

IDC does not currently have open positions

IDC data model

Resources maintained by the IDC team

Other locations for accessing public imaging data

If you did not find the images you need in IDC, you can consider the following resources:

• Imaging Data Commons team has been funded in whole or in part with Federal funds from the National Cancer Institute, National Institutes of Health, under Task Order No. HHSN26110071 under Contract No. HHSN261201500003l.

How to download data from IDC?

Check out the Downloading data documentation page!

How do I get my data into IDC?

Note that currently IDC prioritizes submissions from NCI-funded driving projects and data from special selected projects.

• If you would like to submit images, it will be your responsibility to de-identify them first, documenting the de-identification process and submitting that documentation for the review by IDC stakeholders.

How much does it cost to use the cloud?

What is the status of IDC?

IDC pilot release took place in Fall 2020, followed by the production release in September 2021.

What data is available?

How to acknowledge IDC?

Please cite the latest paper from the IDC team. Please also make sure you acknowledge the specific data collections you used in your analysis.

What is the difference between IDC and TCIA?

IDC and TCIA are partners in providing FAIR data for cancer imaging researchers. While some of the functions between the two resources are similar, there are also key differences. The table below provides a summary of similarities and differences.

Where do I learn more about other components of CRDC?

What about non-imaging data that accompanies IDC collections?

I want to search IDC content using an attribute not available in the portal

IDC Portal gives you access to just a small subset of the metadata accompanying IDC images. If you want to learn more about what is available, you have several options:

Let's start with the overall principles of how we organize data in IDC.

IDC brings you (as of v18) over 60 TB of publicly available DICOM images and image-derived content. We share those with you as DICOM files, and those DICOM files are available in cloud-based storage buckets - both in Google and AWS.

Sharing just the files, however, is not particularly helpful. With that much data, it is no longer practical to just download all of those files to later sort through them to select those you need.

To provide you with a catalog of our data, along with the files, we maintain metadata that makes it possible to understand what is contained within files, and select the files that are of interest for your project, so that you can download just the files you need. We make that metadata available in BigQuery tables searchable using standard SQL.

BigQuery Tables and Views

IDC utilizes BigQuery tables to organize metadata accompanying the files we host. If you have never worked with BigQuery before, you need to understand the basics of data organization in BQ.

BQ tables are organized in BQ datasets. BQ datasets are not unlike folders on your computer, but contain tables related to each other instead of files. BQ datasets, in turn, are organized under Google Cloud projects. GCP projects can be thought of as containers that are managed by a particular organization. To continue with the file system analogy, think about projects as hard drives that contain folders.

Let's map the aforementioned project-dataset-table hierarchy to the concrete locations that contain IDC data.

IDC BigQuery datasets

All of the IDC tables are stored under the bigquery-public-data project. That project is managed by Google Public Datasets Program, and contains many public BQ datasets, beyond those maintained by IDC.

All of the IDC tables are organized into datasets by data release version. If you complete the tutorial mentioned above, open the BQ console, and scroll down the list of datasets, you will find those that are named starting with the idc_v prefix - those are IDC datasets.

Following the prefix, you will find the number that corresponds to the IDC data release version. IDC data releases version numbers start from 1 and are incremented by one for each subsequent release. As of writing this, the most recent version of IDC is 16, and you can find dataset idc_v16 corresponding to this version.

Finally, you will also see two special datasets: idc_current and idc_current_clinical. Those two datasets are essentially aliases, or links, to the versioned datasets corresponding to the latest release of IDC data.

IDC BigQuery tables

BQ views can be very handy when you want to simplify your queries by factoring out the part of the query that is often reused. But a key disadvantage of BQ views over tables is the reduced performance and increased cost due to re-running the query each time you query the view.

As we will discuss further, most of the tables maintained by IDC are created by joining and/or post-processing other tables. Because of this we rely heavily on BQ views to improve transparency of the provenance of those "derived" tables. BQ views can be easily distinguished from the tables in a given dataset by a different icon. IDC datasets also follow a convention that all views in the versioned datasets include suffix _view in the name, and are accompanied by the result of running the query used by the view in a table that has the same name sans the _view suffix. See the figure below for an illustration of this convention.

Now that we reviewed the main concepts behind IDC tables organization, it is time to explain the sources of metadata contained in those tables. Leaving _clinical datasets aside, IDC tables are populated from one of the two sources:

• DICOM metadata extracted from the DICOM files hosted by IDC, and various derivative tables that simplify access to specific DICOM metadata items;

• collection-level and auxiliary metadata, which is not stored in DICOM tags, but is either received by IDC from other sources, or is populated by IDC as part of data curation (these include Digital Object Identifiers, description of the collections, hashsums, etc).

The set of BQ tables and views has grown over time. The enumeration below documents the BQ tables and views as of IDC v14. Some of these tables will not be found in earlier IDC BigQuery datasets.

dicom_metadata

Each row in the dicom_metadata table holds the DICOM metadata of an instance in the corresponding IDC version. There is a single row for each DICOM instance in the corresponding IDC version, and the columns correspond to the DICOM attributes encountered in the data across all of the ingested instances.

dicom_metadata table contains DICOM metadata extract from the files included in the given IDC data release. The amount and variety of the DICOM files grows with the new releases, and the schema of this table reflects the organization of the metadata in each IDC release. Non-sequence attributes, such as Modality or SeriesInstanceUID, once encountered in any one file will result in the corresponding column being introduced to the table schema (i.e., if we have column X in IDC release 11, in all likelihood it will also be present in all of the subsequent releases).

dicom_metadata can be used to conduct detailed explorations of the metadata content, and build cohorts using fine-grained controls not accessible from the IDC portal. Note that the dicom_all table, described below, is probably a better choice for such explorations.

auxiliary_metadata

This table defines the contents of the corresponding IDC version. There is a row for each instance in the version. We group the attributes for convenience:

Collection attributes:

• tcia_api_collection_id: The ID, as accepted by the TCIA API, of the original data collection containing this instance (will be Null for collections not sourced from TCIA)

• idc_webapp_collection_id: The ID, as accepted by the IDC web app, of the original data collection containing this instance

• collection_id: The ID, as accepted by the IDC web app. Duplicate of idc_webapp_collection_id

• collection_timestamp: Datetime when the IDC data in the collection was last revised

• collection_hash: md5 hash of the of this version of the collection containing this instance

• collection_init_idc_version: The IDC version in which the collection containing this instance first appeared

• collection_revised_idc_version: The IDC version in which this version of the collection containing this instance first appeared

Patient attributes:

• submitter_case_id:The Patient ID assigned by the submitter of this data. This is the same as the DICOM PatientID

• idc_case_id:IDC generated UUID that uniquely identifies the patient containing this instance

  This is needed because DICOM PatientIDs are not required to be globally unique

• patient_hash: md5 hash of this version of the patient/case containing this instance

• patient_init_idc_version: The IDC version in which the patient containing this instance first appeared

• patient_revised_idc_version: The IDC version in which this version of the patient/case containing this instance first appeared

Study attributes:

• StudyInstanceUID: DICOM UID of the study containing this instance

• study_uuid:IDC assigned UUID that identifies a version the the study containing this instance.

• study_instances: The number instances in the study containing this instance

• study_hash: md5 hash of the data in this version of the study containing this instance

• study_init_idc_version: The IDC version in which the study containing this instance first appeared

• study_revised_idc_version: The IDC version in which this version of the study containing this instance first appeared

Series attributes:

• SeriesInstanceUID: DICOM UID of the series containing this instance

• series_uuid:IDC assigned UUID that identifies the version of the series containing this instance

• source_doi:The DOI of an information page corresponding to the original data collection or analysis results that is the source of this instance

• source_url:The URL of an information page that describes the original collection or analysis result that is the source of this instance

• series_instances: The number of instances in the series containing this instance

• series_hash: md5 hash of the data in the this version of the series containing this instance

• access: Collection access status: 'Public' or 'Limited'. (Currently all data is 'Public')

• series_init_idc_version: The IDC version in which the series containing this instance first appeared

• series_revised_idc_version: The IDC version in which this version of the series containing this instance first appeared

Instance attributes:

• SOPInstanceUID: DICOM UID of this instance.

• instance_uuid:IDC assigned UUID that identifies the version of this instance.

• gcs_url: The GCS URL of a file containing the version of this instance that is identified by this series_uuid/instance_uuid

• aws_url: The AWS URL of a file containing the version of this instance that is identified by this series_uuid/instance_uuid

• instance_hash: the md5 hash of this version of this instance

• instance_size: the size, in bytes, of this version of this instance

• instance_init_idc_version: The IDC version in which this instance first appeared

• instance_revised_idc_version: The IDC version in which this version of this instance first appeared

• license_url: The URL of a web page that describes the license governing this version of this instance

• license_long_name: A long form name of the license governing this version of this instance

• license_short_name: A short form name of the license governing this version of this instance

mutable_metadata

Some non-DICOM metadata may change over time. This includes the GCS and AWS URLs of instance data, the accessibility of each instance and the URL of an instance's associated description page. BigQuery metadata tables such as the auxiliary_metadata and dicom_all tables are never revised even when such metadata changes. However, tables in the datasets of previous IDC versions can be joined with the mutable_metadata table to obtain the current values of these mutable attributes.

The table has one row for each version of each instances:

• crdc_instance_uuid: The uuid of an instance version

• crdc_series_uuid: The uuid of a series version that contains this instance version

• crdc_study_uuid: The uuid of a study version that contains the series version

• gcs_url: URL to the Google Cloud Storage (GCS) object containing this instance version

• aws_url: URL to the Amazon Web Services (AWS) object containing this instance version

• `access: Current access status of this instance (Public or Limited)

• source_url: The URL of a page that describes the original collection or analysis result that includes this instance

• source_doi: The DOI of a page that describes the original collection or analysis result that includes this instance

original_collections_metadata

• tcia_api_collection_id: The collection ID as is accepted by the TCIA AP

• tcia_wiki_collection_id: The collection ID as on the TCIA wiki page

• idc_webapp_collection_id:The collection ID as accepted by the IDC web app

• Program: The program to which this collection belongs

• Updated: Most recent update date reported by the collection source

• Status:Collection status: "Ongoing" or "Complete"

• Access:Collection access conditions: "Limited" or "Public"

• ImageType: Enumeration of image types/modalities in the collection

• Subjects:Number of subjects in the collection

• DOI:DOI that can be resolved at doi.org to the TCIA wiki page for this collection

• URL:URL of an information page for this collection

• CancerType:Collection source(s) assigned cancer type of this collection

• SupportingData:Type(s) of additional data available

• Species: Species of collection subjects

• Location:Body location that was studied

• Description: Description of the collection (HTML format)

• license_url: The URL of a web page that describes the license governing this collection

• license_long_name: A long form name of the license governing this collection

• license_short_name: A short form name of the license governing this collection

analysis_results_metadata

• ID: Results ID

• Title: Descriptive title

• DOI:DOI that can be resolved at doi.org to the TCIA wiki page for this analysis result

• CancerType:TCIA assigned cancer type of this analysis result

• Location:Body location that was studied

• Subjects:Number of subjects in the analysis result

• Collections: Original collections studied

• AnalysisArtifactsonTCIA: Type(s) of analysis artifacts generated

• Updated: Data when results were last updated

• license_url: The URL of a web page that describes the license governing this collection

• license_long_name: A long form name of the license governing this collection

• license_short_name: A short form name of the license governing this collection

• description: Description of analysis result

version_metadata

Metadata for each IDC version, one row per version:

• idc_version: IDC version number

• version_hash: MD5 hash of hashes of collections in this version

• version_timestamp: Version creation timestamp

The following tables and views consist of metadata derived from one or more other IDC tables tables for convenience of the user. For each such table, <table_name>, there is also a corresponding view, <table_name>_view, that, when queried, generates an equivalent table. These views are intended as a reference; each view's SQL is available to be used for further investigation.

dicom_all, dicom_all_view

All columns from dicom_metadata together with selected date from the auxiliary_metadata, original_collections_metadata, and analysis_results_metadata tables.

segmentations, segmentations_view

This table is derived from dicom_all to simplify access to the attributes of DICOM Segmentation objects available in IDC. Each row in this table corresponds to one DICOM Segmentation instance segment.

measurement_groups, measurement_groups_view

Each row corresponds to one TID1500 measurement group.

qualitative_measurements, qualitative_measurements_view

This table is derived from dicom_all to simplify access to the qualitative measurements in DICOM SR TID1500 objects. It contains coded evaluation results extracted from the DICOM SR TID1500 objects. Each row in this table corresponds to a single qualitative measurement extracted.

quantitative_measurements, quantitative_measurements_view

This table is derived from dicom_all to simplify access to the quantitative measurements in DICOM SR TID1500 objects. It contains quantitative evaluation results extracted from the DICOM SR TID1500 objects. Each row in this table corresponds to a single quantitative measurement extracted.

dicom_metadata_curated, dicom_metadata_curated_view

Curated values of DICOM metadata extracted from dicom_metadata.

dicom_metadata_curated_series_level, dicom_metadata_curated_series_level_view

Curated columns from dicom_metadata that have been aggregated/cleaned up to describe content at the series level. Each row in this table corresponds to a DICOM instance in IDC. The columns are curated by defining queries that apply transformations to the original values of DICOM attributes.

idc_pivot_v<idc version>

A view that is the basis for the queries performed by the IDC web app.

Collection-specific BigQuery tables

TCGA

The following tables contain TCGA-specific metadata:

• tcga_biospecimen_rel9: biospecimen metadata

• tcga_clinical_rel9: clinical metadata

NLST

• nlst_canc: "Lung Cancer"

• nlst_ctab: "SCT Abnormalities"

• nlst_ctabc: "SCT Comparison Abnormalities"

• nlst_prsn: "Participant"

• nlst_screen: "SCT Screening"

Storage Buckets

The object namespace is hierarchical, where, for each version of a DICOM instance having instance UUID <instance_uuid> in a version of a series version having UUID <series_uuid>, the file name is:

<series_uuid>/<instance_uuid>.dcm

Corresponding files have the same object name in GCS and S3, though the name of the containing buckets will be different.

UIDs and UUIDs explained with an example

Consider an instance in the CPTAC-CM collection that has this SOPInstanceUID: 1.3.6.1.4.1.5962.99.1.171941254.777277241.1640849481094.35.0\

It is in a series having this SeriesInstanceUID: 1.3.6.1.4.1.5962.99.1.171941254.777277241.1640849481094.2.0

The instance and series were added to the IDC Data set in IDC version 7. At that point, the instance was assigned UUID: 5dce0cf0-4694-4dff-8f9e-2785bf179267 and the series was assigned this UUID: e127d258-37c2-47bb-a7d1-1faa7f47f47a

In IDC version 10, a revision of this instance was added (keeping its original SOPInstanceUID), and assigned this UUID: 21e5e9ce-01f5-4b9b-9899-a2cbb979b542

Because this instance was revised, the series containing it was implicitly revised. The revised series was thus issued a new UUID: ee34c840-b0ca-4400-a6c8-c605cef17630

Thus, the initial version of this instance has this file name: e127d258-37c2-47bb-a7d1-1faa7f47f47a/5dce0cf0-4694-4dff-8f9e-2785bf179267.dcm and the revised version of the instance has the this file name: ee34c840-b0ca-4400-a6c8-c605cef17630/21e5e9ce-01f5-4b9b-9899-a2cbb979b542.dcm

Both versions of the instance are in both AWS and GCS buckets.

Note that GCS and AWS bucket names are different. In fact, DICOM instance data is distributed across multiple buckets in both GCS and AWS. We will discuss obtaining GCS and AWS URLs more a little later.

Utilities like gsutil, s3 and s5cmd "understand" the implied hierarchy in these file names. Thus the series UUID now acts like the name of a directory that contains all the instance versions in the series version:

and similarly for AWS buckets, thus making it easy to transfer all instances in a series from the cloud.

Because file names are more or less opaque, the user will not typically select files by listing the contents of a bucket. Instead, one should use either the IDC Portal or IDC BigQuery tables to identify items of interest and, then, generate a manifest of objects that can be passed to a utility like s5cmd.

DICOM Stores

Data sources

Most of the data in IDC is received from the data collection initiatives/projects supported by US National Cancer Institute. Whenever source images or image-derived data is not in the DICOM format, it is harmonized into DICOM as part of the ingestion.

As of data release v21, IDC sources of data include:

• • all DICOM files from the public collections are mirrored in IDC

  • a subset of digital pathology collections and analysis results harmonized from vendor-specific representation (as available from TCIA) into DICOM Slide Microscopy (SM) format

• • digital pathology slides harmonized into DICOM SM

• • The Cancer Genome Atlas (TCGA) slides harmonized into DICOM SM

• • release 1 of the HTAN data harmonized into DICOM SM

• • v1 of the Visible Human images harmonized into DICOM MR/CT/XC

• • digital pathology slides harmonized into DICOM SM

Data provenance

Whenever IDC replicates data from a publicly available source, we include the reference to the origin:

• from the IDC Portal Explore page, click on the "i" icon next to the collection in the collections list

Data ingestion process

Simplified workflow for IDC data ingestion is summarized in the following diagram.

IDC V14 introduced important enhancements to IDC data organization. The discussion of the organization of data in earlier versions is preserved here.

Background

By clinical data we refer to the broad spectrum of image-related data that may accompany images. Such data may include demographics of the patients, observations related to their clinical history (therapies, diagnoses, findings), lab tests, surgeries.

Not only the terms used in the clinical data accompanying individual collection are not harmonized, but the format of the spreadsheets is also collection-specific. In order to search and navigate clinical data, one has to parse those collection specific tables, and there is no interface to support searching across collections.

Clinical data BigQuery tables

• collection_id (STRING, NULLABLE) - the collection_id of the collection in the given table. The collection id is in a format used internally by the IDC Web App (with only lowercase letters, numbers and '_' allowed). It is equivalent to the idc_webapp_id field in the dicom_all view in the idc_current dataset.

• table_name (STRING,NULLABLE) - name of the table

• table_description (STRING,NULLABLE) - description of the type of data found in the table. Usually this is set to 'clinical data', unless a description is provided in the source files

• idc_version_table_added (STRING, NULLABLE) - the IDC data version for which this table was first added

• idc_table_added_datetime (STRING,NULLABLE) - the date/time this particular table was first generated

• post_process_src (STRING, NULLABLE) - except for the CPTAC and TCGA collections the tables are curated from ZIP, Excel, and CSV files downloaded from the TCIA wiki. These files do not have a consistent structure and were not meant to be machine readable or to translate directly into BigQuery. A semi-manual curation process results in either a CSV of JSON file that can be directly written into a BigQuery table. post_process_src is the name of the JSON or CSV file that results from this process and is used to create the BigQuery table. This field is not used for the CPTAC- and TCGA-related tables

• post_process_src_add_md5 (STRING, NULLABLE) - the md5 hash of post_process_src when the table was first added

• idc_version_table_prior (STRING, NULLABLE) - the idc version the second most recent time the table was updated

• post_process_src_prior_md5 (STRING, NULLABLE) - the md5 hash of post_process_src the second most recent time the table was updated

• idc_version_table_updated (STRING, NULLABLE) - the idc version when the table was last updated

• table_update_datetime (STRING, NULLABLE) - date and time an update of the table was last recorded

• post_process_src_updated_md5 (STRING, NULLABLE) - the md5 hash of post_process_source when the table was last updated

• number_batches (INTEGER, NULLABLE) - records the number of batches. Within the source data patients are sometimes grouped into different 'batches' (i.e. training vs test, responder vs non-responder etc.) and the batches are placed in different locations (i.e. different files or different sheets in the same Excel file)

• source_info (RECORD, REPEATED) - an array of records with information about the table sources. These sources are either files downloaded from the TCIA wiki or another BigQuery table (as is the case for CPTAC and TCGA collections). There is a source_info record for each source 'batch' described above

• source_info.srcs (STRING, REPEATED) - a source file downloaded from the TCIA wiki may be a ZIP file, and CSV file, or an Excel file. Sometimes the ZIP files contain other ZIP files that must be opened to extract the clinical data. In the source_info.src array the first string is the file that is downloaded from TCIA for this particular source batch. The final string is the CSV or Excel file that contains the clinical data. Any intermediate strings are the names of ZIP files 'in between' the downloaded file and the clinical file. For CPTAC and TCGA collections this field contains the source BigQuery table

• source_info.md5 (STRING, NULLABLE) - md5 hash of the downloaded file from TCIA the most recent time the table was updated

• source_info.table_last_modified (STRING, NULLABLE) - CPTAC and TCGA collections only. The date and time the source BigQuery table was most recently modified, as recorded when last copied

• source_info.table_size (STRING, NULLABLE) - CPTAC and TCGA collections only. The size of the source BigQuery table as recorded when last copied

• collection_id (STRING,NULLABLE) - the collection_id of the collection in the given table. The collection id is in a format used internally by the IDC Web App (with only lowercase letters, numbers and '_' allowed). It is equivalent to the idc_webapp_id field in the dicom_all view in the idc_current dataset.

• case_col (BOOLEAN, NULLABLE) - true if the BigQuery column contains the patient or case id, i.e. if this column is used to determine the value of the dicom_patient_id column

• table_name (STRING, NULLABLE) - table name

• column (STRING, NULLABLE) - the actual column name in the table. For ACRIN collections the column_name is the variable_name from the provided data dictionary. For other collections it is a name constructed by 'normalizing' the column_label (see next) in a format that can be used as a BigQuery field name

• column_label (STRING, NULLABLE) - a 'free form' label for the column that does not need to conform to the BigQuery column format requirements. For ACRIN collections this is the variable_label given by a data dictionary that accompanies the collection. For other collections it is the name or label of the clinical attribute as inferred from the source document during the curation process

• data_type (STRING, NULLABLE) - the type of data in this column. Again for ACRIN collections this is provided in the data dictionary. For other collections it is inferred by analyzing the data during curation

• original_column_headers (STRING, REPEATED) - the name(s) or label(s) in the source document that were used to construct the column_label field. In most cases there is one column label in the source document that perscribes the column_label. In some cases, multiple columns are concantenated and reformated to form the column_label

• values (RECORD, REPEATED) - a structure that is borrowed from the ACRIN data model. This is an array that contains observerd attribute values for this given column. For ACRIN collections these values are reported in the data dictionary. For most other collections these values are determined by analyzing the source data. For simplicity this field is left blank when the number of unique values is greater than 20

• values.option_code (STRING, NULLABLE) - a unique attribute value found in this column

• values.option_description (STRING, NULLABLE) - a description of the option_code as provided by a data dictionary. For collections that do not have a data dictionary this is null.

• values_source (STRING, NULLABLE) - indicates the source of the values records. The text 'provided dictionary' indicates that the records were obtained from a provided data dictionary. The text 'derived from inspection of values' indicates that the records were determined by automated analysis of the source materials during the ETL process that generated the BigQuery tables.

• files (STRING, REPEATED) - names of the files that contain the source data for each batch. These are the Excel or CSV files directly downloaded from TCIA, or the files extracted from downloaded ZIP files

• sheet_names (STRING, REPEATED) - for Excel-sourced files, the sheet names containing this column's values for each batch

• batch (INTEGER, REPEATED) - source batches that contain this particular column. Some columns or attributes may be missing from some batches

• column_numbers (STRING, REPEATED) - for each source batch, the column in the original source corresponding to this column in the BigQuery table

IDC content is organized in Collections: groups of DICOM files that were collected through certain research activity.

Individual DICOM files included in the collection contain attributes that organize content according to the DICOM data model.

Each collection will contain data for one or more case, or patient. Data for the individual patient is organized in DICOM studies, which group images corresponding to a single imaging exam/enconter, and collected in a given session. Studies are composed of DICOM series, which in turn consist of DICOM instances. Each DICOM instance correspond to a single file on disk. As an example, in radiology imaging, individual instances would correspond to image slices in multi-slice acquisitions, and in digital pathology you will see a separate file/instance for each resolution layer of the image pyramid. When using IDC Portal, you will never encounter individual instances - you will only see them if you download data to your computer.

Analysis results collection is a very important concept in IDC. These contain analysis results that were not contributed as part of any specific collection. Such analysis results might be contributed by investigators unrelated to those that submitted the analyzed images, and may span images across multiple collections.

Summary

When you work with IDC data at any given time, you should be aware of the data release version. If you build cohorts using filters or queries, the result of those queries will change as the IDC content is evolving. Building queries that refer to the specific data release version will ensure that the result is the same.

Here is how you can learn what version of IDC data you are interacting with, depending on what interface to the data you are using:

• IDC Portal: data version and release date are displayed in the summary strip
• idc-index: use get_idc_version()function

from idc_index import IDCClient

idc_version = IDCClient.get_idc_version()

• 3D Slicer / SlicerIDCBrowser: version information is provided in the SlicerIDCBrowser module top panel, and in the pop-up window title.

Implementation details

The IDC obtains curated DICOM radiology, pathology and microscopy image and analysis data from The Cancer Imaging Archive (TCIA) and additional sources. Data from all these sources evolves over time as new data is added (common), existing files are corrected (rare), or data is removed (extremely rare).

Users interact with IDC using one of the following interfaces to define cohorts, and then perform analyses on these cohorts:

The goal of IDC versioning is to create a series of "snapshots” over time of the entirety of the evolving IDC imaging dataset, such that searching an IDC version according to some criteria (creating a cohort) will always identify exactly the same set of objects. Here “identify” particularly means providing URLs or other access methods to the corresponding physical data objects.

In order to reproduce the result of such analysis, it must be possible to precisely recreate a cohort. For this purpose an IDC cohort as defined in the Portal is specified and saved as a filter applied against a specified IDC data version. Alternatively, the cohort can be defined as an SQL query, or as a list of unique identifiers selecting specific files within a defined data release version.

Because an IDC version exactly defines the set of data against which the filter/query is applied, and because all versions of all data, except data removed due to PHI/PII concerns, should continue to be available, a cohort is therefore persistent over the course of the evolution of IDC data.

DICOM Entities are versioned

There are various reasons that can cause modification of the existing collections in IDC:

• images for new patients can be added to an existing collections;

• additional DICOM series are sometimes added to a DICOM study over time (i.e., those that contain new annotations or analysis results);

• a series may be added or removed from an existing study;

• metadata of an existing instance might be corrected (which may or may not lead to an update of the DICOM SOPInstanceUID corresponding to the instance).

These and other possible changes mean that DICOM instances, series and studies can change from one IDC data version to the next, while their DICOM UIDs remain unchanged. This motivates the need for maintaining versioning of the DICOM entities.

The data in each IDC version, then, can be thought of as some set of versioned DICOM instances, series and studies. This set is defined in terms of the corresponding set of instance UUIDs, series UUIDs and study UUIDs. This means that if, e.g., some version of an instance having UUID UUIDx that was in IDC version Vm is changed, a new UUID, UUIDy, will be assigned to the new instance version. Subsequent IDC versions, Vm+1, Vm+2, ... will include that new instance version identified by UUIDy unless and until that instance is again changed. Similarly if the composition of some series changes, either because an instance in the series is changed, or an instance is added or removed from that series, a new UUID is assigned to the new version of that series and identifies that version of the series in subsequent IDC versions. Similarly, a study is assigned a new UUID when its composition changes.

A corollary is that only a single version of an instance, series or study is in an IDC version.

Note that instances, series and studies do not have an explicit version number in their metadata. Versioning of an object is implicit in the associated UUIDs.

This is a typical IDC instance UUID: 641121f1-5ca0-42cc-9156-fb5538c14355 of a (version of a) DICOM instance, and this is the corresponding DRS ID: dg.4DFC/641121f1-5ca0-42cc-9156-fb5538c14355

A DRS ID can be resolved by appending it to the following URL, which is the resolution service within CRDC: https://nci-crdc.datacommons.io/ga4gh/drs/v1/objects/ . For example, the following curl command:

>> curl https://nci-crdc.datacommons.io/ga4gh/drs/v1/objects/dg.4DFC/641121f1-5ca0-42cc-9156-fb5538c14355

returns this DrsObject:

AS can be seen, the access_methods component in the returned DrsObject includes a URL for each of the corresponding files in Google GCS and AWS S3.

Storage Buckets

Storage buckets are named using the format idc-tcia-<TCIA_COLLECTION_NAME>, where TCIA_COLLECTION_NAME corresponds to the collection name in the collections table here.

Within the bucket, DICOM files are organized using the following directory naming conventions:

dicom/<StudyInstanceUID>/<SeriesInstanceUID>/<SOPInstanceUID>.dcm

where *InstanceUIDs correspond to the respective value of the DICOM attributes in the stored DICOM files.

BigQuery Tables

IDC users can access this table to conduct detailed exploration of the metadata content, and build cohorts using fine-grained controls not accessible from the IDC portal.

In addition to the DICOM metadata tables, we maintain several additional tables that curate metadata non-DICOM metadata (e.g., attribution of a given item to a specific collection and DOI, collection-level metadata, etc).

DICOM Stores

BigQuery tables external to IDC

In addition to the DICOM data, some of the image-related data hosted by IDC is stored in additional tables. These include the following:

Depending on whether you would like to download data interactively or programmatically, we provide two recommended tools to help you.

Command-line or programmatic download: idc-index python package

Command line download interface

With the idc-index package you get command line scripts that aim to make download simple.

Have a .s5cmd manifest file you downloaded from IDC Portal or from the records in the IDC Zenodo community? Get the corresponding files as follows (you will also get download progress bar and the downloaded files will be organized in the collection/patient/study/series folder hierarchy!):

You can use the same command to download files corresponding to any collection, patient, study or series, referred to by the identifiers you can copy from the portal!

Similarly, you can copy identifiers for patient/study/series and download the corresponding content!

Programmatic download

Interactive download: 3D Slicer SlicerIDCBrowser extension

Once installed, you can use SlicerIDCBrowser in one of the two modes:

1. As an interface to explore IDC data: you can select individual collections, cases and DICOM studies and download items of interest directly into 3D Slicer for subsequent visualization and analysis.

2. As download tool: download IDC content based on the manifest you created using IDC Portal, or identifiers of the individual cases, DICOM studies or series.

Limited access content

In a future release of IDC we will by default exclude limited access items from what you select in the portal, so the portal selection should be more intuitive. But if you access the data via BigQuery queries you will need to know that “Limited” are not accessible and account for this in your query.Storage Buckets

BigQuery Tables

The flat address space of IDC DICOM objects in GCS storage is accompanied by BigQuery tables that allow the researcher to reconstruct the DICOM hierarchy as it exists for any given version. There are also several BQ tables and views in which we keep copies of the metadata exposed via the TCIA interface at the time a version was captured and other pertinent information.

There is an instance of each of the following tables and views per IDC version. The set of tables and views corresponding to an IDC version are collected in a single BQ dataset per IDC version, bigquery-public-data.idc_<idc_version_number> where bigquery-public-data is the project in which the dataset is hosted. As an example, the BQ tables for IDC version 4 are in the bigquery-public-data.idc_v4dataset.

In addition to the per-version datasets, the bigquery-public-data.idc-current dataset consists of a set of BQ views. There is a view for each table or view in the BQ data set corresponding to the current IDC release. Each such view in bigquery-public-data.idc-current is named identically to some table or view in the bigquery-public-data.idc_<idc_version_number> dataset of the current IDC release and can be used to access that table or view.

Several Google BigQuery (BQ) tables support searches against metadata extracted from the data files. Additional BQ tables define the composition of each IDC data version.

We maintain several additional tables that curate metadata non-DICOM metadata (e.g., attribution of a given item to a specific collection and DOI, collection-level metadata, etc).

• • tcia_api_collection_id: The ID, as accepted by the TCIA API, of the original data collection containing this instance

  • idc_webapp_collection_id:The ID, as accepted by the IDC web app, of the original data collection containing this instance

  • collection_timestamp: Datetime when the IDC data in the collection was last revised

  • source_doi:A DOI of the TCIA wiki page corresponding to the original data collection or analysis results that is the source of this instance

  • collection_hash: The md5 hash of the sorted patient_hashes of all patients in the collection containing this instance

  • collection_init_idc_version: The IDC version in which the collection containing this instance first appeared

  • collection_revised_idc_version: The IDC version in which the collection containing this instance was most recently revised

  Patient attributes:

  • submitter_case_id:The submitter’s (of data to TCIA) ID of the patient containing this instance. This is the DICOM PatientID

  • idc_case_id:IDC generated UUID that uniquely identifies the patient containing this instance

    This is needed because DICOM PatientIDs are not required to be globally unique

  • patient_hash: the md5 hash of the sorted study_hashes of all studies in the patient containing this instance

  • patient_init_idc_version: The IDC version in which the patient containing this instance first appeared

  • patient_revised_idc_version: The IDC version in which the patient containing this instance was most recently revised

  Study attributes:

  • StudyInstanceUID: DICOM UID of the study containing this instance

  • study_uuid:IDC assigned UUID that identifies a version of the study containing this instance.

  • study_instances: The number instances in the study containing this instance

  • study_hash: the md5 hash of the sorted series_hashes of all series in study containing this instance

  • study_init_idc_version: The IDC version in which the study containing this instance first appeared

  • study_revised_idc_version: The IDC version in which the study containing this instance was most recently revised

  Series attributes:

  • SeriesInstanceUID: DICOM UID of the series containing this instance

  • series_uuid:IDC assigned UUID that identifies a version of the series containing this instance

  • source_doi:A DOI of the TCIA wiki page corresponding to the original data collection or analysis results that is the source of this instance

  • series_instances: The number of instances in the series containing this instance

  • series_hash: the md5 hash of the sorted instance_hashes of all instance in the series containing this instance

  • series_init_idc_version: The IDC version in which the series containing this instance first appeared

  • series_revised_idc_version: The IDC version in which the series containing this instance was most recently revised

  Instance attributes:

  • SOPInstanceUID: DICOM UID of this instance.

  • instance_uuid:IDC assigned UUID that identifies a version of this instance.

  • gcs_url: The GCS URL of a file containing the version of this instance that is identified by the instance_uuid

  • instance_hash: the md5 hash of the version of this instance that is identified by the instance_uuid

  • instance_size: the size, in bytes, of this version of the instance that is identified by the instance_uuid

  • instance_init_idc_version: The IDC version in which this instance first appeared

  • instance_revised_idc_version: The IDC version in which this instance was most recently revised

  • license_url: The URL of a web page that describes the license governing this instance

  • license_long_name: A long form name of the license governing this instance

  • license_short_name: A short form name of the license governing this instance

• • tcia_api_collection_id: The collection ID as is accepted by the TCIA AP

  • tcia_wiki_collection_id: The collection ID as on the TCIA wiki page

  • idc_webapp_collection_id:The collection ID as accepted by the IDC web app

  • Program: The program to which this collection belongs

  • Updated: Moser recent update date reported by TCIA

  • Status:Collection status" Ongoing or complete

  • Access:Collection access conditions: Limited or Public

  • ImageType: Enumeration of image types/modalities in the collection

  • Subjects:Number of subjects in the collection

  • DOI:DOI that can be resolved at doi.org to the TCIA wiki page for this collection

  • CancerType:TCIA assigned cancer type of this collection

  • SupportingData:Type(s) of additional data available

  • Species: Species of collection subjects

  • Location:Body location that was studied

  • Description:TCIA description of the collection (HTML format)

  • license_url: The URL of a web page that describes the license governing this collection

  • license_long_name: A long form name of the license governing this collection

  • license_short_name: A short form name of the license governing this collection

• • ID: Results ID

  • Title: Descriptive title

  • DOI:DOI that can be resolved at doi.org to the TCIA wiki page for this analysis result

  • CancerType:TCIA assigned cancer type of this analysis result

  • Location:Body location that was studied

  • Subjects:Number of subjects in the analysis result

  • Collections: Original collections studied

  • AnalysisArtifactsonTCIA: Type(s) of analysis artifacts generated

  • Updated: Data when results were last updated

  • license_url: The URL of a web page that describes the license governing this collection

  • license_long_name: A long form name of the license governing this collection

  • license_short_name: A short form name of the license governing this collection

• cancer-idc.idc_v<version_number>.version_metadata (also available via the canceridc-data.idc-current.version_metadata view for the current version of IDC data). Metadata for each IDC version, one row per row:

  • idc_version: IDC version number

  • version_hash: MD5 hash of hashes of collections in this version

  • version_timestamp: Version creation timestamp

• view for the current version of IDC data) Measurement group sequences extracted from the DICOM SR TID1500 objects

The following tables contain TCGA-specific metadata:

• tcga_biospecimen_rel9: biospecimen metadata

• tcga_clinical_rel9: clinical metadata

Collection-specific BigQuery tables

Some of the collections are accompanied by BigQuery tables that have not been harmonized to a single data model. Those tables are available within the BigQuery dataset corresponding to a given release, and will have the name prefix corresponding to the short name of the collection. The list below discusses those collection-specific tables.

NLST

DICOM Stores

BigQuery tables external to IDC

In addition to the DICOM data, some of the image-related data hosted by IDC is stored in additional tables. These include the following:

Background

By clinical data we refer to the broad spectrum of image-related data that may accompany images. Such data may include demographics of the patients, observations related to their clinical history (therapies, diagnoses, findings), lab tests, surgeries.

Not only are the terms used in the clinical data accompanying individual collection not harmonized, but the format of the spreadsheets is also collection-specific. In order to search and navigate clinical data, one has to parse those collection specific tables, and there is no interface to support searching across collections.

Clinical data BigQuery tables

• collection_id (STRING, NULLABLE) - the collection_id of the collection in the given table. The collection id is in a format used internally by the IDC Web App (with only lowercase letters, numbers and '_' allowed). It is equivalent to the idc_webapp_id field in the dicom_all view in the idc_current dataset.

• table_name (STRING,NULLABLE) - name of the table

• table_description (STRING,NULLABLE) - description of the type of data found in the table. Usually this is set to 'clinical data', unless a description is provided in the source files

• idc_version_table_added (STRING, NULLABLE) - the IDC data version for which this table was first added

• idc_table_added_datetime (STRING,NULLABLE) - the date/time this particular table was first generated

• post_process_src (STRING, NULLABLE) - except for the CPTAC and TCGA collections the tables are curated from ZIP, Excel, and CSV files downloaded from the TCIA wiki. These files do not have a consistent structure and were not meant to be machine readable or to translate directly into BigQuery. A semi-manual curation process results in either a CSV of JSON file that can be directly written into a BigQuery table. post_process_src is the name of the JSON or CSV file that results from this process and is used to create the BigQuery table. This field is not used for the CPTAC- and TCGA-related tables

• post_process_src_add_md5 (STRING, NULLABLE) - the md5 hash of post_process_src when the table was first added

• idc_version_table_prior (STRING, NULLABLE) - the idc version the second most recent time the table was updated

• post_process_src_prior_md5 (STRING, NULLABLE) - the md5 hash of post_process_src the second most recent time the table was updated

• idc_version_table_updated (STRING, NULLABLE) - the idc version when the table was last updated

• table_update_datetime (STRING, NULLABLE) - date and time an update of the table was last recorded

• post_process_src_updated_md5 (STRING, NULLABLE) - the md5 hash of post_process_source when the table was last updated

• number_batches (INTEGER, NULLABLE) - records the number of batches. Within the source data patients are sometimes grouped into different 'batches' (i.e. training vs test, responder vs non-responder etc.) and the batches are placed in different locations (i.e. different files or different sheets in the same Excel file)

• source_info (RECORD, REPEATED) - an array of records with information about the table sources. These sources are either files downloaded from the TCIA wiki or another BigQuery table (as is the case for CPTAC and TCGA collections). There is a source_info record for each source 'batch' described above

• source_info.srcs (STRING, REPEATED) - a source file downloaded from the TCIA wiki may be a ZIP file, and CSV file, or an Excel file. Sometimes the ZIP files contain other ZIP files that must be opened to extract the clinical data. In the source_info.src array the first string is the file that is downloaded from TCIA for this particular source batch. The final string is the CSV or Excel file that contains the clinical data. Any intermediate strings are the names of ZIP files 'in between' the downloaded file and the clinical file. For CPTAC and TCGA collections this field contains the source BigQuery table

• source_info.md5 (STRING, NULLABLE) - md5 hash of the downloaded file from TCIA the most recent time the table was updated

• source_info.table_last_modified (STRING, NULLABLE) - CPTAC and TCGA collections only. The date and time the source BigQuery table was most recently modified, as recorded when last copied

• source_info.table_size (STRING, NULLABLE) - CPTAC and TCGA collections only. The size of the source BigQuery table as recorded when last copied

• collection_id (STRING,NULLABLE) - the collection_id of the collection in the given table. The collection id is in a format used internally by the IDC Web App (with only lowercase letters, numbers and '_' allowed). It is equivalent to the idc_webapp_id field in the dicom_all view in the idc_current dataset.

• case_col (BOOLEAN, NULLABLE) - true if the BigQuery column contains the patient or case id, i.e. if this column is used to determine the value of the dicom_patient_id column

• table_name (STRING, NULLABLE) - table name

• column (STRING, NULLABLE) - the actual column name in the table. For ACRIN collections the column_name is the variable_name from the provided data dictionary. For other collections it is a name constructed by 'normalizing' the column_label (see next) in a format that can be used as a BigQuery field name

• column_label (STRING, NULLABLE) - a 'free form' label for the column that does not need to conform to the BigQuery column format requirements. For ACRIN collections this is the variable_label given by a data dictionary that accompanies the collection. For other collections it is the name or label of the clinical attribute as inferred from the source document during the curation process

• data_type (STRING, NULLABLE) - the type of data in this column. Again for ACRIN collections this is provided in the data dictionary. For other collections it is inferred by analyzing the data during curation

• original_column_headers (STRING, REPEATED) - the name(s) or label(s) in the source document that were used to construct the column_label field. In most cases there is one column label in the source document that perscribes the column_label. In some cases, multiple columns are concantenated and reformated to form the column_label

• values (RECORD, REPEATED) - a structure that is borrowed from the ACRIN data model. This is an array that contains observerd attribute values for this given column. For ACRIN collections these values are reported in the data dictionary. For most other collections these values are determined by analyzing the source data. For simplicity this field is left blank when the number of unique values is greater than 20

• values.option_code (STRING, NULLABLE) - a unique attribute value found in this column

• values.option_description (STRING, NULLABLE) - a description of the option_code as provided by a data dictionary. For collections that do not have a data dictionary this is null.

• values_source (STRING, NULLABLE) - indicates the source of the values records. The text 'provided dictionary' indicates that the records were obtained from a provided data dictionary. The text 'derived from inspection of values' indicates that the records were determined by automated analysis of the source materials during the ETL process that generated the BigQuery tables.

• files (STRING, REPEATED) - names of the files that contain the source data for each batch. These are the Excel or CSV files directly downloaded from TCIA, or the files extracted from downloaded ZIP files

• sheet_names (STRING, REPEATED) - for Excel-sourced files, the sheet names containing this column's values for each batch

• batch (INTEGER, REPEATED) - source batches that contain this particular column. Some columns or attributes may be missing from some batches

• column_numbers (STRING, REPEATED) - for each source batch, the column in the original source corresponding to this column in the BigQuery table

With this approach you will follow a a 2-step process covered on this page:

• Step 2: given the manifest, download files to your computer or to a cloud VM using s5cmd command line tool.

Step 1: Create the manifest

Start with the query templates provided below, modify them based on your needs, and save the result in a file query.txt. The specific values for PatientID, SeriesInstanceUID, StudyInstanceUID are chosen to serve as examples.

Queries below demonstrate how to get the Google Storage URLs to download cohort files.

# Select all files for a given PatientID
SELECT DISTINCT(CONCAT(series_aws_url, "* .")) 
FROM `bigquery-public-data.idc_current.dicom_all`
WHERE PatientID = "LUNG1-001"

# Select all files for a given collection
SELECT DISTINCT(CONCAT(series_aws_url, "* .")) 
FROM `bigquery-public-data.idc_current.dicom_all`
WHERE collection_id = "nsclc_radiomics"

# Select all files for a given DICOM series
SELECT DISTINCT(CONCAT(series_aws_url, "* .")) 
FROM `bigquery-public-data.idc_current.dicom_all`
WHERE SeriesInstanceUID = "1.3.6.1.4.1.32722.99.99.298991776521342375010861296712563382046"

# Select all files for a given DICOM study
SELECT DISTINCT(CONCAT(series_aws_url, "* .")) 
FROM `bigquery-public-data.idc_current.dicom_all`
WHERE StudyInstanceUID = "1.3.6.1.4.1.32722.99.99.239341353911714368772597187099978969331"

If you want to download the files corresponding to the cohort from GCP instead of AWS, substitute series_aws_url for series_gcp_url in the SELECT statement of the query, such as in the following SELECT clause:

SELECT DISTINCT(CONCAT(series_gcp_url, "* ."))

Next, use a Google Cloud SDK bq query command (from command line) to run the query and save the result into a manifest file, which will be the list of GCP URLs that can be used to download the data.

bq query --use_legacy_sql=false --format=csv --max_rows=20000000 < query.txt > manifest.txt

# count the number of rows
SELECT COUNT(DISTINCT(crdc_series_uuid)) 
FROM bigquery-public-data.idc_current.dicom_all 
WHERE collection_id = "nsclc_radiomics"

You can also get the total disk space that will be needed for the files that you will be downloading:

# calculate the disk size in GB needed for the files to be downloaded
SELECT ROUND(SUM(instance_size)/POW(1024,3),2) as size_GB 
FROM bigquery-public-data.idc_current.dicom_all 
WHERE collection_id = "nsclc_radiomics"

Step 2: Download the files defined by the manifest

You can verify if your setup was successful by running the following command: it should successfully download one file from IDC.

s5cmd --no-sign-request --endpoint-url https://storage.googleapis.com cp s3://public-datasets-idc/cdac3f73-4fc9-4e0d-913b-b64aa3100977/902b4588-6f10-4342-9c80-f1054e67ee83.dcm .

Once s5cmd is installed, you can use s5cmd run command to download the files corresponding to the manifest.

If you defined manifest that references AWS buckets:

s5cmd --no-sign-request --endpoint-url=https://s3.amazonaws.com run manifest_file_name

If you defined manifest that references GCP buckets, you will need to specify GCS endpoint:

s5cmd --no-sign-request --endpoint-url https://storage.googleapis.com run manifest_file_name

The slides below give a quick guided overview of how you can use IDC Portal.

5. Due to the existing limitations of Google Healthcare API, not all of the DICOM attributes are extracted and are available in BigQuery tables. Specifically:

   • sequences that contain around 1MiB of data are dropped from BigQuery export and RetrieveMetadata output currently. 1MiB is not an exact limit, but it can be used as a rough estimate of whether or not the API will drop the tag (this limitation was not documented as of writing this) - we know that some of the instances in IDC will be affected by this limitation. The fix for this limitation is targeted for sometime in 2021, according to the communication with Google Healthcare support.

In the following subsections you will find notebooks that don't require python programming, or have dependencies that make them not suitable for the python notebook format.

An IDC manifest may include study and/or series GUIDs that can be resolved to the underlying DICOM instance files in GCS. Such use of GUIDs in a manifest enables a much shorter manifest compared to a list of per-instance GCS URLs. Also, as explained below, a GUID is expected to be resolvable even when the data which it represents has been moved.

In IDC, we use the term GUID to mean a persistent identifier that can be resolved to a GA4GH DrsObject. GUID persistence ensures that the data which the GUID represents can continue to be located and accessed even if it has been moved to a different hosting site.

This is a typical UUID: 641121f1-5ca0-42cc-9156-fb5538c14355 of a (version of a) DICOM instance, and this is the corresponding CRDC GUID: dg.4DFC/641121f1-5ca0-42cc-9156-fb5538c14355

>> curl https://nci-crdc.datacommons.io/ga4gh/drs/v1/objects/dg.4DFC/641121f1-5ca0-42cc-9156-fb5538c14355

returns:

{
   "access_methods":[
      {
         "access_id":"gs",
         "access_url":{
            "url":"gs://idc-open/641121f1-5ca0-42cc-9156-fb5538c14355.dcm"
         },
         "region":"",
         "type":"gs"
      }
   ],
   "aliases":[

   ],
   "checksums":[
      {
         "checksum":"f338e8c5e3d8955d222a04d5f3f6e2b4",
         "type":"md5"
      }
   ],
   "contents":[

   ],
   "created_time":"2020-09-18T02:14:02.830862",
   "description":null,
   "form":"object",
   "id":"dg.4DFC/641121f1-5ca0-42cc-9156-fb5538c14355",
   "mime_type":"application/json",
   "name":null,
   "self_uri":"drs://nci-crdc.datacommons.io/dg.4DFC/641121f1-5ca0-42cc-9156-fb5538c14355",
   "size":"135450",
   "updated_time":"2020-09-18T02:14:02.830868",
   "version":"9e13fb30"
}

which is a DrsObject. Because we resolved the GUID of an instance, the access_methods in the returned DrsObject includes a URL at which the corresponding DICOM entity can be accessed.

When the GUID of a series is resolved, the DrsObject that is returned does not include access methods because there are no series file objects. Instead, the contents component of the returned DrsObject contains the URLs that can be accessed to obtain the DrsObjects of the instances in the series. Thus, we see that when we resolve dg.4DFC/cc9c8541-949d-48d9-beaf-7028aa4906dc, the GUID of the series containing the instance above:

curl -o foo https://nci-crdc.datacommons.io/ga4gh/drs/v1/objects/dg.4DFC/cc9c8541-949d-48d9-beaf-7028aa4906dc

we see that the contents component includes the GUID of that instance as well as the GUID of another instance:

{
   "aliases":[

   ],
   "checksums":[
      {
         "checksum":"0512207cb222fa2f085bc110c8474fa2",
         "type":"md5"
      }
   ],
   "contents":[
      {
         "drs_uri":"drs://nci-crdc.datacommons.io/dg.4DFC/ccafd781-ef39-4d39-ad74-e09de1ada476",
         "id":"dg.4DFC/ccafd781-ef39-4d39-ad74-e09de1ada476",
         "name":null
      },
      {
         "drs_uri":"drs://nci-crdc.datacommons.io/dg.4DFC/641121f1-5ca0-42cc-9156-fb5538c14355",
         "id":"dg.4DFC/641121f1-5ca0-42cc-9156-fb5538c14355",
         "name":null
      }
   ],
   "created_time":"2020-12-04T19:11:58.072088",
   "description":"",
   "form":"bundle",
   "id":"dg.4DFC/cc9c8541-949d-48d9-beaf-7028aa4906dc",
   "mime_type":"application/json",
   "name":"dg.4DFCcc9c8541-949d-48d9-beaf-7028aa4906dc",
   "self_uri":"drs://nci-crdc.datacommons.io/dg.4DFC/cc9c8541-949d-48d9-beaf-7028aa4906dc",
   "size":270902,
   "updated_time":"2020-12-04T19:11:58.072094",
   "version":""
}

Similarly, the GUID of a DICOM study resolves to a DrsObject whose contents component consists of the GUIDs of the series in that study.

Load a brightfield (RGB) DICOM slide

Next, open QuPath and select "File > Open".

Choose just one of the .dcm files that belong to the desired dataset, then click Open. The remaining files will be automatically detected and should not be selected.

Zooming and panning in real time:

The Image tab on the left side of the screen shows dimension information, and lists any associated images. In this case, a thumbnail image is present under Associated Images at the bottom of the Image tab. Double-clicking on Series 1 (THUMBNAIL) will open the thumbnail image in a separate window:

Open a fluorescence DICOM dataset

For this part, we will use a slide from the HTAN-OHSU collection identified by SeriesInstanceUID 1.3.6.1.4.1.5962.99.1.1999932010.1115442694.1655562373738.4.0. As before, you can download it as follows:

As in the brightfield case, open QuPath and select File > Open.

Choose just one of the .dcm files in the dataset, as the other files will be automatically detected. It does not matter which file is selected. When prompted, set the image type to Fluorescence, or as appropriate for the dataset:

The Image tab indicates the number of channels (12 in this case). By default, all channels will be displayed at once. This can be changed by selecting View > Brightness/Contrast or the "half-circles" icon in the toolbar:

Unchecking the Show box will hide the channel's data, and update the image.

DICOM SR uses data elements to encode a higher level abstraction that is a tree of content, where nodes of the tree and their relationships are formalized. SR-TID1500 is one of many standard templates that define constraints on the structure of the tree, and is intended for generic tasks involving image-based measurements. DICOM SR uses standard terminologies and codes to deliver structured content. These codes are used for defining both the concept names and values assigned to those concepts (name-value pairs). Measurements include coded concepts corresponding to the quantity being measured, and a numeric value accompanied by coded units. Coded categorical or qualitative values may also be present. In SR-TID1500, measurements are accompanied by additional context that helps interpret and reuse that measurement, such as finding type, location, method and derivation. Measurements computed from segmentations can reference the segmentation defining the region and the image segmented, using unique identifiers of the respective objects.

Tools referenced above can be used to 1) extract qualitative evaluations and quantitative measurements fro the SR-TID1500 document; 2) generate standard-compliant SR-TID1500 objects.

IDC releases summary view

V20 - November 2024

New radiology collections

New pathology collections

Revised radiology collections

Revised pathology collections

Revised analysis results

2. The segmentation of an instance in each of the following series was excluded due to having a DICOM PixelData size greater than or equal to 2GB:

   1. 1.2.826.0.1.3680043.10.511.3.10544506665348704312902213950958190

   2. 1.2.826.0.1.3680043.10.511.3.11183783347037364699862133130586654

   3. 1.2.826.0.1.3680043.10.511.3.11834745481756047014039855874680259

   4. 1.2.826.0.1.3680043.10.511.3.11901667084519361717338400810055642

   5. 1.2.826.0.1.3680043.10.511.3.12041600048156613329793822566495651

   6. 1.2.826.0.1.3680043.10.511.3.12718116375608495830041119776887887

   7. 1.2.826.0.1.3680043.10.511.3.13386724401829265460622415500801368

   8. 1.2.826.0.1.3680043.10.511.3.14042734131864468280344737986870899

   9. 1.2.826.0.1.3680043.10.511.3.17374765903080083648409690755539184

   10. 1.2.826.0.1.3680043.10.511.3.17429002643681869326389465422353495

   11. 1.2.826.0.1.3680043.10.511.3.20359930476040698387716730891020638

   12. 1.2.826.0.1.3680043.10.511.3.28397033639127902823368316410884210

   13. 1.2.826.0.1.3680043.10.511.3.28425539132321749931109935391487352

   14. 1.2.826.0.1.3680043.10.511.3.34574227972763695321794092913087775

   15. 1.2.826.0.1.3680043.10.511.3.36216094237641867532902805456135029

   16. 1.2.826.0.1.3680043.10.511.3.39533936694797964318706337783276378

   17. 1.2.826.0.1.3680043.10.511.3.39900930856460689132625586523683939

   18. 1.2.826.0.1.3680043.10.511.3.41633795217567037218184715094985555

   19. 1.2.826.0.1.3680043.10.511.3.42218106649761752724553401155203874

   20. 1.2.826.0.1.3680043.10.511.3.49098870621170235412220976183110770

   21. 1.2.826.0.1.3680043.10.511.3.50064322235999800062455171235601125

   22. 1.2.826.0.1.3680043.10.511.3.50905421517530127976832505410705816

   23. 1.2.826.0.1.3680043.10.511.3.62935684444056080516153739948364303

   24. 1.2.826.0.1.3680043.10.511.3.73572792121235596011940904319511291

   25. 1.2.826.0.1.3680043.10.511.3.74494366757564543824303304482444570

   26. 1.2.826.0.1.3680043.10.511.3.79988146996803179892075404247166692

   27. 1.2.826.0.1.3680043.10.511.3.80004293150506819482091023564947091

   28. 1.2.826.0.1.3680043.10.511.3.82774274518897141254234567300292686

   29. 1.2.826.0.1.3680043.10.511.3.84202416467561501610598853920808906

   30. 1.2.826.0.1.3680043.10.511.3.86214492184712627544696209982376598

   31. 1.2.826.0.1.3680043.10.511.3.90193069664920622990317347485104073

   32. 1.2.826.0.1.3680043.10.511.3.95666157880521064637011880609274546

   33. 1.2.826.0.1.3680043.10.511.3.96676982370873257329281821215166082

   34. 1.2.826.0.1.3680043.10.511.3.98258035017480972315346136181769675

New Clinical Metadata Tables

v19 - September 2024

New pathology collections