# Welcome! {% hint style="warning" %} **New: query IDC from your LLM agent with the** [**IDC MCP server**](/mcp/mcp) **and the** [**REST API (v3)**](/api/api)**, or with this natural-language** [**agent skill**](https://github.com/ImagingDataCommons/imaging-data-commons-skill)**.** {% endhint %} [**NCI Imaging Data Commons** **(IDC)**](https://imaging.datacommons.cancer.gov) is a cloud-based environment containing publicly available cancer imaging data co-located with analysis and exploration tools. IDC is a node within the broader NCI [Cancer Research Data Commons (CRDC)](https://datacommons.cancer.gov/) infrastructure that provides secure access to a large, comprehensive, and expanding collection of cancer research data.
## Highlights * **>95 TB of data**: IDC contains radiology, brightfield (H\&E) and fluorescence slide microscopy images, along with image-derived data (annotations, segmentations, quantitative measurements) and accompanying clinical data * **free**: all of the data in IDC is publicly available: no registration, no access requests * **commercial-friendly**: >95% of the data in IDC is covered by the permissive CC-BY license, which allows commercial reuse (small subset of data is covered by the CC-NC license); each file in IDC is tagged with the license to make it easier for you to understand and follow the rules * **cloud-based**: all of the data in IDC is available from both Google and AWS public buckets: fast and free to download, no out-of-cloud egress fees * **harmonized**: all of the images and image-derived data in IDC is harmonized into standard DICOM representation ## Functionality IDC is as much about data as it is about what you can do with the data! We maintain and actively develop a variety of tools that are designed to help you efficiently navigate, access and analyze IDC data: * **exploration**: start with the [IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/) to get an idea of the data available * **visualization**: examine images and image-derived annotations and analysis results from the convenience of your browser using integrated OHIF, VolView and Slim open source viewers * **programmatic access**: use [`idc-index` python package](https://github.com/ImagingDataCommons/idc-index) to perform search, download and other operations programmatically, or the language-agnostic [REST API](/api/api) over plain HTTP * **LLM agents**: point Claude or any other agent at the [IDC MCP server](/mcp/mcp) to discover, subset, and download IDC data in plain conversation * **cohort building**: use rich and extensive metadata to build subsets of data programmatically using `idc-index` or BigQuery SQL * **download**: use your favorite S3 API client or `idc-index` to efficiently fetch any of the IDC files from our public buckets * **analysis**: conveniently access IDC files and metadata from the tools that are cloud-native, such as Google Colab; fetch IDC data directly into 3D Slicer using [SlicerIDCBrowser extension](https://github.com/ImagingDataCommons/SlicerIDCBrowser/) {% hint style="info" %} The overview of IDC is available in this open access publication. If you use IDC, please acknowledge us by citing it! > Fedorov, A., Longabaugh, W. J. R., Pot, D., Clunie, D. A., Pieper, S. D., Gibbs, D. L., Bridge, C., Herrmann, M. D., Homeyer, A., Lewis, R., Aerts, H. J. W., Krishnaswamy, D., Thiriveedhi, V. K., Ciausu, C., Schacherer, D. P., Bontempi, D., Pihl, T., Wagner, U., Farahani, K., Kim, E. & Kikinis, R. *National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence*. RadioGraphics (2023). > {% endhint %} # Getting started {% hint style="info" %} **New: query IDC from your LLM agent with the** [**IDC MCP server**](/mcp/mcp) **and the** [**REST API (v3)**](/api/api)**, or with this natural-language** [**agent skill**](https://github.com/ImagingDataCommons/imaging-data-commons-skill)**.** {% endhint %} {% hint style="info" %} **If you are looking for the instructions on how to download data from IDC, check out the** [**Downloading data**](/data/downloading-data) **section of the documentation.** {% endhint %} We want Imaging Data Commons to be your companion in your cancer imaging research activities - from discovering relevant data to sharing your analysis results and showcasing the tools you developed!
### **Explore the data available** Check out [quick instructions](/portal/getting-started) on how to access and use [IDC Portal](https://portal.imaging.datacommons.cancer.gov/) web application that will help you search, subset and visualize data available in IDC. IDC Portal is integrated with powerful visualization tools: just with your web browser you will be able to see IDC images and annotations using OHIF Viewer, Slim viewer and VolView! ### **Subset the content you need** We have many tools to help you search data in IDC, so that you download only what you need! * you can do basic filtering/subsetting of the data using IDC Portal, but if you are developer, you will want to learn how to use [`idc-index` python package](https://github.com/ImagingDataCommons/idc-index) for programmatic access. [This python notebook](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/part2_searching_basics.ipynb) will introduce you to the basics of `idc-index` for interaction with IDC content. * search clinical data: many of the IDC collections are accompanied by clinical data, which we parsed for you into searchable tabular representation - no need to download or parse CSV/Excel/PDF files! Dive into searching clinical data using [this notebook](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/exploring_clinical_data.ipynb). * if advanced content does not scare you, check out [this notebook](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/part3_exploring_cohorts.ipynb) to learn how to search **all** of the metadata accompanying IDC using SQL and Google BigQuery. * prefer plain HTTP, or want an LLM agent to do the searching? The [IDC REST API](/api/api) exposes discovery, cohort building, downloads, SQL and citations over HTTP/JSON, and the [IDC MCP server](/mcp/mcp) exposes the same capabilities as tools for Claude and other agents. ### **Download the data you liked** We provide various tools for downloading data from IDC, as discussed in the [Download documentation page](/data/downloading-data). Access to all data in IDC is free! No registration. No access request forms. No logins. * once you have `idc-index` python package installed, download from the command line is as easy as running `idc download `, or `idc download `. * looking for an interactive "point-and-click" application? [3D Slicer IDC Browser extension](https://github.com/ImagingDataCommons/SlicerIDCBrowser) is for you (note that you will only be able to visualize radiology - not microscopy - images in 3D Slicer) ### **Experiment with analysis tools** We want to make it easier to understand performance of the latest advances in AI on real-world cancer imaging data! * if you have a Google account, you have free access to Google Colab, which allows you to run python notebooks on cloud VMs equipped with GPU - for free! Combined with `idc-index` for data access, this makes it rather easy to experiment with the latest AI tools! As an example, take a look at [this notebook](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/analysis/MedSAM_with_IDC.ipynb) that allows you to apply MedSAM model to IDC data. You will find a growing number of notebooks to help you use IDC in [this repository](https://github.com/ImagingDataCommons/IDC-Tutorials). * use IDC to develop HuggingFace spaces that demonstrate the power of your models on real data: see [this space](https://huggingface.co/spaces/ImagingDataCommons/SegVolOnIDC) we developed for SegVol * growing number of AI medical imaging models is being curated on the [MHub.ai](https://mhub.ai) platform; see [this notebook](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/analysis/mhubai_tutorial.ipynb) to learn how to apply those models on data from IDC How about accompanying your next publication by a working demonstration notebook on relevant samples from IDC? You can see an example how we did this in [this recent publication](https://dx.doi.org/10.1016/j.cmpb.2023.107839). ### **Scale the analysis to thousands of cloud VMs** With the cloud, you can do things that are simply impossible to do with your local resources. * read [this preprint](https://doi.org/10.21203/rs.3.rs-4351526/v1) to learn how we applied TotalSegmentator+pyradiomics to >126,000 of CT scans of the NLST collection using Terra platform, completing the analysis in \~8 hours with the total cost \~$1000 * [this repository](https://github.com/ImagingDataCommons/CloudSegmentator) contains the code we used in the above (this is **really** advanced content!) ### **Share analysis results or annotations** If you have an algorithm, that you evaluated/published, that can enrich data in IDC with analysis results and you want to contribute those, or if you are a domain expert and would like to publish results of manual annotations you prepared - we want to hear from you! * IDC maintains a [Zenodo community](https://zenodo.org/communities/nci-idc) where we curate contributions of analysis results and other datasets produced by IDC (see the [expert annotations of the RMS-Mutations-Prediction microscopy images collection](https://zenodo.org/records/10462858) as one example of such contribution) * through a dedicated Zenodo record you will have a citation and DOI to get credit for your work; your data is ingested from Zenodo into IDC, and citation will be generated for the users of your data in IDC * once your data is in IDC, it should be easier to discover it, combine with other datasets, visualize and use from analysis workflows (as an example, see [this notebook](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/collections_demos/rms_mutation_prediction/RMS-Mutation-Prediction-Expert-Annotations_exploration.ipynb) accompanying the RMS annotations) * email us at [support+submissions@canceridc.dev](https://mail.google.com/mail/?view=cm\&fs=1\&tf=1\&to=support+submissions@canceridc.dev) to inquire about contributing your annotations/analysis results to IDC! ### Questions? If you need support with IDC or have any questions, please open a new topic in [IDC User Forum](https://discourse.canceridc.dev/) (preferred) or send email to . Would you rather discuss your questions in an meeting with an expert from the IDC team? Book a 1-on-1 support session here: # Core functions ## Easy and efficient access to public cancer imaging data We ingest and distribute datasets from variety of sources and contributors, primarily focusing on large data collection initiatives sponsored by US National Cancer Institute. At this time, we do not have resources to prioritize receipt of the imaging data from individual PIs (but we are encouraging submissions of annotations/analysis results for existing IDC data!). Nevertheless, if you feel you might have a compelling dataset, please email us at [support+submissions@canceridc.dev](https://mail.google.com/mail/?view=cm\&fs=1\&tf=1\&to=support+submissions@canceridc.dev). On ingestion, we harmonize images and image-derived data into DICOM format for interoperability, whenever data is represented in a non-DICOM format. Upon conversion, the data undergoes Extract-Transform-Load (ETL), which extracts DICOM metadata to make the data searchable, ingests the DICOM files into public S3 storage buckets and a DICOMweb store. Once the data is released, we provide various interfaces to access data and metadata. {% embed url="" %} Schematic summary of the IDC data ingestion and release process. {% endembed %} ## Tools to simplify the use of the data We are actively developing a variety of capabilities to make it easier for the users to work with the data in IDC. Some of the examples of those tools include * [IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/) provides interactive browser-based interface for exploration of IDC data * we are the maintainers of [Slim](https://github.com/ImagingDataCommons/slim) - an open-source viewer of DICOM digital pathology images; Slim is integrated with IDC Portal for visualizing pathology images and image-derived data available in IDC * we are actively contributing to the [OHIF Viewer](https://github.com/OHIF/Viewers), and rely on it for visualizing radiology images and image-derived data * [`idc-index`](https://github.com/ImagingDataCommons/idc-index) is a python package that provides convenience functions for accessing IDC data, including efficient download from IDC public S3 buckets * [3D Slicer](https://slicer.org) extensions [SlicerIDCBrowser](https://github.com/ImagingDataCommons/SlicerIDCBrowser) can be used for interactive download of IDC data * we are contributing to a variety of tools that aim to simplify the use of DICOM in cancer imaging research; these include [OpenSlide](https://openslide.org/formats/dicom/) and [BioFormats bfconvert](https://bio-formats.readthedocs.io/en/v7.3.1/formats/dicom.html) library that can be used for conversion between DICOM Whole Slide Imaging (WSI) format and other slide microscopy formats, [dcmqi](https://github.com/QIICR/dcmqi) library for converting image analysis results to and from DICOM representation {% embed url="" %} Although IDC data is stored in DICOM format, it can be converted into alternative research representations using open-source tools. {% endembed %} ## Support of continuous enrichment of data We welcome you to apply to contribute analysis results and annotations of the images available in IDC! These can be expert manual annotations, analysis results generated using AI tools, segmentations, contours, metadata attributes describing the data (e.g., annotation of the scan type), expert evaluation of the quality of existing AI-generated annotations in IDC. If you would like your annotations/analysis results to be considered, you must establish the value of your contribution (e.g., describe the qualifications of the experts performing manual annotations, demonstrate robustness of the AI tool you are applying to images with a peer-reviewed publication or other type of evidence), and be willing to share your contribution under a permissive Creative Commons Attribution [CC BY 4.0 license](https://creativecommons.org/licenses/by/4.0/deed.en). See more details on our curation policy [here](https://zenodo.org/communities/nci-idc/curation-policy), and reach out by sending email to [support+submissions@canceridc.dev](https://mail.google.com/mail/?view=cm\&fs=1\&tf=1\&to=support+submissions@canceridc.dev) with any questions or inquries. Every application will be reviewed by IDC stakeholders. If your contribution is accepted by the IDC stakeholders: * we will work with you to choose the appropriate DICOM object type for your data and convert it into DICOM representation * upon conversion, we will create a Zenodo entry under the[NCI Imaging Data Commons Zenodo community](https://zenodo.org/communities/nci-idc/) for your contribution so that you get the Digital Object Identifier (DOI), citation and recognition of your contribution * once published in IDC * your data will become searchable and viewable in IDC Portal, so it is easier for the users of your data to discover and work with your data * files can be downloaded very efficiently using S3 interface and `idc-index` ## Integration of cancer imaging data with other components of CRDC IDC is a component of the broader NCI [Cancer Research Data Commons (CRDC)](https://datacommons.cancer.gov/), giving you access to the following: * [Cancer Data Aggregator (CDA)](https://cda.readthedocs.io/en/latest/) can be used to find data related to the images in IDC in [Genomics Data Commons](https://portal.gdc.cancer.gov/), [Proteomics Data Commons](https://pdc.cancer.gov/pdc/) and [Integrated Canine Data Commons](https://caninecommons.cancer.gov/) * Broad [FireCloud](https://firecloud.terra.bio/) and [Seven Bridges Cancer Genimics Cloud](https://www.cancergenomicscloud.org/) (SB-CGC) can be used to apply analysis tools to the data in IDC (you can read more about how this can be done in [this preprint](https://doi.org/10.21203/rs.3.rs-4351526/v1) from the IDC team) * [MHub.AI](https://mhub.ai/) platform curates a growing number of cancer imaging AI models that can be applied directly to the DICOM data available in IDC # Frequently Asked Questions ## What is the difference between IDC and TCIA? IDC and TCIA are partners in providing FAIR data for cancer imaging researchers. TCIA provides unique service to work with data submitters to de-identify cancer imaging data and make it available for download. The mission of IDC is to support efficient access and use of the cancer imaging data, after it was de-identified and released. Here are some of the highlights that make IDC unique: * **Unique datasets:** while all of the public TCIA DICOM collections are available in IDC, there is a growing amount of data in IDC that is not available anywhere else: * DICOM digital pathology collections from prominent initiatives: Childhood Cancer Data Initiative (CCDI), GTEx, TCGA, CPTAC, HTAN, CMB * image analysis results available only from IDC, such as TotalSegmentator segmentations and radiomics features for most of the CT images in the NLST collection * **Cloud-native:** IDC makes the data available in public cloud buckets, the egress is free (TCIA provides download from on-premises servers at a single institution): chances are your will be able to download data from IDC much faster than from TCIA * **Partnerships with cloud vendors:** IDC collaborates with Public Datasets Programs of Amazon Web Services and Google Cloud to support hosting and free out-of-cloud egress, contributing to improved accessibility, sustainability and longevity of the resource * **State of the art tools:** IDC maintains superior community recognized tools to support the use of the data: * modern OHIF Viewer v3 for radiology data, with support of visualization of annotations and segmentations; * Slim viewer for digital pathology and annotations * highly capable IDC Portal * **Standard access interfaces:** IDC offers standard interfaces for data access: S3 API for file download, DICOMweb for interoperability with DICOM tools, SQL for searching all of the DICOM metadata (TCIA offers various non-standard, in-house interfaces and APIs for data access) * **Harmonized data:** All of the data (radiology and digital pathology images, annotations, segmentations, image-derived features) available in IDC is harmonized into DICOM representation, which means * interoperability: you can use IDC data with any DICOM-compatible tool * metadata: every single file in IDC is accompanied by metadata that follows DICOM data model, and is associated with unique identifiers, allowing you to build reproducible cohorts * uniform representation: you don't need to customize your processing pipelines to a specific collection, and can build cohorts combining data across collections * **Co-location with cloud compute resources:** IDC data is easier to access from cloud computing resources, allowing you to more easily experiment with the new analysis tools and scale your computation * **Versioning:** IDC data is versioned: you will be able to access the exact files you analyzed in a given verison of IDC even if there were any updates to the collection after you accessed it, helping you achieve reproducibility of your analyses * **Open-source tool stack:** all of the tools developed by IDC are shared under permissive licenses to support community contribution, reuse and sustainability ## How to download data from IDC? Check out the [Downloading data](/data/downloading-data) documentation page! ## How do I get my data into IDC? Note that currently IDC prioritizes submissions from NCI-funded driving projects and data from special selected projects. * If you would like to submit images, it will be your responsibility to de-identify them first, documenting the de-identification process and submitting that documentation for the review by IDC stakeholders. * We welcome submissions of image-derived data (expert annotations, AI-generated segmentations) for the images already in IDC, see IDC Zenodo community [Curation policy](https://zenodo.org/communities/nci-idc/curation-policy) to learn about the requirements for such submissions! IDC works closely with [The Cancer Imaging Archive (TCIA)](https://www.cancerimagingarchive.net/) and mirrors TCIA public collections. If you submit your DICOM data to TCIA and your data is released as a public collection, it will be automatically available in IDC in a following release. If you are interested in making your data available within IDC, please contact us by sending email to [support+submissions@canceridc.dev](https://mail.google.com/mail/?view=cm\&fs=1\&tf=1\&to=support+submissions@canceridc.dev). ## How much does it cost to use the cloud? IDC data is stored in the cloud buckets, and you can search and [download data from IDC](/data/downloading-data) for free and without login. If you would like to use the cloud for analysis of the data, we recommend you start with the free tier of [Google Colab](https://colab.research.google.com) to get free access to a cloud-hosted VM with GPU to experiment with analysis workflows for IDC data. If you are an NIH-funded researcher, you may be eligible for a free allocation via [NIH Cloud Lab](https://cloud.nih.gov/resources/cloudlab/). US-based researchers can also access free cloud-based computing resources via [ACCESS program allocations](/cookbook/access-allocations). ## What is the status of IDC? IDC pilot release took place in Fall 2020, followed by the production release in September 2021. IDC team is continuously refining the capabilities of IDC Portal and various tools, and publishes new data releases every 3-4 months. ## What data is available? We host most of the public collections from [The Cancer Imaging Archive (TCIA)](https://www.cancerimagingarchive.net). We also host HTAN and other pathology images not hosted by TCIA. You can review the complete, up-to-date list of [collections included in IDC](https://portal.imaging.datacommons.cancer.gov/collections/). ## How to acknowledge IDC? Please cite the latest paper from the IDC team. Please also make sure you acknowledge the specific data collections you used in your analysis. > Fedorov, A., Longabaugh, W. J. R., Pot, D., Clunie, D. A., Pieper, S. D., Gibbs, D. L., Bridge, C., Herrmann, M. D., Homeyer, A., Lewis, R., Aerts, H. J. W. L., Krishnaswamy, D., Thiriveedhi, V. K., Ciausu, C., Schacherer, D. P., Bontempi, D., Pihl, T., Wagner, U., Farahani, K., Kim, E. & Kikinis, R. National cancer institute imaging data commons: Toward transparency, reproducibility, and scalability in imaging artificial intelligence. *Radiographics* 43, (2023). ## Where do I learn more about other components of CRDC? The main website for the Cancer Research Data Commons (CRDC) is [https://datacommons.cancer.gov/](https://datacommons.cancer.gov) ## What about non-imaging data that accompanies IDC collections? Clinical data that was shared by the submitters is available for a number of imaging collections in IDC. Please see [this tutorial](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/advanced_topics/clinical_data_intro.ipynb) on how to search that data and how to link clinical data with imaging metadata! Many of the imaging collections are also accompanied by the genomics or proteomics data. CRDC [Cancer Data Aggregator (CDA)](https://cda.readthedocs.io/en/latest/) provides the API to locate such related datasets. ## I want to search IDC content using an attribute not available in the portal IDC Portal gives you access to just a small subset of the metadata accompanying IDC images. If you want to learn more about what is available, you have several options: * [this notebook](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/part2_searching_basics.ipynb) from our Getting Started tutorial series explains how to use [`idc-index`](https://github.com/ImagingDataCommons/idc-index) - a python package that aims to simplify access to IDC data * [this more advanced notebook](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/part3_exploring_cohorts.ipynb) will help you get started with searching IDC metadata in BigQuery, which gives you access to all of the DICOM metadata extracted from IDC-hosted files * if you are not comfortable writing queries or coding in pyhon, you can use [this DataStudio dashboard](https://datastudio.google.com/reporting/ab96379c-e134-414f-8996-188e678f1b70/page/KHtxB) to search using some of the attributes that are not available through the portal. You can also [extend this dashboard](/cookbook/data-studio/cohort-dashboard) to include additional attributes. # Support {% hint style="warning" %} **If you need support with IDC or have any questions, please open a new topic in** [**IDC User Forum**](https://discourse.canceridc.dev/) **(preferred) or send email to .** **Would you rather discuss your questions in an meeting with an expert from the IDC team? Book a 1-on-1 support session here:** [**https://tinyurl.com/idc-help-request**](https://tinyurl.com/idc-help-request) {% endhint %} ## Discounted use and training materials for NIH-funded investigators If you are an NIH-funded investigator, you can join the [NIH STRIDES Initiative](https://datascience.nih.gov/strides) that offers significant discounts on the use of cloud resources, and free training courses and materials on the use of the cloud. # Key pointers ## Resources maintained by the IDC team * **Portal:** * Discourse (community forum): * Documentation: * GitHub organization: * Tutorials: * [Publications](/publications) ## Other locations for accessing public imaging data If you did not find the images you need in IDC, you can consider the following resources: * [The Cancer Imaging Archive (TCIA)](https://www.cancerimagingarchive.net/): while most of the public DICOM collections from TCIA are available in IDC, we do not replicate limited access TCIA collections * [Open-Access Medical Imaging Repositories](https://www.aylward.org/notes/open-access-medical-image-repositories): list curated by Stephen Aylward * [Finding Medical Images](https://hslguides.med.nyu.edu/medicalimages): list curated by New York University Health Sciences Library # Publications ## Publications by the IDC team 1. Fedorov, A., Longabaugh, W. J. R., Pot, D., Clunie, D. A., Pieper, S. D., Gibbs, D. L., Bridge, C., Herrmann, M. D., Homeyer, A., Lewis, R., Aerts, H. J. W., Krishnaswamy, D., Thiriveedhi, V. K., Ciausu, C., Schacherer, D. P., Bontempi, D., Pihl, T., Wagner, U., Farahani, K., Kim, E. & Kikinis, R. *National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence*. RadioGraphics (2023). 2. Weiss, J., Bernatz, S., Johnson, J., Thiriveedhi, V., Mak, R. H., Fedorov, A., Lu, M. T. & Aerts, H. J. W. Opportunistic assessment of steatotic liver disease in lung cancer screening eligible individuals. *J. Intern. Med.* (2025). 3. Thiriveedhi, V. K., Krishnaswamy, D., Clunie, D., Pieper, S., Kikinis, R. & Fedorov, A. Cloud-based large-scale curation of medical imaging data using AI segmentation. *Research Square* (2024). 4. Fedorov, A., Longabaugh, W. J. R., Pot, D., Clunie, D. A., Pieper, S., Aerts, H. J. W. L., Homeyer, A., Lewis, R., Akbarzadeh, A., Bontempi, D., Clifford, W., Herrmann, M. D., Höfener, H., Octaviano, I., Osborne, C., Paquette, S., Petts, J., Punzo, D., Reyes, M., Schacherer, D. P., Tian, M., White, G., Ziegler, E., Shmulevich, I., Pihl, T., Wagner, U., Farahani, K. & Kikinis, R. NCI Imaging Data Commons. *Cancer Res.* 81, 4188–4193 (2021). 5. Gorman, C., Punzo, D., Octaviano, I., Pieper, S., Longabaugh, W. J. R., Clunie, D. A., Kikinis, R., Fedorov, A. Y. & Herrmann, M. D. Interoperable slide microscopy viewer and annotation tool for imaging data science and computational pathology. *Nat. Commun.* 14, 1–15 (2023). 6. Bridge, C. P., Gorman, C., Pieper, S., Doyle, S. W., Lennerz, J. K., Kalpathy-Cramer, J., Clunie, D. A., Fedorov, A. Y. & Herrmann, M. D. Highdicom: a Python Library for Standardized Encoding of Image Annotations and Machine Learning Model Outputs in Pathology and Radiology. *J. Digit. Imaging* 35, 1719–1737 (2022). 7. Schacherer, D. P., Herrmann, M. D., Clunie, D. A., Höfener, H., Clifford, W., Longabaugh, W. J. R., Pieper, S., Kikinis, R., Fedorov, A. & Homeyer, A. The NCI Imaging Data Commons as a platform for reproducible research in computational pathology. *Comput. Methods Programs Biomed.* 107839 (2023). doi:[10.1016/j.cmpb.2023.107839](https://dx.doi.org/10.1016/j.cmpb.2023.107839) 8. Krishnaswamy, D., Bontempi, D., Thiriveedhi, V., Punzo, D., Clunie, D., Bridge, C. P., Aerts, H. J., Kikinis, R. & Fedorov, A. Enrichment of the NLST and NSCLC-Radiomics computed tomography collections with AI-derived annotations. arXiv \[cs.CV] (2023). at <> 9. Bontempi, D., Nuernberg, L., Pai, S., Krishnaswamy, D., Thiriveedhi, V., Hosny, A., Mak, R. H., Farahani, K., Kikinis, R., Fedorov, A. & Aerts, H. J. W. L. End-to-end reproducible AI pipelines in radiology using the cloud. *Nat. Commun.* 15, 6931 (2024). 10. Krishnaswamy, D., Bontempi, D., Thiriveedhi, V. K., Punzo, D., Clunie, D., Bridge, C. P., Aerts, H. J. W. L., Kikinis, R. & Fedorov, A. Enrichment of lung cancer computed tomography collections with AI-derived annotations. *Sci. Data* 11, 1–15 (2024). 11. Murugesan, G. K., McCrumb, D., Aboian, M., Verma, T., Soni, R., Memon, F., Farahani, K., Pei, L., Wagner, U., Fedorov, A. Y., Clunie, D., Moore, S. & Van Oss, J. The AIMI Initiative: AI-Generated Annotations for Imaging Data Commons Collections. *arXiv \[eess.IV]* (2023). at ## Publications referencing IDC (a subset) {% hint style="info" %} See the full list, as curated by Google Scholar, [here](https://scholar.google.com/scholar?oi=bibs\&hl=en\&cites=8052604365477078213). {% endhint %} 1. Pai, S., Bontempi, D., Hadzic, I., Prudente, V., Sokač, M., Chaunzwa, T. L., Bernatz, S., Hosny, A., Mak, R. H., Birkbak, N. J. & Aerts, H. J. W. L. Foundation model for cancer imaging biomarkers. *Nature Machine Intelligence* 6, 354–367 (2024). 2. Murugesan, G. K., McCrumb, D., Aboian, M., Verma, T., Soni, R., Memon, F. & Van Oss, J. The AIMI initiative: AI-generated annotations for imaging data commons collections. *arXiv \[eess.IV]* (2023). at <> 3. Kulkarni, P., Kanhere, A., Yi, P. H. & Parekh, V. S. Text2Cohort: Democratizing the NCI Imaging Data Commons with natural language cohort discovery. *arXiv \[cs.LG]* (2023). at <> 4. Jiang, P., Sinha, S., Aldape, K., Hannenhalli, S., Sahinalp, C. & Ruppin, E. Big data in basic and translational cancer research. *Nat. Rev. Cancer* 22, 625–639 (2022). 5. Schapiro, D., Yapp, C., Sokolov, A., Reynolds, S. M., Chen, Y.-A., Sudar, D., Xie, Y., Muhlich, J., Arias-Camison, R., Arena, S., Taylor, A. J., Nikolov, M., Tyler, M., Lin, J.-R., Burlingame, E. A., Human Tumor Atlas Network, Chang, Y. H., Farhi, S. L., Thorsson, V., Venkatamohan, N., Drewes, J. L., Pe’er, D., Gutman, D. A., Herrmann, M. D., Gehlenborg, N., Bankhead, P., Roland, J. T., Herndon, J. M., Snyder, M. P., Angelo, M., Nolan, G., Swedlow, J. R., Schultz, N., Merrick, D. T., Mazzili, S. A., Cerami, E., Rodig, S. J., Santagata, S. & Sorger, P. K. MITI minimum information guidelines for highly multiplexed tissue images. *Nat. Methods* 19, 262–267 (2022). 6. Wahid, K. A., Glerean, E., Sahlsten, J., Jaskari, J., Kaski, K., Naser, M. A., He, R., Mohamed, A. S. R. & Fuller, C. D. Artificial intelligence for radiation oncology applications using public datasets. *Semin. Radiat. Oncol.* 32, 400–414 (2022). 7. Hartley, M., Kleywegt, G. J., Patwardhan, A., Sarkans, U., Swedlow, J. R. & Brazma, A. The BioImage Archive - Building a Home for Life-Sciences Microscopy Data. *J. Mol. Biol.* 167505 (2022). doi:10.1016/j.jmb.2022.167505 8. Diaz-Pinto, A., Alle, S., Nath, V., Tang, Y., Ihsani, A., Asad, M., Pérez-García, F., Mehta, P., Li, W., Flores, M., Roth, H. R., Vercauteren, T., Xu, D., Dogra, P., Ourselin, S., Feng, A. & Cardoso, M. J. MONAI Label: A framework for AI-assisted interactive labeling of 3D medical images. *arXiv \[cs.HC]* (2022). at <> 9.
# IDC team Imaging Data Commons is being developed by a team of engineers and imaging scientists with decades of experience in cancer imaging informatics, cloud computing, imaging standards, security, open source tool development and data sharing.

IDC team during face-to-face project meeting at ISB, May 11, 2023

Our team includes the following sites and project leads: * Brigham and Women's Hospital, Boston, MA, USA (BWH) * Andrey Fedorov, PhD, and Ron Kikinis, MD - Co-PIs of the project * Katie Mastrogiacomo * Maria Loy * Institute for Systems Biology, Seattle, WA, USA (ISB) * David Gibbs, PhD - site PI * William Clifford, MS * Suzanne Paquette, MS * General Dynamics Information Technology, Bethesda, MD, USA (GDIT) * David Pot, PhD - site PI * Fabian Seidl * Fraunhofer MEVIS, Bremen, Germany (Fraunhofer MEVIS) * André Homeyer, PhD - site PI * Daniela Schacherer, MS * Henning Höfener, PhD * Massachusetts General Hospital, Boston, MA, USA (MGH) * Chris Bridge, DPhil - site PI * Radical Imaging LLC, Boston, MA, USA (Radical Imaging) * Rob Lewis, PhD - site PI * Igor Octaviano * PixelMed Publishing, Bangor, PA, USA (PixelMed) * David Clunie, MB, BS - site PI * Isomics Inc, Cambridge, MA, USA (Isomics) * Steve Pieper, PhD - site PI Oversight: * Leidos Biomedical Research * Ulrike Wagner - project manager * Todd Pihl - project manager * National Cancer Institute * Erika Kim - federal lead * Granger Sutton - federal lead ### IDC Alumni We are grateful to the following individuals who contributed to IDC in the past, but are no longer directly involved in the development of IDC. * Deepa Krishnaswamy, PhD (BWH) * William Longabaugh, MS (ISB) * George White (ISB) * Ilya Shmulevich, PhD (ISB) * Poojitha Gundluru (GDIT) * Prema Venkatesun (GDIT) * Chris Gorman, PhD (MGH) * Pedro Kohler (Radical Imaging) * Hugo Aerts, PhD (BWH) * Cosmin Ciausu, MS (BWH) * Keyvan Farahani (NCI) * Markus Herrmann (MGH) * Davide Punzo (Radical Imaging) * James Petts (Radical Imaging) * Erik Ziegler (Radical Imaging) * Gitanjali Chhetri (Radical Imaging) * Rodrigo Basilio (Radical Imaging) * Jose Ulloa (Radical Imaging) * Madelyn Reyes (GDIT) * Derrick Moore (GDIT) * Mark Backus (GDIT) * Rachana Manandhar (BWH) * Rasmus Kiehl (Fraunhofer MEVIS) * Chad Osborne (GDIT) * Afshin Akbarzadeh (BWH) * Dennis Bontempi (BWH) * Vamsi Thiriveedhi (BWH) * Jessica Cienda (GDIT) * Bernard Larbi (GDIT) * Mi Tian (ISB) # Acknowledgments * Imaging Data Commons team has been funded in whole or in part with Federal funds from the National Cancer Institute, National Institutes of Health, under Task Order No. HHSN26110071 under Contract No. HHSN261201500003l. * We gratefully acknowledge [Google Public Data Program](https://console.cloud.google.com/marketplace/product/bigquery-public-data/nci-idc-data) and the [AWS Open Data Sponsorship Program](https://registry.opendata.aws/nci-imaging-data-commons/) that support public hosting of IDC-curated content, and cover out-of-cloud egress fees! * Several of the members of the IDC team utilize [JetStream2](https://jetstream-cloud.org/) compute resources supported via the [ACCESS](https://access-ci.org/) Allocations program, which is in turn funded by the US National Science Foundation. You can see instructions on how to get your allocation at [this documentation page](/cookbook/access-allocations). # Jobs ## IDC does not currently have open positions
# Features The following components and capabilities are being developed by IDC: * [Cloud-hosted imaging data collections](/data/introduction) * [Search and cohort building portal](/portal/getting-started) (IDC Portal) * Visualization of the hosted imaging data supported by the integrated [OHIF Viewer](https://github.com/OHIF/Viewers) (radiology data), [SliM viewer](https://github.com/MGHComputationalPathology/slim) (digital pathology data), and [VolView](https://volview.kitware.com/) (volumetric 3D rendering of CT and MRI data) * [`idc-index` python package](https://github.com/ImagingDataCommons/idc-index) to support programmatic use of the IDC functionality * [REST API and MCP server](/api/api) for language-agnostic HTTP access and for LLM agents (via the [Model Context Protocol](/mcp/mcp)) to query IDC directly * [Interactive self-guided tutorials](https://github.com/ImagingDataCommons/IDC-Tutorials/tree/master/notebooks/getting_started) to help you get started IDC is being built utilizing the Google Cloud Platform (GCP) as the foundation. We use a range of products developed by GCP, most notably Google Healthcare, to implement the capabilities listed above. We discuss the role of the major GCP components used by IDC in [this section](/introduction/google-cloud-platform). IDC data in files is hosted both in Google Cloud Storage (GCS) buckets and (starting from v14) in Amazon AWS S3 buckets. GCS data storage and egress are sponsored by [Google Public Data Program](https://console.cloud.google.com/marketplace/product/bigquery-public-data/nci-idc-data). AWS data storage and egress sponsored by [AWS Open Data Sponsorship Program](https://registry.opendata.aws/nci-imaging-data-commons/). {% hint style="success" %} You can download IDC data (either to your computer or to a cloud VM) from either GCS or AWS buckets without the need to login, and without having to pay for egress! {% endhint %} IDC is relying on the [Digital Imaging and Communications in Medicine (DICOM)](https://www.dicomstandard.org/) standard for data modeling and representation. Most of the data hosted by IDC is in DICOM format. Our goal is to have all of the imaging (including digital pathology, microscopy and other non-radiological image types) and image-derived data (such as results of segmentation, image annotation, or post-processing) represented in the DICOM format. You can read more about our motivation and details of our approach in [this section](/introduction/dicom). # Google Cloud Platform (GCP) [Google Cloud Platform (GCP)](https://cloud.google.com/) is the foundation of the IDC architecture. We rely on GCP to implement the IDC functionality. IDC users can use GCP as the environment to perform the analysis of the data hosted on IDC. Specific GCP components that we utilize and that can be useful to support cancer imaging research use cases include: * [Compute Engine](https://cloud.google.com/compute) * [BigQuery](https://cloud.google.com/bigquery) * [Cloud Storage](https://cloud.google.com/storage) * [Healthcare API](https://cloud.google.com/healthcare/docs) Within CRDC, Cloud Resources are intended to support CRDC users' analysis needs. In the future, we plan to develop use cases and documentation demonstrating best practices for utilizing Cloud Resources for computational tasks. In addition to the GCP components listed above, the following components are not used to support IDC functionality, but can be quite handy: * [Colab Notebooks](https://colab.research.google.com/) * [AI Notebooks](https://cloud.google.com/ai-platform-notebooks) * [DataStudio](http://datastudio.google.com/) * [Datalab](https://cloud.google.com/datalab/docs) # Getting started with GCP You are also encouraged to review the slides in the following presentation that provides an introduction into GCP, and shares some best practices for its usage. > W. Longabaugh. *Introduction to Google Cloud Platform*. Presented at MICCAI 2021. ([slides in Google Slides](https://docs.google.com/presentation/d/1HNZ34xkZCXt6WRDcEtmAUGNq5TM0xzPaK7sojKJfoBc/edit?usp=sharing)) Google Cloud platform provides a range of solutions to better understand and analyze data hosted by IDC. Depending on what you want to do (see the range of options [here](/getting-started-with-idc)), you may need to complete one or more of the following steps below. The steps concerning creating a Google Cloud project and setting up billing are covered in Part 1 of our ["Getting started" tutorial series](https://github.com/ImagingDataCommons/IDC-Tutorials/tree/master/notebooks/getting_started), and in [this short video tutorial](https://youtu.be/i08S0KJLnyw). ### **Obtain a Google identity** Do you have a Google identity? If so, you can proceed to the next step. If not, it only takes a minute to [create a Google account](https://support.google.com/accounts/answer/27441?hl=en). Note that you do NOT need a Gmail email account - [you can use your non-Gmail email address to create one instead](https://support.google.com/accounts/answer/27441?hl=en#existingemail). ### **Set up a Google Cloud Project** To perform queries against IDC BigQuery tables you will need a cloud project. You can get a Google Cloud free project by following these steps (they are also illustrated in [this short video](https://youtu.be/i08S0KJLnyw)): 1. Go to , and accept Terms and conditions. 2. Click "Select a project" button in the upper left corner of the screen, and then click "New project". 3. Open the GCP Dashboard ( **≡** > Cloud overview > Dashboard) and take note of the "Project ID" value - you will need it to perform some of the operations. Additional reading materials: * See Google’s documentation about how to [create a Google Cloud Project](https://cloud.google.com/resource-manager/docs/creating-managing-projects). * Learn about how to [add members and roles to a project](https://cloud.google.com/iam/docs/quickstart). * [Enable Required Google Cloud APIs](https://cloud.google.com/apis/docs/getting-started#enabling_apis). ### Locate IDC metadata tables in Cloud BigQuery console IDC uses BigQuery for managing metadata for the hosted data. In order to locate the tables that contain such metadata, complete the following steps: 1. Open BigQuery console: 2. Click "+ ADD" button, and select "Star a project by name" from the Additional Resource table 3. Type `bigquery-public-data` in the text box and click "PIN" button 4. In the left panel, expand the `bigquery-public-data` drop-down, and navigate to the items called `idc_v1`, `idc_v2`, ..., `idc_current`, which are the datasets containing metadata tables maintained by IDC. Numbered datasets correspond to the IDC data versions documented in [Data Release Notes](/data/data-release-notes). `idc_current` is an alias that always points to the latest IDC version. ### Enable GCP BigQuery API Navigate to the GCP [BigQuery API page](https://console.cloud.google.com/marketplace/product/google/bigquery.googleapis.com). If the BigQuery API has not been enabled, you will see a blue "ENABLE" button that you will need to push to enable that API. This is needed in order to be able to query IDC BigQuery tables using Python API. ### \[skip if using Google Colab] Install and configure Cloud SDK Follow the instructions here to install and configure Google Cloud SDK: . Note that you will need to do this only if you want to interact with IDC data from your computer. If you use Google Colab, or Google Compute Engine VMs, Cloud SDK tools will be pre-installed and ready to use. ### **OPTIONAL: Set up billing for your project** You will not need to set up billing for your project in order to do basic operations with IDC, such as running Colab notebooks, or executing queries, as long as you stay within the [GCP free tier](https://cloud.google.com/free). You will need to set up project billing if you want to launch your own VMs, or use resources beyond the free usage tier. {% hint style="danger" %} Once you set up billing, we can't stress enough how important it is to be diligent in tracking your usage of GCP resources! * **Be sure to shut down anything you aren't using** - free trial credits, IDC-provided credits or your credit card will be charged otherwise for the resources you are not using. * Be careful with your login information. If someone takes over your account they **could run up a huge bill that you will be responsible for paying**. * Unless you are not concerned about billing, remember to **SHUT DOWN THE MACHINE** when you aren't using it! You are billed continuously while the VM instance is running. * Even after you stop the VMs, you keep paying for the disk storage attached to those machines! You can delete the VM instances to stop incurring those costs. {% endhint %} {% hint style="info" %} You can use [Budget Alerts](https://cloud.google.com/billing/docs/how-to/budgets) to monitor your adherence to a budget! {% endhint %} # Amazon AWS platform [Amazon Web Services (AWS)](https://aws.amazon.com/) provide cloud-based scalable solutions to support computation, storage and a range of other services. Starting from release v14, IDC files are replicated in AWS storage buckets (see [IDC dataset entry](https://registry.opendata.aws/nci-imaging-data-commons/) in the AWS Open Data Registry), and their AWS locations are indexed in IDC metadata tables. If you are using AWS to meet your analysis needs, you can download IDC data from AWS. Note that no matter whether you are [downloading IDC data](/data/downloading-data) from GCP or AWS, you do not need to log in or pay for data egress! As of IDC data release v14, in order to build a manifest for downloading IDC files from AWS, you will have to either use the IDC Portal (this can be done without a Google identity, and does not require a GCP project), or by querying GCP BigQuery tables (you will need to complete [GCP prerequisites](/introduction/google-cloud-platform/getting-started-with-gcp) for this step). # DICOM The DICOM standard is one of the key foundation components of the IDC. In the following we summarize the relevant characteristics of the DICOM standard, and provide relevant pointers, to support our decision to use DICOM in IDC both for data representation and as a communication protocol. {% hint style="info" %} DICOM is the comprehensive standard for the management and communication of medical imaging information and imaging-related data. {% endhint %} DICOM is used universally in clinical operations involving acquisition, communication, visualization and interpretation of radiological and cardiological images, and is used increasingly for all enterprise imaging applications, including medical photography, ophthalmology, dermatology, and more recently, for whole slide imaging for histopathology and cytology. DICOM applications, however, extend beyond clinical domains into *clinical trials* as well as *pre-clinical research*. DICOM approaches modeling the complexity of the real world using a hierarchy of abstract objects described by a set of attributes. Real-world objects, such as a patient, or a tissue specimen image, correspond to DICOM *Information Object Definitions (IODs)*. DICOM objects, such as images, are composite IODs that include attributes of different real-world entities (e.g., a CT image would include attributes describing the patient and imaging study, alongside attributes describing details of CT image acquisition). The resulting real-world model is comprehensive, incorporating a variety of objects corresponding to imaging modalities, imaging-derived data (e.g., results of post-processing and annotation of images), and imaging-related data (e.g., clinical information describing relevant patient characteristics or specimen handling). The standard and the data model are not limited to human imaging. The standard includes provisions that harmonize data representation for veterinary and pre-clinical small animal imaging, and includes capabilities to characterize, in detail, image acquisition context in pre-clinical applications (e.g., animal handling, housing or medication, critical for pre- and co-clinical trials, as an example). The DICOM data model is also extensible: there is an established process for modification and refinement of the DICOM standard through participation in working groups that include a range of stakeholders, such as vendors of imaging equipment, pharma, clinical and research users. Importantly, the DICOM standard is not limited to the definition of the data model and object storage representation. It is comprehensive in defining various aspects related to communication and management of imaging and image-related data, such as communication protocols and presentation of the data. The DICOMweb communication component of the standard found broad adoption in commercial and open source tools and, importantly, was chosen by Google Healthcare for cloud deployment of imaging data. By utilizing the DICOM standard, we will leverage decades of development and participation, and harmonization with the existing infrastructure. # Requesting GCP cloud credits Opportunities for free Google Cloud credits provided by Google: * If you are a first-time user of GCP, [you can get $300 of GCP credits ](https://cloud.google.com/free)for a 3-month free trial without any application process * You can apply for TPU credits through the [TensorFlow Research Cloud program](https://www.tensorflow.org/tfrc) * If you are an academic researcher, you can apply for [Google Research Credits](https://edu.google.com/programs/credits/research/) * If you are a faculty, there are opportunities to apply for credits to support students in your courses, see [Google Cloud for Faculty](https://cloud.google.com/edu/faculty) # Requesting AWS cloud credits IDC does not have a mechanism to provide AWS cloud credits to the users. IDC users that are looking for free AWS cloud credit should check the options available from AWS directly: . # Introduction ## Data sources Most of the data in IDC is received from the data collection initiatives/projects supported by US National Cancer Institute. Whenever source images or image-derived data is not in the DICOM format, it is harmonized into DICOM as part of the ingestion. As of data release v23, IDC sources of data include: * [The Cancer Imaging Archive (TCIA) (ongoing)](https://www.cancerimagingarchive.net/) * all DICOM files from the public collections are mirrored in IDC * a subset of digital pathology collections and analysis results harmonized from vendor-specific representation (as available from TCIA) into DICOM Slide Microscopy (SM) format * [Childhood Cancer Data Initiative (CCDI) (ongoing)](https://www.cancer.gov/research/areas/childhood/childhood-cancer-data-initiative) * digital pathology slides harmonized into DICOM SM * [Genomic Data Commons (GDC)](https://portal.gdc.cancer.gov/) * The Cancer Genome Atlas (TCGA) slides harmonized into DICOM SM * [Human Tumor Atlas Network (HTAN)](https://humantumoratlas.org/) * release 1 of the HTAN data harmonized into DICOM SM * [National Library of Medicine Visible Human Project](https://www.nlm.nih.gov/research/visible/visible_human.html) * v1 of the Visible Human images harmonized into DICOM MR/CT/XC * [Genotype-Tissue Expression Project (GTex)](https://commonfund.nih.gov/GTEx) * digital pathology slides harmonized into DICOM SM * [BoneMarrowWSI-PediatricLeukemia](https://doi.org/10.5281/zenodo.14933087) * a comprehensive dataset of bone marrow aspirate smear whole slide images with expert annotations and clinical data in pediatric leukemia The list of all of the IDC collections is available in IDC Portal here: . ## Data provenance Whenever IDC replicates data from a publicly available source, we include the reference to the origin: * from the IDC Portal Explore page, click on the "i" icon next to the collection in the collections list
* `source_doi` metadata column contains Digital Object Identifier (DOI) at the granularity of the individual files and is available both via [python `idc-index` package](https://github.com/ImagingDataCommons/idc-index) (see [this tutorial](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/part2_searching_basics.ipynb) on how to access it) and BigQuery interfaces {% hint style="info" %} Whenever source data is harmonized into DICOM, the DOI will correspond to a Zenodo entry for the result of harmonization, which in turn will reference the location where data can be accessed in the native format (if available). As an example, IDC NLM-Visible-Human-Project collection refers to this DOI that describes the dataset resulting from the original dataset harmonized into DICOM , which in turn references the [NLM Visible Human project page](https://www.nlm.nih.gov/research/visible/visible_human.html) containing information on accessing the original files collected by the project. {% endhint %} Check out [Data release notes](/data/data-release-notes) for information about the collections added in the individual IDC data releases. ## Data ingestion process Simplified workflow for IDC data ingestion is summarized in the following diagram. {% embed url="" %} IDC data ingestion workflow {% endembed %} # Data model IDC relies on the DICOM data model for organizing images and image-derived data. At the same time, IDC includes certain attributes and data types that are outside of the DICOM data model. The *Entity-Relationship (E-R) diagram* and examples below summarize a simplified view of the IDC data model (you will find the explanation of how to interpret the notation used in this E-R diagram in [this page](https://mermaid.js.org/syntax/entityRelationshipDiagram.html) from Mermaid documentation). ```mermaid erDiagram COLLECTION ||--o{ CASE: contains CASE ||--o{ STUDY : contains STUDY ||--o{ SERIES : contains SERIES ||--o{ INSTANCE : contains ANALYSIS_RESULT ||--o{ SERIES : adds ANALYSIS_RESULT }o--o{ COLLECTION : spans CASE |o--o| CLINICAL_DATA : "may have" PROGRAM ||--o{ COLLECTION : contains PROGRAM { string program PK } COLLECTION { string collection_id string source_doi } CASE { string PatientID } STUDY { string StudyInstanceUID } SERIES { string SeriesInstanceUID } INSTANCE { string SOPInstanceUID } ANALYSIS_RESULT { string analysis_result_id string source_doi } CLINICAL_DATA { string CaseID } ``` IDC content is organized in **Collections**: groups of DICOM files that were collected through certain research activity. We sometimes refer to these as **Original Collections** to distinguish them from Analysis Results collections described below. Collections are organized into **Programs**, which group related collections, or those collections that were contributed under the same funding initiative or a consortium. Example: TCGA program contains TCGA-GBM, TCGA-BRCA and other collections. You will see Collections nested under Programs in the upper left section of the [IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/). You will also see the list of collections that meet the filter criteria in the top table on the right-hand side of the portal interface. Individual DICOM files included in the collection contain attributes that organize content according to the [DICOM data model](/dicom/data-model). Each collection will contain data for one or more cases, or **patients**. Data for the individual patient is organized in DICOM **studies**, which group images corresponding to a single imaging exam/encounter, and collected in a given session. Studies are composed of DICOM **series**, which in turn consist of DICOM **instances**. Each DICOM instance corresponds to a single file on disk. As an example, in radiology imaging, individual instances would correspond to image slices in multi-slice acquisitions, and in digital pathology you will see a separate file/instance for each resolution layer of the image pyramid. When using the IDC Portal, you will never encounter individual instances - you will only see them if you download data to your computer. The **Analysis results collection** is a very important concept in IDC. An analysis result is the DICOM encoded result of some analysis performed on data from one or more original collections. Such analysis results are often contributed by investigators unrelated to those that submitted the analyzed images, and may span images across multiple collections. # Organization of data IDC provides a variety of interfaces to access both the data (as files) and metadata (to subset files and build cohorts). The flow of data and the relationship between the various components IDC uses is summarized in the following figure. {% embed url="" %} We maintain the following resources to enable access to IDC data: * [Cloud storage buckets](#files-and-metadata): files maintained by IDC are mirrored between Google and AWS public storage buckets that provide fee-free egress without requiring login. The buckets organize files by DICOM series, each series stored in a separate folder. Given the large overall size of data in IDC, you will likely need to use one of the search interfaces to identify relevant series first. * BigQuery tables: collection-level metadata, DICOM metadata, [clinical data tables](/data/organization-of-data/clinical) available via SQL query interface. * Python API: pip-installable [idc-index package](https://idc-index.readthedocs.io/en/latest/) provides a programmatic interface and command-line tools to search IDC data using most important metadata attributes, and to download files corresponding to the selected cohorts from the cloud buckets * [REST API and MCP server](/api/api): language-independent HTTP API (v3) for discovering, subsetting, and downloading data, plus an [MCP server](/mcp/mcp) exposing the same capabilities to LLM agents * [DICOMweb](/data/organization-of-data/dicom-stores): DICOM files and metadata queries available from Google Healthcare DICOM stores # Files and metadata {% hint style="info" %} We gratefully acknowledge [Google Public Data Program](https://console.cloud.google.com/marketplace/product/bigquery-public-data/nci-idc-data) and the [AWS Open Data Sponsorship Program](https://registry.opendata.aws/nci-imaging-data-commons/) that support public hosting of IDC-curated content, and cover out-of-cloud egress fees! {% endhint %} ```mermaid flowchart TB  subgraph BUCKETS["CLOUD STORAGE BUCKETS (AWS S3 + GCS mirrors)"]     direction LR         B1["gs://idc-open-data
s3://idc-open-data
~90%, CC BY"]         B2["gs://idc-open-idc1
s3://idc-open-data-two
potential head scans, CC BY"]         B3["gs://idc-open-cr
s3://idc-open-data-cr
~4%, CC BY-NC"]   end  subgraph STORES["DICOMweb / DICOM STORES"]     direction LR         PROXY["IDC DICOM store
IDC Public Proxy in front of Google Healthcare DICOM store
No auth, 100% coverage"]         GHC["Google Healthcare DICOM store
Google Healthcare API
Auth required, >95% coverage"]   end  subgraph BQ["BigQuery (GCP auth + billing)
      All 4000+ DICOM tags                    
Tables: dicom_all, dicom_metadata, derived metadata, clinical"]   end  subgraph IDX["idc-index PARQUET FILES (no auth)
~50 key columns per series, bundled in Python package          
Auto-loaded: index, prior_versions_index
On-demand: collections, seg, sm, ann, clinical, contrast"]   end     DCM["DICOM FILES (.dcm)
Named by crdc_instance_uuid, grouped by crdc_series_uuid"] -- stored in --> BUCKETS     B1 -- DICOM instances
imported into --> PROXY     B2 -- DICOM instances
imported into --> PROXY     B3 -- DICOM instances
imported into --> PROXY     B1 -- DICOM instances
imported into --> GHC     PROXY -- DICOM metadata exported to --> BQ     BQ -- ~50 key columns queried via SQL --> IDX     BQ -- tables exported to --> S3BQ["Parquet files in AWS S3"]     IDX -. SeriesInstanceUID for DICOMweb queries .-> STORES     IDX -. series_aws_url / crdc_series_uuid maps to bucket paths .-> BUCKETS     style DCM fill:#fff3e0,stroke:#FF9800,stroke-width:2px,color:#000     style BUCKETS fill:#e8f4fd,stroke:#2196F3,stroke-width:2px,color:#000     style B1 fill:#e8f4fd,stroke:#2196F3,color:#000     style B2 fill:#e8f4fd,stroke:#2196F3,color:#000     style B3 fill:#e8f4fd,stroke:#2196F3,color:#000     style PROXY fill:#f3e5f5,stroke:#9C27B0,color:#000     style GHC fill:#f3e5f5,stroke:#9C27B0,color:#000     style BQ fill:#fce4ec,stroke:#E91E63,stroke-width:2px,color:#000     %% style BQ_DESC fill:#fce4ec,stroke:none,color:#000     style IDX fill:#e8f5e9,stroke:#4CAF50,stroke-width:2px,color:#000     style S3BQ fill:#e8f4fd,stroke:#2196F3,stroke-width:2px,color:#000     %% style IDX_DESC fill:#e8f5e9,stroke:none,color:#000     style STORES fill:#f3e5f5,stroke:#9C27B0,stroke-width:2px,color:#000 ``` Let's start with the overall principles of how we organize data in IDC. IDC brings you (as of v23) over 95 TB of publicly available DICOM images and image-derived content. We share those with you as DICOM files, and those DICOM files are available in cloud-based **storage buckets** - both in Google and AWS. Sharing just the files, however, is not particularly helpful. With that much data, it is no longer practical to just download all of those files to later sort through them to select those you need. {% hint style="success" %} Think of IDC as a library, where each file is a book. With that many books, it is not feasible to read them all, or even open each one to understand what is inside. Libraries are of little use without a catalog! {% endhint %} To provide you with a catalog of our data, along with the files, we maintain *metadata* that makes it possible to understand what is contained within files, and select the files that are of interest for your project, so that you can download just the files you need. In the following we describe organization of both the storage buckets containing the files, and the metadata catalog that you can use to select files that meet your needs. As you go over this documentation, please consider completing our ["Getting started" tutorial](https://github.com/ImagingDataCommons/IDC-Tutorials/tree/master/notebooks/getting_started) - it will give you the opportunity to apply the knowledge you gain by reading this article while interacting with the data, and should help better understand this content. ## Storage Buckets {% hint style="info" %} Storage Buckets are basic containers in Google Cloud Storage and AWS S3 that provide storage for data objects (you can read more about the relevant terms in the Google Cloud Storage documentation [here](https://cloud.google.com/storage/docs/key-terms) and in S3 [here](https://aws.amazon.com/s3/)). {% endhint %} All IDC DICOM file data for all IDC data versions across all of the [collections hosted by IDC](https://imaging.datacommons.cancer.gov/collections/) are mirrored between Google Cloud Storage (GCS) and AWS S3 buckets. Currently all DICOM files are maintained in buckets that allow for free egress within or out of the cloud. This is enabled through the partnership of IDC with [Google Public Data Program](https://console.cloud.google.com/marketplace/product/gcp-public-data-idc/nci-idc-data) and the [AWS Open Data Sponsorship Program](https://registry.opendata.aws/nci-imaging-data-commons/).
Data categoryCloud provider and bucket name
Data covered by a non-restrictive license (CC-BY or like) and not labeled as such that may contain head scans. This category contains >90% of the data in IDC.AWS: idc-open-data
GCS: idc-open-data
(until IDC v19, we utilized GCS bucket public-datasets-idc before it was superseded by idc-open-data)
Collections that may contain head scans. This is done for the collections that were labeled as such by TCIA, in case there is a change in policy and we need to treat such images in any special way in the future.AWS: idc-open-data-two
GCS: idc-open-idc1
Data that is covered by a license that restricts commercial use (CC-NC). Note that the license information is available programmatically at the granularity of the individual files, as explained in this tutorial - you do not need to check the bucket name to get the license information!AWS: idc-open-data-cr
GCS: idc-open-cr
Within each bucket files are organized in folders, each folder containing files corresponding to a single DICOM series. On ingestion, we assign each DICOM series and each DICOM instance a UUID, in order to be able to support [data versioning](/data/data-versioning) (when needed). These UUIDs are available in our metadata indices, and are used to organize the content of the buckets: for each version of a DICOM instance having instance UUID `instance_uuid` in a version of a series version having UUID `series_uuid`, the file name is: `/.dcm` Corresponding files have the same object name in GCS and S3, though the name of the containing buckets will be different. ## Metadata IDC metadata tables are provided to help you navigate IDC content and narrow down to the specific files that meet your research interests. As a step in data ingestion process (summarized [earlier](/data/organization-of-data)), IDC extracts all of the DICOM metadata, merges it with collection-level and some other metadata attributes not available from DICOM, ingests collection-level clinical tables and stores the result in **Google BigQuery tables**. Google [BigQuery (BQ)](https://cloud.google.com/bigquery) is a massively-parallel analytics engine ideal for working with tabular data. Data stored in BQ can be accessed using [standard SQL](https://cloud.google.com/bigquery/docs/reference/standard-sql/enabling-standard-sql) queries. We talk more about those in the subsequent sections of the documentation! Searching BigQuery tables requires you to sign in with a Google Account! If this poses a problem for you, there are several alternatives. {% hint style="danger" %} `idc-index` provides access to the metadata aggregated at the DICOM series level. BigQuery and Parquet files provide metadata at the granularity of individual DICOM instances (files). {% endhint %} #### Python *idc-index* package A small subset of most critical metadata attributes available in IDC BigQuery tables is extracted and made available via the [`idc-index` python package](https://github.com/ImagingDataCommons/idc-index). If you are just starting with IDC, you can skip the details covering the content of BigQuery tables, and proceed to [this tutorial](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/part2_searching_basics.ipynb) that will help you learn the basics of searching IDC metadata using `idc-index`. But for the sake of example, you would select and download MR DICOM series available in IDC as follows. ```bash pip install --upgrade idc-index ``` {% code overflow="wrap" %} ```python from idc_index import IDCClient # instantiate the client client = IDCClient() # define and execute the query selection_query = """ SELECT SeriesInstanceUID FROM index WHERE Modality = 'MR' """ selection_result = client.sql_query(selection_query) # download the first series from the list client.download_dicom_series(seriesInstanceUID=selection_result["SeriesInstanceUID"].values[0],downloadDir=".") ``` {% endcode %} #### Parquet files available via a cloud bucket While the idc-index is based on a series level subset of IDC BQ data, we also export all the content available via BigQuery into Parquet () files, and which are available from our public AWS bucket! Using open-source tools such as DuckDB () you can query those files using SQL queries, without relying on BigQuery (although, running complex queries may require significant resources from your runtime environment!). The exported Parquet files are located in the IDC-maintained AWS `idc-open-metadata` bucket, which is updated every time IDC has a new data release. The exported tables are organized under the folder `bigquery_export` in that bucket, with each sub-folder corresponding to a BigQuery dataset. Assuming you have `s5cmd` installed, you can list the exported datasets as follows. 
$ s5cmd --no-sign-request ls s3://idc-open-metadata/bigquery_export/
                                  DIR  idc_current/
                                  DIR  idc_current_clinical/
                                  DIR  idc_v1/
                                  DIR  idc_v10/
                                  DIR  idc_v11/
                                  DIR  idc_v11_clinical/
                                  DIR  idc_v12/
                                  DIR  idc_v12_clinical/
                                  DIR  idc_v13/
                                  DIR  idc_v13_clinical/
                                  DIR  idc_v14/
                                  DIR  idc_v14_clinical/
                                  DIR  idc_v15/
                                  DIR  idc_v15_clinical/
                                  DIR  idc_v16/
                                  DIR  idc_v16_clinical/
                                  DIR  idc_v17/
                                  DIR  idc_v17_clinical/
                                  DIR  idc_v18/
                                  DIR  idc_v18_clinical/
                                  DIR  idc_v19/
                                  DIR  idc_v19_clinical/
                                  DIR  idc_v2/
                                  DIR  idc_v20/
                                  DIR  idc_v20_clinical/
                                  DIR  idc_v21/
                                  DIR  idc_v21_clinical/
                                  DIR  idc_v22/
                                  DIR  idc_v22_clinical/
                                  DIR  idc_v23/
                                  DIR  idc_v23_clinical/
                                  DIR  idc_v3/
                                  DIR  idc_v4/
                                  DIR  idc_v5/
                                  DIR  idc_v6/
                                  DIR  idc_v7/
                                  DIR  idc_v8/
                                  DIR  idc_v9/
The `idc_current` and `idc_current_clinical` datasets always contain the most recent version of data. As an example, the `dicom_all` table for the latest (current) IDC release can be accessed as `s3://idc-open-metadata/bigquery_export/idc_current/dicom_all` (since the table is quite large, the export result is not a single file, but a folder containing thousands of Parquet files. {% code overflow="wrap" %} ``` $ s5cmd --no-sign-request ls s3://idc-open-metadata/bigquery_export/idc_current/dicom_all/ 2024/11/23 18:01:07 7545045 000000000000.parquet 2024/11/23 18:01:07 7687834 000000000001.parquet 2024/11/23 18:01:07 7409070 000000000002.parquet 2024/11/23 18:01:07 7527558 000000000003.parquet ... ... 2024/11/23 18:00:14 7501451 000000004997.parquet 2024/11/23 18:00:14 7521972 000000004998.parquet 2024/11/23 18:00:14 7575037 000000004999.parquet 2024/09/12 18:20:05 588723 000000005000.parquet ``` {% endcode %} You can query those tables/parquet files without downloading them, as shown in the following snippet. Depending on the query you are trying to execute, you may need a lot of patience! {% code overflow="wrap" %} ```python import duckdb # Connect to DuckDB (in-memory) con = duckdb.connect() # Install and load the httpfs extension for S3 access con.execute("INSTALL httpfs;") con.execute("LOAD httpfs;") # No credentials needed for public buckets # Query all Parquet files in the public S3 folder selection_query = """ SELECT SeriesInstanceUID FROM read_parquet('s3://idc-open-metadata/bigquery_export/idc_current/dicom_all/*.parquet') AS dicom_all WHERE Modality = 'MR' LIMIT 1 """ selection_result = con.execute(selection_query).fetchdf() print(selection_result['SeriesInstanceUID'].values[0]) ``` {% endcode %} # BigQuery tables BQ **tables** are organized in BQ **datasets**. BQ datasets are not unlike folders on your computer, but contain tables related to each other instead of files. BQ datasets, in turn, are organized under Google Cloud Platform (GCP) **projects**. GCP projects can be thought of as containers that are managed by a particular organization. To continue with the file system analogy, think about projects as hard drives that contain folders. This may be a good time for you to complete [Part 1 of the IDC "Getting started" tutorial series](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/part1_prerequisites.ipynb), so that you are able to open the tables and datasets we will be discussing in the following paragraphs! Let's map the aforementioned project-dataset-table hierarchy to the concrete locations that contain IDC data. ### IDC BigQuery datasets All of the IDC tables are stored under the `bigquery-public-data` project. That project is managed by Google Public Datasets Program, and contains many public BQ datasets, beyond those maintained by IDC. All of the IDC tables are organized into datasets by data release version. If you complete the tutorial mentioned above, open the BQ console, and scroll down the list of datasets, you will find those that are named starting with the `idc_v` prefix - those are IDC datasets.

Some of the BigQuery datasets curated by IDC.

Following the prefix, you will find the number that corresponds to the IDC data release version. IDC data has released versions numbered starting from 1 and are incremented by one for each subsequent release. As of writing this, the most recent version of IDC is 23, and you can find dataset `idc_v23` corresponding to this version. In addition to `idc_v23` you will find a dataset named `idc_v23_clinical`. That dataset contains clinical data accompanying IDC collections. We started clinical data ingestion in IDC v11. If you want to learn more about the organization and searching of clinical data, take a look at the [clinical data documentation](/data/organization-of-data/clinical). Finally, you will also see two special datasets: `idc_current` and `idc_current_clinical`. Those two datasets are essentially aliases, or links, to the versioned datasets corresponding to the latest release of IDC data. {% hint style="warning" %} If you want to explore the latest content of IDC - use `current` datasets. If you want to make sure your queries and data selection are reproducible - always use the version numbered datasets! {% endhint %} ### IDC BigQuery tables Before we dive into discussing the individual tables maintained by IDC, there is just one more BigQuery-specific concept you need to learn: the *view*. A BigQuery view is a table that is defined by an SQL query that is run every time you query the view (you can read more about BQ views in [this article](https://cloud.google.com/bigquery/docs/views-intro)). BQ views can be very handy when you want to simplify your queries by factoring out the part of the query that is often reused. But a key disadvantage of BQ views over tables is the reduced performance and increased cost due to re-running the query each time you query the view. As we will discuss further, most of the tables maintained by IDC are created by joining and/or post-processing other tables. Because of this we rely heavily on BQ views to improve transparency of the provenance of those "derived" tables. BQ views can be easily distinguished from the tables in a given dataset by a different icon. IDC datasets also follow a convention that all views in the versioned datasets include suffix `_view` in the name, and are accompanied by the result of running the query used by the view in a table that has the same name *sans* the `_view` suffix. See the figure below for an illustration of this convention.

In this example, dicom_all_view is a BQ view, as indicated by the icon to the left from the table name. dicom_all table is the result of running the query that defines the dicom_all_view.

If you are ever curious (and you should be, at least once in a while!) about the queries behind individual views, you can click on the view in the BQ console, and see the query in the "Details" tab. Try this out yourself to check the query for [`dicom_all_view`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=dicom_all_view\&page=table)

Yes, you can view the query of a view!

Now that we reviewed the main concepts behind IDC tables organization, it is time to explain the sources of metadata contained in those tables. Leaving `_clinical` datasets aside, IDC tables are populated from one of the two sources: * DICOM metadata extracted from the DICOM files hosted by IDC, and various derivative tables that simplify access to specific DICOM metadata items; * collection-level and auxiliary metadata, which is not stored in DICOM tags, but is either received by IDC from other sources, or is populated by IDC as part of data curation (these include Digital Object Identifiers, description of the collections, hashsums, etc). The set of BQ tables and views has grown over time. The enumeration below documents the BQ tables and views as of IDC v23. Some of these tables will not be found in earlier IDC BigQuery datasets. #### Foundation tables #### `dicom_metadata` {% hint style="info" %} Table in BigQuery: [`dicom_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=dicom_metadata\&page=table) {% endhint %} Each row in the `dicom_metadata` table holds the DICOM metadata of an instance in the corresponding IDC version. There is a single row for each DICOM instance in the corresponding IDC version, and the columns correspond to the DICOM attributes encountered in the data across all of the ingested instances. IDC utilizes the standard capabilities of the Google Healthcare API to extract all of the DICOM metadata from the hosted collections into a single BQ table. Conventions of how DICOM attributes of various types are converted into BQ form are covered in the [Understanding the BigQuery DICOM schema](https://cloud.google.com/healthcare/docs/how-tos/dicom-bigquery-schema) Google Healthcare API documentation article. The `dicom_metadata` table contains DICOM metadata extract from the files included in the given IDC data release. I.E., the `dicom_metadata` table in some IDC BQ dataset, e.g. dataset idc\_v22, will contain the metadata of only the DICOM instances in IDC version 22. The amount and variety of the DICOM files grows with the new releases, and the schema of this table reflects the organization of the metadata in each IDC release. Non-sequence attributes, such as `Modality` or `SeriesInstanceUID`, once encountered in any one file will result in the corresponding column being introduced to the table schema (i.e., if we have column `X` in IDC release 11, in all likelihood it will also be present in all of the subsequent releases). Sequence DICOM attributes, however, may have content that is highly variable across different DICOM instances (especially in Structured Reports). Those attributes will map to the [`STRUCT` BQ SQL type](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#struct_type), and it is not unusual to see drastic differences in the corresponding columns of the table between different releases. `dicom_metadata` can be used to conduct detailed explorations of the metadata content, and build cohorts using fine-grained controls not accessible from the IDC portal. Note that the `dicom_all` table, described below, is probably a better choice for such explorations. {% hint style="warning" %} Due to the existing limitations of Google Healthcare API, not all of the DICOM attributes are extracted and are available in BigQuery tables. Specifically: * sequences that have more than 15 levels of nesting are not extracted (see ) - we believe this limitation does not affect the data stored in IDC * sequences that contain around 1MiB of data are dropped from BigQuery export and RetrieveMetadata output currently. 1MiB is not an exact limit, but it can be used as a rough estimate of whether or not the API will drop the tag (this limitation was not documented as of writing this) - we know that some of the instances in IDC will be affected by this limitation. The fix for this limitation is targeted for sometime in 2021, according to the communication with Google Healthcare support. {% endhint %} #### `auxiliary_metadata` {% hint style="info" %} Table in BigQuery: [`auxiliary_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=auxiliary_metadata\&page=table) {% endhint %} This table defines the contents of the corresponding IDC version. There is a row for each instance in the version. We group the attributes for convenience: Collection attributes: * `collection_name:` Collection name as used externally by IDC webapp) * `collection_id:` Collection ID as used internally by IDC webapp and accepted by the IDC API * `collection_timestamp:` Datetime when the IDC data in the collection was last revised * `collection_hash`: md5 hash of the of this version of the collection containing this instance * `collection_init_idc_version:` The IDC version in which the collection containing this instance first appeared * `collection_revised_idc_version:` The IDC version in which this version of the collection containing this instance first appeared Patient attributes: * `submitter_case_id:`The Patient ID assigned by the submitter of this data. This is the same as the DICOM PatientID * `idc_case_id:`IDC generated UUID that uniquely identifies the patient containing this instance This is needed because DICOM PatientIDs are not required to be globally unique * `patient_hash`: md5 hash of this version of the patient/case containing this instance * `patient_init_idc_version:` The IDC version in which the patient containing this instance first appeared * `patient_revised_idc_version:` The IDC version in which this version of the patient/case containing this instance first appeared Study attributes: * `StudyInstanceUID:` DICOM UID of the study containing this instance * `study_uuid:`IDC assigned UUID that identifies a version of the study containing this instance. * `study_instances:` The number of instances in the study containing this instance * `study_hash`: md5 hash of the data in this version of the study containing this instance * `study_init_idc_version:` The IDC version in which the study containing this instance first appeared * `study_revised_idc_version:` The IDC version in which this version of the study containing this instance first appeared Series attributes: * `SeriesInstanceUID:` DICOM UID of the series containing this instance * `series_uuid:`IDC assigned UUID that identifies the version of the series containing this instance * `series_gcs_url`: URL of the Google Cloud Storage (GCS) folder of the series containing this instance * `series_aws_url`: URL of the Amazon Web Services (AWS) folder of the series containing this instance * `source_doi:` The DOI of an information page corresponding to the original data collection or analysis results that is the source of this instance * `source_url:` The URL of an information page that describes the original collection or analysis result that is the source of this instance * `versioned_source_doi`: If non-null, the DOI of a wiki page that describes the original collection or analysis result that includes this version of this instance * `series_instances:` The number of instances in the series containing this instance * `series_hash`: md5 hash of the data in the this version of the series containing this instance * `access:` Collection access status: 'Public' or 'Limited'. (Currently all data is 'Public') * `series_init_idc_version:` The IDC version in which the series containing this instance first appeared * `series_revised_idc_version:` The IDC version in which this version of the series containing this instance first appeared Instance attributes: * `SOPInstanceUID:` DICOM UID of this instance. * `instance_uuid:`IDC assigned UUID that identifies the version of this instance. * `gcs_url:` The GCS URL of a file containing the version of this instance that is identified by this `series_uuid/instance_uuid` * `gcs_bucket` : Name of the Google Cloud Storage (GCS) bucket containing the current version of this instance * `aws_url:` The AWS URL of a file containing the version of this instance that is identified by this `series_uuid/instance_uuid` * `aws_bucket` : Name to the Amazon Web Services (AWS) bucket containing the current version of this instance * `instance_hash`: the md5 hash of this version of this instance * `instance_size:` the size, in bytes, of this version of this instance * `instance_init_idc_version:` The IDC version in which this instance first appeared * `instance_revised_idc_version:` The IDC version in which this version of this instance first appeared * `license_url:` The URL of a web page that describes the license governing this version of this instance * `license_long_name:` A long form name of the license governing this version of this instance * `license_short_name:` A short form name of the license governing this version of this instance #### `mutable_metadata` {% hint style="info" %} Table in BigQuery: [`mutable_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=original_collections_metadata\&page=table) {% endhint %} Some non-DICOM metadata may change over time. This includes the GCS and AWS URLs of instance data, the license of an instance, the accessibility of each instance and the URL of an instance's associated description page. BigQuery metadata tables such as the auxiliary\_metadata and dicom\_all tables in an IDC per-version dataset are never revised even when such metadata changes. However, tables in the datasets of previous IDC versions can be joined with the mutable\_metadata table to obtain the current values of these mutable attributes. The table has one row for each version of each instance: * `crdc_instance_uuid`: The uuid of an instance version * `crdc_series_uuid`: The uuid of a series version that contains this instance version * `crdc_study_uuid`: The uuid of a study version that contains the series version * `gcs_url`: URL to the Google Cloud Storage (GCS) object containing this instance version * `aws_url`: URL to the Amazon Web Services (AWS) object containing this instance version * `access`: Current access status of this instance (Public or Limited) * `source_url`: The URL of a page that describes the original collection or analysis result that includes this instance * `source_doi`: The DOI of a page that describes the original collection or analysis result that includes this instance * `versioned_source_doi`: If non-null, the DOI of a wiki page that describes the original collection or analysis result that includes this version of this instance * `license_long_name`: Long name of license of this analysis result * `license_short_name`: Short name of license of this analysis result * `license_url`: URL of license of this analysis result #### `original_collections_metadata` {% hint style="info" %} Table in BigQuery: [`original_collections_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=original_collections_metadata\&page=table) {% endhint %} This table is comprised of IDC data collection-level metadata for the original TCIA data collections hosted by IDC, for the most part corresponding to the content available in [this table at TCIA](https://www.cancerimagingarchive.net/collections/). One row per collection: * `collection_name`:Collection name as used externally by IDC webapp * `collection_id`: Collection ID as used internally by IDC webapp * `CancerTypes`: Summary of cancer types in this collection * `TumorLocations`: Body location that was studied * `Subjects`: Number of subjects in the collection * `Species:` Species of collection subjects * `Sources`: A repeated record of * `Access`: Limited or Public (currently only Public) * `source_doi`: DOI that can be resolved at doi.org to a collection information page * `source_url`: URL of collection information page * `ImageType`: Enumeration of image types/modalities in this source * `license`: a record of * `license_url`: URL of license of this (sub)collection * `license_long_name`: Long name of license of this (sub)collection * `license_short_name`: Short name of license of this (sub)collection * `Citation`: Citation to be used for this source * `SupportingData:`Type(s) of additional data available * `Program:` The program to which this collection belongs * `Status:`Collection status: "Ongoing" or "Complete" * `Updated:` Most recent update date reported by the collection source * `Description:` Description of the collection (HTML format) #### `analysis_results_metadata` {% hint style="info" %} Table in BigQuery: [`analysis_results_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=analysis_results_metadata\&page=table) {% endhint %} Metadata for the TCIA analysis results hosted by IDC, for the most part corresponding to the content available in [this table at TCIA](https://www.cancerimagingarchive.net/tcia-analysis-results/). One row per analysis result: * `ID:` Results ID * `Title:` Descriptive title * `Access:` Limited of Public (currently all Public) * `source_doi:`DOI that can be resolved at doi.org to the information page for this analysis result * `source_url:` URL of a the information page for this analysis result * `CancerTypes:`Types of cancer analyzed * `TumorLocations:`Body locations that were studied * `Subjects:`Number of subjects whose data was analyzed * `Collections:` collection\_names of original data collections analyzed * `Updated:` Date when results were last updated * `license_url:` URL of license of this analysis result * `license_long_name:` Long name of license of this analysis result * `license_short_name:` Short name of license of this analysis result * `description:` Description of analysis result * `Citation:` Citation to be used for this source #### `version_metadata` {% hint style="info" %} Table in BigQuery: [`version_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=version_metadata\&page=table) {% endhint %} Metadata for each IDC version, one row per version: * `idc_version`: IDC version number * `version_hash`: MD5 hash of hashes of collections in this version * `version_timestamp`: Version creation timestamp #### `idc_pivot` A table that is the basis for the queries performed by the IDC web app. #### Derived tables and views The following tables and views consist of metadata derived from one or more other IDC tables for the convenience of the user. For each such table, ``, there is also a corresponding view, `_view`, that, when queried, generates an equivalent table. Several of these tables/views are discussed more completely [here](/dicom/derived-objects). These views are intended as a reference; each view's SQL is available to be used for further investigation. However, the SQL that defines each IDC view is not available from the BQ console. We therefore include it here. #### `dicom_all`, `dicom_all_view` {% hint style="info" %} Table in BigQuery: [`dicom_all`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=dicom_all\&page=table) {% endhint %} {% hint style="info" %} Table in BigQuery: [`dicom_all_view`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=dicom_all_view\&page=table) {% endhint %} All columns from `dicom_metadata` together with selected date from the `auxiliary_metadata`, `original_collections_metadata`, and `analysis_results_metadata` tables. `dicom_all_view` SQL: ```googlesql WITH aux AS ( SELECT am.*, arm.ID as analysis_result_id FROM `nci-idc-bigquery-data.idc_current.auxiliary_metadata` am LEFT JOIN `nci-idc-bigquery-data.idc_current.analysis_results_metadata` arm ON (LOWER(am.source_doi) = LOWER(arm.source_doi)) ), pre_dicom_all AS ( SELECT aux.collection_name AS collection_name, aux.collection_id AS collection_id, aux.collection_timestamp AS collection_timestamp, aux.collection_hash as collection_hash, aux.collection_init_idc_version AS collection_init_idc_version, aux.collection_revised_idc_version AS collection_revised_idc_version, data_collections.TumorLocations AS collection_tumorLocation, data_collections.Species AS collection_species, data_collections.CancerTypes AS collection_cancerType, aux.access AS access, aux.idc_case_id as idc_case_id, aux.patient_hash as patient_hash, aux.patient_init_idc_version AS patient_init_idc_version, aux.patient_revised_idc_version AS patient_revised_idc_version, aux.study_uuid as crdc_study_uuid, aux.study_hash as study_hash, aux.study_init_idc_version AS study_init_idc_version, aux.study_revised_idc_version AS study_revised_idc_version, aux.series_uuid as crdc_series_uuid, aux.series_gcs_url as series_gcs_url, aux.series_aws_url as series_aws_url, aux.series_hash as series_hash, aux.series_init_idc_version AS series_init_idc_version, aux.series_revised_idc_version AS series_revised_idc_version, aux.SOPInstanceUID AS SOPInstanceUID, aux.instance_uuid as crdc_instance_uuid, aux.gcs_url as gcs_url, aux.gcs_bucket as gcs_bucket, aux.aws_url as aws_url, aux.aws_bucket as aws_bucket, aux.instance_size as instance_size, aux.instance_hash as instance_hash, aux.instance_init_idc_version AS instance_init_idc_version, aux.instance_revised_idc_version AS instance_revised_idc_version, aux.Source_DOI as Source_DOI, aux.Source_URL as Source_URL, aux.versioned_source_doi as Versioned_Source_DOI, aux.analysis_result_id as analysis_result_id, aux.license_url as license_url, aux.license_long_name as license_long_name, aux.license_short_name as license_short_name, -- aux.collection_name AS tcia_api_collection_id, -- aux.collection_id AS idc_webapp_collection_id, -- data_collections.Location as tcia_tumorLocation, -- data_collections.Species as tcia_species, -- data_collections.CancerType as tcia_cancerType FROM aux INNER JOIN `nci-idc-bigquery-data.idc_current.original_collections_metadata` AS data_collections ON aux.collection_id = data_collections.collection_id) SELECT pda.collection_name AS collection_name, pda.collection_id AS collection_id, pda.collection_timestamp AS collection_timestamp, pda.collection_hash as collection_hash, pda.collection_init_idc_version AS collection_init_idc_version, pda.collection_revised_idc_version AS collection_revised_idc_version, pda.collection_tumorLocation AS collection_tumorLocation, pda.collection_species AS collection_species, pda.collection_cancerType AS collection_cancerType, pda.access AS access, dcm.PatientID as PatientID, pda.idc_case_id as idc_case_id, pda.patient_hash as patient_hash, pda.patient_init_idc_version AS patient_init_idc_version, pda.patient_revised_idc_version AS patient_revised_idc_version, dcm.StudyInstanceUID AS StudyInstanceUID, pda.crdc_study_uuid as crdc_study_uuid, pda.study_hash as study_hash, pda.study_init_idc_version AS study_init_idc_version, pda.study_revised_idc_version AS study_revised_idc_version, dcm.SeriesInstanceUID AS SeriesInstanceUID, pda.crdc_series_uuid as crdc_series_uuid, pda.series_gcs_url as series_gcs_url, pda.series_aws_url as series_aws_url, pda.series_hash as series_hash, pda.series_init_idc_version AS series_init_idc_version, pda.series_revised_idc_version AS series_revised_idc_version, dcm.SOPInstanceUID AS SOPInstanceUID, pda.crdc_instance_uuid as crdc_instance_uuid, pda.gcs_url as gcs_url, pda.gcs_bucket as gcs_bucket, pda.aws_url as aws_url, pda.aws_bucket as aws_bucket, pda.instance_size as instance_size, pda.instance_hash as instance_hash, pda.instance_init_idc_version AS instance_init_idc_version, pda.instance_revised_idc_version AS instance_revised_idc_version, pda.Source_DOI as Source_DOI, pda.Source_URL as Source_URL, pda.Versioned_Source_DOI as Versioned_Source_DOI, pda.analysis_result_id as analysis_result_id, pda.license_url as license_url, pda.license_long_name as license_long_name, pda.license_short_name as license_short_name, dcm.* except(SOPInstanceUID, PatientID, StudyInstanceUID, SeriesInstanceUID) FROM pre_dicom_all pda INNER JOIN `nci-idc-bigquery-data.idc_current.dicom_metadata` AS dcm ON pda.SOPInstanceUID = dcm.SOPInstanceUID ``` #### `segmentations`, `segmentations_view` {% hint style="info" %} Table in BigQuery: [`segmentations`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=segmentations\&page=table) {% endhint %} {% hint style="info" %} Table in BigQuery: [`segmentations_view`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=segmentations_view\&page=table) {% endhint %} This table is derived from `dicom_all` to simplify access to the attributes of DICOM Segmentation objects available in IDC. Each row in this table corresponds to one DICOM Segmentation instance segment. `segmentations_view` SQL: ```googlesql WITH segmentations AS ( WITH segs AS ( SELECT PatientID, StudyInstanceUID, SeriesInstanceUID, SOPInstanceUID, FrameOfReferenceUID, SegmentSequence FROM `nci-idc-bigquery-data.idc_current.dicom_metadata` WHERE # more reliable than Modality = "SEG" SOPClassUID = "1.2.840.10008.5.1.4.1.1.66.4" ) SELECT PatientID, StudyInstanceUID, SeriesInstanceUID, SOPInstanceUID, FrameOfReferenceUID, CASE ARRAY_LENGTH(unnested.AnatomicRegionSequence) WHEN 0 THEN NULL ELSE STRUCT( unnested.AnatomicRegionSequence [ OFFSET (0)].CodeValue AS CodeValue, unnested.AnatomicRegionSequence [ OFFSET (0)].CodingSchemeDesignator AS CodingSchemeDesignator, unnested.AnatomicRegionSequence [ OFFSET (0)].CodeMeaning AS CodeMeaning ) END AS AnatomicRegion, CASE ( ARRAY_LENGTH(unnested.AnatomicRegionSequence) > 0 AND ARRAY_LENGTH( unnested.AnatomicRegionSequence [ OFFSET (0)].AnatomicRegionModifierSequence ) > 0 ) WHEN TRUE THEN unnested.AnatomicRegionSequence [ OFFSET (0)].AnatomicRegionModifierSequence [ OFFSET (0)] #unnested.AnatomicRegionSequence[OFFSET(0)].AnatomicRegionModifierSequence, ELSE NULL END AS AnatomicRegionModifier, CASE ARRAY_LENGTH(unnested.SegmentedPropertyCategoryCodeSequence) WHEN 0 THEN NULL ELSE unnested.SegmentedPropertyCategoryCodeSequence [ OFFSET (0)] END AS SegmentedPropertyCategory, CASE ARRAY_LENGTH(unnested.SegmentedPropertyTypeCodeSequence) WHEN 0 THEN NULL ELSE unnested.SegmentedPropertyTypeCodeSequence [ OFFSET (0)] END AS SegmentedPropertyType, #unnested.SegmentedPropertyTypeCodeSequence, #unnested.SegmentedPropertyTypeModifierCodeSequence, unnested.SegmentAlgorithmType, unnested.SegmentAlgorithmName, unnested.SegmentNumber, unnested.TrackingUID, unnested.TrackingID FROM segs CROSS JOIN UNNEST(SegmentSequence) AS unnested), sampled_sops AS ( SELECT SOPInstanceUID AS seg_SOPInstanceUID, ReferencedSeriesSequence[SAFE_OFFSET(0)].ReferencedInstanceSequence[SAFE_OFFSET(0)].ReferencedSOPInstanceUID AS rss_one, ReferencedImageSequence[SAFE_OFFSET(0)].ReferencedSOPInstanceUID AS ris_one, SourceImageSequence[SAFE_OFFSET(0)].ReferencedSOPInstanceUID AS sis_one FROM `nci-idc-bigquery-data.idc_current.dicom_all` WHERE Modality="SEG" AND SOPClassUID = "1.2.840.10008.5.1.4.1.1.66.4" ), coalesced_ref AS ( SELECT *, COALESCE(rss_one, ris_one, sis_one) AS referenced_sop FROM sampled_sops) SELECT segmentations.*, dicom_all.SeriesInstanceUID AS segmented_SeriesInstanceUID, CONCAT("https://viewer.imaging.datacommons.cancer.gov/viewer/", segmentations.StudyInstanceUID,"?seriesInstanceUID=",segmentations.SeriesInstanceUID,",",dicom_all.SeriesInstanceUID) AS viewer_url, FROM coalesced_ref JOIN `nci-idc-bigquery-data.idc_current.dicom_all` AS dicom_all ON coalesced_ref.referenced_sop = dicom_all.SOPInstanceUID RIGHT JOIN segmentations ON segmentations.SOPInstanceUID = coalesced_ref.seg_SOPInstanceUID ``` #### `measurement_groups`, `measurement_groups_view` {% hint style="info" %} Table in BigQuery: [`measurement_groups`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=measurement_groups\&page=table) {% endhint %} {% hint style="info" %} Table in BigQuery: [`measurement_groups_view`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=measurement_groups_view\&page=table) {% endhint %} This table is derived from `dicom_all` to simplify access to the measurement groups encoded in DICOM Structured Report TID 1500 objects available in IDC. Specifically, this table contains measurement groups corresponding to the "Measurement group" content item in the [TID 1500 Measurement report](https://dicom.nema.org/medical/dicom/current/output/chtml/part16/chapter_A.html#sect_TID_1500) DICOM SR objects. Each row corresponds to one TID1500 measurement group. `measurement_groups_view` SQL: ```googlesql WITH measurementGroups AS ( WITH contentSequenceLevel1 AS ( WITH structuredReports AS ( SELECT PatientID, SOPInstanceUID, SeriesInstanceUID, SeriesDescription, ContentSequence FROM `nci-idc-bigquery-data.idc_current.dicom_metadata` WHERE ( SOPClassUID = "1.2.840.10008.5.1.4.1.1.88.11" OR SOPClassUID = "1.2.840.10008.5.1.4.1.1.88.22" OR SOPClassUID = "1.2.840.10008.5.1.4.1.1.88.33" OR SOPClassUID = "1.2.840.10008.5.1.4.1.1.88.34" OR SOPClassUID = "1.2.840.10008.5.1.4.1.1.88.35" ) AND ARRAY_LENGTH(ContentTemplateSequence) <> 0 AND ContentTemplateSequence [ OFFSET (0)].TemplateIdentifier = "1500" AND ContentTemplateSequence [ OFFSET (0)].MappingResource = "DCMR" ) SELECT PatientID, SOPInstanceUID, SeriesInstanceUID, SeriesDescription, contentSequence FROM structuredReports CROSS JOIN UNNEST(ContentSequence) AS contentSequence ) SELECT PatientID, SOPInstanceUID, SeriesInstanceUID, SeriesDescription, contentSequence, measurementGroup_number FROM contentSequenceLevel1 CROSS JOIN UNNEST (contentSequence.ContentSequence) AS contentSequence WITH OFFSET AS measurementGroup_number WHERE contentSequence.ValueType = "CONTAINER" AND contentSequence.ConceptNameCodeSequence [ OFFSET (0)].CodeMeaning = "Measurement Group" ), measurementGroups_withTrackingID AS ( SELECT SOPInstanceUID, PatientID, SeriesInstanceUID, SeriesDescription, measurementGroup_number, unnestedContentSequence.TextValue AS trackingIdentifier, measurementGroups.contentSequence FROM measurementGroups CROSS JOIN UNNEST(contentSequence.ContentSequence) AS unnestedContentSequence WHERE unnestedContentSequence.ValueType = "TEXT" AND ( unnestedContentSequence.ConceptNameCodeSequence [ OFFSET (0)].CodeValue = "112039" AND unnestedContentSequence.ConceptNameCodeSequence [ OFFSET (0)].CodingSchemeDesignator = "DCM" ) ), measurementGroups_withTrackingUID AS ( SELECT SOPInstanceUID, SeriesInstanceUID, measurementGroup_number, unnestedContentSequence.UID AS trackingUniqueIdentifier FROM measurementGroups CROSS JOIN UNNEST(contentSequence.ContentSequence) AS unnestedContentSequence WHERE unnestedContentSequence.ValueType = "UIDREF" AND ( unnestedContentSequence.ConceptNameCodeSequence [ OFFSET (0)].CodeValue = "112040" AND unnestedContentSequence.ConceptNameCodeSequence [ OFFSET (0)].CodingSchemeDesignator = "DCM" ) ), measurementGroups_withSegmentation AS ( SELECT SOPInstanceUID, measurementGroup_number, unnestedContentSequence.ReferencedSOPSequence[ OFFSET (0)].ReferencedSOPInstanceUID AS segmentationInstanceUID, unnestedContentSequence.ReferencedSOPSequence[ OFFSET (0)].ReferencedSegmentNumber AS segmentationSegmentNumber FROM measurementGroups CROSS JOIN UNNEST(contentSequence.ContentSequence) AS unnestedContentSequence WHERE unnestedContentSequence.ValueType = "IMAGE" AND unnestedContentSequence.ReferencedSOPSequence[ OFFSET (0)].ReferencedSOPClassUID = "1.2.840.10008.5.1.4.1.1.66.4" ), measurementGroups_withSourceSeries AS ( SELECT SOPInstanceUID, measurementGroup_number, unnestedContentSequence.UID AS sourceSegmentedSeriesUID FROM measurementGroups CROSS JOIN UNNEST(contentSequence.ContentSequence) AS unnestedContentSequence WHERE unnestedContentSequence.ValueType = "UIDREF" AND ( unnestedContentSequence.ConceptNameCodeSequence [ OFFSET (0)].CodeValue = "121232" AND unnestedContentSequence.ConceptNameCodeSequence [ OFFSET (0)].CodingSchemeDesignator = "DCM" ) ), measurementGroups_withFinding AS ( SELECT SOPInstanceUID, measurementGroup_number, unnestedContentSequence.ConceptCodeSequence [ OFFSET (0)] AS finding FROM measurementGroups CROSS JOIN UNNEST(contentSequence.ContentSequence) AS unnestedContentSequence WHERE unnestedContentSequence.ValueType = "CODE" AND ( unnestedContentSequence.ConceptNameCodeSequence [ OFFSET (0)].CodeValue = "121071" AND unnestedContentSequence.ConceptNameCodeSequence [ OFFSET (0)].CodingSchemeDesignator = "DCM" ) ), measurementGroups_withFindingSite AS ( SELECT SOPInstanceUID, measurementGroup_number, unnestedContentSequence.ConceptCodeSequence [ OFFSET (0)] AS findingSite FROM measurementGroups CROSS JOIN UNNEST(contentSequence.ContentSequence) AS unnestedContentSequence WHERE unnestedContentSequence.ValueType = "CODE" AND ( (unnestedContentSequence.ConceptNameCodeSequence [ OFFSET (0)].CodeValue = "G-C0E3" AND unnestedContentSequence.ConceptNameCodeSequence [ OFFSET (0)].CodingSchemeDesignator = "SRT" ) OR (unnestedContentSequence.ConceptNameCodeSequence [ OFFSET (0)].CodeValue = "363698007" AND unnestedContentSequence.ConceptNameCodeSequence [ OFFSET (0)].CodingSchemeDesignator = "SCT" ) ) ) SELECT mWithUID.SOPInstanceUID, mWithUID.SeriesInstanceUID, mWithUID.measurementGroup_number, mWithUID.trackingUniqueIdentifier, mWithID.trackingIdentifier, mWithID.PatientID, mWithID.SeriesDescription, mWithFinding.finding, mWithFindingSite.findingSite, mWithSourceSeries.sourceSegmentedSeriesUID, mWithSegmentation.segmentationInstanceUID, dicom_metadata.SeriesInstanceUID AS segmentationSeriesUID, mWithSegmentation.segmentationSegmentNumber[SAFE_OFFSET(0)] as segmentationSegmentNumber, mWithID.contentSequence FROM measurementGroups_withTrackingUID AS mWithUID JOIN measurementGroups_withTrackingID AS mWithID --- ON mWithID.SOPInstanceUID = mWithUID.SOPInstanceUID AND mWithID.measurementGroup_number = mWithUID.measurementGroup_number JOIN measurementGroups_withFinding AS mWithFinding ON mWithID.SOPInstanceUID = mWithFinding.SOPInstanceUID AND mWithID.measurementGroup_number = mWithFinding.measurementGroup_number JOIN measurementGroups_withFindingSite AS mWithFindingSite ON mWithID.SOPInstanceUID = mWithFindingSite.SOPInstanceUID AND mWithID.measurementGroup_number = mWithFindingSite.measurementGroup_number JOIN measurementGroups_withSourceSeries AS mWithSourceSeries ON mWithID.SOPInstanceUID = mWithSourceSeries.SOPInstanceUID AND mWithID.measurementGroup_number = mWithSourceSeries.measurementGroup_number JOIN measurementGroups_withSegmentation AS mWithSegmentation ON mWithID.SOPInstanceUID = mWithSegmentation.SOPInstanceUID AND mWithID.measurementGroup_number = mWithSegmentation.measurementGroup_number --- JOIN `nci-idc-bigquery-data.idc_current.dicom_metadata` AS dicom_metadata ON mWithSegmentation.segmentationInstanceUID = dicom_metadata.SOPInstanceUID ORDER BY trackingUniqueIdentifier ``` #### `qualitative_measurements`, `qualitative_measurements_view` {% hint style="info" %} Table in BigQuery: [`qualitative_measurements`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=qualitative_measurements\&page=table) {% endhint %} {% hint style="info" %} Table in BigQuery: [`qualitative_measurements_view`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=qualitative_measurements_view\&page=table) {% endhint %} This table is derived from `dicom_all` to simplify access to the qualitative measurements in DICOM SR TID1500 objects. It contains coded evaluation results extracted from the DICOM SR TID1500 objects. Each row in this table corresponds to a single qualitative measurement extracted. `qualitative_measurements_view` SQL: ```googlesql WITH contentSequenceLevel3 AS ( SELECT PatientID, SOPInstanceUID, SeriesInstanceUID, measurementGroup_number, segmentationInstanceUID, segmentationSeriesUID, segmentationSegmentNumber, sourceSegmentedSeriesUID, trackingIdentifier, trackingUniqueIdentifier, contentSequence.ConceptNameCodeSequence [ OFFSET (0)] AS ConceptNameCodeSequence, contentSequence.ConceptCodeSequence [ OFFSET (0)] AS ConceptCodeSequence FROM `nci-idc-bigquery-data.idc_current.measurement_groups` CROSS JOIN UNNEST (contentSequence.ContentSequence) AS contentSequence WHERE contentSequence.ValueType = "CODE" ), findingsAndFindingSites AS ( WITH findings AS ( SELECT PatientID, SOPInstanceUID, measurementGroup_number, segmentationInstanceUID, segmentationSeriesUID, segmentationSegmentNumber, sourceSegmentedSeriesUID, trackingIdentifier, trackingUniqueIdentifier, ConceptCodeSequence AS finding FROM contentSequenceLevel3 WHERE ConceptNameCodeSequence.CodeValue = "121071" AND ConceptNameCodeSequence.CodingSchemeDesignator = "DCM" ), findingSites AS ( SELECT PatientID, SOPInstanceUID, measurementGroup_number, ConceptCodeSequence AS findingSite FROM contentSequenceLevel3 WHERE ConceptNameCodeSequence.CodeValue = "G-C0E3" AND ConceptNameCodeSequence.CodingSchemeDesignator = "SRT" ) SELECT findings.PatientID, findings.SOPInstanceUID, findings.finding, findings.segmentationInstanceUID, findings.segmentationSeriesUID, findings.segmentationSegmentNumber, findings.sourceSegmentedSeriesUID, findings.trackingIdentifier, findings.trackingUniqueIdentifier, findingSites.findingSite, findingSites.measurementGroup_number FROM findings JOIN findingSites ON findings.SOPInstanceUID = findingSites.SOPInstanceUID AND findings.measurementGroup_number = findingSites.measurementGroup_number ) SELECT contentSequenceLevel3.PatientID, contentSequenceLevel3.SOPInstanceUID, contentSequenceLevel3.SeriesInstanceUID, findingsAndFindingSites.measurementGroup_number, findingsAndFindingSites.segmentationInstanceUID, findingsAndFindingSites.segmentationSeriesUID, findingsAndFindingSites.segmentationSegmentNumber, findingsAndFindingSites.sourceSegmentedSeriesUID, findingsAndFindingSites.trackingIdentifier, findingsAndFindingSites.trackingUniqueIdentifier, contentSequenceLevel3.ConceptNameCodeSequence AS Quantity, contentSequenceLevel3.ConceptCodeSequence AS Value, findingsAndFindingSites.finding, findingsAndFindingSites.findingSite FROM contentSequenceLevel3 JOIN findingsAndFindingSites ON contentSequenceLevel3.SOPInstanceUID = findingsAndFindingSites.SOPInstanceUID AND contentSequenceLevel3.measurementGroup_number = findingsAndFindingSites.measurementGroup_number WHERE # exclude ( ConceptNameCodeSequence.CodeMeaning <> "121071" AND ConceptNameCodeSequence.CodingSchemeDesignator <> "DCM" ) AND # Finding ( ConceptNameCodeSequence.CodeMeaning <> "G-C0E3" AND ConceptNameCodeSequence.CodingSchemeDesignator <> "SRT" ) # Finding Site # correctness check: adding the below should result in a 36 rows column (4 segmented lesions, with 9 evaluations per each) # AND # contentSequenceLevel3.PatientID = "LIDC-IDRI-0001" ``` #### `quantitative_measurements`, `quantitative_measurements_view` {% hint style="info" %} Table in BigQuery: [`quantitative_measurements`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=quantitative_measurements\&page=table) {% endhint %} {% hint style="info" %} Table in BigQuery: [`quantitative_measurements_view`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=quantitative_measurements_view\&page=table) {% endhint %} This table is derived from `dicom_all` to simplify access to the quantitative measurements in DICOM SR TID1500 objects. It contains quantitative evaluation results extracted from the DICOM SR TID1500 objects. Each row in this table corresponds to a single quantitative measurement extracted. `quantitative_measurments_view` SQL: ```googlesql WITH --- contentSequenceLevel3numeric AS ( SELECT PatientID, SOPInstanceUID, SeriesInstanceUID, SeriesDescription, measurementGroup_number, segmentationInstanceUID, segmentationSeriesUID, segmentationSegmentNumber, sourceSegmentedSeriesUID, trackingIdentifier, trackingUniqueIdentifier, contentSequence.ConceptNameCodeSequence [ SAFE_OFFSET (0)] AS ConceptNameCodeSequence, contentSequence.MeasuredValueSequence [ SAFE_OFFSET (0)] AS MeasuredValueSequence, contentSequence.MeasuredValueSequence [ SAFE_OFFSET (0)].MeasurementUnitsCodeSequence [ SAFE_OFFSET (0)] AS MeasurementUnits, contentSequence.ContentSequence FROM `nci-idc-bigquery-data.idc_current.measurement_groups` CROSS JOIN UNNEST (contentSequence.ContentSequence) AS contentSequence WHERE contentSequence.ValueType = "NUM" ), --- contentSequenceLevel3codes AS ( SELECT PatientID, SOPInstanceUID, SeriesDescription, measurementGroup_number, segmentationInstanceUID, segmentationSeriesUID, segmentationSegmentNumber, sourceSegmentedSeriesUID, trackingIdentifier, trackingUniqueIdentifier, contentSequence.ConceptNameCodeSequence [ SAFE_OFFSET (0)] AS ConceptNameCodeSequence, contentSequence.ConceptCodeSequence [ SAFE_OFFSET (0)] AS ConceptCodeSequence, contentSequence.ContentSequence AS ContentSequence FROM `nci-idc-bigquery-data.idc_current.measurement_groups` CROSS JOIN UNNEST (contentSequence.ContentSequence) AS contentSequence WHERE contentSequence.ValueType = "CODE" ), --- contentSequenceLevel3uidrefs AS ( SELECT contentSequence.ConceptNameCodeSequence [ SAFE_OFFSET (0)] AS ConceptNameCodeSequence, contentSequence.ConceptCodeSequence [ SAFE_OFFSET (0)] AS ConceptCodeSequence, measurementGroup_number FROM `nci-idc-bigquery-data.idc_current.measurement_groups` CROSS JOIN UNNEST (contentSequence.ContentSequence) AS contentSequence WHERE contentSequence.ValueType = "UIDREF" AND ConceptCodeSequence [ SAFE_OFFSET (0)].CodeMeaning = "Tracking Unique Identifier" ), --- findings AS ( SELECT PatientID, SOPInstanceUID, SeriesDescription, ConceptCodeSequence AS finding, measurementGroup_number, segmentationInstanceUID, segmentationSeriesUID, segmentationSegmentNumber, sourceSegmentedSeriesUID, trackingIdentifier, trackingUniqueIdentifier, FROM contentSequenceLevel3codes WHERE ConceptNameCodeSequence.CodeValue = "121071" AND ConceptNameCodeSequence.CodingSchemeDesignator = "DCM" ), --- findingSites AS ( SELECT PatientID, SOPInstanceUID, SeriesDescription, ConceptCodeSequence AS findingSite, measurementGroup_number, CASE ( ContentSequence[SAFE_OFFSET(0)].ConceptNameCodeSequence[SAFE_OFFSET(0)].CodeValue = "272741003" AND ContentSequence[SAFE_OFFSET(0)].ConceptNameCodeSequence[SAFE_OFFSET(0)].CodingSchemeDesignator = "SCT") WHEN TRUE THEN STRUCT( contentSequenceLevel3codes.ContentSequence [ SAFE_OFFSET (0)].ConceptCodeSequence [ SAFE_OFFSET (0)].CodeValue AS CodeValue, contentSequenceLevel3codes.ContentSequence [ SAFE_OFFSET (0)].ConceptCodeSequence [ SAFE_OFFSET (0)].CodingSchemeDesignator AS CodingSchemeDesignator, contentSequenceLevel3codes.ContentSequence [ SAFE_OFFSET (0)].ConceptCodeSequence [ SAFE_OFFSET (0)].CodeMeaning AS CodeMeaning ) ELSE STRUCT(NULL as CodeValue,NULL as CodingSchemeDesignator,NULL as CodeMeaning) END AS lateralityModifier, # added FROM contentSequenceLevel3codes WHERE (ConceptNameCodeSequence.CodeValue = "G-C0E3" AND ConceptNameCodeSequence.CodingSchemeDesignator = "SRT" ) OR (ConceptNameCodeSequence.CodeValue = "363698007" AND ConceptNameCodeSequence.CodingSchemeDesignator = "SCT" ) ), --- findingsAndFindingSites AS ( SELECT findings.PatientID, findings.SOPInstanceUID, findings.SeriesDescription, findings.finding, findingSites.findingSite, findingSites.lateralityModifier, findingSites.measurementGroup_number, findings.segmentationInstanceUID, findings.segmentationSeriesUID, findings.segmentationSegmentNumber, findings.sourceSegmentedSeriesUID, findings.trackingIdentifier, findings.trackingUniqueIdentifier FROM findings JOIN findingSites ON findings.SOPInstanceUID = findingSites.SOPInstanceUID AND findings.measurementGroup_number = findingSites.measurementGroup_number ) --- # correctness check: the below should result in 11 rows (this is how many segments/measurement # groups are there for each QIN-HEADNCK-01-0139 segmentation #SELECT # * #FROM # findingsAndFindingSites #WHERE # SOPInstanceUID = "1.2.276.0.7230010.3.1.4.8323329.18336.1440004659.731760" --- SELECT contentSequenceLevel3numeric.PatientID, contentSequenceLevel3numeric.SOPInstanceUID, contentSequenceLevel3numeric.SeriesInstanceUID, contentSequenceLevel3numeric.SeriesDescription, contentSequenceLevel3numeric.measurementGroup_number, findingsAndFindingSites.segmentationInstanceUID, findingsAndFindingSites.segmentationSeriesUID, findingsAndFindingSites.segmentationSegmentNumber, findingsAndFindingSites.sourceSegmentedSeriesUID, findingsAndFindingSites.trackingIdentifier, findingsAndFindingSites.trackingUniqueIdentifier, contentSequenceLevel3numeric.ConceptNameCodeSequence AS Quantity, CASE ( ARRAY_LENGTH(contentSequenceLevel3numeric.ContentSequence) > 0 AND contentSequenceLevel3numeric.ContentSequence [ SAFE_OFFSET (0)].ConceptNameCodeSequence [ SAFE_OFFSET (0)].CodeValue = "121401" AND contentSequenceLevel3numeric.ContentSequence [ SAFE_OFFSET (0)].ConceptNameCodeSequence [ SAFE_OFFSET (0)].CodingSchemeDesignator = "DCM" ) WHEN TRUE THEN STRUCT( contentSequenceLevel3numeric.ContentSequence [ SAFE_OFFSET (0)].ConceptCodeSequence [ SAFE_OFFSET (0)].CodeValue AS CodeValue, contentSequenceLevel3numeric.ContentSequence [ SAFE_OFFSET (0)].ConceptCodeSequence [ SAFE_OFFSET (0)].CodingSchemeDesignator AS CodingSchemeDesignator, contentSequenceLevel3numeric.ContentSequence [ SAFE_OFFSET (0)].ConceptCodeSequence [ SAFE_OFFSET (0)].CodeMeaning AS CodeMeaning ) ELSE STRUCT(NULL as CodeValue,NULL as CodingSchemeDesignator,NULL as CodeMeaning) END AS derivationModifier, findingsAndFindingSites.lateralityModifier, SAFE_CAST( contentSequenceLevel3numeric.MeasuredValueSequence.NumericValue [ SAFE_OFFSET (0)] AS NUMERIC ) AS Value, contentSequenceLevel3numeric.MeasurementUnits AS Units, findingsAndFindingSites.finding, findingsAndFindingSites.findingSite FROM contentSequenceLevel3numeric JOIN findingsAndFindingSites ON contentSequenceLevel3numeric.SOPInstanceUID = findingsAndFindingSites.SOPInstanceUID AND contentSequenceLevel3numeric.measurementGroup_number = findingsAndFindingSites.measurementGroup_number --- # correctness check: for this patient, there should be 12 rows: 4 segmented nodules, with 3 numeric evaluations for each #WHERE # contentSequenceLevel3numeric.PatientID = "LIDC-IDRI-0001" --- # correctness check: for this specific instance, there should be 238 rows (11 segments) #WHERE # contentSequenceLevel3numeric.SOPInstanceUID = "1.2.276.0.7230010.3.1.4.8323329.18336.1440004659.731760" #where contentSequenceLevel3numeric.PatientID LIKE "%QIN%" ``` #### `dicom_metadata_curated`, `dicom_metadata_curated_view` {% hint style="info" %} Table in BigQuery: [`dicom_metadata_curated`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=dicom_metadata_curated\&page=table) {% endhint %} {% hint style="info" %} Table in BigQuery: [`dicom_metadata_curated_view`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=dicom_metadata_curated_view\&page=table) {% endhint %} Curated values of DICOM metadata extracted from `dicom_metadata`. `dicom_metadata_curated_view` SQL: ```googlesql SELECT SOPInstanceUID, SAFE_CAST(SliceThickness AS FLOAT64) AS SliceThickness, CASE WHEN BodyPartExamined IN ('PORT ABDOMEN', 'J BRZUSZNA', 'ABD', 'BD BD MR ABDOME', 'BD CT ABD WO_W', 'J brzuszna') THEN 'ABDOMEN' WHEN BodyPartExamined IN ('Pelvis') THEN 'PELVIS' WHEN BodyPartExamined IN ('PET_ABDOMEN_PEL', 'CT ABD PELVIS', 'ABD PEL', 'ABD PELV', 'ABDOMEN PEL', 'ABDOMEN PELVIS', 'ABDOMEN TO PEL', 'ABDOMEN_PELVIS', 'ABDPEL', 'ABDOMEN_PELVIS C', 'PET_ABDOMEN_PELV') THEN 'ABDOMENPELVIS' WHEN BodyPartExamined IN ('TH CT CHEST WO', 'SPI CHEST 5MM', 'PORT CHEST', 'PET_CT SCAN CHE', 'CTA CHEST', 'CT CHEST W_ENHA', 'CT CHEST WITH C', 'CT CHEST WO CE', 'AP PORTABLE CHE', 'CHEST COMPUTED', 'CHEST INF', 'CHEST NO GRID', 'CHEST PE', 'Chest', 'CT CHEST W_ENHAN', 'CHEST COMPUTED T', 'PET_CT SCAN CHES', 'AP PORTABLE CHES') THEN 'CHEST' WHEN BodyPartExamined IN ('MR BRAIN W CON', 'MR NEURO BRAIN', 'BRAIN W/WO_AH', 'BRAIN W/WO_AH32') THEN 'BRAIN' WHEN BodyPartExamined IN ('CHEST_TO_PELVIS', 'CAP', 'CAP INFUSION', 'CHABDPELV', 'CHEST ABD PEL', 'CHEST ABD PELVI', 'CHEST TO PEL', 'CHEST TO PELVIS', 'CHEST_ABD_PEL', 'CHEST ABD PELVIS') THEN 'CHESTABDPELVIS' WHEN BodyPartExamined IN ('CERVICAL_SPINE') THEN 'CSPINE' WHEN BodyPartExamined IN ('THORAXABD', 'CT_CHABD', 'CHEST ABDOMEN', 'CHEST_ABDOMEN', 'CHEST/ABD') THEN 'CHESTABDOMEN' WHEN BodyPartExamined IN ('CT 3PHASE REN', 'Kidney') THEN 'KIDNEY' WHEN BodyPartExamined IN ('THORAX CT _AH05', 'THORAX CT _OT01', 'CT THORAX W CNT', 'Thorax') THEN 'THORAX' WHEN BodyPartExamined IN ('HEAD_NECK', 'HEAD-AND-NECK', 'HEAD-NECK', 'HEADANDNECK') THEN 'HEADNECK' WHEN BodyPartExamined IN ('LUMBO-SACRAL SP', 'LUMBO-SACRAL SPI') THEN 'LSSPINE' WHEN BodyPartExamined IN ('NECK TO PELVIS', 'NECKCHESTABDPEL') THEN 'NECKCHESTABDPELV' WHEN BodyPartExamined IN ('THORACIC SPINE') THEN 'TSPINE' WHEN BodyPartExamined IN ('WHOLE BODY') THEN 'WHOLEBODY' ELSE BodyPartExamined END AS BodyPartExamined FROM `nci-idc-bigquery-data.idc_current.dicom_metadata` AS dcm ``` #### `dicom_metadata_curated_series_level`, `dicom_metadata_curated_series_level_view` {% hint style="info" %} Table in BigQuery: [`dicom_metadata_curated_series_level`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=dicom_metadata_curated_series_level\&page=table) {% endhint %} {% hint style="info" %} Table in BigQuery: [`dicom_metadata_curated_series_level_view`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=dicom_metadata_curated_series_level_view\&page=table) {% endhint %} Curated columns from `dicom_metadata` that have been aggregated/cleaned up to describe content at the series level. Each row in this table corresponds to a DICOM instance in IDC. The columns are curated by defining queries that apply transformations to the original values of DICOM attributes. `dicom_metadata_curated_series_level_view` SQL: ```googlesql WITH temp_table AS ( SELECT dicom_all.SeriesInstanceUID, ANY_VALUE(Modality) AS Modality, STRING_AGG(DISTINCT(collection_id),",") AS collection_id, ANY_VALUE(OpticalPathSequence[SAFE_OFFSET(0)].ObjectiveLensPower) AS ObjectiveLensPower, MAX(DISTINCT(TotalPixelMatrixColumns)) AS max_TotalPixelMatrixColumns, MAX(DISTINCT(TotalPixelMatrixRows)) AS max_TotalPixelMatrixRows, MAX(DISTINCT(`Columns`)) AS max_Columns, MAX(DISTINCT(`Rows`)) AS max_Rows, MIN(DISTINCT(SAFE_CAST(PixelSpacing[SAFE_OFFSET(0)] AS FLOAT64))) AS min_spacing_0, MIN(SAFE_CAST(SharedFunctionalGroupsSequence[SAFE_OFFSET(0)].PixelMeasuresSequence[SAFE_OFFSET(0)]. PixelSpacing[SAFE_OFFSET(0)] AS FLOAT64)) AS fg_min_spacing_0, ARRAY_AGG(DISTINCT(CONCAT(SpecimenDescriptionSequence[SAFE_OFFSET(0)].PrimaryAnatomicStructureSequence[SAFE_OFFSET(0)].CodingSchemeDesignator,":", SpecimenDescriptionSequence[SAFE_OFFSET(0)].PrimaryAnatomicStructureSequence[SAFE_OFFSET(0)].CodeValue, ":", SpecimenDescriptionSequence[SAFE_OFFSET(0)].PrimaryAnatomicStructureSequence[SAFE_OFFSET(0)].CodeMeaning)) IGNORE NULLS)[SAFE_OFFSET(0)] AS primaryAnatomicStructure_code_str, ARRAY_AGG(DISTINCT(CONCAT(SpecimenDescriptionSequence[SAFE_OFFSET(0)].PrimaryAnatomicStructureSequence[SAFE_OFFSET(0)].PrimaryAnatomicStructureModifierSequence[SAFE_OFFSET(0)].CodingSchemeDesignator,":", SpecimenDescriptionSequence[SAFE_OFFSET(0)].PrimaryAnatomicStructureSequence[SAFE_OFFSET(0)].PrimaryAnatomicStructureModifierSequence[SAFE_OFFSET(0)].CodeValue, ":", SpecimenDescriptionSequence[SAFE_OFFSET(0)].PrimaryAnatomicStructureSequence[SAFE_OFFSET(0)].PrimaryAnatomicStructureModifierSequence[SAFE_OFFSET(0)].CodeMeaning)) IGNORE NULLS)[SAFE_OFFSET(0)] AS primaryAnatomicStructureModifier_code_str, ARRAY_AGG(DISTINCT(CONCAT(OpticalPathSequence[SAFE_OFFSET(0)].IlluminationTypeCodeSequence[SAFE_OFFSET(0)].CodingSchemeDesignator,":", OpticalPathSequence[SAFE_OFFSET(0)].IlluminationTypeCodeSequence[SAFE_OFFSET(0)].CodeValue, ":", OpticalPathSequence[SAFE_OFFSET(0)].IlluminationTypeCodeSequence[SAFE_OFFSET(0)].CodeMeaning)) IGNORE NULLS)[SAFE_OFFSET(0)] AS illuminationType_code_str, FROM `nci-idc-bigquery-data.idc_current.dicom_all` AS dicom_all GROUP BY SeriesInstanceUID ), SpecimenPreparationSequence_unnested AS ( SELECT SeriesInstanceUID, concept_name_code_sequence.CodeMeaning AS cnc_cm, concept_name_code_sequence.CodingSchemeDesignator AS cnc_csd, concept_name_code_sequence.CodeValue AS cnc_val, concept_code_sequence.CodeMeaning AS ccs_cm, concept_code_sequence.CodingSchemeDesignator AS ccs_csd, concept_code_sequence.CodeValue AS ccs_val, FROM `bigquery-public-data.idc_v18.dicom_all`, UNNEST(SpecimenDescriptionSequence[SAFE_OFFSET(0)].SpecimenPreparationSequence) as preparation_unnest_step1, UNNEST(preparation_unnest_step1.SpecimenPreparationStepContentItemSequence) as preparation_unnest_step2, UNNEST(preparation_unnest_step2.ConceptNameCodeSequence) as concept_name_code_sequence, UNNEST(preparation_unnest_step2.ConceptCodeSequence) as concept_code_sequence ), slide_embedding AS ( SELECT SeriesInstanceUID, ARRAY_AGG(DISTINCT(CONCAT(ccs_cm,":",ccs_csd,":",ccs_val))) as embeddingMedium_code_str FROM SpecimenPreparationSequence_unnested WHERE (cnc_csd = 'SCT' and cnc_val = '430863003') -- CodeMeaning is 'Embedding medium' GROUP BY SeriesInstanceUID ), slide_fixative AS ( SELECT SeriesInstanceUID, ARRAY_AGG(DISTINCT(CONCAT(ccs_cm, ":", ccs_csd,":",ccs_val))) as tissueFixative_code_str FROM SpecimenPreparationSequence_unnested WHERE (cnc_csd = 'SCT' and cnc_val = '430864009') -- CodeMeaning is 'Tissue Fixative' GROUP BY SeriesInstanceUID ), slide_staining AS ( SELECT SeriesInstanceUID, ARRAY_AGG(DISTINCT(CONCAT(ccs_cm, ":", ccs_csd,":",ccs_val))) as staining_usingSubstance_code_str, FROM SpecimenPreparationSequence_unnested WHERE (cnc_csd = 'SCT' and cnc_val = '424361007') -- CodeMeaning is 'Using substance' GROUP BY SeriesInstanceUID ) SELECT temp_table.SeriesInstanceUID, temp_table.Modality, -- Embedding Medium ARRAY( SELECT IF(code IS NULL, NULL, SPLIT(code, ':')[SAFE_OFFSET(0)]) FROM UNNEST(embeddingMedium_code_str) AS code ) AS embeddingMedium_CodeMeaning, ARRAY( SELECT IF(code IS NULL, NULL, IF(STRPOS(code, ':') = 0, NULL, SUBSTR(code, STRPOS(code, ':') + 1))) FROM UNNEST(embeddingMedium_code_str) AS code ) AS embeddingMedium_code_designator_value_str, -- Tissue Fixative ARRAY( SELECT IF(code IS NULL, NULL, SPLIT(code, ':')[SAFE_OFFSET(0)]) FROM UNNEST(tissueFixative_code_str) AS code ) AS tissueFixative_CodeMeaning, ARRAY( SELECT IF(code IS NULL, NULL, IF(STRPOS(code, ':') = 0, NULL, SUBSTR(code, STRPOS(code, ':') + 1))) FROM UNNEST(tissueFixative_code_str) AS code ) AS tissueFixative_code_designator_value_str, -- Staining using substance ARRAY( SELECT IF(code IS NULL, NULL, SPLIT(code, ':')[SAFE_OFFSET(0)]) FROM UNNEST(staining_usingSubstance_code_str) AS code ) AS staining_usingSubstance_CodeMeaning, ARRAY( SELECT IF(code IS NULL, NULL, IF(STRPOS(code, ':') = 0, NULL, SUBSTR(code, STRPOS(code, ':') + 1))) FROM UNNEST(staining_usingSubstance_code_str) AS code ) AS staining_usingSubstance_code_designator_value_str, if(COALESCE(min_spacing_0, fg_min_spacing_0) = 0, 0, round(COALESCE(min_spacing_0, fg_min_spacing_0) ,CAST(2 -1-floor(log10(abs(COALESCE(min_spacing_0, fg_min_spacing_0) ))) AS INT64))) AS min_PixelSpacing_2sf, COALESCE(max_TotalPixelMatrixColumns, max_Columns) AS max_TotalPixelMatrixColumns, COALESCE(max_TotalPixelMatrixRows, max_Rows) AS max_TotalPixelMatrixRows, SAFE_CAST(ObjectiveLensPower as INT) as ObjectiveLensPower, CONCAT(SPLIT(primaryAnatomicStructure_code_str,":")[SAFE_OFFSET(0)],":",SPLIT(primaryAnatomicStructure_code_str,":")[SAFE_OFFSET(1)]) as primaryAnatomicStructure_code_designator_value_str, SPLIT(primaryAnatomicStructure_code_str,":")[SAFE_OFFSET(2)] as primaryAnatomicStructure_CodeMeaning, CONCAT(SPLIT(primaryAnatomicStructureModifier_code_str,":")[SAFE_OFFSET(0)],":",SPLIT(primaryAnatomicStructureModifier_code_str,":")[SAFE_OFFSET(1)]) as primaryAnatomicStructureModifier_code_designator_value_str, SPLIT(primaryAnatomicStructureModifier_code_str,":")[SAFE_OFFSET(2)] as primaryAnatomicStructureModifier_CodeMeaning, CONCAT(SPLIT(illuminationType_code_str,":")[SAFE_OFFSET(0)],":",SPLIT(illuminationType_code_str,":")[SAFE_OFFSET(1)]) as illuminationType_code_designator_value_str, SPLIT(illuminationType_code_str,":")[SAFE_OFFSET(2)] as illuminationType_CodeMeaning, FROM temp_table LEFT JOIN slide_embedding on temp_table.SeriesInstanceUID = slide_embedding.SeriesInstanceUID LEFT JOIN slide_fixative on temp_table.SeriesInstanceUID = slide_fixative.SeriesInstanceUID LEFT JOIN slide_staining on temp_table.SeriesInstanceUID = slide_staining.SeriesInstanceUID ``` ### Collection-specific BigQuery tables Most clinical data is found in the [idc\_v\\_clinical datasets](/data/organization-of-data/clinical). However, a few tables of clinical data are found in the idc\_v\ datasets. #### TCGA The following tables contain TCGA-specific metadata: * `tcga_biospecimen_rel9:` biospecimen metadata * `tcga_clinical_rel9:` clinical metadata # DICOM stores If you would like to access IDC data via [DICOMweb interface](https://www.dicomstandard.org/using/dicomweb), you have two options: 1. IDC-maintained DICOM store available via proxy 2. DICOM store maintained by Google Healthcare In the following we provide details for each of those options. ### IDC-maintained DICOM store via proxy This store contains all of the data for the current IDC data release. It does not require authentication and is available via the following DICOMweb URL of the proxy (you can ignore the "viewer-only-no-downloads" part in the URL, it is a legacy constraint that is no longer applicable). DICOMweb URL: {% code overflow="wrap" %} ``` https://proxy.imaging.datacommons.cancer.gov/current/viewer-only-no-downloads-see-tinyurl-dot-com-slash-3j3d9jyp/dicomWeb ``` {% endcode %} **Limitations**: * since all requests go through the proxy before reaching the DICOM store, you may experience reduced performance as compared to direct access you can achieve using the store described in the following section * there are per-IP and overall daily quotas, as described in IDC [Proxy policy](/portal/proxy-policy), that may not be sufficient for your use case ### DICOM store maintained by Google Healthcare This store replicates all of the data from the `idc-open-data` bucket, which contains most of the data in IDC (learn more about the organization of data in IDC buckets from [this documentation article](/data/organization-of-data/files-and-metadata#storage-buckets)). DICOMweb URL (note the store name includes the IDC data release version that corresponds to its content: `idc-store-v23`): {% code overflow="wrap" %} ``` https://healthcare.googleapis.com/v1/projects/nci-idc-data/locations/us-central1/datasets/idc/dicomStores/idc-store-v23/dicomWeb ``` {% endcode %} This DICOM store is documented in . **Limitations**: * most, but not all of the IDC data is available in this store * authentication with a Google account is required (anyone signed in with a Google account can access this interface, no whitelisting is required!) * since this DICOM store is not maintained directly by the IDC team, it may lag behind the latest IDC release in content in the future * If you experience errors accessing this store, fill out this form to whitelist specific account for access. You can see current status from the [IDC health page](https://imagingdatacommons.github.io/idc-health-monitor/). ## DICOMweb usage tutorials Check out [this tutorial](/data/downloading-data/dicomweb-access) and the accompanying Colab notebook to learn more. ## Frequently Asked Questions ### When you say most data is available in the Google Healthcare maintained DICOM store - what is "most"? *TL;DR: as of IDC v21, it is 95.89% of all of the DICOM series available in IDC (IDC-maintained DICOM store has all of the 100%).*
The Google Healthcare maintained DICOM store contains the latest versions of the DICOM series stored in the `idc-open-data` Google Storage bucket (see [this article](/data/organization-of-data/files-and-metadata#storage-buckets) for details on buckets organization). You can get the exact number of DICOM series in each of the buckets with the following python code (before running it, do `pip install --upgrade idc-index`): ```python from idc_index import IDCClient c=IDCClient() query = """ SELECT aws_bucket, COUNT(DISTINCT(SeriesInstanceUID)) AS num_series FROM index GROUP BY aws_bucket ORDER BY num_series DESC """ c.sql_query(query) ``` As of IDC v21, the result of running the code above is the following, showing that 95.89% of DICOM series in IDC are available from the Google Healthcare maintained DICOM store (IDC-maintained DICOM store has all of the 100%). ``` aws_bucket num_series idc-open-data 911781 idc-open-data-cr 34634 idc-open-data-two 4473 ``` ### Are the two DICOM stores kept in sync? *TL;DR: our goal is to have the two stores in sync within 1-2 weeks of each IDC data release.* The DICOM store maintained by IDC is updated by the IDC team with each new release. The DICOM store maintained by Google Healthcare is populated after the release. We hope to have that done within 1-2 weeks after the IDC release. As a new release of IDC data is out, there will be a new DICOM store maintained by Google Healthcare, and the connection to the IDC release version will be indicated in the store name. I.e., when the new IDC data release is out, whenever you are able to access `https://healthcare.googleapis.com/v1/projects/nci-idc-data/locations/us-central1/datasets/idc/dicomStores/` **idc-store-v\**`/dicomWeb` , it is expected to be in sync. # Clinical data {% hint style="info" %} Check out our [IDC clinical data exploration Colab notebook](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/exploring_clinical_data.ipynb) tutorial for a brief hands-on introduction into IDC clinical data! You can also see the high-level summary of the clinical data attributes accompanying IDC data in [this dashboard](https://datastudio.google.com/reporting/04cf5976-4ea0-4fee-a749-8bfd162f2e87/page/p_s7mk6eybqc). {% endhint %} ### Background By clinical data we refer to the broad spectrum of image-related data that may accompany images. Such data may include demographics of the patients, observations related to their clinical history (therapies, diagnoses, findings), lab tests, surgeries. Clinical data is often critical in understanding imaging data, and is essential for the development and validation of imaging biomarkers. However, such data is most often stored in spreadsheets that follow conventions specific to the site that collected the data, may not be accompanied by the dictionary defining the terms used in describing clinical data, and is rarely harmonized. This can be observed on the example of various collections ingested into IDC from The Cancer Imaging Archive (TCIA), such as the [ACRIN 6698 collection](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=50135447). Not only are the terms used in the clinical data accompanying individual collection not harmonized, but the format of the spreadsheets is also collection-specific. In order to search and navigate clinical data, one has to parse those collection specific tables, and there is no interface to support searching across collections. With the release v11 of IDC, we make the attempt to lower the barriers for accessing clinical data accompanying IDC imaging collections. We parse collection-specific tables, and organize the underlying data into BigQuery tables that can be accessed using standard SQL queries. You can also see the summary of clinical data available for IDC collections in [this dashboard](https://datastudio.google.com/u/0/reporting/04cf5976-4ea0-4fee-a749-8bfd162f2e87/page/p_s7mk6eybqc). ### Clinical data BigQuery tables As of Version 11 IDC provides a public [BigQuery dataset](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_clinical_current) with clinical data associated with several of its imaging collections. The clinical data tables associated with a particular version are in the dataset `bigquery-public-data.idc__clinical`. In addition the dataset `bigquery-public-data.idc_current_clinical` has an identically named view for each table in the BQ clinical dataset corresponding to the current IDC release. There are currently 130 tables with clinical data representing 70 different collections. Most of this data was curated from Excel and CSV files downloaded from [The Cancer Imaging Archive (TCIA) wiki](https://wiki.cancerimagingarchive.net/). For most collections, data is placed in a single table named `_clinical`, where `` is the name of the collection in a standardized format (i.e. the `idc_webapp_collection_id` column in the `dicom_all` view in the [idc\_current dataset](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_clinical\&page=dataset)). Collections from the ACRIN project have different types of clinical data spread across CSV files, and so this data is represented by several BigQuery tables. The clinical data for collections in the [CPTAC program](https://proteomics.cancer.gov/programs/cptac) program is not curated from TCIA but instead is copied from a [BigQuery table](https://console.cloud.google.com/bigquery?p=isb-cgc-bq\&d=cptac\&t=clinical_gdc_current\&page=table) in the ISB-CGC project, which in turn was sourced from the [Genomics Data Commons (GDC) api](https://gdc.cancer.gov/developers/gdc-application-programming-interface-api). Similarly clinical data for collections in the [TCGA program](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga) is copied from the table `tcga_clinical_rel9` in the `idc_current` dataset, which was also created using the [Genomics Data Commons (GDC) api](https://gdc.cancer.gov/developers/gdc-application-programming-interface-api). Every clinical data table contains two fields we have introduced, `dicom_patient_id` and `source_batch`. `dicom_patient_id` is identical to the `PatientID` field in the DICOM files that correspond to the given patient. The `dicom_patient_id` value is determined by inspecting the patient column in the clinical data file. In some of the collections' clinical data, the patients are separated into different 'batches' i.e. different source files, or different sheets in the same Excel file. The `source_batch` field is an integer indicating the 'batch' for the given patient. For most collections, in which all patients data is found in the same location, the `source_batch` value is zero. Most of the clinical tables are legible by themselves. Tables from the ACRIN collection are an exception as the column names and some of the column values are coded. To provide for clarity and ease of use of all clinical data, we have created two metadata tables, [`table_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_clinical_current\&t=table_metadata\&page=table) and [`column_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_clinical_current\&t=column_metadata\&page=table) that provide information about the structure and provenance of all data in this dataset. `table_metadata` has table-level metadata about each clinical collection, while `column_metadata` has column-level metadata. Structure of the[`table_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_clinical_current\&t=table_metadata\&page=table) table: * `collection_id` (STRING, NULLABLE) - the collection\_id of the collection in the given table. The collection id is in a format used internally by the IDC Web App (with only lowercase letters, numbers and '\_' allowed). It is equivalent to the `idc_webapp_id` field in the `dicom_all` view in the `idc_current` dataset. * `table_name` (STRING,NULLABLE) - name of the table * `table_description` (STRING,NULLABLE) - description of the type of data found in the table. Usually this is set to 'clinical data', unless a description is provided in the source files * `idc_version_table_added` (STRING, NULLABLE) - the IDC data version for which this table was first added * `idc_table_added_datetime` (STRING,NULLABLE) - the date/time this particular table was first generated * `post_process_src` (STRING, NULLABLE) - except for the CPTAC and TCGA collections the tables are curated from ZIP, Excel, and CSV files downloaded from the TCIA wiki. These files do not have a consistent structure and were not meant to be machine readable or to translate directly into BigQuery. A semi-manual curation process results in either a CSV of JSON file that can be directly written into a BigQuery table. *post\_process\_src* is the name of the JSON or CSV file that results from this process and is used to create the BigQuery table. This field is not used for the CPTAC- and TCGA-related tables * `post_process_src_add_md5` (STRING, NULLABLE) - the md5 hash of post\_process\_src when the table was first added * `idc_version_table_prior` (STRING, NULLABLE) - the idc version the second most recent time the table was updated * `post_process_src_prior_md5` (STRING, NULLABLE) - the md5 hash of post\_process\_src the second most recent time the table was updated * `idc_version_table_updated` (STRING, NULLABLE) - the idc version when the table was last updated * `table_update_datetime` (STRING, NULLABLE) - date and time an update of the table was last recorded * `post_process_src_updated_md5` (STRING, NULLABLE) - the md5 hash of post\_process\_source when the table was last updated * `number_batches` (INTEGER, NULLABLE) - records the number of batches. Within the source data patients are sometimes grouped into different 'batches' (i.e. training vs test, responder vs non-responder etc.) and the batches are placed in different locations (i.e. different files or different sheets in the same Excel file) * `source_info` (RECORD, REPEATED) - an array of records with information about the table sources. These sources are either files downloaded from the TCIA wiki or another BigQuery table (as is the case for CPTAC and TCGA collections). There is a source\_info record for each source 'batch' described above * `source_info.srcs` (STRING, REPEATED) - a source file downloaded from the TCIA wiki may be a ZIP file, and CSV file, or an Excel file. Sometimes the ZIP files contain other ZIP files that must be opened to extract the clinical data. In the `source_info.src` array the first string is the file that is downloaded from TCIA for this particular source batch. The final string is the CSV or Excel file that contains the clinical data. Any intermediate strings are the names of ZIP files 'in between' the downloaded file and the clinical file. For CPTAC and TCGA collections this field contains the source BigQuery table * `source_info.md5` (STRING, NULLABLE) - md5 hash of the downloaded file from TCIA the most recent time the table was updated * `source_info.table_last_modified` (STRING, NULLABLE) - CPTAC and TCGA collections only. The date and time the source BigQuery table was most recently modified, as recorded when last copied * `source_info.table_size` (STRING, NULLABLE) - CPTAC and TCGA collections only. The size of the source BigQuery table as recorded when last copied Structure of [`column_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_clinical_current\&t=column_metadata\&page=table) table: * `collection_id` (STRING,NULLABLE) - the collection\_id of the collection in the given table. The collection id is in a format used internally by the IDC Web App (with only lowercase letters, numbers and '\_' allowed). It is equivalent to the `idc_webapp_id` field in the `dicom_all` view in the `idc_current` dataset. * `case_col` (BOOLEAN, NULLABLE) - true if the BigQuery column contains the patient or case id, i.e. if this column is used to determine the value of the `dicom_patient_id` column * `table_name` (STRING, NULLABLE) - table name * `column` (STRING, NULLABLE) - the actual column name in the table. For ACRIN collections the `column_name` is the `variable_name` from the provided data dictionary. For other collections it is a name constructed by 'normalizing' the `column_label` (see next) in a format that can be used as a BigQuery field name * `column_label` (STRING, NULLABLE) - a 'free form' label for the column that does not need to conform to the BigQuery column format requirements. For ACRIN collections this is the `variable_label` given by a data dictionary that accompanies the collection. For other collections it is the name or label of the clinical attribute as inferred from the source document during the curation process * `data_type` (STRING, NULLABLE) - the type of data in this column. Again for ACRIN collections this is provided in the data dictionary. For other collections it is inferred by analyzing the data during curation * `original_column_headers` (STRING, REPEATED) - the name(s) or label(s) in the source document that were used to construct the `column_label` field. In most cases there is one column label in the source document that perscribes the `column_label`. In some cases, multiple columns are concantenated and reformated to form the `column_label` * `values` (RECORD, REPEATED) - a structure that is borrowed from the ACRIN data model. This is an array that contains observerd attribute values for this given column. For ACRIN collections these values are reported in the data dictionary. For most other collections these values are determined by analyzing the source data. For simplicity this field is left blank when the number of unique values is greater than 20 * `values.option_code` (STRING, NULLABLE) - a unique attribute value found in this column * `values.option_description` (STRING, NULLABLE) - a description of the `option_code` as provided by a data dictionary. For collections that do not have a data dictionary this is null. * `values_source` (STRING, NULLABLE) - indicates the source of the `values` records. The text 'provided dictionary' indicates that the records were obtained from a provided data dictionary. The text 'derived from inspection of values' indicates that the records were determined by automated analysis of the source materials during the ETL process that generated the BigQuery tables. * `files` (STRING, REPEATED) - names of the files that contain the source data for each batch. These are the Excel or CSV files directly downloaded from TCIA, or the files extracted from downloaded ZIP files * `sheet_names` (STRING, REPEATED) - for Excel-sourced files, the sheet names containing this column's values for each batch * `batch` (INTEGER, REPEATED) - source batches that contain this particular column. Some columns or attributes may be missing from some batches * `column_numbers` (STRING, REPEATED) - for each source batch, the column in the original source corresponding to this column in the BigQuery table # UUIDs and GUIDs As described in the [Data Versioning](/data/data-versioning) section, a UUID identifies a particular version of an IDC data object. Thus, there is a UUID for every version of every DICOM instance in IDC hosted data. An IDC BigQuery manifest optionally includes the UUID (called a crdc\_instance\_uuid) of each instance (version) in the cohort. ### UIDs and UUIDs explained with an example Consider an instance in the CPTAC-CM collection that has this `SOPInstanceUID`: `1.3.6.1.4.1.5962.99.1.171941254.777277241.1640849481094.35.0`\\ It is in a series having this `SeriesInstanceUID`:\ `1.3.6.1.4.1.5962.99.1.171941254.777277241.1640849481094.2.0` The instance and series were added to the IDC Data set in IDC version 7. At that point, the instance was assigned UUID:\ `5dce0cf0-4694-4dff-8f9e-2785bf179267`\ and the series was assigned this UUID:\ `e127d258-37c2-47bb-a7d1-1faa7f47f47a` In IDC version 10, a revision of this instance was added (keeping its original `SOPInstanceUID`), and assigned this UUID:\ `21e5e9ce-01f5-4b9b-9899-a2cbb979b542` Because this instance was revised, the series containing it was implicitly revised. The revised series was thus issued a new UUID:\ `ee34c840-b0ca-4400-a6c8-c605cef17630` Thus, the initial version of this instance has this file name:\ `e127d258-37c2-47bb-a7d1-1faa7f47f47a/5dce0cf0-4694-4dff-8f9e-2785bf179267.dcm`\ and the revised version of the instance has the this file name:\ `ee34c840-b0ca-4400-a6c8-c605cef17630/21e5e9ce-01f5-4b9b-9899-a2cbb979b542.dcm` Both versions of the instance are in both AWS and GCS buckets. {% code title="AWS bucket example" overflow="wrap" %} ```bash s5cmd --no-sign-request ls s3://idc-open-data/e127d258-37c2-47bb-a7d1-1faa7f47f47a/5dce0cf0-4694-4dff-8f9e-2785bf179267.dcm 2023-04-09 11:49:55 3308170 5dce0cf0-4694-4dff-8f9e-2785bf179267.dcm ``` {% endcode %} {% code title="GCS bucket example" overflow="wrap" %} ```bash s5cmd --no-sign-request --endpoint-url https://storage.googleapis.com ls s3://idc-open-data/e127d258-37c2-47bb-a7d1-1faa7f47f47a/5dce0cf0-4694-4dff-8f9e-2785bf179267.dcm 3308170 2023-04-01T01:21:31Z gs://idc-open-data/e127d258-37c2-47bb-a7d1-1faa7f47f47a/5dce0cf0-4694-4dff-8f9e-2785bf179267.dcm TOTAL: 1 objects, 3308402 bytes (3.16 MiB) ``` {% endcode %} Note that GCS and AWS bucket names are different. In fact, DICOM instance data is distributed across multiple buckets in both GCS and AWS. We will discuss obtaining GCS and AWS URLs more a little later. {% hint style="info" %} It is possible that a series is revised, but one or more instances in the series are not revised. For example if a single instance in a series (assume the series has a uuid \) is revised, that instance gets a new UUID, and there is implicitly a new version of the series, which gets a new UUID (call it \). If an instance that is not revised has UUID \, then its corresponding file in cloud storage will the have name:\ `/.dcm` in the "old" series. But, because that same instance version is in the revised series, there must also be a file in cloud storage named:\ `/.dcm`\ The result will be two distinct but identical files. {% endhint %} Utilities like gsutil, s3 and s5cmd "understand" the implied hierarchy in these file names. Thus the series UUID now acts like the name of a directory that contains all the instance versions in the series version: {% code overflow="wrap" %} ```bash s5cmd --no-sign-request --endpoint-url https://storage.googleapis.com ls s3://public-datasets-idc/ee34c840-b0ca-4400-a6c8-c605cef17630/ 2023/04/01 03:00:34 1719696 18c206a6-2db4-45cd-89a2-e83273a38f42.dcm 2023/04/01 03:00:36 3308402 21e5e9ce-01f5-4b9b-9899-a2cbb979b542.dcm 2023/04/01 01:50:29 29477804 3cfc3da3-8389-49f6-a6ee-6ba6406f639e.dcm 2023/04/01 01:50:27 214715792 428590a0-816c-4041-a3ae-676a68411794.dcm 2023/04/01 03:00:30 2301902 57ff4432-c29d-4ccf-964c-0b421302add3.dcm 2023/04/01 03:00:33 3540080 77ff406a-a236-4846-83dd-ae3bd7a6bc71.dcm ``` {% endcode %} and similarly for AWS buckets, thus making it easy to transfer all instances in a series from the cloud. Because file names are more or less opaque, the user will not typically select files by listing the contents of a bucket. Instead, one should use either the IDC Portal or IDC BigQuery tables to identify items of interest and, then, generate a manifest of objects that can be passed to a utility like s5cmd. ### Resolving CRDC Globally Unique Identifiers (GUIDs) {% hint style="info" %} From the [GA4GH Data Repository Service API](https://ga4gh.github.io/data-repository-service-schemas/preview/release/drs-1.0.0/docs/#_drs_ids) specification: "The Data Repository Service (DRS) API provides a generic interface to data repositories so data consumers, including workflow systems, can access data objects in a single, standard way regardless of where they are stored and how they are managed. The primary functionality of DRS is to map a logical ID to a means for physically retrieving the data represented by the ID." {% endhint %} Each instance UUID can be used to form a [`DRS ID`](https://ga4gh.github.io/data-repository-service-schemas/preview/release/drs-1.0.0/docs/#_drs_ids) that has been indexed by the [NCI CRDC Data Commons Framework](https://dcf.gen3.org/) (DCF), and can be used to access data that defines that object. In particular this data includes the GCS and AWS URLs of the DICOM instance file. Though the GCS or AWS URL of an instance might change over time, the UUID of an instance can always be resolved to obtain its current URLs. Thus, for long term curation of data, it is recommended to record instance UUIDs. The data object returned by the server is a GA4GH DRS [`DrsObject`](https://ga4gh.github.io/data-repository-service-schemas/preview/release/drs-1.0.0/docs/#_drs_ids)`:` This is a typical IDC instance UUID:\ `641121f1-5ca0-42cc-9156-fb5538c14355`\ of a (version of a) DICOM instance, and this is the corresponding DRS ID:\ `dg.4DFC/641121f1-5ca0-42cc-9156-fb5538c14355` A DRS ID can be resolved by appending it to the following URL, which is the resolution service within CRDC: `https://nci-crdc.datacommons.io/ga4gh/drs/v1/objects/` . For example, the following `curl` command: `>> curl https://nci-crdc.datacommons.io/ga4gh/drs/v1/objects/dg.4DFC/641121f1-5ca0-42cc-9156-fb5538c14355` returns this DrsObject: ``` { "access_methods": [ { "access_id": "gs", "access_url": { "url": "gs://idc-open-data/cc9c8541-949d-48d9-beaf-7028aa4906dc/641121f1-5ca0-42cc-9156-fb5538c14355.dcm" }, "region": "", "type": "gs" }, { "access_id": "s3", "access_url": { "url": "s3://idc-open-data/cc9c8541-949d-48d9-beaf-7028aa4906dc/641121f1-5ca0-42cc-9156-fb5538c14355.dcm" }, "region": "", "type": "s3" } ], "aliases": [], "checksums": [ { "checksum": "f338e8c5e3d8955d222a04d5f3f6e2b4", "type": "md5" } ], "created_time": "2020-06-01T00:00:00", "description": "DICOM instance", "form": "object", "id": "dg.4DFC/641121f1-5ca0-42cc-9156-fb5538c14355", "index_created_time": "2023-06-26T18:27:45.810110", "index_updated_time": "2023-06-26T18:27:45.810110", "mime_type": "application/json", "name": "1.3.6.1.4.1.14519.5.2.1.7695.1700.277743171070833720282648319465", "self_uri": "drs://dg.4DFC:641121f1-5ca0-42cc-9156-fb5538c14355", "size": 135450, "updated_time": "2020-06-01T00:00:00", "version": "IDC version: 1" } ``` As can be seen, the `access_methods` component in the returned DrsObject includes a URL for each of the corresponding files in Google GCS and AWS S3. # Data versioning ## Summary IDC updates its data offering at intervals of 2-4 months, with the data releases timing driven by the availability of new data, updates of existing data, introduction of new capabilities and various priority considerations. You can see the historical summary of IDC releases in [this page](/data/data-release-notes#idc-releases-summary-view). When you work with IDC data at any given time, you should be aware of the data release version. If you build cohorts using filters or queries against the IDC idc\_current and idc\_current\_clinical datasets, the result of those queries may change as the IDC content is evolving. Building queries that refer to the specific data release version will ensure that the result is the same. Here is how you can learn what version of IDC data you are interacting with, depending on what interface to the data you are using: * **IDC Portal**: data version and release date are displayed in the summary strip
* **idc-index**: use `get_idc_version()`function ```python from idc_index import IDCClient idc_version = IDCClient.get_idc_version() ``` * **BigQuery**: the `bigquery-public-data` project, `idc_current` datasets are effectively aliases of the latest IDC BQ datasets. To find the actual IDC data release number, expand the list of datasets under `bigquery-public-data`project, and search for the ones that follow the pattern \`idc\_v\\`. The one with the largest number corresponds to the latest released version, and will match the content in `idc_current` (related Google bug [here](https://issuetracker.google.com/issues/324112186)).
* **3D Slicer / SlicerIDCBrowser**: version information is provided in the SlicerIDCBrowser module top panel, and in the pop-up window title.
## Implementation details The IDC obtains curated DICOM radiology, pathology and microscopy image and analysis data from The Cancer Imaging Archive (TCIA) and additional sources. Data from all these sources evolves over time as new data is added (common), existing files are corrected (rare), or data is removed (very rare). Users interact with IDC using one of the following interfaces to define cohorts, and then perform analyses on these cohorts: * [IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/): while this approach is most convenient, it allows searching using a small subset of attributes, defines cohorts only in terms of cases that meet the defined criteria, and has very limited options for combining multiple search criteria * [IDC REST API](/api/api) and [MCP server](/mcp/mcp): language-independent access to the same structured cohort filters as the Portal, plus a guarded SQL surface for relational and aggregate queries * [IDC BigQuery](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=dicom_all\&page=table) tables via [SQL interface](https://cloud.google.com/bigquery/docs/reference/standard-sql/introduction): this approach is most powerful, as it allows the use of [any of the DICOM metadata attributes](https://cloud.google.com/healthcare-api/docs/how-tos/dicom-bigquery-schema) to define the cohort, while leveraging the expressiveness of SQL in defining the selection logic, and allows to define cohort at any level of the data model hierarchy (i.e., instances, series, studies or cases) The goal of IDC versioning is to create a series of "snapshots” over time of the entirety of the evolving IDC imaging dataset, such that searching with respect to an IDC version according to some criteria (creating a cohort) will always identify exactly the same set of objects. Here “identify” particularly means providing URLs or other access methods to the corresponding physical data objects. In order to reproduce the result of such analysis, it must be possible to precisely recreate a cohort. For this purpose an IDC cohort as defined in the Portal is specified and saved as a filter applied against a specified IDC data version. Alternatively, the cohort can be defined as an SQL query, or as a list of unique identifiers selecting specific files within a defined data release version. Because an IDC version exactly defines the set of data against which the filter/query is applied, and because all versions of all data, except data removed due to PHI/PII concerns, should continue to be available, a cohort is therefore persistent over the course of the evolution of IDC data. There are various reasons that can cause modification of the existing collections in IDC: * images for new patients can be added to an existing collections; * additional DICOM series are sometimes added to a DICOM study over time (i.e., those that contain new annotations or analysis results); * a series may be added or removed from an existing study; * metadata of an existing instance might be corrected (which may or may not lead to an update of the DICOM `SOPInstanceUID` corresponding to the instance). These and other possible changes mean that DICOM instances, series and studies can change from one IDC data version to the next, while their DICOM UIDs remain unchanged. This motivates the need for maintaining versioning of the DICOM entities. Because DICOM `SOPInstanceUIDs`, `SeriesInstanceUIDs` or `StudyInstanceUIDs` can remain invariant even when the composition of an instance, series or study changes, IDC assigns each version of each instance, series or study a [*UUID*](https://en.wikipedia.org/wiki/Universally_unique_identifier) to uniquely identify it and differentiate it from other versions of the same DICOM object. In the same way patients and collections are versioned and each version of each patient and collection is assigned a unique UUID. As with instance, series and study UUIDs, the UUID of the patient and collection to which an instance belongs is available in the auxiliary\_metadata\_table. {% hint style="info" %} It is very important to appreciate the difference between DICOM Unique Identifiers (UIDs) and CRDC Universally Unique Identifiers (UUIDs) assigned at the various levels of the DICOM hierarchy: * **DICOM UID**s are available as DICOM metadata attributes within the DICOM files for each DICOM Study, Series and Instance. Those UIDs follow the conventions of the DICOM UI Value Representation. DICOM UIDs are not versioned. I.e., if a DICOM study is augmented with a new DICOM series, the DICOM `StudyInstanceUID` will not change. If an instance within an existing DICOM series is modified, DICOM `SeriesInstanceUID` or the `SOPInstanceUID` of the modified instance may or may not change. * **IDC UUID**s are **not** available as DICOM metadata attributes - they are generated for the DICOM studies, series and instances at the time of data ingestion, and are available in the IDC BigQuery tables. IDC UUIDs are tied to the content of the entity they correspond to. I.e., if anything within a DICOM study/series/instance is changed in a given IDC data release, a new UUID at the corresponding level of data hierarchy will be generated, while the previous version will be indexed and available via the prior UUID. {% endhint %} The data in each IDC version, then, can be thought of as some set of versioned DICOM instances, series, studies, patients and collections. This set is defined in terms of the corresponding set of instance, series, study, patient and collection UUIDs. This means that if, e.g., some version of an instance having UUID *UUIDx* that was in IDC version *Vm* is changed, a new UUID, *UUIDy*, will be assigned to the new instance version. Subsequent IDC versions, *Vm+1*, *Vm+2, ...* will include that new instance version identified by *UUIDy* unless and until that instance is again changed. Similarly if the composition of some series changes, either because an instance in the series is changed, or an instance is added or removed from that series, a new UUID is assigned to the new version of that series and identifies that version of the series in subsequent IDC versions. Similarly, a study is assigned a new UUID when its composition changes, and similarly for patients and collections. A corollary is that only a single version of an instance, series, study, patient or collection is in an IDC version. Note that the DICOM does not include such version information. Versioning of an object is implicit in the associated UUIDs. As we will see in [Organization of data](/data/organization-of-data/files-and-metadata), the UUID of a (version of an) instance, and the UUID of the (version of a) series to which it belongs, are used in forming the object (file) name of the corresponding GCS and AWS objects. In addition, each instance version has a corresponding GA4GH DRS object, identified by a GUID based on the instance version's UUID. Refer to the [GA4GH DRS Objects](/data/organization-of-data/guids-and-uuids) section for details. # Downloading data {% hint style="info" %} If you have questions or feedback about the download tools provided by IDC, please reach out via our [forum](https://discourse.canceridc.dev/) - we are very interested in hearing your feedback and suggestions! {% endhint %} IDC supports a variety of interfaces for fetching individual images, cohorts (groups of images), or portions of images, using desktop application, command-line interface, or programmatic API. These interfaces are covered in the subsequent pages. You should select the specific approach to accessing IDC data depending on your requirements. * Download **directly from** [**IDC Portal**](/data/downloading-data/idc-portal): no prerequisites other than a Chrome web browser! * [idc-index](https://learn.canceridc.dev/data/downloading-data/downloading-data#command-line-or-programmatic-download-idc-index-python-package) interface: **command-line and Python API interface** to download images corresponding to the specific patient/study/series, or a cohort defined by a manifest * [3D Slicer](/data/downloading-data/downloading-data) interface: **desktop application** to download images corresponding to the specific patient/study/series, or a cohort defined by a manifest * [s5cmd](/data/downloading-data/downloading-data-with-s5cmd): **command-line interface** to download images for a cohort defined by a manifest (unlike `idc-index`, does not organize downloaded images into folders corresponding to IDC data model hierarchy) * [DICOMweb](/data/downloading-data/dicomweb-access) interface: **REST API interface** to access both metadata and pixel data at the granularity of image frames/tiles * [Directly loading DICOM objects from Google Cloud or AWS in Python](/data/downloading-data/direct-loading): **Python API interface** to access both metadata and pixel data at the granularity of image frames/tiles # IDC Portal You can download data at the patient/case, DICOM study or series levels directly from the IDC Portal interface, as demonstrated below! {% embed url="" %} # idc-index and 3D Slicer Depending on whether you would like to download data interactively or programmatically, we provide two recommended tools to help you. ### Command-line or programmatic download: idc-index python package [`idc-index`](https://github.com/ImagingDataCommons/idc-index) is a python package designed to simplify access to IDC data. Assuming you have Python installed on your computer (if for some reason you do not have Python, you can check out legacy download instructions [here](/data/downloading-data/downloading-data-with-s5cmd)), you can get this package with `pip` like this: ```shell-session pip install idc-index --upgrade ``` Once installed, you can use it to explore, search, select and download corresponding files as shown in the examples below. You can also take a look at a short tutorial on using `idc-index` [here](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/part2_searching_basics.ipynb). #### Command line download interface With the `idc-index` package you get command line scripts that aim to make download simple. Have a .s5cmd manifest file you downloaded from IDC Portal or from the records in the IDC Zenodo community? Get the corresponding files as follows (you will also get download progress bar and the downloaded files will be organized in the collection/patient/study/series folder hierarchy!): ```sh idc download manifest_file.s5cmd ``` You can use the same command to download files corresponding to any collection, patient, study or series, referred to by the identifiers you can copy from the portal!

Copy collection ID from the IDC Portal interface

```sh $ idc download pseudo_phi_dicom_data 2024-09-04 17:59:50,944 - Downloading from IDC v18 index 2024-09-04 17:59:50,952 - Identified matching collection_id: ['pseudo_phi_dicom_data'] 2024-09-04 17:59:50,959 - Total size of files to download: 1.27 GB 2024-09-04 17:59:50,959 - Total free space on disk: 29.02233088GB 2024-09-04 17:59:51,151 - Not using s5cmd sync as the destination folder is empty or sync or progress bar is not requested 2024-09-04 17:59:51,156 - Initial size of the directory: 0 bytes 2024-09-04 17:59:51,156 - Approximate size of the files that need to be downloaded: 1274140000.0 bytes Downloading data: 7%|█████ | 86.3M/1.27G [00:13<03:06, 6.36MB/s] ``` Similarly, you can copy identifiers for patient/study/series and download the corresponding content! ```sh # download all files for patient ID 100002 $ idc download 100002 # download all files for DICOM StudyInstanceUID 1.2.840.113654.2.55.192012426995727721871016249335309434385 $ idc download 1.2.840.113654.2.55.192012426995727721871016249335309434385 # download all files for DICOM SeriesInstanceUID 1.2.840.113654.2.55.305538394446738410906709753576946604022 $ idc download 1.2.840.113654.2.55.305538394446738410906709753576946604022 ``` #### Programmatic download ```python from idc_index import IDCClient client = IDCClient() # get identifiers of all collections available in IDC all_collection_ids = client.get_collections() # download files for the specific collection, patient, study or series client.download_from_selection(collection_id="rider_pilot", \ downloadDir="/some/dir") client.download_from_selection(patientId="rider_pilot", \ downloadDir="/some/dir") client.download_from_selection(studyInstanceUID= \ "1.3.6.1.4.1.14519.5.2.1.6279.6001.175012972118199124641098335511", \ downloadDir="/some/dir") client.download_from_selection(seriesInstanceUID=\ "1.3.6.1.4.1.14519.5.2.1.6279.6001.141365756818074696859567662357", \ downloadDir="/some/dir") ``` `idc-index` includes a variety of other helper functions, such as download from the manifest created using IDC portal, automatic generation of the viewer URLs, information about disk space needed for a given collection, and more. We are very interested in your feedback to define the additional functionality to add to this package! Please reach out via [IDC Forum](https://discourse.canceridc.dev/) if you have any suggestions. ### Interactive download: 3D Slicer SlicerIDCBrowser extension [3D Slicer](https://www.slicer.org/) is a free open source, cross-platform, extensible desktop application developed to support a variety of medical imaging research use cases. IDC maintains [SlicerIDCBrowser](https://github.com/ImagingDataCommons/SlicerIDCBrowser), an extension of 3D Slicer, developed to support direct access to IDC data from your desktop. You will need to [install](https://download.slicer.org/) a recent 3D Slicer 5.7.0 preview application (installers are available for Windows, Mac and Linux), and next use 3D Slicer ExtensionManager to install SlicerIDCBrowser extension. Take a look at the quick demo video in [this post](https://discourse.canceridc.dev/t/sliceridcbrowser-extension-released/515) if you have never used 3D Slicer ExtensionManager before. Once installed, you can use SlicerIDCBrowser in one of the two modes: 1. **As an interface to explore IDC data**: you can select individual collections, cases and DICOM studies and download items of interest directly into 3D Slicer for subsequent visualization and analysis. 2. **As download tool**: download IDC content based on the manifest you created using IDC Portal, or identifiers of the individual cases, DICOM studies or series.

Copy identifiers for the studies/series of interest from the IDC Portal

Insert the identifiers in the appropriate fields, or download content defined by the s5cmd manifest

# s5cmd {% hint style="info" %} Make sure you first review the[ ](/data/downloading-data)[Downloading data](/data/downloading-data) section to learn about the simpler interfaces that provide access to IDC data. {% endhint %} SlicerIDCBrowser and `idc-index` discussed in the previous section aim to provide simple interfaces for data access. In some situations, however, you may want to build cohorts using metadata attributes that are not exposed in those tools. In such cases you will need to use BigQuery interface to form your cohort and build a file manifest that you can then use with [`s5cmd`](https://github.com/peak/s5cmd) to download the files. With this approach you will follow a a 2-step process covered on this page: * **Step 1:** create a manifest - a list of the storage bucket URLs of the files to be downloaded. if you want to download the content of the cohort defined in the IDC Portal, [export the `s5cmd` manifest fist](/portal/cohort-manifests), and proceed to Step 2. Alternatively, you can use BigQuery SQL as discussed below to generate the manifest; * **Step 2**: given the manifest, download files to your computer or to a cloud VM using `s5cmd` command line tool. To learn more about using Google BigQuery SQL with IDC, check out part 3 of our ["Getting started" tutorial series](https://github.com/ImagingDataCommons/IDC-Tutorials/tree/master/notebooks/getting_started), which demonstrates how to query and download IDC data! ### Step 1: Create the manifest {% hint style="info" %} You will need to complete prerequisites described in [Getting started with GCP](/introduction/google-cloud-platform/getting-started-with-gcp) in order to be able to execute the manifest generation queries below! {% endhint %} A download manifest can be created using either the IDC Portal, or by executing a BQ query. **If you have generated a manifest using the IDC Portal, as discussed** [**here**](/portal/cohort-manifests)**, proceed to Step 2!** In the remainder of this section we describe creating a manifest from a BigQuery query. The [`dicom_all`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=dicom_all\&page=table) BigQuery table discussed in [this documentation article](https://learn.canceridc.dev/data/organization-of-data/files-and-metadata#bigquery-tables) can be used to subset the files you need based on the DICOM metadata attributes as needed, utilizing the SQL query interface. The `gcs_url` and `aws_url` columns contain Google Cloud Storage and AWS S3 URLs, respectively, that can be used to retrieve the files. Start with the query templates provided below, modify them based on your needs, and save the result in a file `query.txt`. The specific values for `PatientID`, `SeriesInstanceUID`, `StudyInstanceUID` are chosen to serve as examples. You can use IDC Portal to identify items of interest, or you can use SQL queries to subset your data using any of the DICOM attributes. You are encouraged to use the [BigQuery console](https://console.cloud.google.com/bigquery) to test your queries and explore the data first! Queries below demonstrate how to get the Google Storage URLs to download cohort files. {% code overflow="wrap" %} ```sql # Select all files for a given PatientID SELECT DISTINCT(CONCAT(series_aws_url, "* .")) FROM `bigquery-public-data.idc_current.dicom_all` WHERE PatientID = "LUNG1-001" ``` {% endcode %} {% code overflow="wrap" %} ```sql # Select all files for a given collection SELECT DISTINCT(CONCAT(series_aws_url, "* .")) FROM `bigquery-public-data.idc_current.dicom_all` WHERE collection_id = "nsclc_radiomics" ``` {% endcode %} {% code overflow="wrap" %} ```sql # Select all files for a given DICOM series SELECT DISTINCT(CONCAT(series_aws_url, "* .")) FROM `bigquery-public-data.idc_current.dicom_all` WHERE SeriesInstanceUID = "1.3.6.1.4.1.32722.99.99.298991776521342375010861296712563382046" ``` {% endcode %} {% code overflow="wrap" %} ```sql # Select all files for a given DICOM study SELECT DISTINCT(CONCAT(series_aws_url, "* .")) FROM `bigquery-public-data.idc_current.dicom_all` WHERE StudyInstanceUID = "1.3.6.1.4.1.32722.99.99.239341353911714368772597187099978969331" ``` {% endcode %} If you want to download the files corresponding to the cohort from GCP instead of AWS, substitute `series_aws_url` for `series_gcp_url` in the `SELECT` statement of the query, such as in the following SELECT clause: {% code overflow="wrap" %} ```sql SELECT DISTINCT(CONCAT(series_gcp_url, "* .")) ``` {% endcode %} Next, use a Google Cloud SDK `bq query` command (from command line) to run the query and save the result into a manifest file, which will be the list of GCP URLs that can be used to download the data. {% code overflow="wrap" %} ```shell bq query --use_legacy_sql=false --format=csv --max_rows=20000000 < query.txt > manifest.txt ``` {% endcode %} {% hint style="danger" %} Make sure you adjust the `--max_rows` parameter in the queries above to be equal or exceed the number of rows in the result of the query, otherwise your list will be truncated! {% endhint %} For any of the queries, you can get the count of rows to confirm that the `--max_rows` parameter is sufficiently large (use the [BigQuery console](https://console.cloud.google.com/bigquery) to run these queries): ```sql # count the number of rows SELECT COUNT(DISTINCT(crdc_series_uuid)) FROM bigquery-public-data.idc_current.dicom_all WHERE collection_id = "nsclc_radiomics" ``` You can also get the total disk space that will be needed for the files that you will be downloading: ```sql # calculate the disk size in GB needed for the files to be downloaded SELECT ROUND(SUM(instance_size)/POW(1024,3),2) as size_GB FROM bigquery-public-data.idc_current.dicom_all WHERE collection_id = "nsclc_radiomics" ``` ### Step 2: Download the files defined by the manifest [`s5cmd`](https://github.com/peak/s5cmd) is a very fast S3 and local filesystem execution tool that can be used for accessing IDC buckets and downloading files both from GCS and AWS. Install `s5cmd` following the instructions in , or if you have Python pip on you system you can just do `pip install s5cmd --upgrade`. You can verify if your setup was successful by running the following command: it should successfully download one file from IDC. 
s5cmd --no-sign-request --endpoint-url https://storage.googleapis.com cp s3://idc-open-data/cdac3f73-4fc9-4e0d-913b-b64aa3100977/902b4588-6f10-4342-9c80-f1054e67ee83.dcm .
Once `s5cmd` is installed, you can use `s5cmd run` command to download the files corresponding to the manifest. If you defined manifest that references AWS buckets: {% code overflow="wrap" %} ```bash s5cmd --no-sign-request --endpoint-url=https://s3.amazonaws.com run manifest_file_name ``` {% endcode %} If you defined manifest that references GCP buckets, you will need to specify GCS endpoint: 
s5cmd --no-sign-request --endpoint-url https://storage.googleapis.com run manifest_file_name
[^1]: Use this endpoint for accessing GCS buckets # DICOMweb {% hint style="info" %} We welcome your questions or comments on this documentation page content! Please start a thread on [IDC forum](https://discourse.canceridc.dev), and we will be happy to help you! {% endhint %} ## Background The [DICOMweb](https://www.dicomstandard.org/using/dicomweb) interface is available for accessing IDC data. This interface could be especially useful for efficiently downloading small(er) parts of large digital pathology images. While the entire pathology whole-slide image (WSI) pyramid can reach gigabytes in size, the part that is needed for a specific visualization or analysis task can be rather small and localized to the specific image tiles at a given resolution. {% hint style="info" %} New to DICOM WSI? Check out our [introductory tutorial](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/pathomics/getting_started_with_digital_pathology.ipynb) to learn how slide microscopy images are organized in DICOM. {% endhint %} Detailed information on the DICOMweb endpoints that are available to access IDC data is provided here. In brief, there are two DICOM stores available - the [IDC-maintained DICOM store](https://learn.canceridc.dev/data/organization-of-data/dicom-stores#idc-maintained-dicom-store-via-proxy) and the [Google-maintained DICOM store](https://learn.canceridc.dev/data/organization-of-data/dicom-stores#dicom-store-maintained-by-google-healthcare) - we recommend that you familiarize yourself with the documentation to learn about the differences between the two, and select the option that is optimal for your use case. {% hint style="success" %} Code snippets included in this article are also replicated in [this Google Colab tutorial notebook](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/advanced_topics/idc_dicomweb_access.ipynb) for your convenience. {% endhint %} ## Unique identifiers: locating the relevant slides IDC uses DICOM for data organization, and every image contains metadata organized following the data model documented [here](https://learn.canceridc.dev/data/data-model). Each slide corresponds to a DICOM Series, uniquely identified by the `SeriesInstanceUID`, which in turn belongs to a DICOM Study identified by the `StudyInstanceUID`. You will need these two identifiers to access any DICOM slide using DICOMweb! Since IDC contains many terabytes of images, you will typically want to first select images/slides that meet your needs. IDC offers various interfaces to explore and subset the data, starting from the [IDC Portal](https://learn.canceridc.dev/tutorials/portal-tutorial), to the [Python package `idc-index`](https://github.com/ImagingDataCommons/idc-index) (covered in [this tutorial](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/pathomics/slide_microscopy_metadata_search.ipynb)) and BigQuery SQL interfaces (see [this tutorial](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/part3_exploring_cohorts.ipynb)). We strongly recommend you work through the referenced tutorials, but for the purposes of this tutorial, we will demonstrate how you can locate UIDs of a slide that corresponds to pancreas tissue. First, install `idc-index` with `pip install —upgrade idc-index` (`--upgrade` part is very important to make sure you are working with the latest data release of IDC!). Next, the following snippet demonstrates how to select slides of pancreas tissue (you can also select by the lens magnification, stain, and many other attributes - see [this tutorial](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/pathomics/slide_microscopy_metadata_search.ipynb) for details). ```python from idc_index import IDCClient # Instantiate the client idc_client = IDCClient() idc_client.fetch_index('sm_index') # Filter the slides query = """ SELECT index.StudyInstanceUID, sm_index.SeriesInstanceUID FROM sm_index JOIN index ON sm_index.SeriesInstanceUID = index.SeriesInstanceUID WHERE Modality = 'SM' AND primaryAnatomicStructure_CodeMeaning = 'Pancreas' """ pancreas_slides = idc_client.sql_query(query) ``` Next, we select the first slide and will use its StudyInstanceUID and SeriesInstanceUID in the subsequent sections of the code. ```python sample_study_uid = pancreas_slides['StudyInstanceUID'][0] sample_series_uid = pancreas_slides['SeriesInstanceUID'][0] sample_study_uid, sample_series_uid ``` `('2.25.25332367070577326639024635995523878122', '1.3.6.1.4.1.5962.99.1.3380245274.1362068963.1639762817818.2.0')` ## Reading slide regions via DICOMweb We recommend the following two Python libraries that facilitate access to a DICOM store via DICOMweb: * [wsidicom](https://github.com/imi-bigpicture/wsidicom) * [ez-wsi-dicomweb](https://github.com/GoogleCloudPlatform/EZ-WSI-DICOMweb) Both libraries can be installed using pip: ```python pip install wsidicom pip install ez-wsi-dicomweb ``` `wsidicom` is based upon the [dicomweb\_client](https://github.com/ImagingDataCommons/dicomweb-client) Python library, while `ez-wsi-dicomweb` includes its own DICOMweb implementation. {% hint style="danger" %} Note that you can use `wsidicom` with both the IDC-maintained and the Google-maintained DICOM store, while `ez-wsi-dicomweb` only works with the Google-maintained store. {% endhint %} The following code snippets show exemplarily how to use each of the libraries to access a subregion from a DICOM slide identified by the following UIDs we selected earlier: * `sample_study_uid = 2.25.25332367070577326639024635995523878122` * `sample_series_uid = 1.3.6.1.4.1.5962.99.1.3380245274.1362068963.1639762817818.2.0` ### wsidicom When you work with `wsidicom`, the first step requires setting up [dicomweb\_client](https://github.com/ImagingDataCommons/dicomweb-client)’s `DICOMwebClient`: ```python from dicomweb_client.api import DICOMwebClient from dicomweb_client.ext.gcp.session_utils import create_session_from_gcp_credentials ``` If you are accessing the Google-maintained DICOM store, you need to authenticate with your Google credentials first and set up an authorized session for the DICOMwebClient. {% hint style="danger" %} As discussed in the corresponding documentation page we mentioned earlier, Google-hosted DICOM store may not contain the latest version of IDC data! You will encounter access issues for slides that are not present. If this the case, you will need to use the IDC-hosted DICOM store instead! {% endhint %} {% code overflow="wrap" %} ```python from google.colab import auth auth.authenticate_user() # Create authorized session session = create_session_from_gcp_credentials() # Set-up a DICOMwebClient using the dicomweb_client library google_dicom_store_url = 'https://healthcare.googleapis.com/v1/projects/nci-idc-data/locations/us-central1/datasets/idc/dicomStores/idc-store-v21/dicomWeb' dw_client = DICOMwebClient( url=dicom_store_url, session=session ) ``` {% endcode %} Otherwise, if you prefer using IDC-maintained proxied DICOM store, you can skip ahead and just set up your DICOMwebClient using the proxy URL. {% code overflow="wrap" %} ```python # Set-up a DICOMwebClient using the dicomweb_client library idc_dicom_store_url = 'https://proxy.imaging.datacommons.cancer.gov/current/viewer-only-no-downloads-see-tinyurl-dot-com-slash-3j3d9jyp/dicomWeb' dw_client = DICOMwebClient(url=idc_dicom_store_url) ``` {% endcode %} #### Slide access with wsidicom You now need to wrap the previously set-up `DICOMwebClient` into `wsidicom`’s `WsiDicomWebClient`. Then you can use the `open_web()` functionality to find, open and navigate the content of the selected slide: ```python import wsidicom import matplotlib.pyplot as plt wsidicom_client = wsidicom.WsiDicomWebClient(dw_client) slide = wsidicom.WsiDicom.open_web(wsidicom_client, study_uid=sample_study_uid, series_uids=sample_series_uid ) print(slide) ``` {% code overflow="wrap" %} ``` [0]: Pyramid of levels: [0]: Level: 0, size: Size(width=171359, height=74498) px, mpp: SizeMm(width=0.2472, height=0.2472) um/px Instances: [0]: default z: 0.0 default path: 1 ImageData [1]: Level: 2, size: Size(width=42839, height=18624) px, mpp: SizeMm(width=0.988817311328, height=0.988817311328) um/px Instances: [0]: default z: 0.0 default path: 1 ImageData [2]: Level: 4, size: Size(width=10709, height=4656) px, mpp: SizeMm(width=3.955546250817, height=3.955546250817) um/px Instances: [0]: default z: 0.0 default path: 1 ImageData [3]: Level: 6, size: Size(width=2677, height=1164) px, mpp: SizeMm(width=15.823662607396, height=15.823662607396) um/px Instances: [0]: default z: 0.0 default path: 1 ImageData ``` {% endcode %} To access a certain part of a slide, `wsidicom` offers the `read_region()` functionality: ```python # Access and visualize 500x500px subregion at level 4, starting from pixel (1000,1000) region = slide.read_region(location=(1000, 1000), level=4, size=(500, 500)) plt.imshow(region) plt.show() ```
Screenshot of slide region
### ez-wsi-dicomweb The following code shows how to set-up an interface for DICOMweb with `ez-wsi-dicomweb`. You can only use this interface for accessing data from the Google-maintained DICOM store, which means authentication with you Google account is required. {% code overflow="wrap" %} ```python from ez_wsi_dicomweb import dicomweb_credential_factory from ez_wsi_dicomweb import dicom_slide from ez_wsi_dicomweb import local_dicom_slide_cache_types from ez_wsi_dicomweb import dicom_web_interface from ez_wsi_dicomweb import patch_generator from ez_wsi_dicomweb import pixel_spacing from ez_wsi_dicomweb.ml_toolkit import dicom_path from google.colab import auth auth.authenticate_user() google_dicom_store_url = 'https://healthcare.googleapis.com/v1/projects/nci-idc-data/locations/us-central1/datasets/idc/dicomStores/idc-store-v20/dicomWeb' series_path_str = ( f'{google_dicom_store_url}' f'/studies/{sample_study_uid}' f'/series/{sample_series_uid}' ) series_path = dicom_path.FromString(series_path_str) dcf = dicomweb_credential_factory.CredentialFactory() dwi = dicom_web_interface.DicomWebInterface(dcf) ``` {% endcode %} The slide, slide level information and slide regions can be accessed as follows. To accelerate image retrieval, `ez-wsi-dicomweb` can be configured to fetch frames in blocks and cache them for subsequent use. For more information, check out [this notebook](https://github.com/GoogleCloudPlatform/EZ-WSI-DICOMweb/blob/main/ez_wsi_demo.ipynb), section “Enabling EZ-WSI DICOMweb Frame Cache”. {% code overflow="wrap" %} ```python ds = dicom_slide.DicomSlide( dwi=dwi, path=series_path, enable_client_slide_frame_decompression = True ) # More information: https://github.com/GoogleCloudPlatform/EZ-WSI-DICOMweb/blob/main/ez_wsi_demo.ipynb ds.init_slide_frame_cache( optimization_hint=local_dicom_slide_cache_types.CacheConfigOptimizationHint.MINIMIZE_LATENCY ) ``` {% endcode %} {% code overflow="wrap" %} ```python # Investigate existing levels and their dimensions for level in ds.levels: print(f'Level {level.level_index} has pixel dimensions (row, col): {level.height, level.width}') ``` {% endcode %} ``` Level 1 has pixel dimensions (row, col): (74498, 171359) Level 2 has pixel dimensions (row, col): (18624, 42839) Level 3 has pixel dimensions (row, col): (4656, 10709) Level 4 has pixel dimensions (row, col): (1164, 2677) ``` {% code overflow="wrap" %} ```python # Access and visualize 500x500px subregion at level 3, starting from pixel (1000,1000) level = ds.get_level_by_index(3) region = ds.get_patch(level=level, x=1000, y=1000, width=500, height=500).image_bytes() plt.imshow(region) plt.show() ``` {% endcode %}
Screenshot of slide region
## Iterating through tiles using DICOMweb To iterate over image tiles you can simply wrap the functionality presented above into your own function that iterates over the coordinates of interest to you. In case you prefer to iterate over the frames as they are stored within the DICOM file, wsidicom does also offer a [`read_tile()`](https://github.com/imi-bigpicture/wsidicom/blob/8372612cbbcce972c70bfc0fd2922655ec886c5c/wsidicom/wsidicom.py#L716) method.\ Iteration over a slide and accessing tiles from an area defined by a tissue mask can be quite easily achieved using `ez-wsi-dicomweb`’s DICOMPatchGenerator as described in [this notebook](https://github.com/GoogleCloudPlatform/EZ-WSI-DICOMweb/blob/main/ez_wsi_demo.ipynb) in section “Generating patches from a level image”. ## Recommendations Both libraries — `ez-wsi-dicomweb` and `wsidicom`— can be recommended for reliable DICOMweb access to IDC data. Based on our experience, `ez-wsi-dicomweb` is often faster, likely due to its caching capabilities, and customizations for efficient access to image patches from a Google DICOM store for AI model training. `wsidicom`, on the other hand, is a more general-purpose tool offering extensive functionality for accessing DICOM files (images as well as annotation files) both from local disk or from the cloud via DICOMweb. It is important to note that when running code locally, access times may be slightly longer compared to cloud-based (such as in a Colab notebook) execution. # Directly loading DICOM objects from Google Cloud or AWS in Python DICOM files in the IDC are stored as "blobs" on the cloud, with one copy housed on Google Cloud Storage (GCS) and another on Amazon Web Services (AWS) S3 storage. By using the right tools, these blobs can be wrapped to appear as "file-like" objects to Python DICOM libraries, enabling intelligent loading of DICOM files directly from cloud storage as if they were local files without having to first download them onto a local drive. {% hint style="success" %} Code snippets included in this article are also replicated in this Google Colab tutorial notebook for you convenience: {% endhint %} ### Reading files with Pydicom [Pydicom](https://pydicom.github.io/pydicom/stable/index.html) is popular library for working with DICOM files in Python. Its [dcmread](https://pydicom.github.io/pydicom/stable/reference/generated/pydicom.filereader.dcmread.html#pydicom.filereader.dcmread) function is able to accept any "file-like" object, meaning you can read a file straight from a cloud blob if you know its path. See [this page](/data/organization-of-data/files-and-metadata#storage-buckets) for information on finding the paths of the blobs for DICOM objects in IDC. The `dcmread` function also has some other options that allow you to control what is read. For example you can choose to read only the metadata and not the pixel data, or read only certain attributes. In the following two sections, we demonstrate these abilities using first Google Cloud Storage blobs and then AWS S3 blobs. **Mapping IDC DICOM series to bucket URLs** All of the image data available from IDC is replicated between public Google Cloud Storage (GCS) and AWS buckets. pip-installable [idc-index](https://github.com/imagingdatacommons/idc-index) package provides convenience functions to get URLs of the files corresponding to a given DICOM series. ```python from idc_index import IDCClient # Create IDCClient for looking up bucket URLs idc_client = IDCClient() # Get the list of GCS file URLs in Google bucket from SeriesInstanceUID gcs_file_urls = idc_client.get_series_file_URLs( seriesInstanceUID="1.3.6.1.4.1.14519.5.2.1.131619305319442714547556255525285829796", source_bucket_location="gcs", ) # Get the list of AWS file URLs in AWS bucket from SeriesInstanceUID aws_file_urls = idc_client.get_series_file_URLs( seriesInstanceUID="1.3.6.1.4.1.14519.5.2.1.131619305319442714547556255525285829796", source_bucket_location="aws", ) ``` **From Google Cloud Storage blobs** The [official Python SDK for Google Cloud Storage](https://cloud.google.com/python/docs/reference/storage/latest/) (installable from pip and PyPI as `google-cloud-storage`) provides a "file-like" interface allowing other Python libraries, such as Pydicom, to work with blobs as if they were "normal" files on the local filesystem. To read from a GCS blob with Pydicom, first create a storage client and blob object, representing a remote blob object stored on the cloud, then simply use the `.open('rb')` method to create a readable file-like object that can be passed to the `dcmread` function. ```python from pydicom import dcmread from pydicom.datadict import keyword_dict from google.cloud import storage from idc_index import IDCClient # Create IDCClient for looking up bucket URLs idc_client = IDCClient() # Create a client and bucket object representing the IDC public data bucket gcs_client = storage.Client.create_anonymous_client() # This example uses a CT series in the IDC. # get the list of file URLs in Google bucket from the SeriesInstanceUID file_urls = idc_client.get_series_file_URLs( seriesInstanceUID="1.3.6.1.4.1.14519.5.2.1.131619305319442714547556255525285829796", source_bucket_location="gcs", ) # URLs will look like this: # s3://idc-open-data/668029cf-41bf-4644-b68a-46b8fa99c3bc/f4fe9671-0a99-4b6d-9641-d441f13620d4.dcm (_, _, bucket_name, folder_name, file_name) = file_urls[0].split("/") blob_key = f"{folder_name}/{file_name}" # These objects represent the bucket and a single image blob within the bucket bucket = gcs_client.bucket(bucket_name) blob = bucket.blob(blob_key) # Read the whole file directly from the blob with blob.open("rb") as reader: dcm = dcmread(reader) # Read metadata only (no pixel data) with blob.open("rb", chunk_size=5_000) as reader: dcm = dcmread(reader, stop_before_pixels=True) # Read only specific attributes, identified by their tag # (here the Manufacturer and ManufacturerModelName attributes) with blob.open("rb", chunk_size=5_000) as reader: dcm = dcmread( reader, specific_tags=[keyword_dict['Manufacturer'], keyword_dict['ManufacturerModelName']], ) print(dcm) ``` Reading only metadata or only specific attributes will reduce the amount of data that needs to be pulled down under some circumstances and therefore make the loading process faster. This depends on the size of the attributes being retrieved, the `chunk_size` (a parameter of the `open()` method that controls how much data is pulled in each HTTP request to the server), and the position of the requested element within the file (since it is necessary to seek through the file until the requested attributes are found, but any data after the requested attributes need not be pulled). If you are not retrieving entire images, we strongly recommend specifying a `chunk_size` (in bytes) because the default value is around 40MB, which is typically far larger than the optimal value for accessing metadata attributes or individual frames (see later). This works because running the [open](https://cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.blob.Blob#google_cloud_storage_blob_Blob_open) method on a Blob object returns a [BlobReader](https://cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.fileio.BlobReader) object, which has a "file-like" interface (specifically the `seek`, `read`, and `tell` methods). **From AWS S3 blobs** The `s3fs` [package](https://s3fs.readthedocs.io/en/latest/) provides "file-like" interface for accessing S3 blobs. It can be installed with `pip install s3fs`. The following example repeats the above example using the counterpart of the same blob on AWS S3. ```python import s3fs from pydicom import dcmread from pydicom.datadict import keyword_dict from idc_index import IDCClient # Create IDCClient for looking up bucket URLs idc_client = IDCClient() # This example uses a CT series in the IDC (same as above). # Get the list of file URLs in AWS bucket from SeriesInstanceUID file_urls = idc_client.get_series_file_URLs( seriesInstanceUID="1.3.6.1.4.1.14519.5.2.1.131619305319442714547556255525285829796", source_bucket_location="aws", ) # Configure a client to avoid the need for AWS credentials s3_client = s3fs.S3FileSystem( anon=True, # no credentials needed to access public data default_block_size=50_000, # ~50kB data pulled in each request use_ssl=False # disable encryption for a speed boost ) with s3_client.open(file_urls[0], 'rb') as reader: dcm = dcmread(reader) # Read metadata only (no pixel data) with s3_client.open(file_urls[0], 'rb') as reader: dcm = dcmread(reader, stop_before_pixels=True) # Read only specific attributes, identified by their tag # (here the Manufacturer and ManufacturerModelName attributes) with s3_client.open(file_urls[0], 'rb') as reader: dcm = dcmread( reader, specific_tags=[keyword_dict['Manufacturer'], keyword_dict['ManufacturerModelName']], ) print(dcm) ``` Similar to the `chunk_size` parameter in GCS, the `default_block_size` is crucially important for determining how efficient this is. Its default value is around 50MB, which will result in orders of magnitude more data than necessary being pulled than is needed to retrieve metadata. In the above example, we set it to 50kB. ### Frame-level access with Highdicom [Highdicom](https://highdicom.readthedocs.io) is a higher-level library providing several features to work with images and image-derived DICOM objects. As of the release 0.25.1, its various reading methods (including [imread](https://highdicom.readthedocs.io/en/latest/package.html#highdicom.imread), [segread](https://highdicom.readthedocs.io/en/latest/package.html#highdicom.seg.segread), [annread](https://highdicom.readthedocs.io/en/latest/package.html#highdicom.ann.annread), and [srread](https://highdicom.readthedocs.io/en/latest/package.html#highdicom.sr.srread)) can read any file-like object, including Google Cloud blobs and S3 blobs opened with `s3fs`. A particularly useful feature when working with blobs is ["lazy" frame retrieval](https://highdicom.readthedocs.io/en/latest/image.html#lazy) for images and segmentations. This downloads only the image metadata when the file is initially loaded, uses it to create a frame-level index, and downloads specific frames as and when they are requested by the user. This is especially useful for large multiframe files (such as those found in slide microscopy or multi-segment binary or fractional segmentations) as it can significantly reduce the amount of data that needs to be downloaded to access a subset of the frames. In this first example, we use lazy frame retrieval to load only a specific spatial patch from a large whole slide image from the IDC using GCS. ```python import numpy as np import highdicom as hd import matplotlib.pyplot as plt from google.cloud import storage from pydicom import dcmread from idc_index import IDCClient # Create IDCClient for looking up bucket URLs idc_client = IDCClient() # Install additional component of idc-index to resolve SM instances to file URLs idc_client.fetch_index("sm_instance_index") # Given SeriesInstanceUID of an SM series, find the instance that corresponds to the # highest resolution base layer of the image pyramid query = """ SELECT SOPInstanceUID, TotalPixelMatrixColumns FROM sm_instance_index WHERE SeriesInstanceUID = '1.3.6.1.4.1.5962.99.1.1900325859.924065538.1719887277027.4.0' ORDER BY TotalPixelMatrixColumns DESC LIMIT 1 """ result = idc_client.sql_query(query) # Get URL corresponding to the base layer instance in the Google Storage bucket base_layer_file_url = idc_client.get_instance_file_URL( sopInstanceUID=result.iloc[0]["SOPInstanceUID"], source_bucket_location="gcs" ) # Create a storage client and use it to access the IDC's public data bucket gcs_client = storage.Client.create_anonymous_client() (_,_, bucket_name, folder_name, file_name) = base_layer_file_url.split("/") blob_key = f"{folder_name}/{file_name}" bucket = gcs_client.bucket(bucket_name) base_layer_blob = bucket.blob(blob_key) # Read directly from the blob object using lazy frame retrieval with base_layer_blob.open(mode="rb", chunk_size=500_000) as reader: im = hd.imread(reader, lazy_frame_retrieval=True) # Grab an arbitrary region of tile full pixel matrix region = im.get_total_pixel_matrix( row_start=15000, row_end=15512, column_start=17000, column_end=17512, dtype=np.uint8 ) # Show the region plt.imshow(region) plt.show() ``` Running this code should produce an output that looks like this:
Screenshot of slide region
The next example repeats this on the same image in AWS S3: ```python import numpy as np import highdicom as hd import matplotlib.pyplot as plt from pydicom import dcmread import s3fs from idc_index import IDCClient # Create IDCClient for looking up bucket URLs idc_client = IDCClient() # Install additional component of idc-index to resolve SM instances to file URLs idc_client.fetch_index("sm_instance_index") # Given SeriesInstanceUID of an SM series, find the instance that corresponds to the # highest resolution base layer of the image pyramid query = """ SELECT SOPInstanceUID, TotalPixelMatrixColumns FROM sm_instance_index WHERE SeriesInstanceUID = '1.3.6.1.4.1.5962.99.1.1900325859.924065538.1719887277027.4.0' ORDER BY TotalPixelMatrixColumns DESC LIMIT 1 """ result = idc_client.sql_query(query) # Get URL corresponding to the base layer instance in the AWS S3 bucket base_layer_file_url = idc_client.get_instance_file_URL( sopInstanceUID=result.iloc[0]["SOPInstanceUID"], source_bucket_location="aws" ) # Create a storage client and use it to access the IDC's public data bucket # Configure a client to avoid the need for AWS credentials s3_client = s3fs.S3FileSystem( anon=True, # no credentials needed to access public data default_block_size=500_000, # ~500kB data pulled in each request use_ssl=False # disable encryption for a speed boost ) # Read directly from the blob object using lazy frame retrieval with s3_client.open(base_layer_file_url, 'rb') as reader: im = hd.imread(reader, lazy_frame_retrieval=True) # Grab an arbitrary region of tile full pixel matrix region = im.get_total_pixel_matrix( row_start=15000, row_end=15512, column_start=17000, column_end=17512, dtype=np.uint8 ) # Show the region plt.imshow(region) plt.show() ``` In both cases, we set the `chunk_size`/`default_block_size` to around 500kB, which should be enough to ensure each frame can be retrieved in a single request while minimizing further unnecessary data retrieval. As a further example, we use lazy frame retrieval to load only a specific set of segments from a large multi-organ segmentation of a CT image in the IDC stored in binary format (in binary segmentations, each segment is stored using a separate set of frames) using GCS. ```python import highdicom as hd from google.cloud import storage from idc_index import IDCClient # Create IDCClient for looking up bucket URLs idc_client = IDCClient() # Get the file URL corresponding to the segmentation of a CT series # containing a large number of different organs - the same one as used in the # IDC Portal front page file_urls = idc_client.get_series_file_URLs( seriesInstanceUID="1.2.276.0.7230010.3.1.3.313263360.15787.1706310178.804490", source_bucket_location="gcs" ) (_, _, bucket_name, folder_name, file_name) = file_urls[0].split("/") # Create a storage client and use it to access the IDC's public data package gcs_client = storage.Client.create_anonymous_client() bucket = gcs_client.bucket(bucket_name) blob_name = f"{folder_name}/{file_name}" blob = bucket.blob(blob_name) # Open the blob with "segread" using the "lazy frame retrieval" option with blob.open(mode="rb", chunk_size=500_000) as reader: seg = hd.seg.segread(reader, lazy_frame_retrieval=True) # Find the segment number corresponding to the liver segment selected_segment_numbers = seg.get_segment_numbers(segment_label="Liver") # Read in the selected segments lazily volume = seg.get_volume( segment_numbers=selected_segment_numbers, combine_segments=True, ) # Print dimensions of the liver segment volume print(volume.shape) ``` See [this](https://highdicom.readthedocs.io/en/latest/image.html) page for more information on highdicom's `Image` class, and [this](https://highdicom.readthedocs.io/en/latest/seg.html) page for the `Segmentation` class. ### The importance of offset tables for slide microscopy (SM) images Achieving good performance for the Slide Microscopy frame-level retrievals requires the presence of either a "Basic Offset Table" or "Extended Offset Table" in the file. These tables specify the starting positions of each frame within the file's byte stream. Without an offset table being present, libraries such as highdicom have to parse through the pixel data to find markers that tell it where frame boundaries are, which involves pulling down significantly more data and is therefore very slow. This mostly eliminates the potential speed benefits of frame-level retrieval. Unfortunately there is no simple way to know whether a file has an offset table without downloading the pixel data and checking it. If you find that an image takes a long time to load initially, it is probably because highdicom is constucting the offset table itself because it wasn't included in the file. Most IDC images do include an offset table, but some of the older pathology slide images do not. [This page](https://github.com/ImagingDataCommons/idc-wsi-conversion?tab=readme-ov-file#overview) contains some notes about whether individual collections include offset tables. You can also check whether an image file (including pixel data) has an offset table using pydicom like this: ```python import pydicom dcm = pydicom.dcmread("...") # Any method to read from file/cloud storage if not dcm.file_meta.TransferSyntaxUID.is_encapsulated: print( "This image does not use an encapsulated (compressed) transfer " "syntax, so offset tables are not required." ) else: # Check metadata for the extended offset table print("Has Extended Offset Table:", "ExtendedOffsetTable" in dcm) # The start of the PixelData element will be a 4 byte item tag for the offset table, # which should always be present. The following 4 bytes gives the length of the offset # table. If it is non-zero, the offset table is present has_basic_offset_table = dcm.PixelData[4:8] != b'\x00\x00\x00\x00' print("Has Basic Offset Table:", has_basic_offset_table) ``` To do this from a remote Google Cloud Storage blob without needing to pull all the pixel data, you can do something like this: ```python import os from pydicom import dcmread from google.cloud import storage def check_offset_table(blob_key: str): """Print information on the offset table in an IDC blob.""" # Create a storage client and use it to access the IDC's public data package gcs_client = storage.Client.create_anonymous_client() # Blob object for the particular file you want to check blob = gcs_client.bucket("idc-open-data").blob(blob_key) # Open the blob object for remote reading with a ~500kB chunk size with blob.open(mode="rb", chunk_size=500_000) as reader: # Read the file with stop_before_pixels=True, this moves the cursor # position to the start of the pixel data attribute dcm = dcmread(reader, stop_before_pixels=True) if not dcm.file_meta.TransferSyntaxUID.is_encapsulated: print( "This image does not use an encapsulated (compressed) transfer " "syntax, so offset tables are not required." ) else: # The presence of the extended offset table in the loaded metadata can be # checked straightforwardly has_extended_offset_table = "ExtendedOffsetTable" in dcm print("Has Extended Offset Table:", has_extended_offset_table) # Read the next tag, should be the pixel data tag tag = reader.read(4) assert tag == b'\xe0\x7f\x10\x00', "Expected pixel data tag" # Skip over VR (2 bytes), reserved (2 bytes), and pixel data length (4 # bytes), giving 8 bytes total. Refer to # https://dicom.nema.org/medical/dicom/current/output/chtml/part05/sect_A.4.html#table_A.4-2 reader.seek(8, os.SEEK_CUR) # Read the item tag for the offset table item item_tag = reader.read(4) assert item_tag == b'\xfe\xff\x00\xe0', "Expected item tag" # Read the 32bit length of the pixel data's basic offset table length = reader.read(4) # If the length of the offset table is non-zero, the offset table exists has_basic_offset_table = (length != b'\x00\x00\x00\x00') print("Has Basic Offset Table:", has_basic_offset_table) # Example with no offset table (NLST-LSS collection) check_offset_table("4a30ffd2-8489-427b-9a83-03f4cf28534d/ad46e1e3-b37c-434b-a67a-5bacbcc608d9.dcm") # Example with basic offset table (CCDI-MCI collection) check_offset_table("763fe058-7d25-4ba7-9b29-fd3d6c41dc4b/210f0529-c767-4795-9acf-bad2f4877427.dcm") # Example with extended offset table (CMB-MML collection) check_offset_table("79f38b50-4df4-4358-9271-f28aeac573d7/23b9272a-34ef-49ca-833f-84329a18c1e4.dcm") ``` # Additional tools Since IDC data is available via standard interfaces, you can use any of the tool supporting those interfaces to access the data. This page provides pointers to some of such tools that you might find useful. If you are aware of any other tool that is not listed here, but is helpful for accessing IDC data, please let us know on the [IDC forum](https://discourse.canceridc.dev/), and we will be happy to add it here! ### S3 API * [**obstore**](https://developmentseed.org/obstore/latest/): open-source Python interface to [Amazon S3](https://aws.amazon.com/s3/), [Google Cloud Storage](https://cloud.google.com/storage), [Azure Storage](https://learn.microsoft.com/en-us/azure/storage/common/storage-introduction), & other S3-compliant APIs, powered by Rust # Data release notes {% hint style="info" %} Data hosted by IDC is ingested from several sources, including [The Cancer Imaging Archive (TCIA)](https://www.cancerimagingarchive.net/), [Genomics Data Commons (GDC)](https://gdc.cancer.gov/), [Clinical Proteomic Tumor Analysis Consortium (CPTAC)](https://gdc.cancer.gov/about-gdc/contributed-genomic-data-cancer-research/clinical-proteomic-tumor-analysis-consortium-cptac) and [Human Tumor Atlas Network (HTAN)](https://humantumoratlas.org/). Please refer to the license and terms of use, which are defined in the `license_url` and `source_doi` or `source_url` of the IDC BigQuery [`dicom_all` table](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=dicom_all\&page=table). You can filter the data by license type in the [IDC Portal](https://imaging.datacommons.cancer.gov/), or programmatically, as demonstrated in [this tutorial](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/part2_searching_basics.ipynb). {% endhint %} ## IDC releases summary view ##
## V24 - May 2026 ### New Collections | Collection | Program | Cancer Type | Modality | Patients | Series | Source | | --------------------------- | --------- | ------------------------------------ | --------------------------------- | -------- | ------ | ----------------- | | CDDP-EAGLE-1 | NIH | Lung Adenocarcinoma | SM | 49 | 49 | GDC | | CGCI-BLGSP | CGCI | Burkitt Lymphoma | SM | 388 | 1,933 | GDC | | CGCI-HTMCP-CC | CGCI | Cervical Squamous Cell Carcinoma | SM | 211 | 525 | GDC | | CGCI-HTMCP-DLBCL | CGCI | Diffuse Large B-Cell Lymphoma | SM | 43 | 496 | GDC | | CGCI-HTMCP-LC | CGCI | Non-Small Cell Carcinoma | SM | 27 | 84 | GDC | | HCMI-CMDC | HCMI | Various (pan-cancer) | SM | 382 | 810 | GDC | | PDXNet | NIH | Various (pan-cancer PDX) | SM | 919 | 919 | PDXNet Consortium | | UW-CIRP-Mouse-PET-CT-NSCLC | NIH | Lung Squamous Cell Carcinoma | PT, CT, RTSTRUCT | 14 | 75 | UW Co-CIRP | | HTAN-TNP-SARDANA | HTAN | Colon Mucinous Adenocarcinoma | SM, PR | 1 | 3 | HTAN/Synapse | | CPTAC-STAD | CPTAC | Stomach Adenocarcinoma | CT, US | 20 | 99 | TCIA | | CATCH | Community | Canine cutaneous tumors (7 subtypes) | SM | 282 | 350 | TCIA | | LDCT-and-Projection-data | Community | Lung Cancer | CT | 200 | 698 | TCIA | | PSMA-PET-CT-Lesions | Community | Prostate Cancer | CT, PT, SEG | 378 | 1,791 | TCIA | | Spinal-Multiple-Myeloma-SEG | Community | Multiple Myeloma | CT, SEG | 67 | 720 | TCIA | | EAY131 | NCTN | Pan-cancer (46 types) | CT, MR, PT, RTSTRUCT, SEG, NM, XA | 2,813 | 30,293 | TCIA | **CDDP-EAGLE-1** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=cddp_eagle_1) | [DOI 10.5281/zenodo.17372206](https://doi.org/10.5281/zenodo.17372206)) — 49 H\&E-stained frozen section whole slide images of primary lung adenocarcinoma from the EAGLE population-based case-control study (Lombardy, Italy). Converted from GDC Aperio SVS to DICOM SM. **CGCI-BLGSP** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=cgci_blgsp) | [DOI 10.5281/zenodo.17381396](https://doi.org/10.5281/zenodo.17381396)) — 1,933 whole slide images from 388 subjects in the Burkitt Lymphoma Genome Sequencing Project. Includes H\&E and immunohistochemistry stains (BCL2, BCL6, CD10, Ki-67, CD20, CD3, CD5, CD79a), EBER in situ hybridization, and Wright-Giemsa. Both FFPE and frozen sections. Converted from GDC Aperio SVS to DICOM SM. **CGCI-HTMCP-CC** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=cgci_htmcp_cc) | [DOI 10.5281/zenodo.17381404](https://doi.org/10.5281/zenodo.17381404)) — 525 whole slide images from 211 subjects in the HIV+ Tumor Molecular Characterization Project - Cervical Cancer, part of CGCI. Includes H\&E and p16 stains for most subjects, plus additional IHC markers (BER-EP4, MOC31, P40, P63, CEA, ER, PR, VIM, TP53, CD56, CHR, SYN) for subsets. Converted from GDC Aperio SVS to DICOM SM. **CGCI-HTMCP-DLBCL** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=cgci_htmcp_dlbcl) | [DOI 10.5281/zenodo.17381412](https://doi.org/10.5281/zenodo.17381412)) — 496 whole slide images from 43 subjects in the HIV+ Tumor Molecular Characterization Project - Diffuse Large B-Cell Lymphoma, part of CGCI. Includes H\&E and IHC/ISH stains (BCL2, BCL6, CD10, CD20, CD3, CD79a, EBER, Ki-67, MUM1, TP53) for most subjects, plus INT and CD138 for subsets. Converted from GDC Aperio SVS to DICOM SM. **CGCI-HTMCP-LC** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=cgci_htmcp_lc) | [DOI 10.5281/zenodo.17381428](https://doi.org/10.5281/zenodo.17381428)) — 84 whole slide images from 27 subjects in the HIV+ Tumor Molecular Characterization Project - Lung Cancer, part of CGCI. Includes H\&E, P40, and TTF1 stains for most subjects, plus CHR, SYN, and P16 for subsets. Converted from GDC Aperio SVS to DICOM SM. **HCMI-CMDC** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=hcmi_cmdc) | [DOI 10.5281/zenodo.17381441](https://doi.org/10.5281/zenodo.17381441)) — 810 H\&E-stained whole slide images from 382 subjects in the Human Cancer Models Initiative Cancer Model Development Center. Pan-cancer collection spanning 20+ cancer types (colorectal, pancreatic, esophageal, breast, brain, melanoma, lung, and others) with both FFPE and frozen tumor and normal sections. Converted from GDC Aperio SVS to DICOM SM. **PDXNet** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=pdxnet) | [DOI 10.5281/zenodo.16967600](https://doi.org/10.5281/zenodo.16967600)) — A pan-cancer repository of >1,000 patient-derived xenograft (PDX) and paired parental tumor H\&E whole slide images from the NCI PDXNet Consortium. Covers 29+ cancer types across 22 anatomic sites, with associated genomic, clinical, and pathologic annotation data. Contributed by BCM, Huntsman, MDACC, Wistar, WUSTL, and JAX. Converted from TIFF/SVS to DICOM SM. **UW-CIRP-Mouse-PET-CT-NSCLC** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=uw_cirp_mouse_pet_ct_nsclc) | [DOI 10.5281/zenodo.17257735](https://doi.org/10.5281/zenodo.17257735)) — 18F-FDG PET/CT imaging of 14 genetically-engineered mouse models of lung squamous cell carcinoma from the NCI Co-CIRP program. 25 imaging sessions with 75 series (25 PT, 25 CT, 25 RTSTRUCT segmentations) on Siemens Inveon PET/CT. 11 animals imaged pre- and post-therapy (anti-PD-L1 + CXCR2 antagonist), 3 baseline-only. **HTAN-TNP-SARDANA** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_tnp_sardana) | [DOI 10.5281/zenodo.18488943](https://doi.org/10.5281/zenodo.18488943)) — CyCIF and H\&E whole slide images of a single colorectal cancer specimen from the HTAN SARDANA Trans-Network Project. 265 DICOM instances across 3 series: 240 CyCIF SM, 6 H\&E SM, and 19 presentation state (PR) instances. \~20 CyCIF markers targeting immune, epithelial, stromal, and proliferation cell populations. **CPTAC-STAD** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=cptac_stad) | [DOI 10.7937/jw9a-8k71](https://doi.org/10.7937/jw9a-8k71)) — 99 radiology series (98 CT, 1 US) from 20 patients in the Clinical Proteomic Tumor Analysis Consortium Stomach Adenocarcinoma cohort. Images acquired as standard-of-care radiology (predominantly abdominal/abdominopelvic CT) prior to pathological diagnosis, made publicly available by TCIA. Sourced as-is in DICOM from TCIA. **CATCH** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=catch) | [DOI 10.5281/zenodo.18526942](https://doi.org/10.5281/zenodo.18526942)) — 350 H\&E-stained whole slide images of seven canine cutaneous tumor subtypes (melanoma, mast cell tumor, squamous cell carcinoma, peripheral nerve sheath tumor, trichoblastoma, histiocytoma, plasmacytoma) from 282 patients. Includes 12,424 polygon annotations for 13 histologic classes. Original Aperio SVS images from TCIA converted to DICOM SM. **LDCT-and-Projection-data** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=ldct_and_projection_data) | [DOI 10.7937/9npb-2637](https://doi.org/10.7937/9npb-2637)) — Low-dose CT images and projection data from 200 patients at Mayo Clinic, including non-contrast head CT, low-dose chest CT for pulmonary nodule screening, and contrast-enhanced abdominal CT. CT projection data provided in the open DICOM-CT-PD format. Sourced as-is in DICOM from TCIA. **PSMA-PET-CT-Lesions** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=psma_pet_ct_lesions) | [DOI 10.7937/r7ep-3x37](https://doi.org/10.7937/r7ep-3x37)) — 597 whole-body PSMA-PET/CT studies from 378 male patients with suspected or diagnosed prostate carcinoma, acquired at LMU University Hospital Munich (2014-2022). All PSMA-avid tumor lesions manually segmented on PET images in 3D. Includes CT, PET, and DICOM SEG segmentation masks. Used in the autoPET III and IV Grand Challenges. Sourced as-is in DICOM from TCIA. **Spinal-Multiple-Myeloma-SEG** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=spinal_multiple_myeloma_seg) | [DOI 10.7937/k4qv-hh78](https://doi.org/10.7937/k4qv-hh78)) — Dual-energy low-dose CT scans from 67 patients with multiple myeloma acquired at University Hospital Brno (2020-2023). Includes conventional CT, virtual monoenergetic images (40/80/120 keV), calcium-suppressed images, and DICOM SEG segmentation masks of vertebrae (with type classification) and myeloma lesions. 576 CT series plus 144 SEG series. Supporting clinical and demographic data provided as TSV. Sourced as-is in DICOM from TCIA. (Note: 67 unique PatientIDs with 72 studies — 5 patients have 2 scans each.) **EAY131** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=eay131) | [DOI 10.7937/c5ke-yx42](https://doi.org/10.7937/c5ke-yx42)) — Imaging and clinical data for 2,813 "unmatched" patients from the NCI MATCH Screening Trial (NCT02465060), performed by the ECOG-ACRIN Cancer Research Group. 30,293 series across CT (13,166), RTSTRUCT (14,395), SEG (1,404), MR (1,100), PT (222), NM (5), and XA (1) modalities. Covers 46 cancer types. Includes accompanying clinical/demographic data. Sourced as-is in DICOM from TCIA. ### New Analysis Results | Analysis Result | Program | Cancer Type | Modality | Subjects | Series | Source | | ------------------------ | ------- | --------------------- | ------------- | -------- | ------ | ------ | | EAY131-Tumor-Annotations | NCTN | Pan-cancer (46 types) | RTSTRUCT, SEG | 2,487 | 15,799 | TCIA | **EAY131-Tumor-Annotations** ([IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=eay131) | [DOI 10.7937/q9rn-m510](https://doi.org/10.7937/q9rn-m510)) — Tumor segmentations, seed points, and negative findings assessments for 2,487 subjects from the EAY131 collection. Annotations follow RECIST 1.1 (CT/MR/USG) and PERCIST (PET) criteria, created by an international team of radiologists and reviewed by US board-certified radiologists. Includes longitudinal lesion tracking via SNOMED-CT codes and Tracking UID tags. Also includes a CSV metadata report with lesion volumes. Created by Petr Jordan and Michael Rozenfeld. ### Revised Collections | Collection | Change Summary | Patients (delta) | Series (delta) | Source | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------- | -------------- | ---------------------------------------------------------------------- | | BoneMarrowWSI-PediatricLeukemia | Replaced all 1,033 ANN series with 1,027 new ANN; removed 1 SM series from 1 patient | -1 | -7 | BMDeep/Fraunhofer MEVIS | | ACRIN-NSCLC-FDG-PET | modality was changed from SC to OT in 4 series | 0 | 0 | | | Anti-PD-1\_Lung | modality was changed from SC to OT in 3 series | 0 | 0 | | | Phantom FDA | Phantom FDA DICOM instance data moved to gs\://idc-open-data and s3://idc-open-data buckets from gs\://idc-open-cr and s3://idc-open-data-cr buckets respectively. The Phantom FDA data in gs\://idc-open-cr and s3://idc-open-data-cr will be available through IDC versions v24 and v25. | 0 | 0 | | ### New metadata BQ Tables (non-clinical) | Table Name | Description | | ----------------- | ------------------------------- | | program\_metadata | Metadata about each IDC program | ### Revised BQ Tables (non-clinical) * analysis\_results\_metadata * added columns: * analysis\_result\_id * column id changes: * ID --> analysis\_result\_name * Title --> analysis\_result\_title * CancerTypes --> cancer\_types * TumorLocations --> tumor\_locations * Subjects -->s ubjects * Collections --> collections * Modalities --> modalities * Updated --> updated * Description --> description * deprecated columns * ID * Title * CancerTypes * TumorLocations * auxiliary\_metadata * column id changes * submitter\_case\_id --> Patient\_id * deprecated columns * submitter\_case\_id * Access * dicom\_metadata\_curated * added columns\ cancer\_types * original\_collection\_metadata * added columns * sources.source\_id * sources.source\_name * sources.source\_type * column id changes * CancerTypes --> cancer\_types * TumorLocations --> tumor\_locations * Subject --> subjects * Species --> species * Sources --> sources * Sources.ImageTypes --> sources.modalities * Sources.Citation --> source.citation * SupportingData --> supporting\_data * Program --> program\_id * Status --> status * Updated --> updated * Description --> description * Deprecated columns * Sources.Access * Sources.ImageTypes * CancerTypes * TumorLocations * SupportingData * Program ### New clinical metadata BQ tables * ldct\_and\_projection\_data\_abdomen\_v9 * ldct\_and\_projection\_data\_chest\_v9 ## V23 - Nov 2025 {% hint style="info" %} There are two rows, not one, for every instance in the *DICOM converted Slide Microscopy images for the TCGA-BRCA* collection in the `dicom_all` and `auxiliary_metadata` BigQuery tables in the `bigquery-public-data.idc_current` and `bigquery-public-data.idc_v23` datasets. The collection's DOI is `doi.org/10.5281/zenodo.12689962` . The rows in each pair are identical except that one row has the `Creative Commons Attribution 3.0 Unported License` (`CC BY 3.0`) and the other row the `Creative Commons Attribution 4.0 International License` (`CC BY 4.0`). The correct license for this collection is `CC BY 3.0`. The rows having the `CC BY 4.0` license can be ignored. This error will be corrected in the next IDC release. {% endhint %} {% hint style="success" %} Release counts Files: 46,870,903 (+175,736) Series: 994,073 (+28,666) Studies: 160,199 (+606) Cases: 79,889 (+355) Collections: 161 (no change) Analysis results collections: 23 (+6) Disk size: 95.33 TB (+2.22 TB) {% endhint %} #### New radiology collections #### New pathology collections #### New analysis results 1. [Lung-PET-CT-Dx-Annotations](https://doi.org/10.5281/zenodo.16989819)\ Collections analyzed: 1. Lung-PET-CT-Dx 2. [NLST-Sybil](https://doi.org/10.5281/zenodo.15643334)\ Collections analyzed: 1. NLST 3. [NLSTSeg](https://doi.org/10.5281/zenodo.17362624)\ Collections analyzed 1. NLST 4. [PROSTATEx-Targets](https://doi.org/10.5281/zenodo.15643312)\ Collections analyzed: 1. ProstateX 5. [TCGA-GBM360](https://doi.org/10.5281/zenodo.17470190)\ Collections analyzed: 1. TCGA-GBM 6. [TCGA-SBU-TIL-Maps](https://doi.org/10.5281/zenodo.16966285)\ Collections analyzed: 1. [TCGA-ACC](https://doi.org/10.5281/zenodo.12690035) 2. [TCGA-BLCA](https://doi.org/10.5281/zenodo.12690045) 3. [TCGA-BRCA](https://doi.org/10.5281/zenodo.12689962) 4. [TCGA-CESC](https://doi.org/10.5281/zenodo.12689956) 5. [TCGA-COAD](https://doi.org/10.5281/zenodo.12689970) 6. [TCGA-ESCA](https://doi.org/10.5281/zenodo.12690012) 7. [TCGA-HNSC](https://doi.org/10.5281/zenodo.12690033) 8. [TCGA-KIRC](https://doi.org/10.5281/zenodo.12689952) 9. [TCGA-LUAD](https://doi.org/10.5281/zenodo.12689915) 10. [TCGA-LUSC](https://doi.org/10.5281/zenodo.12690008) 11. [TCGA-MESO](https://doi.org/10.5281/zenodo.12689966) 12. [TCGA-OV](https://doi.org/10.5281/zenodo.12689954) 13. [TCGA-PAAD](https://doi.org/10.5281/zenodo.12689972) 14. [TCGA-PRAD](https://doi.org/10.5281/zenodo.12689935) 15. [TCGA-READ](https://doi.org/10.5281/zenodo.12689998) 16. [TCGA-SARC](https://doi.org/10.5281/zenodo.12689917) 17. [TCGA-SKCM](https://doi.org/10.5281/zenodo.12690039) 18. [TCGA-STAD](https://doi.org/10.5281/zenodo.12690037) 19. [TCGA-TGCT](https://doi.org/10.5281/zenodo.12689995) 20. [TCGA-THYM](https://doi.org/10.5281/zenodo.12689913) 21. [TCGA-UCEC](https://doi.org/10.5281/zenodo.12689968) 22. [TCGA-UVM](https://doi.org/10.5281/zenodo.12690041) #### Revised radiology collections 1. [CMB-BRCA](https://www.cancerimagingarchive.net/collection/cmb-brca/) 2. [CMB-CRC](https://www.cancerimagingarchive.net/collection/cmb-crc/) 3. [CMB-LCA](https://www.cancerimagingarchive.net/collection/cmb-lca/) 4. [CMB-MEL](https://www.cancerimagingarchive.net/collection/cmb-mel/) 5. [CMB-MML](https://www.cancerimagingarchive.net/collection/cmb-mml/) 6. [CMB-OV](https://www.cancerimagingarchive.net/collection/cmb-ov/) 7. [CMB-PCA](https://doi.org/10.5281/zenodo.13993798) 8. [VAREPOP-APOLLO](https://www.cancerimagingarchive.net/collection/varepop-apollo/) 9. [CPTAC-CCRCC](https://www.cancerimagingarchive.net/collection/cptac-ccrcc/) 10. [CPTAC-LUAD](https://www.cancerimagingarchive.net/collection/cptac-luad/) 11. [VAREPOP-APOLLO](https://www.cancerimagingarchive.net/collection/varepop-apollo/) #### Revised pathology collections 1. [BoneMarrowWSI-PediatricLeukemia](https://doi.org/10.5281/zenodo.14933087) 2. [CCDI-MCI](https://doi.org/10.5281/zenodo.11099086) 3. [TCGA-BRCA](https://doi.org/10.5281/zenodo.17486654) #### Revised analysis results #### New clinical metadata tables #### Revised clinical metadata tables 1. varepop\_apollo\_clinical ## V22 - Sept 2025 {% hint style="success" %} Release counts Files: 46,695,167 (+910,713) Series: 965,407 (+14,519) Studies: 159,593 (+10,016) Cases: 79,214 (+8,132) Collections: 161 (+11) Analysis results collections: 17 (no change) Disk size: 93.11 TB (+5.62 TB) {% endhint %} #### New radiology collections 1. [CBIS-DDSM](https://www.cancerimagingarchive.net/collection/cbis-ddsm/) 2. [CC-Radiomics-Phantom](https://www.cancerimagingarchive.net/collection/cc-radiomics-phantom/) 3. [CC-Radiomics-Phantom-2](https://www.cancerimagingarchive.net/collection/cc-radiomics-phantom-2/) 4. [CC-Radiomics-Phantom-3](https://www.cancerimagingarchive.net/collection/cc-radiomics-phantom-3/) 5. [CT4Harmonization-Multicentric](https://www.cancerimagingarchive.net/collection/ct4harmonization-multicentric/) 6. [QIBA-CT-Liver-Phantom](https://www.cancerimagingarchive.net/collection/qiba-ct-liver-phantom/) 7. [QIN-BREAST-02](https://www.cancerimagingarchive.net/collection/qin-breast-02/) 8. [QIN PET Phantom](https://www.cancerimagingarchive.net/collection/qin-pet-phantom/) 9. [RIDER PHANTOM MRI](https://www.cancerimagingarchive.net/collection/rider-phantom-mri/) 10. [RIDER PHANTOM PET-CT](https://www.cancerimagingarchive.net/collection/rider-phantom-pet-ct/) #### New pathology collections 1. [BoneMarrowWSI-PediatricLeukemia](https://doi.org/10.5281/zenodo.14933087) #### Revised radiology collections 1. [CMB-AML](https://www.cancerimagingarchive.net/collection/cmb-aml/) 2. [CMB-BRCA](https://www.cancerimagingarchive.net/collection/cmb-brca/) 3. [CMB-CRC](https://www.cancerimagingarchive.net/collection/cmb-crc/) 4. [CMB-GEC](https://www.cancerimagingarchive.net/collection/cmb-gec/) 5. [CMB-LCA](https://www.cancerimagingarchive.net/collection/cmb-lca/) 6. [CMB-MEL](https://www.cancerimagingarchive.net/collection/cmb-mel/) 7. [CMB-MML](https://www.cancerimagingarchive.net/collection/cmb-mml/) 8. [CMB-OV](https://www.cancerimagingarchive.net/collection/cmb-ov/) 9. [CMB-PCA](https://www.cancerimagingarchive.net/collection/cmb-pca/) 10. [CPTAC-CCRCC](https://www.cancerimagingarchive.net/collection/cptac-ccrcc/) 11. [CPTAC-LUAD](https://www.cancerimagingarchive.net/collection/cptac-luad/) 12. [VAREPOP-APOLLO](https://www.cancerimagingarchive.net/collection/varepop-apollo/) #### Revised pathology collections 1. [CCDI-MCI](https://doi.org/10.5281/zenodo.11099086) 2. [CMB-AML](https://doi.org/10.5281/zenodo.13993759) 3. [CMB-BRCA](https://doi.org/10.5281/zenodo.13993761) 4. [CMB-CRC](https://doi.org/10.5281/zenodo.13993769) 5. [CMB-GEC](https://doi.org/10.5281/zenodo.13993773) 6. [CMB-LCA](https://doi.org/10.5281/zenodo.13993776) 7. [CMB-MEL](https://doi.org/10.5281/zenodo.13993787) 8. [CMB-MML](https://doi.org/10.5281/zenodo.13993792) 9. [CMB-OV](https://doi.org/10.5281/zenodo.13993796) 10. [CMB-PCA](https://doi.org/10.5281/zenodo.13993798) 11. [NLST](https://testing-portal.canceridc.dev/collections/) #### New clinical metadata tables 1. bonemarrowwsi\_pediatricleukemia\_clinical 2. cbis\_ddsm\_calc\_case\_description\_test\_set 3. cbis\_ddsm\_calc\_case\_description\_train\_set 4. cbis\_ddsm\_mass\_case\_description\_test\_set 5. cbis\_ddsm\_mass\_case\_description\_train\_set 6. cc\_radiomics\_phantom\_3\_chest\_settings 7. cc\_radiomics\_phantom\_3\_head\_settings 8. cc\_radiomics\_phantom\_3\_manufacturer #### Revised clinical metadata tables 1. varepop\_apollo\_clinical ## V21 - May 2025 {% hint style="success" %} **Release counts** Files: 45,784,454 (+174,244) Series: 950,888 (+3,308) Studies: 149,577 (+2,070) Cases: 71,082 (+1,893) Collections: 150 (+1) Analysis results collections: 17 (no change) Disk size: 87.49 TB (+1.94 TB) {% endhint %} #### New radiology collections 1. [CMB-BRCA](https://www.cancerimagingarchive.net/collection/cmb-brca/) 2. [VAREPOP-APOLLO](https://www.cancerimagingarchive.net/collection/varepop-apollo/) #### Revised radiology collections 1. [CMB-LCA](https://www.cancerimagingarchive.net/collection/cmb-lca/) 2. [CMB-MEL](https://www.cancerimagingarchive.net/collection/cmb-mel/) 3. [CMB-MML](https://www.cancerimagingarchive.net/collection/cmb-mml/) 4. [CMB-PCA](https://www.cancerimagingarchive.net/collection/cmb-pca/) 5. [CPTAC-CCRCC](https://www.cancerimagingarchive.net/collection/cptac-ccrcc/) 6. [CPTAC-AML](https://www.cancerimagingarchive.net/collection/cptac-aml/) 7. [CPTAC-PDA](https://www.cancerimagingarchive.net/collection/cptac-pda/) #### Revised pathology collections 1. [CCDI-MCI](https://doi.org/10.5281/zenodo.11099086) #### Revised analysis results 1. [RMS-Mutation-Prediction-Expert-Annotations](https://doi.org/10.5281/zenodo.10462857) #### New clinical metadata tables 1. bamf\_aimi\_annotations\_brain\_mr\_qa\_results 2. bamf\_aimi\_annotations\_breast\_fdg\_pet\_ct\_qa\_results 3. bamf\_aimi\_annotations\_breast\_mr\_qa\_results 4. bamf\_aimi\_annotations\_kidney\_ct\_qa\_results 5. bamf\_aimi\_annotations\_liver2\_ct\_qa\_results 6. bamf\_aimi\_annotations\_liver\_ct\_qa\_results 7. bamf\_aimi\_annotations\_liver\_mr\_qa\_results 8. bamf\_aimi\_annotations\_lung2\_ct\_qa\_results 9. bamf\_aimi\_annotations\_lung\_ct\_qa\_results 10. bamf\_aimi\_annotations\_lung\_fdg\_pet\_ct\_qa\_results 11. bamf\_aimi\_annotations\_prostate\_mr\_qa\_results 12. cptac\_aml\_demographic\_classification 13. varepop\_apollo\_clinical #### Renamed clinical metadata tables 1. nlst\_canc 1. Previously nlst\_clinical #### Retired clinical metadata tables 1. acrin\_nsclc\_fdg\_pet\_bamf\_lung\_pet\_ct\_segmentation 1. Subsumed by bamf\_aimi\_annotations\_lung\_fdg\_pet\_ct\_qa\_results 2. anti\_pd\_1\_lung\_bamf\_lung\_ct\_segmentation 1. Subsumed by bamf\_aimi\_annotations\_lung\_ct\_qa\_results 3. anti\_pd\_1\_lung\_bamf\_lung\_fdg\_pet\_ct\_segmenation 1. Subsumed by bamf\_aimi\_annotations\_lung\_fdg\_pet\_ct\_qa\_results 4. lung\_pet\_ct\_dx\_bamf\_lung\_ct\_segmentation 1. Subsumed by bamf\_aimi\_annotations\_lung\_ct\_qa\_results 5. lung\_pet\_ct\_dx\_bamf\_lung\_fdg\_pet\_ct\_segmenation 1. Subsumed by bamf\_aimi\_annotations\_lung\_fdg\_pet\_ct\_qa\_results 6. nsclc\_radiogenomics\_bamf\_lung\_ct\_segmentation 1. Subsumed by bamf\_aimi\_annotations\_lung\_ct\_qa\_results 7. nsclc\_radiogenomics\_bamf\_lung\_fdg\_pet\_ct\_segmenation 1. Subsumed by bamf\_aimi\_annotations\_lung\_fdg\_pet\_ct\_qa\_results 8. prostatex\_bamf\_segmentations 1. Subsumed by bamf\_aimi\_annotations\_prostate\_mr\_qa\_results 9. qin\_breast\_bamf\_breast\_segmentation 1. Subsumed by bamf\_aimi\_annotations\_breast\_fdg\_pet\_ct\_qa\_results 10. rider\_lung\_pet\_ct\_bamf\_lung\_ct\_segmentation 1. Subsumed by bamf\_aimi\_annotations\_lung\_ct\_qa\_results 11. rider\_lung\_pet\_ct\_bamf\_lung\_fdg\_pet\_ct\_segmenation 1. Subsumed by bamf\_aimi\_annotations\_lung\_fdg\_pet\_ct\_qa\_results 12. tcga\_kirc\_bamf\_kidney\_segmentation 1. Subsumed by bamf\_aimi\_annotations\_kidney\_ct\_qa\_results 13. tcga\_lihc\_bamf\_liver\_ct\_segmentation 1. Subsumed by bamf\_aimi\_annotations\_liver\_ct\_qa\_results 14. tcga\_lihc\_bamf\_liver\_mr\_segmentation 1. Subsumed by amf\_aimi\_annotations\_liver\_mr\_qa\_results 15. tcga\_luad\_bamf\_lung\_ct\_segmentation 1. Subsumed by bamf\_aimi\_annotations\_lung\_ct\_qa\_results 16. tcga\_luad\_bamf\_lung\_mr\_segmentation 1. Subsumed by bamf\_aimi\_annotations\_lung\_fdg\_pet\_ct\_qa\_results 17. tcga\_lusc\_lung\_ct\_segmentation 1. Subsumed by bamf\_aimi\_annotations\_lung\_ct\_qa\_results 18. tcga\_lusc\_lung\_mr\_segmentation 1. Subsumed by bamf\_aimi\_annotations\_lung\_fdg\_pet\_ct\_qa\_results ## V20 - November 2024 #### New radiology collections 1. [Mediastinal-Lymph-Node-SEG](https://www.cancerimagingarchive.net/collection/mediastinal-lymph-node-seg/) 2. [Spine-Mets-CT-SEG](https://doi.org/10.7937/kh36-ds04) #### New pathology collections 1. [CMB-BRCA](https://doi.org/10.5281/zenodo.13993761) 2. [CMB-OV](https://doi.org/10.5281/zenodo.13993796) #### Revised radiology collections 1. [CMB-AML](https://www.cancerimagingarchive.net/collection/cmb-aml/) 2. [CMB-CRC](https://www.cancerimagingarchive.net/collection/cmb-crc/) 3. [CMB-GEC](https://www.cancerimagingarchive.net/collection/cmb-gec/) 4. [CMB-LCA](https://www.cancerimagingarchive.net/collection/cmb-lca/) 5. [CMB-MEL](https://www.cancerimagingarchive.net/collection/cmb-mel/) 6. [CMB-MML](https://www.cancerimagingarchive.net/collection/cmb-mml/) 7. [CMB-PCA](https://www.cancerimagingarchive.net/collection/cmb-ov/) #### Revised pathology collections 1. [CCDI-MCI](https://doi.org/10.5281/zenodo.11099086) 2. [CMB-AML](https://doi.org/10.5281/zenodo.13993759) 3. [CMB-CRC](https://doi.org/10.5281/zenodo.13993769) 4. [CMB-GEC](https://doi.org/10.5281/zenodo.13993773) 5. [CMB-LCA](https://doi.org/10.5281/zenodo.13993776) 6. [CMB-MEL](https://doi.org/10.5281/zenodo.13993787) 7. [CMB-MML](https://doi.org/10.5281/zenodo.13993792) 8. [CMB-PCA](https://doi.org/10.5281/zenodo.13993798) #### Revised analysis results 1. [BAMF-AIMI-Annotations
](https://doi.org/10.5281/zenodo.8345959)\ Collections analyzed: 1. [UPENN-GBM](https://doi.org/10.5281/zenodo.8345959) 2. [Pan-Cancer-Nuclei-Seg-DICOM
](https://doi.org/10.5281/zenodo.11099004)\ Collections analyzed: 1. [TCGA-BLCA](https://doi.org/10.5281/zenodo.12690045) 2. [TCGA-BRCA](https://doi.org/10.5281/zenodo.12689962) 3. [TCGA-CESC](https://doi.org/10.5281/zenodo.12689956) 4. [TCGA-COAD](https://doi.org/10.5281/zenodo.12689970) 5. [TCGA-GBM](https://doi.org/10.5281/zenodo.12690010) 6. [TCGA-LUAD](https://doi.org/10.5281/zenodo.12689915) 7. [TCGA-LUSC](https://doi.org/10.5281/zenodo.12690008) 8. [TCGA-PAAD](https://doi.org/10.5281/zenodo.12689972) 9. [TCGA-PRAD](https://doi.org/10.5281/zenodo.12689935) 10. [TCGA-READ](https://doi.org/10.5281/zenodo.12689998) 11. [TCGA-SKCM](https://doi.org/10.5281/zenodo.12690039) 12. [TCGA-STAD](https://doi.org/10.5281/zenodo.12690037) 13. [TCGA-UCEC](https://doi.org/10.5281/zenodo.12689968) 14. [TCGA-UVM](https://doi.org/10.5281/zenodo.12690041) \ The segmentation of an instance in each of the following series was excluded due to having a DICOM PixelData size greater than or equal to 2GB: 1. 1.2.826.0.1.3680043.10.511.3.10544506665348704312902213950958190 2. 1.2.826.0.1.3680043.10.511.3.11183783347037364699862133130586654 3. 1.2.826.0.1.3680043.10.511.3.11834745481756047014039855874680259 4. 1.2.826.0.1.3680043.10.511.3.11901667084519361717338400810055642 5. 1.2.826.0.1.3680043.10.511.3.12041600048156613329793822566495651 6. 1.2.826.0.1.3680043.10.511.3.12718116375608495830041119776887887 7. 1.2.826.0.1.3680043.10.511.3.13386724401829265460622415500801368 8. 1.2.826.0.1.3680043.10.511.3.14042734131864468280344737986870899 9. 1.2.826.0.1.3680043.10.511.3.17374765903080083648409690755539184 10. 1.2.826.0.1.3680043.10.511.3.17429002643681869326389465422353495 11. 1.2.826.0.1.3680043.10.511.3.20359930476040698387716730891020638 12. 1.2.826.0.1.3680043.10.511.3.28397033639127902823368316410884210 13. 1.2.826.0.1.3680043.10.511.3.28425539132321749931109935391487352 14. 1.2.826.0.1.3680043.10.511.3.34574227972763695321794092913087775 15. 1.2.826.0.1.3680043.10.511.3.36216094237641867532902805456135029 16. 1.2.826.0.1.3680043.10.511.3.39533936694797964318706337783276378 17. 1.2.826.0.1.3680043.10.511.3.39900930856460689132625586523683939 18. 1.2.826.0.1.3680043.10.511.3.41633795217567037218184715094985555 19. 1.2.826.0.1.3680043.10.511.3.42218106649761752724553401155203874 20. 1.2.826.0.1.3680043.10.511.3.49098870621170235412220976183110770 21. 1.2.826.0.1.3680043.10.511.3.50064322235999800062455171235601125 22. 1.2.826.0.1.3680043.10.511.3.50905421517530127976832505410705816 23. 1.2.826.0.1.3680043.10.511.3.62935684444056080516153739948364303 24. 1.2.826.0.1.3680043.10.511.3.73572792121235596011940904319511291 25. 1.2.826.0.1.3680043.10.511.3.74494366757564543824303304482444570 26. 1.2.826.0.1.3680043.10.511.3.79988146996803179892075404247166692 27. 1.2.826.0.1.3680043.10.511.3.80004293150506819482091023564947091 28. 1.2.826.0.1.3680043.10.511.3.82774274518897141254234567300292686 29. 1.2.826.0.1.3680043.10.511.3.84202416467561501610598853920808906 30. 1.2.826.0.1.3680043.10.511.3.86214492184712627544696209982376598 31. 1.2.826.0.1.3680043.10.511.3.90193069664920622990317347485104073 32. 1.2.826.0.1.3680043.10.511.3.95666157880521064637011880609274546 33. 1.2.826.0.1.3680043.10.511.3.96676982370873257329281821215166082 34. 1.2.826.0.1.3680043.10.511.3.98258035017480972315346136181769675 3. [RMS-Mutation-Prediction-Expert-Annotations](https://doi.org/10.5281/zenodo.10462857) **WARNING**: After the release of v20, it was discovered that a mistake had been made during data conversion that affected the newly-released segmentations accompanying the "RMS-Mutation-Prediction" collection. Segmentations released in v20 for this collection have the segment labels for alveolar rhabdomyosarcoma (ARMS) and embryonal rhabdomyosarcoma (ERMS) switched in the metadata relative to the correct labels. Thus segment 3 in the released files is labelled in the metadata (the SegmentSequence) as ARMS but should correctly be interpreted as ERMS, and conversely segment 4 in the released files is labelled as ERMS but should be correctly interpreted as ARMS. We apologize for the mistake and any confusion that it has caused, and will be releasing a corrected version of the files in the next release as soon as possible.[
](https://doi.org/10.5281/zenodo.10462857)\ Collections analyzed: 1. [RMS-Mutation-Prediction](https://doi.org/10.5281/zenodo.8225131) #### New Clinical Metadata Tables 1. [mediastinal\_lymph\_node\_seg\_clinical](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=mediastinal_lymph_node_seg) 2. [spine\_mets\_ct\_seg\_clinical](https://testing-portal.canceridc.dev/explore/filters/?collection_id=spine_mets_ct_seg) ## v19 - September 2024 **New pathology collections** 1. [CCDI-MCI](https://doi.org/10.5281/zenodo.11099087) 2. [CMB-AML](https://doi.org/10.5281/zenodo.11099111) 3. [CMB-CRC](https://doi.org/10.5281/zenodo.11099111) 4. [CMB-GEC](https://doi.org/10.5281/zenodo.11099111) 5. [CMB-LCA](https://doi.org/10.5281/zenodo.11099111) 6. [CMB-MEL](https://doi.org/10.5281/zenodo.11099111) 7. [CMB-MML](https://doi.org/10.5281/zenodo.11099111) 8. [CMB-PCA](https://doi.org/10.5281/zenodo.11099111) 9. [GTEx](https://doi.org/10.5281/zenodo.11099099) **New analysis results** 1. [Pancreas-CT-SEG](https://doi.org/10.5281/zenodo.12130275)\ Collections analyzed: 1. [Pancreas-CT](https://doi.org/10.7937/k9/tcia.2016.tnb1kqbu) 2. [Pan-Cancer-Nuclei-Seg-DICOM](https://doi.org/10.5281/zenodo.11099004)\ Collections analyzed: 1. [TCGA-BLCA](https://doi.org/10.5281/zenodo.12690045) 2. [TCGA-BRCA](https://doi.org/10.5281/zenodo.12689962) 3. [TCGA-CESC](https://doi.org/10.5281/zenodo.12689956) 4. [TCGA-COAD](https://doi.org/10.5281/zenodo.12689970) 5. [TCGA-GBM](https://doi.org/10.5281/zenodo.12690010) 6. [TCGA-LUAD](https://doi.org/10.5281/zenodo.12689915) 7. [TCGA-LUSC](https://doi.org/10.5281/zenodo.12690008) 8. [TCGA-PAAD](https://doi.org/10.5281/zenodo.12689972) 9. [TCGA-PRAD](https://doi.org/10.5281/zenodo.12689935) 10. [TCGA-READ](https://doi.org/10.5281/zenodo.12689998) 11. [TCGA-SKCM](https://doi.org/10.5281/zenodo.12690039) 12. [TCGA-STAD](https://doi.org/10.5281/zenodo.12690037) 13. [TCGA-UCEC](https://doi.org/10.5281/zenodo.12689968) 14. [TCGA-UVM](https://doi.org/10.5281/zenodo.12690041) **Revised radiology collections** 1. [Advanced-MRI-Breast-Lesions](https://www.cancerimagingarchive.net/collection/advanced-mri-breast-lesions/) 2. [CMB-AML](https://www.cancerimagingarchive.net/collection/cmb-aml/) 3. [CMB-CRC](https://www.cancerimagingarchive.net/collection/cmb-crc/) 4. [CMB-GEC](https://www.cancerimagingarchive.net/collection/cmb-gec/) 5. [CMB-LCA](https://www.cancerimagingarchive.net/collection/cmb-lca/) 6. [CMB-MEL](https://www.cancerimagingarchive.net/collection/cmb-mel/) 7. [CMB-MML](https://www.cancerimagingarchive.net/collection/cmb-mml/) 8. [CMB-PCA](https://www.cancerimagingarchive.net/collection/cmb-pca/) 9. [CPTAC-CCRCC](https://www.cancerimagingarchive.net/collection/cptac-ccrcc/) 10. [CPTAC-LSCC](https://www.cancerimagingarchive.net/collection/cptac-lscc/) 11. [CPTAC-UCEC](https://www.cancerimagingarchive.net/collection/cptac-ucec/) 12. [NLM-Visible-Human-Project](https://doi.org/10.5281/zenodo.12690049) 13. [RIDER Lung CT](https://www.cancerimagingarchive.net/collection/rider-lung-ct/) Cancer Moonshot Biobank (CMB) radiology images were updated to fix incorrect values assigned to `PatientID` (see details on the collection pages linked above). The updated images have different DICOM Study/Series/SOPInstanceUIDs. **Revised analysis results** 1. [BAMF-AIMI-Annotations](https://doi.org/10.5281/zenodo.8345959)\ Collections analyzed: 1. [ACRIN-NSCLC-FDG-PET](https://doi.org/10.7937/tcia.2019.30ilqfcl) 2. [Anti-PD-1\_Lung](https://doi.org/10.7937/tcia.2019.zjjwb9ip) 3. [Colorectal-Liver-Metastases](https://doi.org/10.7937/qxk2-qg03) 4. [CPTAC-CCRCC](https://doi.org/10.7937/k9/tcia.2018.oblamn27) 5. [Duke-Breast-Cancer-MRI](https://doi.org/10.7937/tcia.e3sv-re93) 6. [HCC-TACE-Seg](https://doi.org/10.7937/tcia.5fna-0924) 7. [Lung-PET-CT-Dx](https://doi.org/10.7937/tcia.2020.nnc2-0461) 8. [NLST](https://doi.org/10.7937/tcia.hmq8-j677) 9. [NSCLC Radiogenomics](https://doi.org/10.7937/k9/tcia.2017.7hs46erv) 10. [Prostate-MRI-US-Biopsy](https://doi.org/10.7937/tcia.2020.a61ioc1a) 11. [PROSTATEx](https://doi.org/10.7937/k9tcia.2017.murs5cl) 12. [QIN-BREAST](https://doi.org/10.7937/k9/tcia.2016.21juebh0) 13. [QIN LUNG CT](https://doi.org/10.7937/k9/tcia.2015.npgzyzbz) 14. [RIDER Lung PET-CT](https://doi.org/10.7937/k9/tcia.2015.ofip7tvm) 15. [SPIE-AAPM Lung CT Challenge](https://doi.org/10.7937/k9/tcia.2015.uzlsu3fl) 16. [TCGA-KICH](https://doi.org/10.7937/k9/tcia.2016.yu3rbczn) 17. [TCGA-KIRC](https://doi.org/10.7937/k9/tcia.2016.v6pbvtdr) 18. [TCGA-KIRP](https://doi.org/10.7937/k9/tcia.2016.acwogbef) 19. [TCGA-LIHC](https://doi.org/10.7937/k9/tcia.2016.immqw8uq) 20. [TCGA-LUAD](https://doi.org/10.7937/k9/tcia.2016.jgnihep5) 21. [TCGA-LUSC](https://doi.org/10.7937/k9/tcia.2016.tygkkfmq) 22. [UPENN-GBM](https://doi.org/10.7937/tcia.709x-dn49) **New clinical metadata tables** 1. [acrin\_contralateral\_breast\_mr\_A0](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 2. [acrin\_contralateral\_breast\_mr\_AB](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 3. [acrin\_contralateral\_breast\_mr\_F1](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 4. [acrin\_contralateral\_breast\_mr\_I1](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 5. [acrin\_contralateral\_breast\_mr\_IA](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 6. [acrin\_contralateral\_breast\_mr\_IM](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 7. [acrin\_contralateral\_breast\_mr\_IS](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 8. [acrin\_contralateral\_breast\_mr\_KS](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 9. [acrin\_contralateral\_breast\_mr\_MS](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 10. [acrin\_contralateral\_breast\_mr\_M4](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 11. [acrin\_contralateral\_breast\_mr\_P8](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 12. [acrin\_contralateral\_breast\_mr\_PA](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 13. [acrin\_contralateral\_breast\_mr\_PD](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 14. [acrin\_contralateral\_breast\_mr\_PE](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 15. [acrin\_contralateral\_breast\_mr\_PR](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 16. [acrin\_contralateral\_breast\_mr\_QA](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_contralateral_breast_mr) 17. [advanced\_mri\_breast\_lesions\_clinical](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=advanced_mri_breast_lesions) 18. [upenn\_gbm](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=upenn_gbm) ## v18 - April 2024 **New radiology collections** 1. [Advanced-MRI-Breast-Lesions](https://doi.org/10.7937/C7X1-YN57) **New analysis results** 1. [RMS-Mutation-Prediction-Expert-Annotations](https://doi.org/10.5281/zenodo.10462857)\*\ Collections analyzed: 1. [RMS-Mutation-Prediction](https://doi.org/10.5281/zenodo.8225131) 2. [TotalSegmentator-CT-Segmentations](https://doi.org/10.5281/zenodo.8347011)\*\*\ Collections analyzed: 1. [NLST](https://doi.org/10.7937/TCIA.HMQ8-J677) **Revised radiology collections** (starred collections are revised due to new or revised analysis results) 1. [Breast-Cancer-Screening-DBT](https://doi.org/10.7937/E4WT-CD02) (revisions only to clinical data) 2. [NLST](https://doi.org/10.7937/TCIA.HMQ8-J677)\*\* **Revised pathology collections** (starred collections are revised due to new or revised analysis results) 1. [CPTAC-BRCA](https://doi.org/10.7937/TCIA.CAEM-YS80) (fix PatientAges > 090Y) 2. [CPTAC-COAD](https://doi.org/10.7937/TCIA.YZWQ-ZZ63) (fix PatientAges > 090Y) 3. [RMS-Mutation-Prediction](https://doi.org/10.5281/zenodo.8225131)\* 1. Also added missing instance\ SOPInstanceUID: 1.3.6.1.4.1.5962.99.1.3459553143.523311062.1687086765943.9.0 2. Removed corrupted instances 1. SOPInstanceUID: 1.3.6.1.4.1.5962.99.1.2164023716.1899467316.1685791236516.37.0 2. SOPInstanceUID: 1.3.6.1.4.1.5962.99.1.2411736851.773458418.1686038949651.37.0 3. SOPInstanceUID: 1.3.6.1.4.1.5962.99.1.2411736851.773458418.16860389 4. [TCGA-BLCA](https://doi.org/10.7937/K9/TCIA.2016.8LNG8XDR) (All TCGA revisions are to correct multiple manufacturer values within same series) 5. [TCGA-BRCA](https://doi.org/10.7937/K9/TCIA.2016.AB2NAZRP) 6. [TCGA-CHOL](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/cholangiocarcinoma) 7. [TCGA-COAD](https://doi.org/10.7937/K9/TCIA.2016.HJJHBOXZ) 8. TCGA-DLBC (No description page) 9. [TCGA-ESCA](https://doi.org/10.7937/K9/TCIA.2016.VPTNRGFY) 10. [TCGA-HNSC](https://doi.org/10.7937/K9/TCIA.2016.LXKQ47MS) 11. [TCGA-KIRC](https://doi.org/10.7937/K9/TCIA.2016.V6PBVTDR) 12. [TCGA-KIRP](https://doi.org/10.7937/K9/TCIA.2016.ACWOGBEF) 13. [TCGA-LIHC](https://doi.org/10.7937/K9/TCIA.2016.IMMQW8UQ) 14. [TCGA-LUAD](https://doi.org/10.7937/K9/TCIA.2016.JGNIHEP5) 15. [TCGA-LUSC](https://doi.org/10.7937/K9/TCIA.2016.TYGKKFMQ) 16. [TCGA-PAAD](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/pancreatic) 17. [TCGA-PRAD](https://doi.org/10.7937/K9/TCIA.2016.YXOGLM4Y) 18. [TCGA-READ](https://doi.org/10.7937/K9/TCIA.2016.F7PPNPNU) 19. [TCGA-SARC](https://doi.org/10.7937/K9/TCIA.2016.CX6YLSUX) 20. [TCGA-SKCM](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/melanoma-skin) 21. [TCGA-STAD](https://doi.org/10.7937/K9/TCIA.2016.GDHL9KIM) 22. [TCGA-TGCT](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/testicular-germ-cell) 23. [TCGA-THCA](https://doi.org/10.7937/K9/TCIA.2016.9ZFRVF1B) 24. [TCGA-THYM](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/thymoma) 25. [TCGA-UCEC](https://doi.org/10.7937/K9/TCIA.2016.GKJ0ZWAC) 26. [TCGA-UCS](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/uterine-carcinosarcoma) **New clinical metadata tables** 1. [acrin\_nsclc\_fdg\_pet\_bamf\_lung\_pet\_ct\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=acrin_nsclc_fdg_pet) 2. [anti\_pd\_1\_lung\_bamf\_lung\_ct\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=anti_pd_1_lung) 3. [anti\_pd\_1\_lung\_bamf\_lung\_fdg\_pet\_ct\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=anti_pd_1_lung) 4. [lung\_pet\_ct\_dx\_bamf\_lung\_ct\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=lung_pet_ct_dx) 5. [lung\_pet\_ct\_dx\_bamf\_lung\_fdg\_pet\_ct\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=lung_pet_ct_dx) 6. [nsclc\_radiogenomics\_bamf\_lung\_ct\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=nsclc_radiogenomics) 7. [nsclc\_radiogenomics\_bamf\_lung\_fdg\_pet\_ct\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=nsclc_radiogenomics) 8. [prostatex\_bamf\_segmentations](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=prostatex) 9. [qin\_breast\_bamf\_breast\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=qin_breast) 10. [rider\_lung\_pet\_ct\_bamf\_lung\_ct\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=rider_lung_pet_ct) 11. [rider\_lung\_pet\_ct\_bamf\_lung\_fdg\_pet\_ct\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=rider_lung_pet_ct) 12. [tcga\_kirc\_bamf\_kidney\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=tcga_kirc) 13. [tcga\_lihc\_bamf\_liver\_ct\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=tcga_lihc) 14. [tcga\_lihc\_bamf\_liver\_mr\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=tcga_lihc) 15. [tcga\_luad\_bamf\_lung\_ct\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=tcga_luad) 16. [tcga\_luad\_bamf\_lung\_mr\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=tcga_luad) 17. [tcga\_lusc\_bamf\_lung\_ct\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=tcga_lusc) 18. [tcga\_lusc\_bamf\_lung\_mr\_segmentation](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=tcga_lusc) **Notes** The deprecated columns `tcia_api_collection_id` and `idc_webapp_collection_id` have been removed from the `auxiliary_metadata` table in the `idc_v18` BQ dataset. These columns were duplicates of columns `collection_name` and `collection_id` respectively. ## v17 - December 2023 New radiology collections 1. [CMB-AML](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CMB\&collection_id=cmb_aml) 2. [CT-Phantom4Radiomics](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=ct_phantom4radiomics) 3. [EA1141](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=NCI_Trials\&collection_id=ea1141) 4. [ReMIND](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=remind) 5. [Vestibular-Schwannoma-MC-RC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=vestibular_schwannoma_mc_rc) New analysis results 1. [BAMF-AIMI-Annotations](https://portal.imaging.datacommons.cancer.gov/explore/filters/?analysis_results_id=BAMF-AIMI-Annotations) Collections analyzed: 1. [ACRIN-NSCLC-FDG-PET](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=NCI_Trials\&collection_id=acrin_nsclc_fdg_pet) 2. [Anti-PD-1-Lung](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=anti_pd_1_lung) 3. [LUNG-PET-CT-Dx](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=lung_pet_ct_dx) 4. [NSCLC Radiogenomics](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=nsclc_radiogenomics) 5. [ProstateX](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=prostatex) 6. [QIN-Breast](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=QIN\&collection_id=qin_breast) 7. [RIDER Lung PET-CT](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=RIDER\&collection_id=rider_lung_pet_ct) 8. [TCGA-KIRC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=TCGA\&collection_id=tcga_kirc) 9. [TCGA-LIHC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=TCGA\&collection_id=tcga_lihc) 10. [TCGA-LUAD](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=TCGA\&collection_id=tcga_luad) 11. [TCGA-LUSC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=TCGA\&collection_id=tcga_lusc) 2. [Prostate-MRI-US-Biopsy-DICOM-Annotations](https://portal.imaging.datacommons.cancer.gov/explore/filters/?analysis_results_id=Prostate-MRI-US-Biopsy-DICOM-Annotations)\ Collections analyzed: 1. [Prostate-MRI-US-Biopsy](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=prostate_mri_us_biopsy) Revised radiology collections 1. [Prostate-MRI-US-Biopsy](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=prostate_mri_us_biopsy) 2. [CMB-CRC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CMB\&collection_id=cmb_crc) 3. [CMB-GEC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CMB\&collection_id=cmb_cgec) 4. [CMB-LCA](https://doi.org/10.7937/3CX3-S132) 5. [CMB-MEL](https://doi.org/10.7937/GWSP-WH72) 6. [CMB-MML](https://doi.org/10.7937/SZKB-SW39) 7. [CMB-PCA](https://doi.org/10.7937/25T7-6Y12) 8. [CPTAC-CCRCC](https://doi.org/10.7937/K9/TCIA.2018.OBLAMN27) 9. [CPTAC-PDA](https://doi.org/10.7937/K9/TCIA.2018.SC20FO18) New clinical metadata tables 1. [ea1141\_demographics](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=ea1141) 2. [ea1141\_mri](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=ea1141) 3. [ea1141\_risk\_model](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=ea1141) 4. [ea1141\_screening](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=ea1141) 5. [ea1141\_status\_12mo](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=ea1141) 6. [ea1141\_status\_6mo](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=ea1141) 7. [ea1141\_tomosynthesis](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=ea1141) 8. [htan\_ohsu\_demographics](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_ohsu) 9. [htan\_vanderbilt\_demographics](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_vanderbilt) 10. [htan\_vanderbilt\_diagnosis](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_vanderbilt) 11. [htan\_vanderbilt\_exposure](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_vanderbilt) 12. [htan\_vanderbilt\_familyhistory](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_vanderbilt) 13. [htan\_vanderbilt\_followup](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_vanderbilt) 14. [htan\_vanderbilt\_moleculartest](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_vanderbilt) 15. [htan\_vanderbilt\_therapy](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_vanderbilt) 16. [remind\_clinical](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=remind) ## v16 - September 2023 New radiology collections 1. [PDMR-Texture-Analysis](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=PDMR\&collection_id=pdmr_texture_analysis) New pathology collections 1. [RMS-Mutation-Prediction](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CCDI\&collection_id=rms_mutation_prediction) Revised radiology collections 1. [Breast-MRI-NACT-Pilot](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=breast_mri_nact_pilot) (TCIA description: (Repair of DICOM tag(0008,0005) to value "ISO\_IR 100" in 79 series) 2. [CPTAC-CRCC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CPTAC\&collection_id=cptac_crcc) (Revised because results from CPTAC-CRCC-Tumor-Annotations were added) 3. [CPTAC-UCEC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CPTAC\&collection_id=cptac_ucec) (Revised because results from CPTAC-UCEC-Tumor-Annotations were added) 4. [CPTAC-PDA](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CPTAC\&collection_id=cptac_pda) (Revised because results from CPTAC-PDA-Tumor-Annotations were added) New analysis results 1. [CPTAC-CRCC-Tumor-Annotations](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=157288300) 2. [CPTAC-UCEC-Tumor-Annotations](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=157288358) 3. [CPTAC-PDA-Tumor-Annotations](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=157288334) New clinical metadata tables 1. [htan\_hms\_demographics](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_hms) 2. [htan\_hms\_diagnosis](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_hms) 3. [htan\_hms\_exposure](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_hms) 4. [htan\_hms\_familyhistory](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_hms) 5. [htan\_hms\_followup](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_hms) 6. [htan\_hms\_moleculartheraphy](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_hms) 7. [htan\_ohsu\_demographics](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_ohsu) 8. [htan\_ohsu\_diagnosis](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_ohsu) 9. [htan\_ohsu\_exposure](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_ohsu) 10. [htan\_ohsu\_familyhistory](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_ohsu) 11. [htan\_ohsu\_followup](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_ohsu) 12. [htan\_ohsu\_moleculartheraphy](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_ohsu) 13. [htan\_wustl\_demographics](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_wustl) 14. [htan\_wustl\_diagnosis](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_wustl) 15. [htan\_wustl\_exposure](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_wustl) 16. [htan\_wustl\_familyhistory](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_wustl) 17. [htan\_wustl\_followup](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_wustl) 18. [htan\_wustl\_moleculartheraphy](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=htan_wustl) 19. [rms\_mutation\_prediction\_demographics](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=rms_mutation_prediction) 20. [rms\_mutation\_prediction\_diagnosis](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=rms_mutation_prediction) 21. [rms\_mutation\_prediction\_sample](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=rms_mutation_prediction) ## v15 - July 2023 New radiology collections 1. [Adrenal-ACC-Ki67-Seg](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=adrenal_acc_ki67_seg) 2. [CC-Tumor-Heterogeneity](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=cc_tumor_heterogeneity) 3. [Colorectal-Liver-Metastases](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=cc_tumor_heterogeneity) 4. [NLM-Visible-Human-Project](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=nlm_visible_human_project) 5. [Prostate-Anatomical-Edge-Cases](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=prostate_anatomical_edge_cases) 6. [RIDER Pilot](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=RIDER\&collection_id=rider_pilot) New pathology collections 1. [HTAN-VANDERBILT](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=HTAN\&collection_id=htan_vanderbilt) 2. [ICDC-Glioma](https://portal.imaging.datacommons.cancer.gov/explore/filters/?Modality_op=OR\&Modality=SM\&collection_id=icdc_glioma) (ICDC-Glioma radiology added in a previous version) Revised radiology collections 1. [CPTAC-CCRCC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CPTAC\&collection_id=cptac_ccrcc) (TCIA description: “Radiology modality data cleanup to remove extraneous scans.”) 2. [CPTAC-CM](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CPTAC\&collection_id=cptac_cm) (“TCIA description: Radiology modality data cleanup to remove extraneous scans.”) 3. [CPTAC-LSCC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CPTAC\&collection_id=cptac_lscc) (TCIA description: “Radiology modality data cleanup to remove extraneous scans.”) 4. [CPTAC-LUAD](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CPTAC\&collection_id=cptac_luad) (TCIA description: “Radiology modality data cleanup to remove extraneous scans.”) 5. [CPTAC-PDA](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CPTAC\&collection_id=cptac_pda) (TCIA description: TCIA description: “Radiology modality data cleanup to remove extraneous scans.”) 6. [CPTAC-SAR](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CPTAC\&collection_id=cptac_sar) (TCIA description: “Radiology modality data cleanup to remove extraneous scans.”) 7. [CPTAC-UCEC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CPTAC\&collection_id=cptac_ucec) (TCIA description: “Radiology modality data cleanup to remove extraneous scans.”) 8. [CT Lymph Nodes](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=ct_lymph_nodes) (TCIA description: “Added DICOM version of MED\_ABD\_LYMPH\_MASKS.zip segmentations that were previously available”) 9. [RIDER Lung CT](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=RIDER\&collection_id=rider_lung_ct) (Revised because QIBA-VolCT-1B analysis results were added) 10. [NLST](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=RIDER\&collection_id=nlst) (Revised because analysis results from nnU-Net-BPR-Annotations were revised) 11. [NSCLC-Radiomics](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=nsclc_radiomics) (Revised because analysis results from nnU-Net-BPR-Annotations were revised) Revised pathology collections 1. [CPTAC-GBM](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=cptac_gbm) (11 pathology-only patients removed at request of data owner) 2. [CPTAC-SAR](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=CPTAC\&collection_id=cptac_sar) (1 pathology-only patient removed at request of data owner) New analysis results 1. [QIBA-VolCT-1B](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=qiba_ct_1c) (Analysis of NLST and NSCLC-Radiomics) Revised analysis results 1. [nnU-Net-BPR-Annotations](https://portal.imaging.datacommons.cancer.gov/explore/filters/?analysis_results_id=nnU-Net-BPR-annotations) (Annotations of NLST and NSCLC-Radiomics radiology) New clinical metadata tables 1. [adrenal\_acc\_ki67\_seg\_clinical](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=adrenal_acc_ki67_seg) 2. [cc\_tumor\_heterogeneity\_clinical](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=cc_tumor_heterogeneity) 3. [colorectal\_liver\_metastases\_clinical](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=colorectal_liver_metastases) 4. [duke\_breast\_cancer\_mri\_clinical](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=duke_breast_cancer_mri) 5. [nlst\_clinical](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=NCI_Trials\&collection_id=nlst) 6. [nlst\_ctab](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=NCI_Trials\&collection_id=nlst) 7. [nlst\_ctabc](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=NCI_Trials\&collection_id=nlst) 8. [nlst\_prsn](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=NCI_Trials\&collection_id=nlst) 9. [nlst\_screen](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=NCI_Trials\&collection_id=nlst) ## v14 - May 2023 This release does not introduce any new data, but changes the bucket organization and introduces replication of IDC files in Amazon AWS storage buckets, as described in [this section](/data/organization-of-data/files-and-metadata). ## v13 - Mar 2023 New analysis results collection: 1. [nnU-Net-BPR-annotations](https://portal.imaging.datacommons.cancer.gov/explore/filters/?analysis_results_id=nnU-Net-BPR-annotations) New clinical data collections: 1. [PROSTATEx](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=prostatex) ## v12 - Nov 2022 New collections: 1. [CT-vs-PET-Ventilation-Imaging](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=ct_vs_pet_ventilation_imaging) 2. [CTpred-Sunitinib-panNET](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=ctpred_sunitinib_pannet) Updated collections: 1. [CMB-CRC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?access=Public\&collection_id=CMB\&collection_id=cmb_crc) 2. [CMB-LCA](https://portal.imaging.datacommons.cancer.gov/explore/filters/?access=Public\&collection_id=CMB\&collection_id=cmb_lca) 3. [CMB-MEL](https://portal.imaging.datacommons.cancer.gov/explore/filters/?access=Public\&collection_id=CMB\&collection_id=cmb_mel) 4. [CMB-PCA](https://portal.imaging.datacommons.cancer.gov/explore/filters/?access=Public\&collection_id=CMB\&collection_id=cmb_pca) 5. [Pancreatic-CT-CBCT-SEG](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=pancreatic_ct_cbct_seg) Other: Metadata corresponding to "limited" access collections are removed. New clinical data collections: 1. [CTpred-Sunitinib-panNET](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=ctpred_sunitinib_pannet) Other clinical data updates: Limited access collections are removed. Clinical metadata for the COVID-19-NY-SUB and ACRIN 6698/I-SPY2 Breast DWI collections now includes information ingested from data dictionaries associated with these collections. In v11 the string value 'NA' was being changed to null during the ETL process for some columns/collections. This is now fixed in v12 and the value 'NA' is preserved. ## v11 - Sept 2022 This release introduces clinical data ingested for a subset of collections, and now available via a dedicated BigQuery dataset. New collections: 1. [CMB-CRC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?access=Public\&collection_id=CMB\&collection_id=cmb_crc) 2. [CMB-GEC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?access=Public\&collection_id=CMB\&collection_id=cmb_gec) 3. [CMB-LCA](https://portal.imaging.datacommons.cancer.gov/explore/filters/?access=Public\&collection_id=CMB\&collection_id=cmb_lca) 4. [CMB-MEL](https://portal.imaging.datacommons.cancer.gov/explore/filters/?access=Public\&collection_id=CMB\&collection_id=cmb_mel) 5. [CMB-MML](https://portal.imaging.datacommons.cancer.gov/explore/filters/?access=Public\&collection_id=CMB\&collection_id=cmb_mml) 6. [CMB-PCA](https://portal.imaging.datacommons.cancer.gov/explore/filters/?access=Public\&collection_id=CMB\&collection_id=cmb_pca) 7. [GBM-DSC-MRI-DRO](https://portal.imaging.datacommons.cancer.gov/explore/filters/?access=Public\&collection_id=QIN\&collection_id=gbm_dsc_mri_dro) 8. [HCC-TACE-Seg](https://portal.imaging.datacommons.cancer.gov/explore/filters/?access=Public\&collection_id=Community\&collection_id=hcc_tace_seg) 9. [PDMR-521955-158-R4](https://portal.imaging.datacommons.cancer.gov/explore/filters/?access=Public\&collection_id=PDMR\&collection_id=pdmr_521955_158_r4) ## v10 - Aug 2022 In this release we introduce a new HTAN program including currently three collections release by the [Human Tumor Atlas Network](https://humantumoratlas.org/). New collections: 1. [ACRIN-6698](https://dx.doi.org/10.7937/tcia.kk02-6d95) 2. [HTAN-HMS](https://humantumoratlas.org/hta7/) 3. [HTAN-OHSU](https://humantumoratlas.org/hta9/) 4. [HTAN-WUSTL](https://humantumoratlas.org/hta12/) 5. [ISPY2](https://dx.doi.org/10.7937/TCIA.D8Z0-9T85) 6. [UPENN-GBM](https://dx.doi.org/10.7937/TCIA.709X-DN49) Updated collections: CPTAC, TCGA and NLST collections have been reconverted due to a technical issue identified with a subset of images included in v9. 1. [CPTAC-AML](https://dx.doi.org/10.7937/tcia.2019.b6foe619) 2. [CPTAC-BRCA](https://dx.doi.org/10.7937/TCIA.CAEM-YS80) \* 3. [CPTAC-CCRCC](https://dx.doi.org/10.7937/K9/TCIA.2018.OBLAMN27) 4. [CPTAC-CM](https://dx.doi.org/10.7937/K9/TCIA.2018.ODU24GZE) 5. [CPTAC-COAD](https://dx.doi.org/10.7937/TCIA.YZWQ-ZZ63) 6. [CPTAC-GBM](https://dx.doi.org/10.7937/K9/TCIA.2018.3RJE41Q1) 7. [CPTAC-HNSCC](https://dx.doi.org/10.7937/K9/TCIA.2018.UW45NH81) 8. [CPTAC-LSCC](https://dx.doi.org/10.7937/K9/TCIA.2018.6EMUB5L2) 9. [CPTAC-LUAD](https://dx.doi.org/10.7937/K9/TCIA.2018.PAT12TBS) 10. [CPTAC-OV](https://dx.doi.org/10.7937/TCIA.ZS4A-JD58) 11. [CPTAC-PDA](https://dx.doi.org/10.7937/K9/TCIA.2018.SC20FO18) 12. [CPTAC-SAR](https://dx.doi.org/10.7937/TCIA.2019.9bt23r95) 13. [CPTAC-UCEC](https://dx.doi.org/10.7937/K9/TCIA.2018.3R3JUISW) 14. [Duke-Breast-Cancer-MRI](https://dx.doi.org/10.7937/TCIA.e3sv-re93) 15. [NLST](https://dx.doi.org/10.7937/TCIA.hmq8-j677) 16. [TCGA-ACC](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/adrenocortical) 17. [TCGA-BLCA](https://dx.doi.org/10.7937/K9/TCIA.2016.8LNG8XDR) 18. [TCGA-BRCA](https://dx.doi.org/10.7937/K9/TCIA.2016.AB2NAZRP) 19. [TCGA-BRCA](https://dx.doi.org/10.7937/TCIA.2019.wgllssg1) 20. [TCGA-CESC](https://dx.doi.org/10.7937/K9/TCIA.2016.SQ4M8YP4) 21. [TCGA-CHOL](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/cholangiocarcinoma) 22. [TCGA-COAD](https://dx.doi.org/10.7937/K9/TCIA.2016.HJJHBOXZ) 23. TCGA-DLBC 24. [TCGA-ESCA](https://dx.doi.org/10.7937/K9/TCIA.2016.VPTNRGFY) 25. [TCGA-GBM](https://dx.doi.org/10.7937/K9/TCIA.2016.RNYFUYE9) 26. [TCGA-GBM](https://dx.doi.org/10.7937/TCIA.2018.ow6ce3ml) 27. [TCGA-HNSC](https://dx.doi.org/10.7937/K9/TCIA.2016.LXKQ47MS) 28. [TCGA-KICH](https://dx.doi.org/10.7937/K9/TCIA.2016.YU3RBCZN) 29. [TCGA-KIRC](https://dx.doi.org/10.7937/K9/TCIA.2016.V6PBVTDR) 30. [TCGA-KIRP](https://dx.doi.org/10.7937/K9/TCIA.2016.ACWOGBEF) \* 31. [TCGA-LGG](https://dx.doi.org/10.7937/K9/TCIA.2016.L4LTD3TK) 32. [TCGA-LGG](https://dx.doi.org/10.7937/TCIA.2018.ow6ce3ml) 33. [TCGA-LIHC](https://dx.doi.org/10.7937/K9/TCIA.2016.IMMQW8UQ) 34. [TCGA-LUAD](https://dx.doi.org/10.7937/K9/TCIA.2016.JGNIHEP5) 35. [TCGA-LUSC](https://dx.doi.org/10.7937/K9/TCIA.2016.TYGKKFMQ) 36. [TCGA-MESO](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/mesothelioma) 37. [TCGA-OV](https://dx.doi.org/10.7937/K9/TCIA.2016.NDO1MDFQ) 38. [TCGA-PAAD](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/pancreatic) 39. [TCGA-PCPG](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/paraganglioma) 40. [TCGA-PRAD](https://dx.doi.org/10.7937/K9/TCIA.2016.YXOGLM4Y) 41. [TCGA-READ](https://dx.doi.org/10.7937/K9/TCIA.2016.F7PPNPNU) 42. [TCGA-SARC](https://dx.doi.org/10.7937/K9/TCIA.2016.CX6YLSUX) 43. [TCGA-SKCM](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/melanoma-skin) 44. [TCGA-STAD](https://dx.doi.org/10.7937/K9/TCIA.2016.GDHL9KIM) 45. [TCGA-TGCT](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/testicular-germ-cell) 46. [TCGA-THCA](https://dx.doi.org/10.7937/K9/TCIA.2016.9ZFRVF1B) 47. [TCGA-THYM](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/thymoma) 48. [TCGA-UCEC](https://dx.doi.org/10.7937/K9/TCIA.2016.GKJ0ZWAC) 49. [TCGA-UCS](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/uterine-carcinosarcoma) 50. [TCGA-UVM](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/melanoma-eye) Note that the TCGA-KIRP and TCGA-BRCA collections (marked with the asterisk in the list above) are currently missing SM high resolution layer files/instances due to a [known limitation](https://issuetracker.google.com/issues/238920874) of Google Healthcare that makes it not possible to ingest datasets that exceed some internal limits. Specifically, the following patient/studies are affected: * TCGA-KIRP: `PatientID` TCGA-5P-A9KA, `StudyInstanceUID` 2.25.191236165605958868867890945341011875563 * TCGA-BRCA: `PatientID` TCGA-OL-A66H, `StudyInstanceUID` 2.25.82800314486527687800038836287574075736 The affected files will be included in IDC when the infrastructure limitation is addressed. Collection access level change: 1. [Vestibular-Schwannoma-SEG](https://doi.org/10.7937/TCIA.9YTJ-5Q73) is now available as public access collection ## v9 - May 2022 This data release introduces the concept of differential license to IDC: some of the collections maintained by IDC contain items that have different licenses. As an example, radiology component of the TCGA-GBM collection is covered by the TCIA limited access license, and is not available in IDC, while the digital pathology component is covered by CC-BY. With this release, we complete sharing in full of the digital pathology component of the datasets released by the CPTAC and TCGA programs. New collections: 1. [ACRIN-Contralateral-Breast-MR](https://dx.doi.org/10.7937/Q1EE-J082) 2. [StageII-Colorectal-CT](https://dx.doi.org/10.7937/p5k5-tg43) Updated collections: 1. [B-mode-and-CEUS-Liver](https://dx.doi.org/10.7937/TCIA.2021.v4z7-tc39) 2. [CPTAC-GBM](https://dx.doi.org/10.7937/K9/TCIA.2018.3RJE41Q1) 3. [CPTAC-HNSCC](https://dx.doi.org/10.7937/K9/TCIA.2018.UW45NH81) 4. [Pediatric-CT-SEG](https://dx.doi.org/10.7937/TCIA.X0H0-1706) 5. [TCGA-GBM](https://dx.doi.org/10.7937/K9/TCIA.2016.RNYFUYE9) 6. [TCGA-HNSC](https://dx.doi.org/10.7937/K9/TCIA.2016.LXKQ47MS) 7. [TCGA-LGG](https://dx.doi.org/10.7937/K9/TCIA.2016.L4LTD3TK) ## v8 - April 2022 The main highlight of this release is the addition of the NLST and TCGA Slide Microscopy imaging data. New TCGA content includes introduction of new (to IDC) TCGA collections that have only slide microscopy component, and addition of the slide microscopy component to those IDC collections that were available earlier and included only the radiology component. New collections 1. [TCGA-ACC](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/adrenocortical) 2. [TCGA-CHOL](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/cholangiocarcinoma) 3. TCGA-DLBC (TCGA-DLBC collection does not have a description page) 4. [TCGA-MESO](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/mesothelioma) 5. [TCGA-PAAD](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/pancreatic) 6. [TCGA-PCPG](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/paraganglioma) 7. [TCGA-SKCM](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/melanoma-skin) 8. [TCGA-TGCT](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/testicular-germ-cell) 9. [TCGA-THYM](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/thymoma) 10. [TCGA-UCS](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/uterine-carcinosarcoma) 11. [TCGA-UVM](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/studied-cancers/melanoma-eye) Updated collections 1. [NLST](https://dx.doi.org/10.7937/TCIA.hmq8-j677) 2. [TCGA-BLCA](https://dx.doi.org/10.7937/K9/TCIA.2016.8LNG8XDR) 3. [TCGA-BRCA](https://dx.doi.org/10.7937/K9/TCIA.2016.AB2NAZRP) 4. [TCGA-BRCA](https://dx.doi.org/10.7937/TCIA.2019.wgllssg1) 5. [TCGA-CESC](https://dx.doi.org/10.7937/K9/TCIA.2016.SQ4M8YP4) 6. [TCGA-COAD](https://dx.doi.org/10.7937/K9/TCIA.2016.HJJHBOXZ) 7. [TCGA-ESCA](https://dx.doi.org/10.7937/K9/TCIA.2016.VPTNRGFY) 8. [TCGA-KICH](https://dx.doi.org/10.7937/K9/TCIA.2016.YU3RBCZN) 9. [TCGA-KIRC](https://dx.doi.org/10.7937/K9/TCIA.2016.V6PBVTDR) 10. [TCGA-KIRP](https://dx.doi.org/10.7937/K9/TCIA.2016.ACWOGBEF) 11. [TCGA-LIHC](https://dx.doi.org/10.7937/K9/TCIA.2016.IMMQW8UQ) 12. [TCGA-LUAD](https://dx.doi.org/10.7937/K9/TCIA.2016.JGNIHEP5) 13. [TCGA-LUSC](https://dx.doi.org/10.7937/K9/TCIA.2016.TYGKKFMQ) 14. [TCGA-OV](https://dx.doi.org/10.7937/K9/TCIA.2016.NDO1MDFQ) 15. [TCGA-PRAD](https://dx.doi.org/10.7937/K9/TCIA.2016.YXOGLM4Y) 16. [TCGA-READ](https://dx.doi.org/10.7937/K9/TCIA.2016.F7PPNPNU) 17. [TCGA-SARC](https://dx.doi.org/10.7937/K9/TCIA.2016.CX6YLSUX) 18. [TCGA-STAD](https://dx.doi.org/10.7937/K9/TCIA.2016.GDHL9KIM) 19. [TCGA-THCA](https://dx.doi.org/10.7937/K9/TCIA.2016.9ZFRVF1B) 20. [TCGA-UCEC](https://dx.doi.org/10.7937/K9/TCIA.2016.GKJ0ZWAC) ## v7 - February 2022 The main highlight of this release is the addition of the Slide Microscopy imaging component to the remaining CPTAC collections. New collections 1. [APOLLO-5-ESCA](https://dx.doi.org/10.7937/n69a-7a26) 2. [APOLLO-5-LUAD](https://dx.doi.org/10.7937/BDM9-4623) 3. [APOLLO-5-PAAD](https://dx.doi.org/10.7937/tcia.1yeg-5740) 4. [APOLLO-5-THYM](https://dx.doi.org/10.7937/tcia.0pg4-nh82) 5. [CPTAC-AML](https://dx.doi.org/10.7937/tcia.2019.b6foe619) 6. [CPTAC-BRCA](https://dx.doi.org/10.7937/TCIA.CAEM-YS80) 7. [CPTAC-COAD](https://dx.doi.org/10.7937/TCIA.YZWQ-ZZ63) 8. [CPTAC-OV](https://dx.doi.org/10.7937/TCIA.ZS4A-JD58) 9. [Pancreatic-CT-CBCT-SEG](https://dx.doi.org/10.7937/TCIA.ESHQ-4D90) 10. [Pediatric-CT-SEG](https://dx.doi.org/10.7937/TCIA.X0H0-1706) Updated collections 1. [CPTAC-CCRCC](https://dx.doi.org/10.7937/K9/TCIA.2018.OBLAMN27) 2. [CPTAC-CM](https://dx.doi.org/10.7937/K9/TCIA.2018.ODU24GZE) 3. [CPTAC-LSCC](https://dx.doi.org/10.7937/K9/TCIA.2018.6EMUB5L2) 4. [CPTAC-LUAD](https://dx.doi.org/10.7937/K9/TCIA.2018.PAT12TBS) 5. [CPTAC-PDA](https://dx.doi.org/10.7937/K9/TCIA.2018.SC20FO18) 6. [CPTAC-SAR](https://dx.doi.org/10.7937/TCIA.2019.9bt23r95) 7. [CPTAC-UCEC](https://dx.doi.org/10.7937/K9/TCIA.2018.3R3JUISW) ## v6 - January 2022 The following collections became limited access due to the [change in policy by TCIA](https://discourse.canceridc.dev/t/tcia-collections-of-brain-head-or-head-neck-cancers-are-changing-to-limited-access/241), which is the original source of those collections. Original collections: 1. [AAPM-RT-MAC](https://doi.org/10.7937/tcia.2019.bcfjqfqb) 2. [ACRIN-DSC-MR-Brain](https://doi.org/10.7937/tcia.2019.zr1pjf4i) 3. [ACRIN-FMISO-Brain](https://doi.org/10.7937/K9/TCIA.2018.vohlekok) 4. [ACRIN-HNSCC-FDG-PET-CT](https://doi.org/10.7937/K9/TCIA.2016.JQEJZZNG) 5. [Anti-PD-1\_MELANOMA](https://doi.org/10.7937/tcia.2019.1ae0qtcu) 6. [Brain-Tumor-Progression](https://doi.org/10.7937/K9/TCIA.2018.15quzvnb) 7. [CPTAC-GBM](https://doi.org/10.7937/K9/TCIA.2018.3RJE41Q1) 8. [CPTAC-HNSCC](https://doi.org/10.7937/K9/TCIA.2018.UW45NH81) 9. [HEAD-NECK-RADIOMICS-HN1](https://doi.org/10.7937/tcia.2019.8kap372n) 10. [HNSCC](https://doi.org/10.7937/k9/tcia.2020.a8sh-7363) 11. [HNSCC-3DCT-RT](https://doi.org/10.7937/K9/TCIA.2018.13upr2xf) 12. [Head-Neck Cetuximab](https://doi.org/10.7937/K9/TCIA.2015.7AKGJUPZ) 13. [Head-Neck-PET-CT](https://doi.org/10.7937/K9/TCIA.2017.8oje5q00) 14. [IvyGAP](https://doi.org/10.7937/K9/TCIA.2016.XLwaN6nL) 15. [LGG-1p19qDeletion](https://doi.org/10.7937/K9/TCIA.2017.dwehtz9v) 16. [MRI-DIR](https://doi.org/10.7937/K9/TCIA.2018.3f08iejt) 17. [OPC-Radiomics](https://doi.org/10.7937/tcia.2019.8dho2gls) 18. [QIN GBM Treatment Response](https://doi.org/10.7937/K9/TCIA.2016.nQF4gpn2) 19. [QIN-BRAIN-DSC-MRI](https://doi.org/10.7937/K9/TCIA.2016.5DI84Js8) 20. [QIN-HEADNECK](https://doi.org/10.7937/K9/TCIA.2015.K0F5CGLI) 21. [REMBRANDT](https://doi.org/10.7937/K9/TCIA.2015.588OZUZB) 22. [RIDER NEURO MRI](https://doi.org/10.7937/K9/TCIA.2015.VOSN3HN1) 23. [TCGA-GBM](https://doi.org/10.7937/K9/TCIA.2016.RNYFUYE9) 24. [TCGA-HNSC](https://doi.org/10.7937/K9/TCIA.2016.LXKQ47MS) 25. [TCGA-LGG](https://doi.org/10.7937/K9/TCIA.2016.L4LTD3TK) 26. [Vestibular-Schwannoma-SEG](https://doi.org/10.7937/TCIA.9YTJ-5Q73) Analysis results collections: 1. [DICOM-SEG Conversions for TCGA-LGG and TCGA-GBM Segmentation Datasets](https://doi.org/10.7937/TCIA.2018.ow6ce3ml) 2. [Outcome Prediction in Patients with Glioblastoma by Using Imaging, Clinical, and Genomic Biomarkers: Focus on the Nonenhancing Component of the Tumor](https://doi.org/10.7937/K9/TCIA.2014.FAB7YRPZ) ## v5 - December 2021 New collections: * [COVID-19-NY-SBU](https://dx.doi.org/10.7937/TCIA.BBAG-2923) * [B-mode-and-CEUS-Liver](https://dx.doi.org/10.7937/TCIA.2021.v4z7-tc39) * [APOLLO-5-LSCC](https://dx.doi.org/10.7937/TCIA.QQ0G-EB24) * [CMMD](https://dx.doi.org/10.7937/tcia.eqde-4b16) * [ACRIN-HNSCC-FDG-PET-CT](https://dx.doi.org/10.7937/K9/TCIA.2016.JQEJZZNG) * [Duke-Breast-Cancer-MRI](https://dx.doi.org/10.7937/TCIA.e3sv-re93) New analysis results collections: * Outcome Prediction in Patients with Glioblastoma by Using Imaging, Clinical, and Genomic Biomarkers: Focus on the Nonenhancing Component of the Tumor ([GBM-MR-NER-Outcomes](https://dx.doi.org/10.7937/K9/TCIA.2014.FAB7YRPZ)) * DICOM-SEG Conversions for TCGA-LGG and TCGA-GBM Segmentation Datasets ([DICOM-Glioma-SEG](https://dx.doi.org/10.7937/TCIA.2018.ow6ce3ml)) Updated collections: * [TCGA-GBM](https://dx.doi.org/10.7937/K9/TCIA.2014.FAB7YRPZ) * [TCGA-LGG](https://dx.doi.org/10.7937/TCIA.2018.ow6ce3ml) * [QIN-HEADNECK](https://dx.doi.org/10.7937/K9/TCIA.2015.K0F5CGLI) * [Breast-Cancer-Screening-DBT](https://dx.doi.org/10.7937/e4wt-cd02) * [NSCLC Radiogenomics](https://dx.doi.org/10.7937/K9/TCIA.2017.7hs46erv) * [QIN-HEADNECK](https://dx.doi.org/10.7937/K9/TCIA.2015.K0F5CGLI) * [Pseudo-PHI-DICOM-Data](https://dx.doi.org/10.7937/s17z-r072) ## v4 - September 2021 [National Lung Screening Trial (NLST) collection](https://doi.org/10.7937/tcia.hmq8-j677) is added. The data included consists of the following components: 1\) CT images available as any other imaging collection (via IDC Portal, BigQuery metadata tables, and storage buckets); 2\) a subset of clinical data available in the BigQuery tables starting with `nlst_` under the `idc_v4` dataset, as documented in the [Collection-specific BigQuery Tables](/data/organization-of-data/files-and-metadata) section. 3\) One instance is missing from patient/study/series:\ `126153/1.2.840.113654.2.55.319335498043274792486636919135185299851/1.2.840.113654.2.55.262421043240525317038356381369289737801` 4\) Three instances are missing from patient/study/series:\ `215303/1.3.6.1.4.1.14519.5.2.1.7009.9004.337968382369511017896638591276/1.3.6.1.4.1.14519.5.2.1.7009.9004.180224303090109944523368212991` ## v3 - August 2021 The following radiology collections were updated to include DICOM Slide Microscopy (SM) images converted from the original vendor-specific representation into [dual personality DICOM-TIFF format](/dicom/dicom-tiff-dual-personality-files). 1. [CPTAC-LUAD](https://doi.org/10.7937/K9/TCIA.2018.PAT12TBS) 2. [CPTAC-LSCC](https://doi.org/10.7937/K9/TCIA.2018.6EMUB5L2) {% hint style="warning" %} The DICOM Slide Microscopy (SM) images included in the collections above in IDC are not available in TCIA. TCIA only includes images in the vendor-specific SVS format! {% endhint %} ## v2 - June 2021 Listed below are all of the [original](https://www.cancerimagingarchive.net/collections/) and [analysis results](https://www.cancerimagingarchive.net/tcia-analysis-results/) collections of [The Cancer Imaging Archive](https://www.cancerimagingarchive.net) currently hosted by IDC, with the links to the Digital Object Identifiers (DOIs) of those collections. New original collections: 1. [IvyGAP](https://doi.org/10.7937/K9/TCIA.2016.XLwaN6nL) 2. [QIN LUNG CT](https://doi.org/10.7937/K9/TCIA.2015.NPGZYZBZ) 3. [LungCT-Diagnosis](https://doi.org/10.7937/K9/TCIA.2015.A6V7JIWX) 4. [HEAD-NECK-RADIOMICS-HN1](https://doi.org/10.7937/tcia.2019.8kap372n) 5. [Prostate Fused-MRI-Pathology](https://doi.org/10.7937/K9/TCIA.2016.TLPMR1AM) 6. [APOLLO](https://wiki.cancerimagingarchive.net/x/N4NyAQ) 7. [LGG-1p19qDeletion](https://doi.org/10.7937/K9/TCIA.2017.dwehtz9v) 8. [Soft-tissue-Sarcoma](https://doi.org/10.7937/K9/TCIA.2015.7GO2GSKS) 9. [NSCLC-Radiomics-Genomics](https://doi.org/10.7937/K9/TCIA.2015.L4FRET6Z) 10. [Brain-Tumor-Progression](https://doi.org/10.7937/K9/TCIA.2018.15quzvnb) 11. [Head-Neck Cetuximab](https://doi.org/10.7937/K9/TCIA.2015.7AKGJUPZ) 12. [CPTAC-GBM](https://doi.org/10.7937/K9/TCIA.2018.3RJE41Q1) 13. [CPTAC-SAR](https://doi.org/10.7937/TCIA.2019.9bt23r95) 14. [CPTAC-LUAD](https://doi.org/10.7937/K9/TCIA.2018.PAT12TBS) 15. [CPTAC-LSCC](https://doi.org/10.7937/K9/TCIA.2018.6EMUB5L2) 16. [Head-Neck-PET-CT](https://doi.org/10.7937/K9/TCIA.2017.8oje5q00) 17. [C4KC-KiTS](https://doi.org/10.7937/TCIA.2019.IX49E8NX) 18. [Breast-MRI-NACT-Pilot](https://doi.org/10.7937/K9/TCIA.2016.QHSYHJKY) 19. [4D-Lung](https://doi.org/10.7937/K9/TCIA.2016.ELN8YGLE) 20. [Mouse-Mammary](https://doi.org/10.7937/K9/TCIA.2015.9P42KSE6) 21. [CT Lymph Nodes](https://doi.org/10.7937/K9/TCIA.2015.AQIIDCNM) 22. [HNSCC](https://doi.org/10.7937/k9/tcia.2020.a8sh-7363) 23. [Breast-Cancer-Screening-DBT](https://doi.org/10.7937/e4wt-cd02) 24. [MRI-DIR](https://doi.org/10.7937/K9/TCIA.2018.3f08iejt) 25. [Lung-PET-CT-Dx](https://doi.org/10.7937/TCIA.2020.NNC2-0461) 26. [NSCLC-RADIOMICS-INTEROBSERVER1](https://doi.org/10.7937/tcia.2019.cwvlpd26) 27. [PDMR-BL0293-F563](https://doi.org/10.7937/tcia.2019.b6u7wmqw) 28. [CT COLONOGRAPHY](https://doi.org/10.7937/K9/TCIA.2015.NWTESAY1) 29. [Phantom FDA](https://doi.org/10.7937/K9/TCIA.2015.ORBJKMUX) 30. [QIN-PROSTATE-Repeatability](https://doi.org/10.7937/K9/TCIA.2018.MR1CKGND) 31. [PROSTATEx](https://doi.org/10.7937/K9TCIA.2017.MURS5CL) 32. [AAPM-RT-MAC](https://doi.org/10.7937/tcia.2019.bcfjqfqb) 33. [ICDC-Glioma](https://doi.org/10.7937/TCIA.SVQT-Q016) 34. [RIDER Breast MRI](https://doi.org/10.7937/K9/TCIA.2015.H1SXNUXL) 35. [Anti-PD-1\_MELANOMA](https://doi.org/10.7937/tcia.2019.1ae0qtcu) 36. [COVID-19-AR](https://doi.org/10.7937/tcia.2020.py71-5978) 37. [PROSTATE-MRI](https://doi.org/10.7937/K9/TCIA.2016.6046GUDv) 38. [NaF PROSTATE](https://doi.org/10.7937/K9/TCIA.2015.ISOQTHKO) 39. [Mouse-Astrocytoma](https://doi.org/10.7937/K9TCIA.2017.SGW7CAQW) 40. [ACRIN-DSC-MR-Brain](https://doi.org/10.7937/tcia.2019.zr1pjf4i) 41. [ACRIN-NSCLC-FDG-PET](https://doi.org/10.7937/tcia.2019.30ilqfcl) 42. [QIN Breast DCE-MRI](https://doi.org/10.7937/K9/TCIA.2014.A2N1IXOX) 43. [RIDER NEURO MRI](https://doi.org/10.7937/K9/TCIA.2015.VOSN3HN1) 44. [MIDRC-RICORD-1A](https://doi.org/10.7937/VTW4-X588) 45. [MIDRC-RICORD-1C](https://doi.org/10.7937/91ah-v663) 46. [REMBRANDT](https://doi.org/10.7937/K9/TCIA.2015.588OZUZB) 47. [NSCLC Radiogenomics](https://doi.org/10.7937/K9/TCIA.2017.7hs46erv) 48. [HNSCC-3DCT-RT](https://doi.org/10.7937/K9/TCIA.2018.13upr2xf) 49. [VICTRE](https://doi.org/10.7937/TCIA.2019.ho23nxaw) 50. [CPTAC-CM](https://doi.org/10.7937/K9/TCIA.2018.ODU24GZE) 51. [CPTAC-PDA](https://doi.org/10.7937/K9/TCIA.2018.SC20FO18) 52. [CPTAC-UCEC](https://doi.org/10.7937/K9/TCIA.2018.3R3JUISW) 53. [CPTAC-CCRCC](https://doi.org/10.7937/K9/TCIA.2018.OBLAMN27) 54. [CPTAC-HNSCC](https://doi.org/10.7937/K9/TCIA.2018.UW45NH81) 55. [OPC-Radiomics](https://doi.org/10.7937/tcia.2019.8dho2gls) 56. [Vestibular-Schwannoma-SEG](https://doi.org/10.7937/TCIA.9YTJ-5Q73) 57. [SPIE-AAPM Lung CT Challenge](https://doi.org/10.7937/K9/TCIA.2015.UZLSU3FL) 58. [Lung Phantom](https://doi.org/10.7937/K9/TCIA.2015.08A1IXOO) 59. [Pseudo-PHI-DICOM-Data](https://doi.org/10.7937/s17z-r072) 60. [Pancreas-CT](https://doi.org/10.7937/K9/TCIA.2016.tNB1kqBU) 61. [QIN GBM Treatment Response](https://doi.org/10.7937/K9/TCIA.2016.nQF4gpn2) 62. [Pelvic-Reference-Data](https://doi.org/10.7937/TCIA.2019.woskq5oo) 63. [Lung-Fused-CT-Pathology](https://doi.org/10.7937/K9/TCIA.2018.SMT36LPN) 64. [Anti-PD-1\_Lung](https://doi.org/10.7937/tcia.2019.zjjwb9ip) 65. [BREAST-DIAGNOSIS](https://doi.org/10.7937/K9/TCIA.2015.SDNRQXXR) 66. [RIDER Lung PET-CT](https://doi.org/10.7937/K9/TCIA.2015.OFIP7TVM) 67. [RIDER Lung CT](https://doi.org/10.7937/K9/TCIA.2015.U1X8A5NR) 68. [PDMR-292921-168-R](https://doi.org/10.7937/TCIA.2020.PCAK-8Z10) 69. [PDMR-833975-119-R](https://doi.org/10.7937/TCIA.0ECK-C338) 70. [PDMR-997537-175-T](https://doi.org/10.7937/TCIA.2020.BRY9-4N29) 71. [LCTSC](https://doi.org/10.7937/K9/TCIA.2017.3r3fvz08) 72. [Prostate-3T](https://doi.org/10.7937/K9/TCIA.2015.QJTV5IL5) 73. [ACRIN-FLT-Breast](https://doi.org/10.7937/K9/TCIA.2017.ol20zmxg) 74. [ACRIN-FMISO-Brain](https://doi.org/10.7937/K9/TCIA.2018.vohlekok) 75. [PDMR-425362-245-T](https://doi.org/10.7937/TCIA.2020.7YRS-7J97) 76. [Prostate-MRI-US-Biopsy](https://doi.org/10.7937/TCIA.2020.A61IOC1A) 77. [MIDRC-RICORD-1B](https://doi.org/10.7937/31V8-4A40) 78. [DRO-Toolkit](https://doi.org/10.7937/t062-8262) New analysis results collections: 1. [PROSTATEx Zone Segmentations](https://doi.org/10.7937/tcia.nbb4-4655) 2. [High Resolution Prostate Segmentations for the ProstateX-Challenge](https://doi.org/10.7937/TCIA.2019.DEG7ZG1U) 3. [RIDER Lung CT Segmentation Labels from: Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach](https://doi.org/10.7937/tcia.2020.jit9grk8) ## v1 - October 2020 Listed below are all of the [original](https://www.cancerimagingarchive.net/collections/) and [analysis results](https://www.cancerimagingarchive.net/tcia-analysis-results/) collections of [The Cancer Imaging Archive](https://www.cancerimagingarchive.net) currently hosted by IDC, with the links to the Digital Object Identifiers (DOIs) of those collections. Original collections included: 1. [TCGA-PRAD](https://doi.org/10.7937/K9/TCIA.2016.YXOGLM4Y) 2. [TCGA-BLCA](https://doi.org/10.7937/K9/TCIA.2016.8LNG8XDR) 3. [TCGA-UCEC](https://doi.org/10.7937/K9/TCIA.2016.GKJ0ZWAC) 4. [TCGA-HNSC](https://doi.org/10.7937/K9/TCIA.2016.LXKQ47MS) 5. [TCGA-LUSC](https://doi.org/10.7937/K9/TCIA.2016.TYGKKFMQ) 6. [TCGA-KIRP](https://doi.org/10.7937/K9/TCIA.2016.ACWOGBEF) 7. [TCGA-THCA](https://doi.org/10.7937/K9/TCIA.2016.9ZFRVF1B) 8. [TCGA-SARC](https://doi.org/10.7937/K9/TCIA.2016.CX6YLSUX) 9. [TCGA-ESCA](https://doi.org/10.7937/K9/TCIA.2016.VPTNRGFY) 10. [TCGA-CESC](https://doi.org/10.7937/K9/TCIA.2016.SQ4M8YP4) 11. [TCGA-STAD](https://doi.org/10.7937/K9/TCIA.2016.GDHL9KIM) 12. [TCGA-COAD](https://doi.org/10.7937/K9/TCIA.2016.HJJHBOXZ) 13. [TCGA-KICH](https://doi.org/10.7937/K9/TCIA.2016.YU3RBCZN) 14. [TCGA-READ](https://doi.org/10.7937/K9/TCIA.2016.F7PPNPNU) 15. [TCGA-LUAD](https://doi.org/10.7937/K9/TCIA.2016.JGNIHEP5) 16. [TCGA-LIHC](https://doi.org/10.7937/K9/TCIA.2016.IMMQW8UQ) 17. [TCGA-BRCA](https://doi.org/10.7937/K9/TCIA.2016.AB2NAZRP) 18. [TCGA-OV](https://doi.org/10.7937/K9/TCIA.2016.NDO1MDFQ) 19. [TCGA-KIRC](https://doi.org/10.7937/K9/TCIA.2016.V6PBVTDR) 20. [TCGA-LGG](https://doi.org/10.7937/K9/TCIA.2016.L4LTD3TK) 21. [TCGA-GBM](https://doi.org/10.7937/K9/TCIA.2016.RNYFUYE9) 22. [ISPY1 (ACRIN 6657)](https://doi.org/10.7937/K9/TCIA.2016.HdHpgJLK) 23. [QIN-HeadNeck](https://doi.org/10.7937/K9/TCIA.2015.K0F5CGLI) 24. [LIDC-IDRI](https://doi.org/10.7937/K9/TCIA.2015.LO9QL9SX) 25. [NSCLC-Radiomics](https://wiki.cancerimagingarchive.net/display/Public/NSCLC-Radiomics) Analysis collections included: 1. [Standardized representation of the TCIA LIDC-IDRI annotations using DICOM](https://doi.org/10.7937/TCIA.2018.h7umfurq) 2. [QIN multi-site collection of Lung CT data with Nodule Segmentations](https://doi.org/10.7937/K9/TCIA.2015.1BUVFJR7) (only items corresponding to the LIDC-IDRI original collection are included) 3. [DICOM SR of clinical data and measurement for breast cancer collections to TCIA](https://doi.org/10.7937/TCIA.2019.wgllssg1) (only items corresponding to the ISPY1 original collection are included) # Data known issues 1. Indexing of the collection of [NSCLC-Radiomics](https://wiki.cancerimagingarchive.net/display/Public/NSCLC-Radiomics) by the Data Commons Framework is pending. 2. [QIN multi-site collection of Lung CT data with Nodule Segmentations](https://doi.org/10.7937/K9/TCIA.2015.1BUVFJR7): only items corresponding to the LIDC-IDRI original collection are included 3. [DICOM SR of clinical data and measurement for breast cancer collections to TCIA](https://doi.org/10.7937/TCIA.2019.wgllssg1): only items corresponding to the ISPY1 original collection are included 4. [ISPY1 (ACRIN 6657)](https://doi.org/10.7937/K9/TCIA.2016.HdHpgJLK): Some of the segmentations in this collection are empty (as an example, SeriesNumber 42100 with SeriesDescription "VOI PE Segmentation thresh=70" in [this study](https://viewer.imaging.datacommons.cancer.gov/viewer/1.3.6.1.4.1.14519.5.2.1.7695.1700.186130315530827043916665138479) is empty). 5. Due to the existing limitations of Google Healthcare API, not all of the DICOM attributes are extracted and are available in BigQuery tables. Specifically: * sequences that have more than 15 levels of nesting are not extracted (see ) - we believe this limitation does not affect the data stored in IDC * sequences that contain around 1MiB of data are dropped from BigQuery export and RetrieveMetadata output currently. 1MiB is not an exact limit, but it can be used as a rough estimate of whether or not the API will drop the tag (this limitation was not documented as of writing this) - we know that some of the instances in IDC will be affected by this limitation. The fix for this limitation is targeted for sometime in 2021, according to the communication with Google Healthcare support. # Portal tutorial The[ Imaging Data Commons Portal ](https://portal.imaging.datacommons.cancer.gov/)provides a web-based interactive interface to browse the data hosted by IDC, visualize images, build manifests describing selected cohorts, and download images defined by the manifests. The slides below give a quick guided overview of how you can use IDC Portal. {% hint style="success" %} **No login is required to use the portal, to visualize images, or to download data from IDC!** {% endhint %} {% embed url="" %} # Slide microscopy This section of the documentation complements the [digital pathology self-guided python notebooks](https://github.com/ImagingDataCommons/IDC-Tutorials?tab=readme-ov-file#pathomics) tutorials available in our notebooks repository: * [Getting started with pathology images in IDC](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/pathomics/getting_started_with_digital_pathology.ipynb): all of the pathology images in IDC are in DICOM Slide Microscopy format; this notebook will help you get started with using this representation and also searching IDC pathology images. * [Exploring IDC slide microscopy images metadata](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/pathomics/slide_microscopy_metadata_search.ipynb): introduction to the key metadata accompanying IDC slide microscopy images that can be used for subsetting data and building cohorts. In the following subsections you will find notebooks that don't require python programming, or have dependencies that make them not suitable for the python notebook format. # Using QuPath for visualization [QuPath](https://qupath.github.io/) is a popular open-source desktop application for visualizing and annotating slide microscopy images. It is integrated with both OpenSlide and BioFormats libraries, and as of the current QuPath 0.5.1 version supports direct loading of DICOM Slide Microscopy images. In this tutorial you will learn how to use DICOM SM images from IDC with QuPath. ### Load a brightfield (RGB) DICOM slide First you will need to download a sample SM image from IDC to your desktop. To identify a sample image, you can navigate to the IDC Portal, copy `SeriesInstanceUID` value for a sample SM series you want to download. Given that UID, you can download the corresponding files using `idc-index` python package (see details in the documentation section describing data d[ownload instructions](/data/downloading-data)). In this tutorial, we will use the series identified by SeriesInstanceUID from the [TCGA-ACC](https://portal.imaging.datacommons.cancer.gov/explore/filters/?Modality_op=OR\&Modality=SM\&collection_id=tcga_acc) collection `1.3.6.1.4.1.5962.99.1.3140643155.174517037.1639523215699.2.0`, which you can download as follows: ```sh idc download 1.3.6.1.4.1.5962.99.1.3140643155.174517037.1639523215699.2.0 ``` Next, open QuPath and select `"File > Open"`.
Choose just one of the `.dcm` files that belong to the desired dataset, then click `Open`. The remaining files will be automatically detected and should not be selected. When prompted for an image type, select `Brightfield H&E` (or whatever is appropriate for the dataset being opened), then click `Apply`. This is a QuPath feature intended to aid in analysis, and is further described in the [QuPath documentation](https://qupath.readthedocs.io/en/stable/docs/starting/first_steps.html#setting-the-image-type).
The image should now display, and can be navigated by zooming/panning as described in the [QuPath documentation](https://qupath.readthedocs.io/en/stable/docs/starting/first_steps.html#zooming-in-out). Zooming and panning in real time: {% embed url="" %} The `Image` tab on the left side of the screen shows dimension information, and lists any associated images. In this case, a thumbnail image is present under `Associated Images` at the bottom of the `Image` tab. Double-clicking on `Series 1 (THUMBNAIL)` will open the thumbnail image in a separate window:
### Open a fluorescence DICOM dataset For this part, we will use a slide from the HTAN-OHSU collection identified by SeriesInstanceUID `1.3.6.1.4.1.5962.99.1.1999932010.1115442694.1655562373738.4.0`. As before, you can download it as follows: ```sh idc download 1.3.6.1.4.1.5962.99.1.1999932010.1115442694.1655562373738.4.0 ``` As in the brightfield case, open QuPath and select `File > Open`.
Choose just one of the `.dcm` files in the dataset, as the other files will be automatically detected. It does not matter which file is selected. When prompted, set the image type to `Fluorescence`, or as appropriate for the dataset:
The image should then display, and can be navigated by zooming/panning as described in the [QuPath documentation](https://qupath.readthedocs.io/en/stable/docs/starting/first_steps.html#zooming-in-out). {% embed url="" %} The `Image` tab indicates the number of channels (12 in this case). By default, all channels will be displayed at once. This can be changed by selecting `View > Brightness/Contrast` or the "half-circles" icon in the toolbar:
Unchecking the `Show` box will hide the channel's data, and update the image. # Introduction to DICOM IDC relies on DICOM for data modeling, representation and communication. Most of the data stored in IDC is in DICOM format. If you want to use IDC, you (hopefully!) do not need to become a DICOM expert, but you do need to have a basic understanding of how DICOM data is structured, and how to transform DICOM objects into alternative representations that can be used by the tools familiar to you. This section is not intended to be a comprehensive introduction to the standard, but rather a very brief overview of some of the concepts that you will need to understand to better use IDC data. {% hint style="info" %} If you are looking for an introduction to the DICOM standard, consider this book: *Digital Imaging and Communications in Medicine (DICOM): A Practical Introduction and Survival Guide*, 2nd Edition by Pianykh, Oleg S. published by Springer (2011). {% endhint %} As discussed in [Data organization](/data/organization-of-data), the main mechanism for accessing the data stored in IDC is by using the storage buckets that contain individual files indexed through other interfaces. Each of the files in the IDC-maintained storage buckets encodes a DICOM object. Each DICOM object is a collection of *data elements* or *attributes*. Below is an example of a subset of attributes in a DICOM object, as generated by the IDC OHIF Viewer (which can be toggled by clicking the "Tag browser" icon in the IDC viewer toolbar): ![](/files/-MIyelUo1wlXF5Pe4wDV) The standard defines constraints on what kind of data each of the attributes can contain. Every single attribute defined by the standard is listed in the [DICOM Data Dictionary](http://dicom.nema.org/medical/dicom/current/output/chtml/part06/chapter_6.html), which defines those constraints: * Value Representation (VR) defines the type of the data that data element can contain. There are 27 DICOM VRs, and they are defined in [Part 5 of the standard](http://dicom.nema.org/medical/dicom/current/output/chtml/part05/sect_6.2.html). * Value Multiplicity (VM) defines the number of items of the prescribed VR that can be contained in a given data element. What attributes are included in a given object is determined by the type of object (or, to follow the DICOM nomenclature, *Information Object*). [Part 3 of the DICOM standard](http://dicom.nema.org/medical/dicom/current/output/chtml/part03/PS3.3.html) is dedicated to the definitions (IODs) of those objects. {% hint style="warning" %} It is critical to recognize that while all of the DICOM files at the high level are structured exactly in the same way and follow the same syntax and encoding rules, interpretation of the content of an individual file is dependent on the specific type of object it encodes! {% endhint %} How do you know what object is encoded in a given file (or *instance* of the object, using the DICOM lingo)? For this purpose there is an attribute `SOPClassUID` that uniquely identifies the class of the encoded object. The content of this attribute is not easy to interpret, since it is a unique identifier. To map it to the specific object class name, you can consult the complete list of object classes available in [Part 4 here](http://dicom.nema.org/medical/dicom/current/output/chtml/part04/sect_B.5.html). When you use the IDC portal to build your cohort, unique identifiers for the object classes are mapped to their names, which are available under the "Object class" group of facets in the search interface. A somewhat related attribute that hints at the type of object is `Modality`, which is defined by the standard as "Type of equipment that originally acquired the data used to create the images in this Series", and is expected to take one of the values from [this list](http://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_C.7.3.html#sect_C.7.3.1.1.1). However, `Modality` is not equivalent to `SOPClassUID`, and should not be used as a substitute. As an example it is possible that data derived from the original modality could be saved as a different object class, but keep the value of modality identical. {% hint style="info" %} You can see all of the attributes of a DICOM instance using IDC Viewer by clicking the "Tag browser" button in the viewer toolbar, or you can use one of the many publicly available tools to examine the DICOM attributes of a downloaded DICOM file. {% endhint %} # DICOM data model DICOM defines its own model to map relevant entities from the real world. That model, as [defined in Part 3 of the standard](http://dicom.nema.org/medical/dicom/current/output/chtml/part03/chapter_7.html), is shown in the figure below. {% hint style="warning" %} The DICOM data model is implicit, and is not defined in a machine-readable structured form by the standard! {% endhint %} ![DICOM model of the real world](/files/-MFfFpSnvYc78nwjs7-v) DICOM data model entities do not always map to DICOM objects! In fact, every DICOM object you will ever encounter in IDC will contain attributes describing various properties of the entries at different levels of the real world data model. Such objects are called *Composite* Information Objects. The [information model](http://dicom.nema.org/medical/dicom/current/output/chtml/part03/chapter_A.html) of the Composite Information Object Definitions is shown below, and covers all of the composite objects defined by the standard. ![DICOM Composite Instance IOD Information Model](/files/-MFflAlswuIUkMF-CiEU) As can be observed from this diagram, *"each Composite Instance IOD \[Entity-Relationship] Model requires that all Composite Instances that are part of a specific Study shall share the same context. That is, all Composite Instances within a specific Patient Study share the same Patient and Study information; all Composite Instances within the same Series share the same Series information; etc."* ([ref](http://dicom.nema.org/medical/dicom/current/output/chtml/part03/chapter_A.html)). Each of the boxes in the diagram above corresponds to *Information Entities (IEs)*, which in turn are composed from *Information Modules*. Information Modules group attributes that are related. As an example, [Patient IE included in the MR object](http://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_A.4.3.html) will include [Patient Information Module](http://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_C.7.html#sect_C.7.1.1), which in turn will include such attributes as PatientID, PatientName, and PatientSex. {% hint style="info" %} To recap, individual files in IDC buckets are *instances* of DICOM composite objects. The specific type of object is defined by the`SOPClassUID.`Metadata included in a given object is captured by the *Information Object Definition* as a list of *Information Entities*. Object definition also contains the list of specific Information Modules for the individual IEs. Note that modules included in a given IE are object-specific, while list of attributes included in a given *Information Module* is fixed! {% endhint %} # Original objects We differentiate between the *original* and *derived* DICOM objects in the IDC portal and discussions of the IDC-hosted data. By Original objects we mean DICOM objects that are produced by image acquisition equipment - MR, CT, or PET images fall into this category. By Derived objects we mean those objects that were generated by means of analysis or annotation of the original objects. Those objects can contain, for example, volumetric segmentations of the structures in the original images, or quantitative measurements of the objects in the image. ## Original objects Most of the images stored on IDC are saved as objects that store individual slices of the image in separate instances of a series, with the image stored in the `PixelData` attribute. As of production release, IDC contains both radiology and digital pathology images. The following publication can serve as a good introduction into the use of DICOM for digital pathology. > Herrmann, M. D., Clunie, D. A., Fedorov, A., Doyle, S. W., Pieper, S., Klepeis, V., Le, L. P., Mutter, G. L., Milstone, D. S., Schultz, T. J., Kikinis, R., Kotecha, G. K., Hwang, D. H., Andriole, K. P., John Lafrate, A., Brink, J. A., Boland, G. W., Dreyer, K. J., Michalski, M., Golden, J. A., Louis, D. N. & Lennerz, J. K. Implementing the DICOM standard for digital pathology. J. Pathol. Inform. **9,** 37 (2018). [PubMed](https://pubmed.ncbi.nlm.nih.gov/30533276/) Open source libraries such as DCMTK, GDCM, ITK, and pydicom can be used to parse such files and load pixel data of the individual slices. Recovering geometry of the individual slices (spatial location and resolution) and reconstruction of the individual slices into a volume requires some extra consideration. {% hint style="info" %} You can use one of the existing tools to reconstruct image volume instead of implementing sorting of the slices on your own: {% endhint %} * [dcm2niix](https://github.com/rordenlab/dcm2niix): command-line tool to convert neuroimaging data from the DICOM format to the NIfTI format * [Plastimatch](https://plastimatch.org/): open source software for image computation, which includes [command-line file format converter](https://plastimatch.org/plastimatch.html) * [dicom2nifti](https://github.com/icometrix/dicom2nifti): python library providing API and command-line tools for converting DICOM images into NIfTI format * [SimpleITK](https://simpleitk.readthedocs.io/en/master/index.html): python interface to the [Insight Toolkit](https://github.com/InsightSoftwareConsortium/ITK), includes [DICOM read functionality](https://simpleitk.readthedocs.io/en/master/link_DicomSeriesReader_docs.html). # Derived objects {% hint style="info" %} Make sure you complete IDC ["Getting started" tutorial](https://github.com/ImagingDataCommons/IDC-Tutorials/tree/master/notebooks/getting_started) notebooks to get introduced into IDC data organization, download, visualization and other first-order topics. {% endhint %} In this section we discuss derived DICOM objects, including annotations, that are stored in IDC. It is important to recognize that, in practice, annotations are often shared in non-standard formats. When IDC ingests a dataset where annotations are available in such a non-standard representation, those need to be harmonized into a suitable DICOM object to be available in IDC. Due to the complexity of this task, we are unable to perform such harmonization for all of the datasets. If you want to check if there are annotations in non-DICOM format available for a given collection, you should locate the original source of the data, and examine the accompanying documentation for available non-DICOM annotations. As an example, the [Breast-Cancer-Screening-DBT](https://portal.imaging.datacommons.cancer.gov/explore/filters/?collection_id=Community\&collection_id=breast_cancer_screening_dbt) collection is available in IDC. If you mouse over the name of that collection in the IDC Portal, the tooltip will provide the overview of the collection and the link to the source. ![](/files/xHL2xIKSzH0CbRV07Sqq) You will also find the link to the source in the [list of collections available in IDC](https://portal.imaging.datacommons.cancer.gov/collections/). ![](/files/ypsqj54SZF2JyFW0gISH) Finally, if you select data using SQL, you can use the `source_DOI` and/or the source\_URL column to identify the source of each file in the subset you selected (learn more about `source_DOI`, licenses and attribution in the part 3 of our [Getting started tutorial](https://github.com/ImagingDataCommons/IDC-Tutorials/tree/master/notebooks/getting_started)). For the collection in question, the source DOI is , and on examining that page you will see a pointer to the CSV file with the coordinates of the bounding boxes defining regions containing lesions. Non-standard annotations are not searchable, usually are not possible to visualize in off-the-shelf tools, and require custom code to interpret and parse. The situation is different for the DICOM derived objects that we discuss in the following sections. ### DICOM derived objects In IDC we define "derived" DICOM objects as those that are obtained by analyzing or post-processing the "original" image objects. Examples of derived objects can be annotations of the images to define image regions, or to describe findings about those regions, or voxel-wise parametric maps calculated for the original images. Although DICOM standard provides a variety of mechanisms that can be used to store specific types of derived objects, most of the image-derived objects currently stored in IDC fall into the following categories: * voxel segmentations stored as [DICOM Segmentation objects (SEG)](/dicom/derived-objects/dicom-segmentations) * segmentations defined as a set of planar regions stored as [DICOM Radiotherapy Structure Set objects (RTSTRUCT)](#radiotherapy-structure-sets) * quantitative measurements and qualitative evaluations for the regions defined by DICOM Segmentations, those will be stored as a specific type of [DICOM Structured Reporting (SR)](/dicom/derived-objects/dicom-structured-reports) objects that follows DICOM SR template [TID 1500 "Measurements report"](http://dicom.nema.org/medical/dicom/current/output/chtml/part16/chapter_A.html#sect_TID_1500) (SR-TID1500) The type of the object is defined by the object class unique identifier stored in the `SOPClassUID` attribute of each DICOM object. In the IDC Portal we allow the user to define the search filter based on the human-readable name of the class instead of the value of that identifier. You can find detailed descriptions of these objects applied to specific datasets in TICA in the following open access publications: > Fedorov, A., Clunie, D., Ulrich, E., Bauer, C., Wahle, A., Brown, B., Onken, M., Riesmeier, J., Pieper, S., Kikinis, R., Buatti, J. & Beichel, R. R. DICOM for quantitative imaging biomarker development: a standards based approach to sharing clinical data and structured PET/CT analysis results in head and neck cancer research. PeerJ **4,** e2057 (2016). > > Fedorov, A., Hancock, M., Clunie, D., Brochhausen, M., Bona, J., Kirby, J., Freymann, J., Pieper, S., J W L Aerts, H., Kikinis, R. & Prior, F. DICOM re-encoding of volumetrically annotated Lung Imaging Database Consortium (LIDC) nodules. Med. Phys. (2020). [doi:10.1002/mp.14445](https://pubmed.ncbi.nlm.nih.gov/32772385/) # DICOM Segmentations DICOM Segmentation object (SEG) can be identified by `SOPClassUID`= `1.2.840.10008.5.1.4.1.1.66.4` Unlike most "original" image objects that you will find in IDC, SEG belongs to the family of *enhanced multiframe* image objects, which means that it stores all of the frames (slices) in a single object. SEG can contain multiple *segments*, a segment being a separate label/entity being segmented, with each segment containing one or more frames (slices). All of the frames for all of the segments are stored in the `PixelData` attribute of the object. If you use the IDC Portal, you can select cases that include SEG objects by selecting "Segmentations" in the "Modality" section ([filter link](https://portal.imaging.datacommons.cancer.gov/explore/filters/?Modality_op=OR\&Modality=SEG)) under the "Original" tab . Here is [a sample study](https://viewer.imaging.datacommons.cancer.gov/viewer/1.3.6.1.4.1.14519.5.2.1.7311.5101.170561193612723093192571245493?seriesInstanceUID=1.3.6.1.4.1.14519.5.2.1.7311.5101.206828891270520544417996275680,1.2.276.0.7230010.3.1.3.1070885483.15960.1599120307.701) that contains a SEG series. You can further explore segmentations available in IDC via the "Derived" tab of the Portal by filtering those by specific types and anatomic locations. As an example, [this filter](https://portal.imaging.datacommons.cancer.gov/explore/filters/?SegmentedPropertyTypeCodeSequence=27925004) will select cases that contain segmentations of a nodule. ```sql # get the viewer URL for a random study that # contains SEG modality SELECT ANY_VALUE(CONCAT("https://viewer.imaging.datacommons.cancer.gov/viewer/", StudyInstanceUID)) as viewer_url FROM `bigquery-public-data.idc_current.dicom_all` WHERE StudyInstanceUID IN ( # select a random DICOM study that includes a SEG object SELECT StudyInstanceUID FROM `bigquery-public-data.idc_current.dicom_all` WHERE SOPClassUID = "1.2.840.10008.5.1.4.1.1.66.4" ORDER BY RAND() LIMIT 1) ``` ### Metadata Metadata describing the segments is contained in the `SegmentSequence` of the DICOM object, and is also available in the BigQuery table view maintained by IDC in the [`bigquery-public-data.idc_current.segmentations`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_current\&t=segmentations\&page=table) BigQuery table. That table contains one row per segment, and for each segment includes metadata such as algorithm type and structure segmented. ### Conversion into alternative representations We recommend you use one of the following tools to interpret the content of the DICOM SEG and convert it into alternative representations: * [dcmqi](https://github.com/QIICR/dcmqi): open source DCMTK-based C++ library and command line converters that aim to help with the conversion between imaging research formats and the standard DICOM representation for image analysis results * [highdicom](https://github.com/MGHComputationalPathology/highdicom): high-level DICOM abstractions for the Python programming language * [DCMTK](https://dicom.offis.de/dcmtk.php.en): C++ library that provides API abstractions for reading and writing SEG objects Tools referenced above can be used to 1) extract volumetrically reconstructed mask images corresponding to the individual segments stored in DICOM SEG; 2) extract segment-specific metadata describing its content; 3) generate standard-compliant DICOM SEG objects from research formats. {% hint style="info" %} SEG-specific metadata attributes are available in the table views maintained by IDC. See details [here](/data/organization-of-data/bigquery-tables). {% endhint %} # DICOM Radiotherapy Structure Sets DICOM Radiotherapy Structure Sets (RTSS, or RTSTRUCT) define regions of interest by a set of planar contours. RTSS objects can be identified by the `RTSTRUCT` value assigned to the `Modality` attribute, or by `SOPClassUID` = `1.2.840.10008.5.1.4.1.1.481.3`. If you use the IDC Portal, you can select cases that include RTSTRUCT objects by selecting "Radiotherapy Structure Set" in the "Original" tab, "Modality" section ([filter link](https://portal.imaging.datacommons.cancer.gov/explore/filters/?Modality_op=OR\&Modality=RTSTRUCT)). Here is [a sample study](https://viewer.imaging.datacommons.cancer.gov/viewer/1.3.6.1.4.1.14519.5.2.1.77129820837198295687264213920427872580) that contains an RTSS series. As always, you get most of the power in exploring IDC metadata when using SQL interface. As an example, the query below will select a random study that contains a RTSTRUCT series, and return a URL to open that study in the viewer: ```sql # get the viewer URL for a random study that # contains RTSTRUCT modality SELECT ANY_VALUE(CONCAT("https://viewer.imaging.datacommons.cancer.gov/viewer/", StudyInstanceUID)) as viewer_url FROM `bigquery-public-data.idc_current.dicom_all` WHERE StudyInstanceUID IN ( # select a random DICOM study that includes an RTSTRUCT object SELECT StudyInstanceUID FROM `bigquery-public-data.idc_current.dicom_all` WHERE SOPClassUID = "1.2.840.10008.5.1.4.1.1.481.3" ORDER BY RAND() LIMIT 1) ``` ### Metadata RTSTRUCT relies on unstructured text in describing the semantics of the individual regions segmented. This information is stored in the `StructureSetROISequence.ROIName` attribute. The following query will return the list of all distinct values of `ROIName` and their frequency. ```sql SELECT structureSetROISequence.ROIName AS ROIName, COUNT(DISTINCT(SeriesInstanceUID)) AS ROISeriesCount FROM `bigquery-public-data.idc_current.dicom_all` CROSS JOIN UNNEST (StructureSetROISequence) AS structureSetROISequence WHERE SOPClassUID = "1.2.840.10008.5.1.4.1.1.481.3" GROUP BY ROIName ORDER BY ROISeriesCount DESC ``` ### Conversion into alternative representations We recommend [Plastimatch convert](https://plastimatch.org/plastimatch.html#plastimatch-convert) tool for converting planar contours of the individual structure sets into volumetric representation. # DICOM Structured Reports DICOM SR uses data elements to encode a higher level abstraction that is a tree of content, where nodes of the tree and their relationships are formalized. SR-TID1500 is one of many standard templates that define constraints on the structure of the tree, and is intended for generic tasks involving image-based measurements. DICOM SR uses standard terminologies and codes to deliver structured content. These codes are used for defining both the concept names and values assigned to those concepts (name-value pairs). Measurements include coded concepts corresponding to the quantity being measured, and a numeric value accompanied by coded units. Coded categorical or qualitative values may also be present. In SR-TID1500, measurements are accompanied by additional context that helps interpret and reuse that measurement, such as finding type, location, method and derivation. Measurements computed from segmentations can reference the segmentation defining the region and the image segmented, using unique identifiers of the respective objects. {% hint style="warning" %} At this time, only the measurements that accompany regions of interest defined by segmentations are exposed in the IDC Portal, and in the measurements views maintained by IDC! {% endhint %} Open source DCMTK [dsr2html](https://support.dcmtk.org/docs/dsr2html.html) tool can be used to render the content of the DICOM SR tree in a human-readable form (you can see one example of such rendering [here](https://peerj.com/articles/2057/#fig-7)). Reconstructing this content using tools that operate with DICOM content at the level of individual attributes can be tedious. We recommend the tools referenced above that also provide capabilities for reading and writing SR-TID1500 content: * [highdicom](https://github.com/MGHComputationalPathology/highdicom): high-level DICOM abstractions for the Python programming language * [dcmqi](https://github.com/QIICR/dcmqi): open source DCMTK-based C++ library and command line converters that aim to help with the conversion between imaging research formats and the standard DICOM representation for image analysis results * [DCMTK](https://dicom.offis.de/dcmtk.php.en): C++ library that provides API abstractions for reading and writing SR-TID1500 documents Tools referenced above can be used to 1) extract qualitative evaluations and quantitative measurements fro the SR-TID1500 document; 2) generate standard-compliant SR-TID1500 objects. {% hint style="info" %} SR-TID1500-specific metadata attributes are available in the table views maintained by IDC. See details [here](/data/organization-of-data/bigquery-tables). {% endhint %} # Coding schemes One of the fundamental principles of DICOM is the use of *controlled terminologies*, or *lexicons*, or *coding schemes* (for the purposes of this guide, these can be used interchangeably). While using the DICOM data stored in IDC, you will encounter various situations where the data is captured using coded terms. Controlled terminologies define a set of codes, and sometimes their relationships, that are carefully curated to describe entities for a certain application domain. Consistent use of such terminologies helps with uniform data collection and is critical for harmonization of activities conducted by independent groups. When codes are used in DICOM, they are saved as triplets that consist of * CodeValue: unique identifier for a term * CodingSchemeDesignator: code for the authority that issued this code * CodeMeaning: human-readable code description DICOM relies on various sources of codes, all of which are listed in [PS3.16 Section 8](http://dicom.nema.org/medical/dicom/current/output/chtml/part16/chapter_8.html) of the standard. As an example, if you query the [`canceridc-data:idc_views.segmentations`](https://console.cloud.google.com/bigquery?project=canceridc-data\&p=canceridc-data\&d=idc_views\&t=segmentations\&page=table) view with the following query in the BQ console: ```sql SELECT * FROM `canceridc-data.idc_views.segmentations` LIMIT 10 ``` You will see columns that contain coded attributes of the segment. In the example below, the value of `AnatomicRegion` corresponding to the segment is assigned the value (T-04000, SRT, Breast), where "SRT" is the coding scheme designator corresponding to the [SNOMED-CT](http://www.snomed.org/) coding scheme. ![](/files/-MGnmmgy4y8dFAj5sJf7) As another example, quantitative and qualitative measurements extracted from the SR-TID1500 objects are stored in the [`canceridc-data:idc_views.quantitative_measurements`](https://console.cloud.google.com/bigquery?project=canceridc-data\&p=canceridc-data\&d=idc_views\&t=quantitative_measurements\&page=table) and [`canceridc-data:idc_views.qualitative_measurements`](https://console.cloud.google.com/bigquery?project=canceridc-data\&p=canceridc-data\&d=idc_views\&t=qualitative_measurements\&page=table) views, respectively. If we query those views to see the individual measurements, they also show up as coded items. Each of the quantitative measurements includes a code describing the quantity being measured, the actual numeric value, and a code describing the units of measurement: ![](/files/-MGnrCyj76z_BAIoBGoq) ![](/files/-MGnrHv-yuvS7YvEOOQy) # DICOM-TIFF dual personality files DICOM and TIFF are two different image file formats that share many similar characteristics, and are capable of encoding exactly the same pixel data, whether uncompressed, or compressed with common lossy schemes (including JPEG and JPEG 2000). This allow the pixel data to be losslessly transformed from one format to the other and back. The DICOM file format was also deliberately designed to allow the two formats (TIFF and DICOM) to peacefully co-exist in the same file, sharing the same pixel data without expanding the file size significantly. This is achieved by leaving some unused space at the front of the DICOM file ("preamble"), which allows for the presence of a TIFF format recognition code ("magic number") and a pointer to its Image File Directory (IFD), which in turn contains pointers into the shared DICOM Pixel Data element. The dual-personality mechanism supports both traditional strip-based TIFF organization, such as might be used to encode a single frame image, as well as the tile-based format, which is commonly used for Whole Slide Images (WSI), and which is encoded in DICOM with each tile as a frame of a "multi-frame" image. Unlike TIFF files, which allow multiple different sized images to be encoded in the same file, DICOM does not, so there are limits to this approach. For example, **though an entire WSI pyramid can be encoded in a TIFF file, the DICOM WSI definition requires each pyramid layer to be in a separate file, and all frames (tiles) within the same file to be the same size**. Most of the structural metadata that describes the organization and encoding of the pixel data is similar in DICOM and TIFF. It is copied into the tags (data elements) encoded in the respective format "headers". Biomedical-specific information, such as patient, specimen and anatomical identifiers and descriptions, as well as acquisition technique, is generally only encoded in the DICOM data elements, their being no corresponding standard TIFF tags for it. Limited spatial information (such as physical pixel size) can be encoded in TIFF tags, but more complex multi-dimensional spatial location is standardized only in the DICOM data elements. The dictionary of TIFF tags can be extended with application-specific entries. This has been done for various non-medical and medical applications (e.g., GeoTIFF, DNG, DEFF). Other tools have used alternative mechanisms, such as defining text string (Leica/Aperio SVS) or structured metadata in other formats (such as XML for OME) buried within a TIFF string tag (e.g, ImageDescription). This approach can be used with DICOM-TIFF dual-personality files as well, since DICOM does not restrict the content of the TIFF tags; it does require updating or crafting of the textual metadata to actually reflect the characteristics of the encoded pixel data. It is hoped that the dual-personality approach may serve to mitigate the impact of limited support of one format or the other in different clinical and research tools for acquisition, analysis, storage, indexing, distribution, viewing and annotation. {% hint style="info" %} For further information and an example open source implementation, see Clunie, D. A. [Dual-Personality DICOM-TIFF for whole slide images: A migration technique for legacy software](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6489422/). J. Pathol. Inform. **10,** 12 (2019). {% endhint %} # IDC DICOM white papers The following white papers are intended to provide explanation and clarification into applying DICOM to encoding specific types of data. Comments and questions regarding those white papers are welcomed from the community! Please ask any related questions on [IDC Discourse](https://discourse.canceridc.dev), or by adding comments directly in the documents referenced below: * [DICOM SR for communicating planar annotations](https://docs.google.com/document/d/1bR6m7foTCzofoZKeIRN5YreBrkjgMcBfNA7r9wXEGR4/edit?usp=sharing), 2020 # Getting started ## Overview The[ Imaging Data Commons Portal ](https://portal.imaging.datacommons.cancer.gov/)provides a web-based interactive interface to browse the data hosted by IDC, visualize images, build manifests describing selected cohorts, and download images defined by the manifests. The slides below give a quick guided overview of how you can use IDC Portal. {% hint style="success" %} **No login is required to use the portal, to visualize images, or to download data from IDC!** {% endhint %} {% embed url="" %} # Exploring and subsetting data
## Overview
Components on the left side of the page give you controls for configuring your selection: * **Search scope** allows you to limit your search to just the specific programs, collections and analysis results (as discussed in the documentation of the [IDC Data model](/data/data-model)). * **Search configuration** gives you access to a small set of metadata attributes to select DICOM studies (where "DICOM studies" fit into IDC data model is also discussed in the [IDC Data model](/data/data-model) page) that contain data that meets the search criteria. Panels on the right side will automatically update based on what you select on the left side! * **Selection configuration** reflects the active search scope/filters in the **Cohort Filters** section. You can download all of the studies that match your filters. Below you will see the **Cart** section. Cart is helpful when selecting data by individual filters is too imprecise, and you want to have more granular control over your selection by selecting specific collections/patients/studies/series. * **Filtering results** section consists of the tables containing matching content that you can navigate following IDC Data model: first table shows the matching collections, selecting a collection will list matching cases (patients), selection of a case will populate the next table listing matching studies for the patient, and finally selecting a study will expand the final table with the list of series included in the study. In the following sections of the documentation you will learn more about each of the items we just discussed. # Configuring your search

Note how Cohort Filters section is updated as your selection changes

### **Noteworthy details about the search scope and configuration** * Numbers in the grayed ovals next to the search filters indicate the **total number of cases** (patients) that have the specific attribute
* Click on the "i" button to **toggle information panel** about the individual items in the search panels
* Cohort filters panel: get the **shareable URL** for the current selection by clicking "URL" button in the Cohort Filters panel
* Get the manifest for downloading all of the matching studies by clicking "Manifest" button in the Cohort Filters panel # Exploring search results

Drill down the results of the selection by expanding the content of the individual collections, cases and studies

Search results are updated dynamically based on the search configuration. At any time you can expand the items on the right to explore the selected collections, cases, studies and series. ### **Noteworthy details about navigating search results** * Studies and series tables include the button to open those in the browser-based **image viewer**.
* You can copy identifiers of the individual collections, cases, studies or series to the clipboard - those can be used to download corresponding files as discussed in the [Downloading data](/data/downloading-data) section - using command-line download tool or 3D Slicer IDC extension
# Data selection and download IDC Portal offers lots of flexibility in selecting items to download. In all cases, download of data from IDC Portal is a two step process: 1. Select items and export a manifest corresponding to your selection. 2. Use command-line python tool or 3D Slicer IDC browser extension to download the files for your selection, as discussed in [Downloading data](/data/downloading-data) . {% hint style="success" %} "IDC manifest" is a text file that contains URLs to the files in cloud buckets that correspond to your selection. It will contain one line for each DICOM series, as IDC files are organized in series-level folders in the cloud storage. {% endhint %} ## Downloading content using Cart You will see "Cart" icon in the search results collections/cases/studies/series tables. Any of the items in these tables can be added to the cart for subsequent downloading of the corresponding files. Get the manifest for the cart content using "Manifest" button in the Cart panel.
## Downloading all of the files for the current search configuration Clicking "Manifest" button in the "Cohort Filters" panel will given you the manifest for all of the studies that match your current selection criteria.
## Download individual studies or series Studies table contains a button for downloading manifest that will contain references to the files in the given study. To download a single series, no manifest is needed. You will see the command line to run to do the download.
## Downloading images in the viewers If you would like to download the entire study, or the specific image you see in the image viewer, you can use the download button in the viewer interface.

Button to toggle download instructions in IDC radiology (OHIF v3) viewer

Button to toggle download instructions in IDC microscopy (Slim) viewer

# Manifests: selecting data subsets Manifest contains references that can be used to download the files corresponding to your data selection. You can also archive manifests of items you selected in the IDC Portal for subsequent retrieval, or for sharing with your collaborators or readers of your paper. You can create manifests using To export a cohort manifest, [Create a cohort](/portal/data-exploration-and-cohorts/understanding-cohorts) or click **Cohorts** on the top menu bar and select a cohort you previously created. Click the **Export Cohort Manifest** button: the Export Cohort Manifest dialog box appears. There are three options for exporting the cohort manifest, which serve different needs: 1. **`s5cmd` manifest**: this option is recommended if you want to download the files included in the cohort. You can choose the source of data while downloading the files between Google GCP and Amazon AWS buckets, depending on your needs (in either case, data egress is free). 2. **CSV/TSV/JSON**: use this option if you want to archive the manifest of your cohort. The information contained in this manifest will be sufficient to access the content cohort even after the IDC data is updated. 3. **BigQuery**: this option is most appropriate if you want to further refine your cohort using SQL interface. ## s5cmd cohort manifest `s5cmd` manifest lists the commands needed to copy the files corresponding to the cohort using `s5cmd run` command from either GCP or AWS buckets (based on user preference). The header of the manifest includes the instructions to download the content of the cohort. {% code overflow="wrap" %} ``` # To download the files in this manifest, first install s5cmd (https://github.com/peak/s5cmd), # then run the following command: # s5cmd --no-sign-request --endpoint-url https://storage.googleapis.com run cohorts_996_20230504_80755_gcs.s5cmd cp s3://idc-open-data/6079cc3b-4b0f-41ec-bdbb-c6e754e88229/* . cp s3://idc-open-data/a1e77e5c-299f-49ef-a659-f4484e0cedd2/* . ... ``` {% endcode %} Note that if you chose to download the cohort content from AWS buckets, the manifest header will contain AWS-specific instructions, and the source bucket location will be in AWS, as shown below. You can see that folder names, defined by `crdc_series_uuid`, are the same as in the GCP manifest, but the bucket is different. {% code overflow="wrap" %} ``` # To download the files in this manifest, first install s5cmd (https://github.com/peak/s5cmd), # then run the following command: # s5cmd --no-sign-request --endpoint-url https://s3.amazonaws.com run cohorts_996_20230504_80755_aws.s5cmd cp s3://idc-open-data/6079cc3b-4b0f-41ec-bdbb-c6e754e88229/* . cp s3://idc-open-data/a1e77e5c-299f-49ef-a659-f4484e0cedd2/* . ... ``` {% endcode %} ## CSV/TSV/JSON cohort manifest Comma-Separated Values (CSV) and Tab-Separated Values (TSV) files include header fields, so you can customize which of those fields you want to include in the export. You can also select which columns to include in the file. The header of the CSV/TSV manifest contains the name of cohort, user, filters used, the date generated, and the total number of records. Data is separated by multiple files until the limit of 65,000 files is reached. If your data set is larger than this, you must export it using BigQuery. ![Example cohort manifest](/files/-MO2Stk3oWiPmxHLFMVW) {% hint style="warning" %} Manifests exported as files are defined at the series level. There will be a single row for each DICOM series included in the manifest. This is due to the limitation on the maximum size of the file that can be exported. To download the individual DICOM files corresponding to the instances included in the series the CRDC UUID corresponding to the series object (`series_uuid`) will need to be [resolved to the URL of the underlying objects](/data/organization-of-data/guids-and-uuids). Manifests exported into BigQuery are defined at the level of DICOM instances, with one row per instance. {% endhint %} The fields provided in a cohort manifest are: * `PatientID`: value of the corresponding DICOM attribute * `collection_id`: abbreviated identifier of the source data collection * `StudyInstanceUID`: value of the corresponding DICOM attribute * `SeriesInstanceUID`: value of the corresponding DICOM attribute * `SOPInstanceUID`: value of the corresponding DICOM attribute (NB: included only when manifest is exported into BigQuery!) * `source_DOI`: Digital Object Identifier (DOI) of the source data collection. Pre-pending `source_DOI` with `https://doi.org/` will give you the URL of the collection dataset * `study_uuid`: CRDC UUID of the object maintained by CRDC IndexD corresponding to the DICOM study, which [can be resolved to the URL of the underlying objects](/data/organization-of-data/guids-and-uuids) * `series_uuid`: CRDC UUID of the object maintained by CRDC IndexD corresponding to the DICOM series, which [can be resolved to the URL of the underlying objects](/data/organization-of-data/guids-and-uuids) * `instance_uuid`: CRDC UUID of the object maintained by CRDC IndexD corresponding to the DICOM instance, which [can be resolved to the URL of the underlying objects](/data/organization-of-data/guids-and-uuids) (NB: included only when the manifest is exported into BigQuery!) * `gcs_url`: `gs://` URL that can be used to access the object using the [GCP `gsutil` tool](https://cloud.google.com/storage/docs/gsutil) An example of how you can use an IDC cohort manifest to retrieve the manifest-defined cohort files is shown in [colab notebooks](https://github.com/ImagingDataCommons/IDC-Examples/tree/master/notebooks). {% hint style="info" %} A multipart file export can have a maximum of ten files with 65,000 rows each. If your export is larger than this, you must export the manifest via BigQuery. {% endhint %} JSON files do not include any header information but like CSV and TSV files, require that you identify which columns to include in your export. You can import the JSON file to a BigQuery table for further analysis. ## BigQuery cohort manifest {% hint style="warning" %} You must have a Google account to use BigQuery. {% endhint %} Exporting a manifest to BigQuery allows you to run complex, analytical, SQL-based queries on large sets of data by combining metadata available in IDC BigQuery tables with the content of your manifest. After you start the export, the following window appears on the cohort details page, showing your unique link. ![](/files/-MR0cNQzryKP06fAQNRe) The exported cohort manifest table is intended for short-term use, and **will be deleted after seven days**. However, you can always re-export the cohort manifest. # Visualizing images IDC integrates two different viewers, which will be used depending on the type of images being opened. Visualization of radiology images uses the open-source [Open Health Imaging Foundation (OHIF) Viewer](https://github.com/OHIF/Viewers) v3. The [SliM Viewer](https://github.com/MGHComputationalPathology/slim) is used for visualization of pathology and slide microscopy images. We customized both of those viewers slightly to add features specific to IDC. You can find all of those modifications in the respective forks under the IDC GitHub organization for OHIF and SliM viewers: [OHIF Viewer fork](https://github.com/ImagingDataCommons/Viewers) and [SliM Viewer fork](https://github.com/ImagingDataCommons/slim). IDC Viewer is opened every time you click the "eye" icon in the study or series table of the IDC Portal. {% hint style="danger" %} **The OHIF and SliM viewers do not support 32 bit browsers.** {% endhint %} IDC Viewer is a "zero-footprint" client-side viewer: before you can see the image in the viewer, it has to be downloaded to your browser from the IDC DICOM stores. IDC Viewer communicates the data it receives through a proxy via the [DICOMweb](https://www.dicomstandard.org/using/dicomweb) interface implemented in GCP [Cloud Healthcare API](https://cloud.google.com/healthcare/docs/concepts/dicom). {% hint style="info" %} Currently, IDC Viewer proxy limits the amount of data that can be downloaded in one day to **137 GB per IP address**, and enforces a total quota per day over all of the IP addresses. If the quota is exhausted, you will not be able to see any images in IDC Viewer until the limit is reset and instead will be redirected to [this](https://portal.imaging.datacommons.cancer.gov/quota/index.html)[ page](https://portal.imaging.datacommons.cancer.gov/quota/index.html)! We may adjust the current proxy limits in the future, and you are welcome to provide your feedback on the appropriateness of the current quota in [IDC Discourse](https://discourse.canceridc.dev/c/support/feedback-and-features/7). {% endhint %} ## IDC radiology viewer functionality The main functions of the viewer are available via the toolbar controls shown below.
The functionality supported by those tools should be self-explanatory, or can be discovered via quick experimentation. {% hint style="warning" %} If you want to report a problem related to visualization of a specific study in the IDC Viewer, please use the "Debug Info" tool to collect debugging information. Please report the issue on the [IDC Discourse](https://discourse.canceridc.dev/c/support/feedback-and-features/7), including the entire content of the debugging information to help us investigate the issue. {% endhint %} ### Visualizing radiology annotations IDC Viewer supports visualization of annotations stored as DICOM Segmentation objects (SEG), DICOM Radiotherapy Structure Sets (RTSTRUCT), and certain annotations stored in DICOM TID1500 Structured Reports. When available in a given study, you will see those modalities labeled as such in the left-hand panel of the viewer, as shown below. To load, double-click on the corresponding thumbnail in the series list in the left panel. After that you can open the navigation panel in the upper right corner to jump to the locations of the specific structure sets or segments, and to control their individual visibility.
{% hint style="warning" %} Note that certain modalities, such as Segmentation (SEG) and Real World Value Mapping (RWVM) objects, cannot be selected for visualization from the IDC Portal. SEG can only be viewed in the context of the image series segmented, and RWVM series are not viewable and will not show up in the left panel of the viewer. {% endhint %} Below is an example of series objects that are not viewable at the series level. ![Selected Series panel showing series objects not viewable at the series level](/files/-MJhJ34E6l39jYyx4zoS) ## IDC pathology viewer functionality The IDC pathology viewer allows for interactive visualization of digital slide microscopy (SM) images. Left panel will show all digital slides available in a given study. Click on the thumbnail to open a specific slide. Right panel will summarize the information about slide image channels, and will list annotations, analysis results, and presentation states when available. IDC viewer support visualization of DICOM Segmentations (binary and fractional), Parametric Maps, planar annotations stored as DICOM TID1500 Structured Reports (SR modality) or bulk annotations (ANN modality). ### Visualizing slide microscopy annotations Whenever annotations or segmentations are available for the slide you opened, you will see the corresponding sections populated in the bottom-right portion of the window. Expand those to see what is available and to toggle visualization. {% hint style="warning" %} Note that sometime only small regions of the image are annotated. You can double-click on the specific annotation group in the "Annotation Groups" section to zoom into the section of the image where that group is located! {% endhint %}
## Configuring the IDC Viewer URL You can use IDC Viewer to visualize any of the suitable data in IDC. To configure the IDC Viewer URL, simply append `StudyInstanceUID` of a study available in IDC to the following prefix: (for the radiology viewer) and[ https://viewer.imaging.datacommons.cancer.gov/slim/studies](https://viewer.imaging.datacommons.cancer.gov/slim/studies/)/ (for the digital pathology viewer). This will open the entire study in the viewer. You can also configure the URL to open specific series of the study, as defined by the list of `SeriesInstanceUID` items. When you open the IDC Viewer from the IDC Portal, the URLs of the pages will be populated following those conventions. Here are some specific examples, taken from the IDC Portal dashboard: * open entire study with the `StudyInstanceUID`1.3.6.1.4.1.14519.5.2.1.6279.6001.224985459390356936417021464571: [https://viewer.imaging.datacommons.cancer.gov/viewer/1.3.6.1.4.1.14519.5.2.1.6279.6001.224985459390356936417021464571](https://viewer.imaging.datacommons.cancer.gov/viewer/1.3.6.1.4.1.14519.5.2.1.6279.6001.224985459390356936417021464571?seriesInstanceUID=1.2.276.0.7230010.3.1.3.0.57823.1553343864.578877,1.3.6.1.4.1.14519.5.2.1.6279.6001.273525289046256012743471155680). * open the specified subset of series from the study above: Digital pathology viewer uses a slightly different convention, as should be evident from this example URL: ## Deploying your own viewer You can share the viewer URLs if you want to refer to visualizations of the specific items from IDC. You can also use this functionality if you want to visualize specific items from your notebook or a custom dashboard (e.g., a Google DataStudio dashboard). If you want to visualize your own images, or if you would like to combine IDC images with the analysis results or annotations you generated, you do have several options: * You can use Google FireCloud to deploy [OHIF](https://github.com/OHIF/Viewers) v2 radiology or [Slim](https://github.com/ImagingDataCommons/slim) microscopy viewers as web applications, without having to use virtual machines or docker, and for free! * [OHIF FireCloud deployment tutorial](https://tinyurl.com/idc-ohif-gcp) * [Slim FireCloud deployment tutorial](https://tinyurl.com/idc-slim-gcp) * If you want to visualize images inside a Colab/Jupyter notebook - you can use [itkWidgets](https://github.com/InsightSoftwareConsortium/itkwidgets) - details in [this tutorial](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/part3_exploring_cohorts.ipynb) * You can use open source [VolView](https://volview.kitware.com/) zero-footprint viewer to visualize and volume render any image series by simply pointing it to the cloud bucket with the files - see details in [this tutorial](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/part3_exploring_cohorts.ipynb) # Proxy policy {% hint style="info" %} TL;DR: if you want to download images from IDC, you can do it without charge, limits or sign-ins from our cloud storage buckets. See instructions in [this article](/data/downloading-data). {% endhint %} The primary mechanism for accessing data from IDC is by searching the metadata using the idc-index python package or BigQuery tables, and downloading the binary files from public cloud buckets, as discussed in [this article](/data/downloading-data). There is no limit, quota or fees associated with downloading IDC files from the buckets. Effective March 2024, as a pilot project, IDC also provides access to the DICOM data via the DICOMweb interface available at this endpoint: . This endpoint is read-only. It will route the requests to the Google Healthcare API DICOM store containing IDC data. Our DICOMWeb endpoint should only be used when data access needs cannot be satisfied using other mechanisms (e.g., when accessing individual frames of the microscopy images without having to download the entire binary file). Egress of data via the DICOMweb interface is capped at a non-disclosed limit that is tracked per IP. It is not acceptable to “IP hop” in an attempt to circumvent individual daily quotas, since there is also a global daily cap as well to prevent full egress of the imaging collection. Note that if this global cap is hit, all other users of the site would be unable to use the viewers for the rest of the day (using the UTC clock). Thus, IP hopping against the proxy that causes the global quota to be hit will be considered a denial-of-service attack. If you reach your daily quota, but feel you have a compelling cancer imaging research use case to request an exception to the policy and an increase in your daily quota, please reach out to us at to discuss the situation. We are continuously monitoring the usage of the proxy. Depending on the actual costs and usage, this policy may be revisited in the future to restrict access via the DICOMweb interface for any uses other than IDC viewers.
# Portal release notes {% hint style="warning" %} The version of the portal is shown at the bottom of the portal page. The semantics of the version is the following: *canceridc.\.\*, where revision hash corresponds to that of the [IDC WebApp repo](https://github.com/ImagingDataCommons/IDC-WebApp). {% endhint %} **Portal release notes are maintained at** [**https://github.com/ImagingDataCommons/IDC-WebApp/releases**](https://github.com/ImagingDataCommons/IDC-WebApp/releases)**.** *** ## 18.0 April 9, 2024 (canceridc.202404091237.841769c) * on the Explore Images page the IDC internal id for each collection can now be copied from the Collections table by clicking the corresponding copy icon * on the Explore Images page the IDC case id can now be copied from the Selected Cases table by clicking the corresponding copy icon ## 17.0 December 19, 2023 (canceridc.202312191019.53f66ba) Main highlights of this release include: * add a choice of several viewers (OHIF v2, OHIF v3, VolView, Slim) for viewing image files ## 16.0 September 15, 2023 (canceridc.202309151254.a6dfd6a) Main highlights of this release include: * s5cmd file manifests can now be generated from the Explore images page for individual studies and series ## 15.0 July 14, 2023 (canceridc.202307141313.c80a691) Main highlights of this release include: * The file manifest for a filter can be downloaded without logging into the portal and creating a persistent cohort ## 14.0 May 3, 2023 (canceridc.202305031458.443ea83) Main highlights of this release include: * The Export Cohort Manifest popup now includes options to download manifests that can be used by [s5cmd](https://github.com/peak/s5cmd) to download image files from IDC's s3 buckets in GCP or AWS. Instructions are provided for using s5cmd with these manifests ## 13.0 March 7, 2023 (canceridc.202303071044.57def9a) Main highlights of this release include: * Three new Original Image attributes Max Total Pixel Matrix Columns, Max Total Pixel Matrix Rows, and Min Pixel Spacing are added. * Two new Quantitative Analysis attributes Sphericity (Quant) and Volume of Mesh are added. * Default attribute value order is changed from alphanumeric (by value name) to value count. ## 12.0 - November 2, 2022 (canceridc.202211092039.87ca478) Main highlights of this release include: * As limited access collections have been removed from IDC, the portal is now simplified by removing the option of selecting different access levels. All collections in the portal are public. * A warning message appears on the cohort browser page when a user views a cohort that used the Access filter attribute. That attribute is no longer applied if the user migrates the cohort to the current version. * On the explorer page the reset button has been moved to improve viewability. ## 11.0 - September 8, 2022 (canceridc.202209081302.acb8ce3) This was primarily a data release. There were no significant changes to the portal. ## 10.0 - August 3, 2022 (canceridc.202208040944.6c798a2) Main highlights of this release include: * User control over how selection of multiple filter modalities defines the cohort. Previously when multiple modalities were selected the cohort would include the cases that had ANY of the selected modalities. Now the user can choose if the cohort includes the cases that contain ANY of the selected modaltiies or just those that have ALL of the selected modalities. ## 9.0 - May 19, 2022 (canceridc.202205191051) Main highlights of this release include: * Ability to select specific Analysis Results collections with segmentation and radiomic features * Text boxes added to the slider panels to allow the user to input upper and lower slider bounds * Pie chart tooltips updated to improve viewability ## 8.0 - April 4, 2022 (canceridc.202204050856.2920c81) Main highlights of this release include: * Eleven new collections added * Number of cases, studies, and series in a cohort are reported in the filter de finition * On the Exploration page the Access attribute is placed in the Search Scope * On the Exploration page users are warned when they create a cohort that includes Limited Access collections * Series Instance UID is reported in the Selected Series table ## 7.0 - February 7, 2022 (canceridc.202202071117.164252a) Main highlights of this release include: * The BigQuery query string corresponding to a cohort can now be displayed in user-readable format by pressing a button on either the cohort or cohort list pages * On the exploration page collections can now be sorted alphabetically or by the number of cases. Selected cases are ordered at the top of the collection list * Table rows can be selected by clicking anywhere within the row, not just on the checkbox * The BigQuery export cohort manifest includes the IDC data version as an optional column ## 6.0 - January 10, 2022 (canceridc.202201101504.eb0e309) Main highlights of this release include: * Collections which have limited access are now denoted as such in the Collection tab on the Exploration page * Links to image files belonging to limited collections have been removed from the Studies and Series tables on the Exploration page * The quota of image file data that can be served per user per day has been reduced from 137 to 40 GB ## 5.0 - December 9, 2021 (canceridc.202112091128.eb0e309) Main highlights of this release include: * New attributes including Manufacturer, Manufacturer Model Name, and Slice Thickness added * Checked attribute values are now shown at the top of the attribute value lists * Ability to search by CaseID added to the Selected Cases table * Ability to search by StudyID added to the Selected Studies table * Study Date added to the Studies Table * Changed the persistence of the StudyID tooltip in the tables so that the StudyID can be copied from the tooltip * Specific columns can now be selected in the BigQuery cohort export ## 2.1.0 - August 2021 (canceridc.202108261153.70f59e0) The Imaging Data Commons Explore Image Data portal is a platform that allows users to explore, filter, create cohorts, and view image studies and series using cutting-edge technology viewers. Main highlights of this release include: * Support for slide microscopy series from the CPTAC-LSCC and CPTAC-LUAD collections is now included. * The [Slim viewer](https://github.com/MGHComputationalPathology/slim) is now configured to view slide microscopy series * Search boxes are included for very attribute to search for specific attribute values by name. ## 2.0.0 - June 2021 - (canceridc.202106250849.876f912) The Imaging Data Commons Explore Image Data portal is a platform that allows users to explore, filter, create cohorts, and view image studies and series using cutting-edge technology viewers. Main highlights of this release include: * 112 data collections are now included * Cohort data version is reported * Cohort statistics - ie the number the cases, studies, and series per cohort are now reported * Mechanism included to update a version cohort * Species Attribute is included * Checkbox and plus/minus icons are now used to select table rows ## 1.3.0 - March 2021 (canceridc.202103011131.27ce3b3) The Imaging Data Commons Explore Image Data portal is a platform that allows users to explore, filter, create cohorts, and view image studies and series using cutting-edge technology viewers. Main highlights of this release include: * The user details page will no longer return a 500 error when selected * Sorting of studies panel is now active for all fields * Re-sending of an unreceived verification email is now more clearly explained. * IDC identity login header and column selection is disabled for the exportation of a cohort manifest to BigQuery * Detailed information panel added to efficiently describe why some pie charts have multiple facets even when a filter is selected * Cohort manifest export popup can be scrolled down * Use of Shift or Control (Command for Mac) selection of studies will now behave as expected: Shift-select for a contiguous series of rows, Control/Command-select for individual rows. * All filter selections are now sorted by alphabetical characters ## 1.2.0 - January 2021 (canceridc.202101111506.0a8af57) The Imaging Data Commons Explore Image Data portal is a platform that allows users to explore, filter, create cohorts, and view image studies and series using cutting-edge technology viewers. Main highlights of this release include: * Consistent number of files will be returned between the portal and BigQuery * When the user clicks a non-gov link a popup will appear * Cohort manifest export information now has clickable URLs to take you to the BigQuery console * Collections list displays by default 100 entries * Any empty search criteria is now highlighted in grey and no data will be listed * The user will no longer need to scroll to see search criteria in the left search configuration panel * Portal footer is now in compliance with NCI requirements * Check/uncheck in the collections panel added for collection TCGA ## 1.1.0 - December 2020 (canceridc.202012091728.674fff0) The Imaging Data Commons Explore Image Data portal is a platform that allows users to explore, filter, create cohorts, and view image studies and series using cutting-edge technology viewers. Main highlights of this release include: * Case-level table is added to the portal * Cohorts can now be exported into BigQuery tables using the Export Cohort Manifest button * Cohorts less than 650k rows can now be downloaded as a multipart file. Cohorts larger that 600k rows can only be be exported to BigQuery (for users that are logged in with Google Accounts) * Quantitative filter ranges are updated dynamically with the updates to filter selection * Pie charts will display "No data available" message when zero cases are returned for the given filter selection * RTPLAN and Real World Mapping Attribute values are now disabled at the series level, since they cannot be visualized in the IDC Viewer * Various bug fixes in both the IDC Portal and IDC Viewer ## 1.0.0 - October 2020 (canceridc.202010190226.4e8597) The Imaging Data Commons Explore Image Data portal is a platform that allows users to explore, filter, create cohorts, and view image studies and series using cutting-edge technology viewers. Main features in this initial release include: * The ability to search for data in BigQuery and Solr * The ability to search by multiple attributes: * Collection * Original attributes e.g., Modality * Derived attributes e.g., Segmentations * Qualitative analysis e.g., Lobular pattern * Quantitative analysis e.g., Volume * related attributes e.g., Country * Display of collections results in a tabular format with the following information: * Collection Name * Total Number of Cases * Number of Cases(this cohort) * Display of the Selected Studies results in tabular format with the following information: * Project Name * Case ID * Study ID * Study Description * Display of the Selected Series results in tabular format with the following information: * Study ID * Series Number * Modality * Body Part Examined * Series Description * The ability to hide attributes with zero cases present * The ability to save cohorts * The ability to download the manifest of any cohort created * The ability to promote, filter, and load multiple series instances in the OHIF viewer # Viewer release notes {% hint style="danger" %} The version of the viewer is available from the "About" menu for the OHIF (radiology) viewer, and "Get app info" menu for the Slim (pathology) viewers. Both of those menus are in the upper right corner of the window. radiology viewer version locationpathology viewer version location {% endhint %} IDC viewers release notes are maintained via GitHub Releases. Once you identified the version of the deployed viewer using the instructions above, you can locate the corresponding release notes in the following: * **OHIF v3** release notes: * **Slim** release notes: * a significant portion of functionality available in Slim is implemented in the **dicom-microscopy-viewer** package, its releases are available here: *** ## v2-legacy - 8 Jun 2023 - OHIF The final OHIF v2 published version is 4.12.45. Upstream changes based on v2 will be accessible through the v2-legacy branch (will not be published to NPM). Main highlights from v2-legacy since 4.12.45: * Fix high and critical dependency issues reported by dependabot * Update SEG tolerance popup and update SEG thumbnail warning: Jump to first segment item image and show warning message only once on onChange events * Update to issues and PR templates * Address segmentation visibility toggle applied to all segmentations instead of the active one only * Update dcmjs version so it throws 'Failed to find the reference image in the source data. Cannot load this segmentation' error instead of logging a warning to console * Address eye icon for segment not shown when segment name is long * Change message for segmentation when it fails to load duo to orientation tolerance ## 4.12.45 - Oct 2022 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Handle missing ReferencedInstanceSequence attribute: Update parsing logic to consider attribute as optional. ## 4.12.44 - Oct 2022 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Remove unused code from DICOM SR parsing: Remove referecenedImages attribute from SR display sets. Within TID 1500, sub-template TID 1600 (Image Library) is not required while parsing SR for image references for annotations and planar measurements. The same is obtained from sub template TID 1501>TID 300>TID 320. ## 4.12.43 - Oct 2022 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Update message for segmentation error loading due to orientation tolerance ## 4.12.42 - Oct 2022 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Correct Parsing Logic for Qualitative Instance Level SR ## 4.12.41 - Sep 2022 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Fix 2d MPR rendering issue for the sagittal view ## 0.13.0 - April 2023 - Slim **Slim** is a lightweight server-less single-page application for interactive visualization of digital slide microscopy (SM) images and associated image annotations in standard DICOM format. The application is based on the [dicom-microscopy-viewer](https://github.com/ImagingDataCommons/dicom-microscopy-viewer) library and can simply be placed in front of a [DICOMweb](https://www.dicomstandard.org/using/dicomweb) compatible Image Management System (IMS), Picture Archiving and Communication (PACS), or Vendor Neutral Archive (VNA). New Features * Support configuration of multiple origin servers for different types of DICOM objects (SOP Storage Classes) Enhancements * Improved error handling * Check Pyramid UID (if available) when grouping images into digital slides Bug Fixes * Use Acquisition UID (if available) to group images into digital slides ## 0.11.2 - September 2022 - Slim **The Slim Viewer** is a lightweight server-less single-page application for interactive visualization of digital slide microscopy (SM) images and associated image annotations in standard DICOM format. The application is based on the [dicom-microscopy-viewer](https://github.com/MGHComputationalPathology/dicom-microscopy-viewer) library and can simply be placed in front of a [DICOMweb](https://www.dicomstandard.org/using/dicomweb) compatible Image Management System (IMS), Picture Archiving and Communication (PACS), or Vendor Neutral Archive (VNA). Main highlights of this release include: New features * Add new tool to go to specific slide position; * Show mouse position in slide coordinate system. Enhancements * Improve performance of translations between image and slide coordinates; * Automatically adjust size of overview image to size of browser window. Bug fixes * Fix rendering of label image; * Show error message when creation of viewer fails; * Fix resolution of overview image; * Fix styling of point annotations; * Ensure bounding box annotations are axis aligned; * Add missing keyboard shortcut for navigation tool. ## 4.12.40 - September 2022 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Fix parsing of qualitative slice annotation; * Disable measurements panel interactions in MPR mode; * Fix parsing of segmentation when orientation values are close to zero; * Raise error if a frame StudyInstanceUID, SeriesInstanceUID and SOPInstanceUID are not conforming with the UID (DICOM UI VR) character repertoire; * Implements runtime tolerance for SEGs loading retry; * Fixed popup notifications behavior; * Update cornerstoneWADOImageLoader. ## 0.8.1 - June 2022 - Slim **The Slim Viewer** is a lightweight server-less single-page application for interactive visualization of digital slide microscopy (SM) images and associated image annotations in standard DICOM format. The application is based on the [dicom-microscopy-viewer](https://github.com/MGHComputationalPathology/dicom-microscopy-viewer) library and can simply be placed in front of a [DICOMweb](https://www.dicomstandard.org/using/dicomweb) compatible Image Management System (IMS), Picture Archiving and Communication (PACS), or Vendor Neutral Archive (VNA). Main highlights of this release include: **New features** * Add panel for clinical trial information to case viewer; * Sort digital slides by Container Identifier attribute. **Enhancements** * Reset style of optical paths to default when deactivating presentation state. **Bug fixes** * Fix rendering of ROI annotations by upgrading to React version 1; * Correctly update UIDs of visible/active optical paths; * Fix type declarations of DICOMweb search resources. ## 4.12.30 - June 2022 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Add support for SR qualitative annotation per instance. ## 0.7.2 - June 2022 - Slim **The Slim Viewer** is a lightweight server-less single-page application for interactive visualization of digital slide microscopy (SM) images and associated image annotations in standard DICOM format. The application is based on the [dicom-microscopy-viewer](https://github.com/MGHComputationalPathology/dicom-microscopy-viewer) library and can simply be placed in front of a [DICOMweb](https://www.dicomstandard.org/using/dicomweb) compatible Image Management System (IMS), Picture Archiving and Communication (PACS), or Vendor Neutral Archive (VNA). Main highlights of this release include: **New features** * Support DICOM Advanced Blending Presentation State to parametrize the display of multiplexed IF microscopy images; * Add key bindings for annotations tools; * Enable configuration of tile preload; * Enable configuration of annotation geometry type per finding; * Expose equipment metadata in user interface. **Enhancements** * Improve default presentation of multiplexed IF microscopy images in the absence of presentation state instances; * Correctly configure DCM4CHEE Archive to use reverse proxy URL prefix for BulkDataURI in served metadata; * Enlarge display settings interfaces and add input fields for opacity, VOI limits, and colors; * Update dicom-microscopy-viewer version to use web workers for frame decoding/transformation operations; * Add button for user logout; * Disable optical path selection when a presentation state has been selected. **Bug fixes** * Fix parsing of URL path upon redirect after successful authentication/authorization; * Fix configuration of optical path display settings when switching between presentation states; * Fix caching of presentation states and for selection via drop-down menu. **Security** * Update dependencies with critical security issues. ## 0.5.1 - April 2022 - Slim **The Slim Viewer** is a lightweight server-less single-page application for interactive visualization of digital slide microscopy (SM) images and associated image annotations in standard DICOM format. The application is based on the [dicom-microscopy-viewer](https://github.com/MGHComputationalPathology/dicom-microscopy-viewer) library and can simply be placed in front of a [DICOMweb](https://www.dicomstandard.org/using/dicomweb) compatible Image Management System (IMS), Picture Archiving and Communication (PACS), or Vendor Neutral Archive (VNA). Main highlights of this release include: **Enhancements** * Make overview panel collapsible and hide it entirely if lowest-resolution image is too large. **Bug fixes** * Fix update of optical path settings when switching between slides. ## 4.12.26 - April 2022 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Fix regression in logic for finding segmentations referenced source image; * Fix segmentations loading issues; * Fix thumbnail series type for unsupported SOPClassUID; * Fix toolbar error when getDerivedDatasets finds no referenced series are found. ## 0.5.0 - March 2022 - Slim **The Slim Viewer** is a lightweight server-less single-page application for interactive visualization of digital slide microscopy (SM) images and associated image annotations in standard DICOM format. The application is based on the [dicom-microscopy-viewer](https://github.com/MGHComputationalPathology/dicom-microscopy-viewer) library and can simply be placed in front of a [DICOMweb](https://www.dicomstandard.org/using/dicomweb) compatible Image Management System (IMS), Picture Archiving and Communication (PACS), or Vendor Neutral Archive (VNA). Main highlights of this release include: **New features** * Display of analysis results stored as DICOM Segmentation, Parametric Map, or Microscopy Bulk Simple Annotations instances; * Dynamic selection of DICOMweb server by user (can be enabled by setting AppConfig.enableServerSelection to true); * Dark app mode for fluorescence microscopy (can be enabled by setting App.mode to "dark"); * Support display of segments stored in DICOM Segmentation instances; * Support display of parameter mappings stored in DICOM Parametric Map instances; * Support display of annotation groups stored in DICOM Microscopy Bulk Simple Annotations instances; * Implement color transformations using ICC Profiles to correct color images client side in a browser-independent manner; * Implement grayscale transformations using Palette Color Lookup Tables to pseudo-color grayscale images. **Improvements** * Unify handling of optical paths for color and grayscale images; * Add loading indicator; * Improve styling of overview map; * Render specimen metadata in compacter form; * Improve fetching of WASM library code; * Improve styling of slide viewer sidebar; * Sort slides by Series Number; * Work around common standard compliance issues; * Update docker-compose configuration; * Upgrade dependencies; * Show examples in README; * Decode JPEG, JPEG 2000, and JPEG-LS compressed image frames client side in a browser-independent manner; * Improve performance of transformation and rendering operations using WebGL for both grayscale as well as color images; * Optimize display of overview images and keep overview image fixed when zooming or panning volume images; * Optimize HTTP Accept header field for retrieval of frames to work around issues with various server implementations. **Bug fixes** * Ensure ROI annotations are re-rendered upon modification; * Clean up memory and recreate viewers upon page reload; * Fix selection of volume images; * Fix color space conversion during decoding of JPEG 2000 compressed image frames; * Fix unit of area measurements for ROI annotations; * Publish events when bulkdata loading starts and ends. ## 4.12.22 - March 2022 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Improve logic for finding segmentations referenced source image; * Improve debug dialog: fix text overflow and adding active viewports referenced SEGs and RTSTRUCT series. ## 4.12.17 - February 2022 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Fix fail to load SEG related to geometry assumptions; * Fix fail to load SEG related to tolerance; * Add initial support for SR planar annotations. ## 0.4.5 - January 2022 - Slim **The Slim Viewer** is a lightweight server-less single-page application for interactive visualization of digital slide microscopy (SM) images and associated image annotations in standard DICOM format. The application is based on the [dicom-microscopy-viewer](https://github.com/MGHComputationalPathology/dicom-microscopy-viewer) library and can simply be placed in front of a [DICOMweb](https://www.dicomstandard.org/using/dicomweb) compatible Image Management System (IMS), Picture Archiving and Communication (PACS), or Vendor Neutral Archive (VNA). Main highlights of this release include: **Bug fixes** * Fix selection of VOLUME or THUMBNAIL images with different Photometric Interpretation. ## 4.12.12 - January 2022 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Fix RTSTRUCT right panel updates; * Fix SEG loading regression. ## 4.12.7 - December 2021 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Fix handling of datasets with unsupported modalities; * Fix backward fetch of images for the current active series. * Fix tag browser slider. ## 0.4.3 - November 2021 - Slim **The Slim Viewer** is a lightweight server-less single-page application for interactive visualization of digital slide microscopy (SM) images and associated image annotations in standard DICOM format. The application is based on the [dicom-microscopy-viewer](https://github.com/MGHComputationalPathology/dicom-microscopy-viewer) library and can simply be placed in front of a [DICOMweb](https://www.dicomstandard.org/using/dicomweb) compatible Image Management System (IMS), Picture Archiving and Communication (PACS), or Vendor Neutral Archive (VNA). Main highlights of this release include: **Bug fixes** * Rotate box in overview map outlining the extent of the current view together with the image. ## 4.12.5 - November 2021 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Fix segmentation/rtstruct menu badge update when switching current displayed series; * Add to series thumbnail link icon if they are connected to any annotation (segmentation, etc...); * Fix problems opening series when the study includes many series; * Fix segments visibility handler. ## 0.4.1 - October 2021 - Slim **The Slim Viewer** is a lightweight server-less single-page application for interactive visualization of digital slide microscopy (SM) images and associated image annotations in standard DICOM format. The application is based on the [dicom-microscopy-viewer](https://github.com/MGHComputationalPathology/dicom-microscopy-viewer) library and can simply be placed in front of a [DICOMweb](https://www.dicomstandard.org/using/dicomweb) compatible Image Management System (IMS), Picture Archiving and Communication (PACS), or Vendor Neutral Archive (VNA). Main highlights of this release include: **Improvements** * Include images with new flavor THUMBNAIL in image pyramid; * Properly fit overview map into HTML element and disable re-centering of overview map when user navigates main map; * Allow drawing of ROIs that extent beyond the slide coordinate system (i.e., allow negative ROI coordinates). **Bug fixes** * Prevent display of annotation marker when ROI is deactivated ## 4.11.2 - October 2021 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Fix issues with segmentation orientations; * Fix display of inconsistencies warning for segmentation thumbnails; * Fix throttle thumbnail progress updates. ## 0.3.1 - September 2021 - Slim **The Slim Viewer** is a lightweight server-less single-page application for interactive visualization of digital slide microscopy (SM) images and associated image annotations in standard DICOM format. The application is based on the [dicom-microscopy-viewer](https://github.com/MGHComputationalPathology/dicom-microscopy-viewer) library and can simply be placed in front of a [DICOMweb](https://www.dicomstandard.org/using/dicomweb) compatible Image Management System (IMS), Picture Archiving and Communication (PACS), or Vendor Neutral Archive (VNA). Main highlights of this release include: **Bug fixes** * Set PUBLIC\_URL in Dockerfile. ## 0.3.0 - September 2021 - Slim **The Slim Viewer** is a lightweight server-less single-page application for interactive visualization of digital slide microscopy (SM) images and associated image annotations in standard DICOM format. The application is based on the [dicom-microscopy-viewer](https://github.com/MGHComputationalPathology/dicom-microscopy-viewer) library and can simply be placed in front of a [DICOMweb](https://www.dicomstandard.org/using/dicomweb) compatible Image Management System (IMS), Picture Archiving and Communication (PACS), or Vendor Neutral Archive (VNA). Main highlights of this release include: **Improvements** * Add button to display information about application and environment; * Add ability to include logo; * Verify content of SR documents before attempting to load annotations; * Improve re-direction after authentication; * Add retry logic and error handlers for DICOMweb requests; * Improve documentation of application configuration in README; * Add unit tests. **Bug fixes** * Disable zoom of overview map; * Fix pagination of worklist; * Prevent delay in tile rendering. ## 4.10.1 - September 2021 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Handle uncaught exception for non TID 1500 sr; * Added display of badge numbers in the segmentation / rtstruct panel tabs; * Study prefetcher with loading bar. ## 0.2.0 - August 2021 - Slim **The Slim Viewer** is a lightweight server-less single-page application for interactive visualization of digital slide microscopy (SM) images and associated image annotations in standard DICOM format. The application is based on the [dicom-microscopy-viewer](https://github.com/MGHComputationalPathology/dicom-microscopy-viewer) library and can simply be placed in front of a [DICOMweb](https://www.dicomstandard.org/using/dicomweb) compatible Image Management System (IMS), Picture Archiving and Communication (PACS), or Vendor Neutral Archive (VNA). Main highlights of this release include: **New features** * Support for multiplexed immunofluorescence slide microscopy imaging; * Client-side additive blending of multiple channels using WebGL; * Client-side decoding of compressed frame items using WebAssembly based on Emscripten ports of libjpeg-turbo, openjpeg, and charls C/C++ libraries. **Improvements** * Continuous integration testing pipeline using circle CI; * Deploy previews for manual regression testing. **Major changes** * Introduce new configuration parameter renderer. ## 4.9.20 - June 2021 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Add exponential backoff and retry after 500 error; * Update to HTML SR viewport to display missing header tags. ## 0.1.0 - May 2021 - Slim **The Slim Viewer** is a lightweight server-less single-page application for interactive visualization of digital slide microscopy (SM) images and associated image annotations in standard DICOM format. The application is based on the [dicom-microscopy-viewer](https://github.com/MGHComputationalPathology/dicom-microscopy-viewer) library and can simply be placed in front of a [DICOMweb](https://www.dicomstandard.org/using/dicomweb) compatible Image Management System (IMS), Picture Archiving and Communication (PACS), or Vendor Neutral Archive (VNA). Inital Release. ## 4.9.17 - May 2021 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Add disable server cache feature; * Additional improvements on series inconsistencies report UI. ## 4.9.13 - April 2021 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Add acquisition storage SR sopclass to SR html ext; * Fix missing items in the segmentation combobox items at loading; * Fix slices are not sorted in geometrical order; * Extend series inconsistencies checks to segmentation and improve UI. ## 4.9.7 - March 2021 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Add new log service to be used by debugger extension; * Add UI to communicate to the users inconsistencies within a single series; * Add time in the dates of the items of the segmentation combobox list; * Order segmentation combobox list in reverse time order; * Fix failure to load a valid SEG object because of incorrect expectations about ReferencedSegmentNumber; * Fix RSTRUCT menu visibility when loading a series; * Fix image load slowness regression; * Fix choppy scrolling in 2D mod; * Fix failure to load segmentations when filtering study with '?seriesInstanceUID=' syntax. ## 4.8.10 - February 2021 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Replace instance dropdown to slider for dicom tag browser; * Add error page and not found pages if failed to retrieve study data. ## 4.8.5 - January 2021 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Add UI error report for MPR buffer limit related errors; * Add UI error report for hardware acceleration turned off errors; * Add IDC funding acknowledgment; * Fix RSTRUCT menu panel undefined variables; * Fix RSTRUCT menu visibility when loading a series; * Fix segments visibility control (SEG menu) bugs . ## 4.8.0 - December 2020 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Visualize overlapping segments; * Use runtime value configuration to get pkg version; * Fix navigation issues in the crosshair tool. ## 4.5.22 - October 2020 - OHIF **The OHIF Viewer** is a zero-footprint medical image viewer provided by the [Open Health Imaging Foundation (OHIF)](http://ohif.org). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support [DICOMweb](https://www.dicomstandard.org/using/dicomweb). Main highlights of this release include: * Add MPR crosshair tool. # Portal known issues ## 1.2.0 - January 2021 (canceridc.202101111506.0a8af57) The Imaging Data Commons Explore Image Data portal is a platform that allows users to explore, filter, create cohorts, and view image studies and series using cutting-edge technology viewers. * Some filter selections are not saved for cohort creation after logging into the interface - [ImagingDataCommons/IDC-WebApp#519](https://github.com/ImagingDataCommons/IDC-WebApp/issues/519) * Entire page will scroll when using the cohort creation confirmation popup - [ImagingDataCommons/IDC-WebApp#514](https://github.com/ImagingDataCommons/IDC-WebApp/issues/514) * Error exporting cohort into BigQuery table with integer filter values - [ImagingDataCommons/IDC-WebApp#492](https://github.com/ImagingDataCommons/IDC-WebApp/issues/492) * 500 error returned in developers console, problem fetching data - [ImagingDataCommons/IDC-WebApp#462](https://github.com/ImagingDataCommons/IDC-WebApp/issues/462) * Age at diagnosis None selection is not listed on the cohort creation confirmation pop-up - [ImagingDataCommons/IDC-WebApp#420](https://github.com/ImagingDataCommons/IDC-WebApp/issues/420) * Capitalization of collection names should be consistent with TCIA conventions in all places - [ImagingDataCommons/IDC-WebApp#414](https://github.com/ImagingDataCommons/IDC-WebApp/issues/414) * "Show more" shows a negative number on occasion - [ImagingDataCommons/IDC-WebApp#374](https://github.com/ImagingDataCommons/IDC-WebApp/issues/374) * Duplicate entries under object class with the name "Digital Mammography X-Ray Image" - [ImagingDataCommons/IDC-WebApp/issues#233](https://github.com/ImagingDataCommons/IDC-WebApp/issues/233) * Series and Study Tables need to account for 'unfetched data' - [ImagingDataCommons/IDC-WebApp#186](https://github.com/ImagingDataCommons/IDC-WebApp/issues/186) * Clicking on the 2D MPR freezes the viewer when working on a Safari browser - [ImagingDataCommons/IDC-WebApp#213](https://github.com/ImagingDataCommons/IDC-WebApp/issues/213) ## 1.1.0 - December 2020 (canceridc.202012091728.674fff0) The Imaging Data Commons Explore Image Data portal is a platform that allows users to explore, filter, create cohorts, and view image studies and series using cutting-edge technology viewers. * Error exporting cohort into BigQuery table with integer filter values - [ImagingDataCommons/IDC-WebApp#492](https://github.com/ImagingDataCommons/IDC-WebApp/issues/492) * Series and Study Tables need to account for 'unfetched data' - [ImagingDataCommons/IDC-WebApp#186](https://github.com/ImagingDataCommons/IDC-WebApp/issues/186) * Clicking on the 2D MPR freezes the viewer when working on a Safari browser - [ImagingDataCommons/IDC-WebApp#213](https://github.com/ImagingDataCommons/IDC-WebApp/issues/213) * Duplicate entries under object class with the name "Digital Mammography X-Ray Image" - [ImagingDataCommons/IDC-WebApp/issues#233](https://github.com/ImagingDataCommons/IDC-WebApp/issues/233) * "Show more" shows a negative number on occasion - [ImagingDataCommons/IDC-WebApp#374](https://github.com/ImagingDataCommons/IDC-WebApp/issues/374) * Capitalization of collection names should be consistent with TCIA conventions in all places - [ImagingDataCommons/IDC-WebApp#414](https://github.com/ImagingDataCommons/IDC-WebApp/issues/414) * Age at diagnosis None selection is not listed on the cohort creation confirmation pop-up - [ImagingDataCommons/IDC-WebApp#420](https://github.com/ImagingDataCommons/IDC-WebApp/issues/420) * Check all / uncheck all button below collection list does not work - [ImagingDataCommons/IDC-WebApp#445](https://github.com/ImagingDataCommons/IDC-WebApp/issues/445) * The collection name is labeled Project Name in the studies section of the search results page - [ImagingDataCommons/IDC-WebApp#452](https://github.com/ImagingDataCommons/IDC-WebApp/issues/452) * 500 error returned in developers console, problem fetching data - [ImagingDataCommons/IDC-WebApp#462](https://github.com/ImagingDataCommons/IDC-WebApp/issues/462) * -1 as value range error for standardized added metabolic activity filter - [ImagingDataCommons/IDC-WebApp#493](https://github.com/ImagingDataCommons/IDC-WebApp/issues/493) ## 1.0.0 - October 2020 (canceridc.202010190226.4e8597) The Imaging Data Commons Explore Image Data portal is a platform that allows users to explore, filter, create cohorts, and view image studies and series using cutting-edge technology viewers. ### **Data Exploration** * Lack of consistent punctuation throughout the web app - [ImagingDataCommons/IDC-WebApp#414](https://github.com/ImagingDataCommons/IDC-WebApp/issues/414) * Attribute values appear as blank when zero cases are present for the Search Configuration panel - [ImagingDataCommons/IDC-WebApp#423](https://github.com/ImagingDataCommons/IDC-WebApp/issues/423) * An attribute selection combined with a Quantitative Analysis attribute selection will display all value options when no cases are available in certain use cases - [ImagingDataCommons/IDC-WebApp#341](https://github.com/ImagingDataCommons/IDC-WebApp/issues/341) * Attribute slider for Quantitative values will display with zero cases present in attribute selection - [ImagingDataCommons/IDC-WebApp#425](https://github.com/ImagingDataCommons/IDC-WebApp/issues/425) * The selection of the Age at Diagnosis None attribute is not reflected in the cohorts confirmation pop-up or the cohort details page - [ImagingDataCommons/IDC-WebApp#420](https://github.com/ImagingDataCommons/IDC-WebApp/issues/420) * The download of the cohort manifest is truncated at 65,000 records ordered by PatientID, StudyID, SeriesID, and InstanceID -[ ImagingDataCommons/IDC-WebApp#394](https://github.com/ImagingDataCommons/IDC-WebApp/issues/394) * Duplication of value Digital Mammography X-Ray Image for Object Class attribute present in the data portal under the original tab -[ ImagingDataCommons/IDC-WebApp#233](https://github.com/ImagingDataCommons/IDC-WebApp/issues/233) * Data summary ribbon needs to be updated to account for NSCLC-Radiomics - [ImagingDataCommons/IDC-WebApp#437](https://github.com/ImagingDataCommons/IDC-WebApp/issues/437) * Selection of a collection attribute on the no login required data exploration page doesn't save the selection after login - [ImagingDataCommons/IDC-WebApp#439](https://github.com/ImagingDataCommons/IDC-WebApp/issues/439) More detailed information can be found on GitHub under the ImagingDataCommons/Web-App repository. All tickets related to known issues within the data portal are labeled with either a [bug](https://github.com/ImagingDataCommons/IDC-WebApp/issues?q=is%3Aissue+is%3Aopen+label%3Abug) or an [enhancement](https://github.com/ImagingDataCommons/IDC-WebApp/issues?q=is%3Aissue+is%3Aopen+label%3Aenhancement) label. ### **Image Visualization** * The about box of the OHIF viewer reports the incorrect version number - [OHIF/Viewers#2023](https://github.com/OHIF/Viewers/issues/2023) * Selection of the Object class real-world mapping attribute will cause the OHIF viewer not to load * [ImagingDataCommons/IDC-WebApp#332](https://github.com/ImagingDataCommons/IDC-WebApp/issues/332) * When working with the Safari browser, the 2D MPR functionality causes the viewer to freeze. We recommend using Google Chrome or Firefox when working with the Imaging Data Commons - [ImagingDataCommons/IDC-WebApp#213](https://github.com/ImagingDataCommons/IDC-WebApp/issues/213) * Tag viewer version is currently broken for RT Struct series in the OHIF viewer - [OHIF/Viewers#2122](https://github.com/OHIF/Viewers/issues/2122) * The occasional return of a Chunk Load Error on the viewer load not handled correctly and will cause the viewer to fail - [OHIF/Viewers#2021](https://github.com/OHIF/Viewers/issues/2021) * The scroll bar is missing in the Tag viewer window - [OHIF/Viewers#2029](https://github.com/OHIF/Viewers/issues/2029) * When working with RT Structure sets, the RightSidePanel: Cannot read property 'SeriesInstanceUID' of undefined error returned in the viewer - [OHIF/Viewers#2123](https://github.com/OHIF/Viewers/issues/2123) * On occasion, the RTSTRUCT tab is shown when not applicable, does not load when opened - [OHIF/Viewers#2017](https://github.com/OHIF/Viewers/issues/2117) * The OHIF viewer doesn't load kidney and tumor plane segmentations found in TCIA - [OHIF/Viewers#1345](https://github.com/OHIF/Viewers/issues/1345) * Zoom does not Update Crosshairs in MPR Mode - [OHIF/Viewers#1419](https://github.com/OHIF/Viewers/issues/1419) * Unable to display the same scan in two viewports and displaying different RTSTRUCTs, SEGs, and cornerstone planar measurements in each - [OHIF/Viewers#1522](https://github.com/OHIF/Viewers/issues/1522) * SVG Crosshair issues in VTK - [OHIF/Viewers#1696](https://github.com/OHIF/Viewers/issues/1696) More detailed information can be found in GitHub within the OHIF/Viewers repository. All tickets related to the Imaging Data Commons are labeled with either an [IDC:candidate](https://github.com/OHIF/Viewers/labels/IDC%3Acandidate) or an [IDC:priority](https://github.com/OHIF/Viewers/labels/IDC%3Apriority) label. ### Black screen while opening IDC Viewer You may occasionally experience black screen while opening images in IDC Viewer, such as in the image below: ![](/files/-MLmYLnwNpL8D46IT3J4) If this happens, try reloading the page several times, or clearing cache and doing hard reload, as shown below. ![](/files/-MLt--0iUjiN-xIGmSZR) To track the status of this issue in GitHub please see [OHIF/Viewers#2121](https://github.com/OHIF/Viewers/issues/2121). # API overview The IDC API provides programmatic access to IDC metadata and data: discover what is in IDC, build cohorts with attribute filters, retrieve download manifests, run read-only SQL queries, explore clinical data, and generate citations and license summaries for your selections. The API is live in production at: ``` https://api.imaging.datacommons.cancer.gov/v3 ``` All IDC data is public and open — **no authentication, account, or credentials are required** to use the API. {% hint style="info" %} The current API release is version 3 (v3), in **beta**. The `/v3` contract may still change in response to feedback before the final `3.0.0` release. To see exactly which beta build is deployed, call [`GET /v3/version`](/api/endpoint-details) — the `api_version` field reports the running release. If you have feedback, please let us know on the [IDC support forum](https://discourse.canceridc.dev)! {% endhint %} The same capabilities are available through two surfaces, which share a single backend: * **REST API** (documented in this section) — plain HTTP/JSON for scripts, applications, and notebooks. Interactive Swagger UI at , OpenAPI specification at [/v3/openapi.json](https://api.imaging.datacommons.cancer.gov/v3/openapi.json). * **MCP server** — the same capabilities exposed as [Model Context Protocol](https://modelcontextprotocol.io/) tools, so LLM agents (Claude and others) can query IDC directly. See the [MCP section](/mcp/mcp) of this documentation. Because both surfaces are implemented over one core, anything you can do over REST you can also do over MCP, and vice versa. ## What you can do * **Discover** what's in IDC — collections, derived analysis results (segmentations, annotations), filterable attributes and their valid values, headline statistics. * **Build cohorts** — turn attribute filters into distinct patient/study/series counts, a page of matching series, and a ready-to-use download payload. * **Retrieve** — public `s3://` URLs, a full `manifest.txt`, and `idc` CLI commands; files transfer directly from public S3/GCS buckets, never through the API server. * **Run SQL** — guarded read-only queries against the series-level `index` table, specialized per-modality indices, and per-collection clinical tables — joins, aggregations, anything the structured filters can't express. * **Explore clinical data** — discover and read the per-collection clinical tables (demographics, staging, therapies, outcomes) and join them to imaging. * **Publish responsibly** — viewer URLs for visual inspection, per-cohort license breakdowns (CC BY vs CC BY-NC), and ready-to-use citations. ## In this section * [Getting Started](/api/getting-started) — your first API calls. * [IDC API Concepts](/api/idc-api-concepts) — the data model, the query surfaces, and the recommended workflow. * [Endpoint Details](/api/endpoint-details) — the endpoint reference and worked examples. * [Querying with SQL](/api/querying-with-sql) — the guarded SQL surface and the tables available to it. * [Getting the data](/api/getting-data) — manifests, download commands, licenses, and citations. The API is developed in the open at [ImagingDataCommons/IDC-REST-MCP](https://github.com/ImagingDataCommons/IDC-REST-MCP), which also hosts the developer-facing documentation. {% hint style="warning" %} Looking for documentation of the earlier API versions? The V1 and V2 APIs are superseded by v3; their documentation is preserved in the [Archive](/archive/archive). {% endhint %} # Getting Started The IDC REST API is available at the base URL `https://api.imaging.datacommons.cancer.gov`, with all endpoints under the `/v3` prefix. No authentication is required — you can try every example on this page from any terminal with `curl`. {% hint style="info" %} The interactive [Swagger UI](https://api.imaging.datacommons.cancer.gov/v3/docs) documents every endpoint with a filled-in request/response example, and lets you execute requests directly from the browser. The machine-readable OpenAPI specification is at [/v3/openapi.json](https://api.imaging.datacommons.cancer.gov/v3/openapi.json). {% endhint %} ## Your first calls Check which IDC data release the API is serving, and the headline totals: ```bash curl -s https://api.imaging.datacommons.cancer.gov/v3/version curl -s https://api.imaging.datacommons.cancer.gov/v3/stats ``` List the collections (datasets) available in IDC, or look at one in detail: ```bash curl -s https://api.imaging.datacommons.cancer.gov/v3/collections curl -s https://api.imaging.datacommons.cancer.gov/v3/collections/nlst ``` ## Build your first cohort Before filtering, discover the valid values of the attribute you want to filter on — don't guess: ```bash curl -s 'https://api.imaging.datacommons.cancer.gov/v3/attributes/Modality/values?limit=10' ``` Then check how big your selection is (cheap), and request a page of matching series together with a ready-to-use download payload: ```bash # distinct patient/study/series counts for the filter curl -s https://api.imaging.datacommons.cancer.gov/v3/cohort/counts \ -H 'content-type: application/json' \ -d '{"terms": {"Modality": ["MR"], "BodyPartExamined": ["BREAST"]}}' # counts + a page of series + download payload curl -s https://api.imaging.datacommons.cancer.gov/v3/cohort/manifest \ -H 'content-type: application/json' \ -d '{"filters": {"terms": {"Modality": ["MR"], "BodyPartExamined": ["BREAST"]}}, "page_size": 3}' ``` The `manifest` response includes `idc` CLI commands you can run as-is to download the matching files directly from public cloud buckets — see [Getting the data](/api/getting-data). ## Where to go next * [IDC API Concepts](/api/idc-api-concepts) explains the data model, the query surfaces, and the recommended workflow — worth reading before you go beyond simple filters. * [Endpoint Details](/api/endpoint-details) lists every endpoint with worked examples. * [Querying with SQL](/api/querying-with-sql) covers questions that attribute filters can't express — joins, aggregations, and clinical data. * Prefer to have an LLM agent do the querying? Point it at the [IDC MCP server](/mcp/mcp) — the same capabilities, exposed as agent tools. {% hint style="info" %} If you have feedback about the desired features of the IDC API, please let us know via the IDC [support forum](https://discourse.canceridc.dev). {% endhint %} # IDC API Concepts This page explains the data model behind the API, the distinct "surfaces" the API exposes, and the recommended workflow for getting from a question to the data. ## The data model IDC stores public cancer imaging as **DICOM**, organized as a hierarchy: ``` Patient → Study → Series (the DICOM hierarchy; the `index` table has one row per Series) labelled by two independent grouping axes: collection_id the source dataset (e.g. `nlst`, `tcga_luad`); a patient belongs to exactly one collection analysis_result_id a derived dataset (segmentations / annotations / radiomics); a single analysis result can span multiple collections ``` The main queryable table is **`index`** — **one row per series**, the unit you filter, count, and download. `collection_id` and `analysis_result_id` are **orthogonal**: an analysis result is not nested under one collection, so filtering by a `collection_id` will not necessarily capture all of an analysis result's series, and vice versa. Filter on whichever axis you actually mean. (`analysis_results_index` lists each result's source collections in its plural `collections` field.) IDC is large (100+ TB total), so **always check counts and size before downloading.** ## The query surfaces, and how they relate The API exposes a few distinct surfaces that build on each other: ``` DISCOVERY ───▶ COHORT ───▶ RETRIEVAL ⟍ what exists? how big is download links ⟍ SQL (escape hatch: what can I my filtered (URLs, manifest, ⟋ anything the structured filter on? selection? idc commands) ⟋ surfaces can't express) │ ▲ └─ provides the vocabulary (attributes + valid values) ─┘ ``` | Surface | Answers | Key endpoints | | -------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | **Discovery** | "What exists? What can I filter on?" | `GET /v3/version`, `/v3/stats`, `/v3/collections`, `/v3/collections/{id}`, `/v3/analysis_results`, `/v3/attributes`, `/v3/attributes/{attr}/values` | | **Cohort** | "How big is my selection, and what's in it?" | `POST /v3/cohort/counts`, `POST /v3/cohort/manifest` | | **Retrieval** | "Give me the download links" | `POST /v3/cohort/manifest.txt` | | **SQL** | "Run my custom query" (+ schema) | `GET /v3/tables`, `/v3/tables/{table}`, `POST /v3/sql` | | **Side tools** | View / cite / license-check a cohort | `GET /v3/viewer-url`, `POST /v3/citations`, `POST /v3/licenses` | In one paragraph: **Discovery** hands you the lay of the land *and the vocabulary* — the attribute names and valid values you'll filter on. **Cohort** turns a chosen combination of that vocabulary into distinct counts, a page of matching series, and a ready-to-use download payload. **Retrieval** is the download half on its own — public `s3://` URLs, a full `manifest.txt`, and `idc` CLI commands. **SQL** is the bypass: when your selection needs a `GROUP BY`, a join, or an aggregation that structured cohort filters can't express, you write a read-only `SELECT` (see [Querying with SQL](/api/querying-with-sql)). The side tools (viewer / citations / licenses) all operate on the *same* cohort filters. ## Filter syntax Cohort filters (used by `cohort/counts`, `cohort/manifest`, `cohort/manifest.txt`, `licenses`, and `citations`) have two parts: * **`terms`** — `{attribute: [values]}` for equality / membership. Values are **OR**'d *within* an attribute and **AND**'d *across* attributes. For example, `{"Modality": ["CT", "MR"], "collection_id": ["nlst"]}` means "(CT OR MR) AND in the nlst collection." * **`ranges`** — `{attribute: {"gte": x, "lte": y}}` for the numeric and date attributes (e.g. `instanceCount`, `series_size_MB`, `StudyDate`). Either bound may be omitted for an open-ended range. ```json { "terms": {"Modality": ["CT"], "collection_id": ["nlst"]}, "ranges": {"instanceCount": {"gte": 100, "lte": 200}} } ``` {% hint style="info" %} This is a different, simpler filter syntax than the earlier V2 API (which used per-attribute suffixes like `_btw`). If you are migrating from V2, note that ranges are now expressed with `gte`/`lte` inside a `ranges` object. {% endhint %} These structured filters are not SQL — they operate only on the `index` table's filterable attributes. For a range on a property that isn't a filterable attribute (for example a clinical value such as patient age, which lives in the [clinical tables](/api/querying-with-sql#clinical-non-imaging-data)), use the [SQL surface](/api/querying-with-sql) instead. Always **ground your values first** with `GET /v3/attributes` (what you can filter on — including which attributes are `term` vs `range`) and `GET /v3/attributes/{attr}/values` (the real values and their correct casing). Don't guess values. ## Recommended workflow Start from the shape of your question — there are two entry points: **A. Simple attribute filter** (e.g. *"breast MRI from NLST"*): 1. **Ground values** — `GET /v3/attributes`, then `GET /v3/attributes/{attr}/values`. If the property you need isn't among the attributes (e.g. *what anatomy a segmentation contains*), it lives in a specialized index — switch to path B. 2. **Build and size** — `POST /v3/cohort/counts` (cheap) to sanity-check size, then `POST /v3/cohort/manifest` for the series page and download payload. **B. Relational or aggregate question** (e.g. *"modalities present per collection"*, *"series matching a joined condition"*) → **go straight to SQL**: 1. **Ground the schema** — `GET /v3/tables`, then `GET /v3/tables/{table}`. Don't guess table/column names. 2. **Query** — `POST /v3/sql`. Select `series_aws_url` (or `SeriesInstanceUID`) if you want a manifest out of it. **Both paths then:** get the data — prefer the returned `idc` commands / `manifest.txt` (direct from S3/GCS; see [Getting the data](/api/getting-data)) — and **be a good citizen** — check `licenses` (CC BY vs CC BY-NC) and include `citations` when you publish. {% hint style="info" %} **Explore narrow, then widen.** While you're still figuring out a query, keep result sizes small — a low `page_size` / `max_rows` / `limit`, or a `COUNT`/`GROUP BY` instead of raw rows — and raise the limit only once you know you need the full set. Size-capped responses include a `truncated` boolean: `false` means the result is complete; `true` means raise the limit and re-check (or narrow/aggregate). {% endhint %} # Endpoint Details This page is the endpoint reference for the IDC REST API, with a worked `curl` example for each surface. Every endpoint is also documented interactively at the [Swagger UI](https://api.imaging.datacommons.cancer.gov/v3/docs), which shows a filled-in request/response for each route and lets you execute it in the browser. All endpoints are under the base URL `https://api.imaging.datacommons.cancer.gov` with the `/v3` prefix. The examples below use `localhost:8000` only where noted; against the hosted service, substitute the base URL above. ## Endpoint reference | Method & path | Purpose | | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET /v3/version` | IDC data release served (e.g. `v24`) + pinned index version, and this server's own software version (`api_version`, plus `build` if the deploy stamped one) | | `GET /v3/stats` | Headline totals (collections, patients, studies, series, `size_TB`) | | `GET /v3/collections` | List collections (datasets) | | `GET /v3/collections/{id}` | Collection detail: counts, modalities, license breakdown | | `GET /v3/analysis_results` | Derived datasets (segmentations / annotations) | | `GET /v3/attributes` | Filterable attributes (name, type, term/range, categorical) | | `GET /v3/attributes/{attr}/values?limit=` | Distinct values + counts for an attribute, plus a `note` caveat when one applies (e.g. `BodyPartExamined` ≠ segmented anatomy) | | `GET /v3/tables` | Tables available to SQL | | `GET /v3/tables/{table}` | Column schema for a table | | `GET /v3/clinical/tables?collection_id=` | Per-collection clinical tables (optionally one collection) | | `GET /v3/clinical/tables/{table}` | Clinical table columns + human-readable labels | | `GET /v3/clinical/tables/{table}/rows?max_rows=` | Clinical table rows (capped) | | `POST /v3/cohort/counts` | Distinct counts for a filter (cheap) | | `POST /v3/cohort/manifest` | Counts + a page of series + download payload | | `POST /v3/cohort/manifest.txt` | Full manifest as `text/plain` (`s3://`; `source=gcs` reaches GCS's S3-compatible endpoint) | | `POST /v3/sql` | Guarded read-only SQL (DuckDB) | | `GET /v3/viewer-url` | OHIF / Slim viewer link for a study or series | | `POST /v3/citations` | Citations for a cohort | | `POST /v3/licenses` | License breakdown for a cohort | ## Worked examples **Read-only lookups (GET)** — no body, just the URL: ```bash curl -s https://api.imaging.datacommons.cancer.gov/v3/version # data release + this server's build curl -s https://api.imaging.datacommons.cancer.gov/v3/stats # headline totals curl -s https://api.imaging.datacommons.cancer.gov/v3/collections # list datasets curl -s https://api.imaging.datacommons.cancer.gov/v3/collections/nlst # one collection's detail curl -s https://api.imaging.datacommons.cancer.gov/v3/analysis_results # derived datasets curl -s https://api.imaging.datacommons.cancer.gov/v3/attributes # filterable attributes curl -s https://api.imaging.datacommons.cancer.gov/v3/tables # tables available to SQL curl -s https://api.imaging.datacommons.cancer.gov/v3/tables/index # one table's column schema ``` **Discover valid values before filtering:** ```bash curl -s 'https://api.imaging.datacommons.cancer.gov/v3/attributes/Modality/values?limit=10' ``` **Cheap size check** — the `counts` body is the filter object directly: ```bash curl -s https://api.imaging.datacommons.cancer.gov/v3/cohort/counts \ -H 'content-type: application/json' \ -d '{"terms": {"Modality": ["MR"], "BodyPartExamined": ["BREAST"]}}' ``` **Build a cohort** — `manifest` wraps the filter in a request with paging: ```bash curl -s https://api.imaging.datacommons.cancer.gov/v3/cohort/manifest \ -H 'content-type: application/json' \ -d '{"filters": {"terms": {"Modality": ["MR"], "BodyPartExamined": ["BREAST"]}}, "page": 0, "page_size": 3}' ``` **Get the full manifest as plain text** (for `idc download-from-manifest` / `s5cmd`): ```bash curl -s https://api.imaging.datacommons.cancer.gov/v3/cohort/manifest.txt \ -H 'content-type: application/json' \ -d '{"filters": {"terms": {"collection_id": ["nlst"]}}, "source": "gcs"}' ``` **Custom query via SQL** (anything the structured filters can't express — see [Querying with SQL](/api/querying-with-sql)): ```bash curl -s https://api.imaging.datacommons.cancer.gov/v3/sql \ -H 'content-type: application/json' \ -d '{"sql": "SELECT Modality, count(*) n FROM index GROUP BY 1 ORDER BY n DESC", "max_rows": 20}' ``` **License check** — like `counts`, the body is the filter object directly: ```bash curl -s https://api.imaging.datacommons.cancer.gov/v3/licenses \ -H 'content-type: application/json' \ -d '{"terms": {"collection_id": ["nlst"]}}' ``` **Citations for a cohort** — body wraps the filter, like `manifest`: ```bash curl -s https://api.imaging.datacommons.cancer.gov/v3/citations \ -H 'content-type: application/json' \ -d '{"filters": {"terms": {"collection_id": ["nlst"]}}, "citation_format": "apa"}' ``` **Viewer link** for a study (or pass `series_instance_uid=`): ```bash curl -s 'https://api.imaging.datacommons.cancer.gov/v3/viewer-url?study_instance_uid=1.3.6.1.4.1.14519.5.2.1.7695.4164.129908397467389975396031099306' ``` **Clinical data** — discover and read the per-collection clinical tables: ```bash curl -s 'https://api.imaging.datacommons.cancer.gov/v3/clinical/tables?collection_id=nlst' # clinical tables for a collection curl -s https://api.imaging.datacommons.cancer.gov/v3/clinical/tables/nlst_canc # clinical table columns + labels curl -s 'https://api.imaging.datacommons.cancer.gov/v3/clinical/tables/nlst_canc/rows?max_rows=100' # clinical rows (capped) ``` For filtering by, or joining on, clinical attributes, use SQL against `clinical.` — see [Querying with SQL](/api/querying-with-sql). # Querying with SQL Attribute filters (the [cohort surface](/api/idc-api-concepts#the-query-surfaces-and-how-they-relate)) cover equality and range conditions on the series-level `index` table. For anything *relational or aggregate* — joins, `GROUP BY`, "X that *also has* Y", per-group counts — or for properties that live only in a specialized index (e.g. the anatomy a segmentation contains), use the SQL surface: `POST /v3/sql` (`run_sql` over MCP). ```bash curl -s https://api.imaging.datacommons.cancer.gov/v3/sql \ -H 'content-type: application/json' \ -d '{"sql": "SELECT Modality, count(*) n FROM index GROUP BY 1 ORDER BY n DESC", "max_rows": 20}' ``` Ground the schema before you write a query: `GET /v3/tables` lists the available tables and `GET /v3/tables/{table}` returns a table's columns. Don't guess table or column names. {% hint style="info" %} **Cohort or SQL?** Use **Cohort** when your selection is attribute filters over series metadata (equality/IN + ranges on the one `index` table) — it's structured and validated. Use **SQL** for anything relational or aggregate, and for properties only a specialized index holds. Anything you can `SELECT series_aws_url FROM index WHERE …` for *is* a manifest, so SQL can also produce download URLs directly. {% endhint %} ## Limits of the SQL endpoint `POST /v3/sql` accepts arbitrary SQL, but the data is public and the connection is read-only — you can't break anything. What you *will* run into are the guardrails: only single read-only `SELECT`/`WITH` statements are accepted, and a server row cap and per-query timeout apply. Size-capped results carry a `truncated` flag — for bulk *series*, use the cohort/manifest surface rather than dumping rows through SQL. For the full threat model and hardening details, see the [API security documentation](https://github.com/ImagingDataCommons/IDC-REST-MCP/blob/main/SECURITY.md). ## Tables available to SQL SQL can reach the main `index` table plus a set of specialized indices. Specialized indices are named `_index` after the DICOM Modality of the series they describe — if a Modality value is central to your question (SEG, CT, SM, …), check its index. Join them to `index` on `SeriesInstanceUID`. | Table(s) | Granularity / what it adds | | ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `index` | one row per **series** — the main table | | `collections_index` | one row per collection (curated metadata) | | `analysis_results_index` | one row per analysis result | | `version_metadata_index` / `prior_versions_index` | IDC release versions / removed series | | `seg_index`, `ann_index`, `ann_group_index`, `rtstruct_index` | segmentations / annotations / RT structures: **what was segmented** (`SegmentedPropertyType_CodeMeanings` — note `BodyPartExamined` reflects the source acquisition, not this) and the **reference** to the image series they derive from (`segmented_SeriesInstanceUID` / `referenced_SeriesInstanceUID`) | | `ct_index`, `mr_index`, `pt_index` | per-modality acquisition parameters (slice thickness, kVp, TE/TR, injected dose…) | | `sm_index`, `sm_instance_index` | slide-microscopy (pathology) series / instance metadata | | `contrast_index`, `volume_geometry_index` | contrast agent / 3D volume geometry | | `clinical_index` | per-collection clinical-table data dictionary | This is what makes **relational** questions answerable. For example, *"pathology slides that have a segmentation of a specific structure"* — impossible against `index` alone — is a join of `index` (the slides) to `seg_index` (the segmentations) on the segmented image series: ```sql SELECT i.collection_id, count(DISTINCT i.SeriesInstanceUID) AS slides FROM index i JOIN seg_index seg ON seg.segmented_SeriesInstanceUID = i.SeriesInstanceUID WHERE i.Modality = 'SM' -- slide microscopy (pathology) AND list_contains(seg.SegmentedPropertyType_CodeMeanings, 'Nucleus') -- the segmented structure GROUP BY 1 ORDER BY slides DESC ``` {% hint style="info" %} **Array columns:** columns whose schema type is `STRING[]` (e.g. the `*_CodeMeanings` columns above) hold a *list* of values per row — match elements with `list_contains(col, 'value')`, not `=` or `LIKE`. If a query is invalid, the error response carries DuckDB's own message (including its "Did you mean …?" suggestions), so you can fix and retry. {% endhint %} {% hint style="warning" %} **Still BigQuery-only.** A handful of things remain outside these indices: *per-individual-segment* detail (each segment rather than the series-level aggregated code lists in `seg_index`), DICOM SR quantitative/qualitative measurements (radiomics), and private DICOM elements. For those, query the full metadata with [`idc-index`](https://github.com/ImagingDataCommons/idc-index) and [BigQuery](/data/organization-of-data/bigquery-tables). {% endhint %} ## Clinical (non-imaging) data Many collections ship clinical data — demographics, diagnoses, cancer staging, therapies, labs, outcomes — alongside the images. It comes in two layers: * **`clinical_index`** — a *data dictionary*: one row per (collection, table, column) with a human-readable `column_label` and an array of coded `values`. Use it to discover *what* clinical attributes a collection has and what their codes mean. It's a normal table — query it with SQL (it joins to `index` on `collection_id`). * **Per-collection clinical tables** (e.g. `nlst_canc`) — the actual clinical rows. These live under a separate **`clinical` schema** and are queried as `clinical.
`. There are \~150 of them, so they are kept out of `GET /v3/tables` and discovered with the [clinical endpoints](/api/endpoint-details#worked-examples) instead. Each joins to imaging on **`dicom_patient_id = index.PatientID`** (not `SeriesInstanceUID`). Clinical data is *not harmonized* across collections — table and column names vary, so always discover before querying. For relational questions — filtering by a clinical attribute, or joining clinical data to imaging — use SQL against `clinical.
`. For example, *"NLST patients imaged with CT whose cancer is stage IV (code `400`)"*: ```sql SELECT count(DISTINCT i.PatientID) AS patients FROM index i JOIN clinical.nlst_canc c ON c.dicom_patient_id = i.PatientID WHERE i.collection_id = 'nlst' AND i.Modality = 'CT' AND c.clinical_stag = '400' ``` See [Clinical data](/data/organization-of-data/clinical) for more on IDC's clinical data model. # Getting the data The API never moves image bytes. Every series URL it returns points at **public AWS S3 and GCS buckets — no credentials needed** — so the transfer goes directly from cloud storage to you, works the same against the hosted or a local instance, and scales to whole collections. The API's job is to hand you a **manifest** (or the equivalent download commands); you pull the files with a client. Install the `idc` command with: ```bash pip install idc-index ``` ## Two ways to download **1. Whole collection** — the simplest path. `cohort/manifest`'s download payload emits this for you when your filter is a single `collection_id`: ```bash idc download nlst --download-dir ./idc-data ``` **2. From a manifest** — save the `manifest.txt` output (or any SQL result's `series_aws_url` column) and feed it to the CLI: ```bash # get the manifest curl -s https://api.imaging.datacommons.cancer.gov/v3/cohort/manifest.txt \ -H 'content-type: application/json' \ -d '{"filters": {"terms": {"collection_id": ["nlst"]}}}' > idc_manifest.txt # download it idc download-from-manifest idc_manifest.txt --download-dir ./idc-data ``` You can also drive the raw URLs yourself with `s5cmd --no-sign-request` (anonymous access). ### `source`: `aws` vs `gcs` Manifest and URL requests take a `source` of `aws` (default) or `gcs`. **Both return `s3://` URLs** — GCS is reached through its S3-compatible endpoint rather than a `gs://` URL. This matches how `idc-index` itself works, and is why `idc download-from-manifest` only recognizes `s3://` lines. For `source=gcs`, add `--endpoint-url https://storage.googleapis.com` when driving `s5cmd` directly. {% hint style="info" %} IDC is large (100+ TB total). Always check counts and `size_TB` (from `cohort/counts` or `cohort/manifest`) before downloading a broad selection. {% endhint %} For more download options and tools, see the [Downloading data](/data/downloading-data) section. ## Licenses IDC data is open, but licenses vary per series — typically **CC BY** (commercial use allowed) vs **CC BY-NC** (non-commercial only). Before reusing or redistributing a cohort, check the split for your filter: ```bash curl -s https://api.imaging.datacommons.cancer.gov/v3/licenses \ -H 'content-type: application/json' \ -d '{"terms": {"collection_id": ["nlst"]}}' ``` `licenses` returns the series count and size per license, so you can see at a glance whether the selection is commercial-friendly. The filter is an ordinary [cohort filter](/api/idc-api-concepts#filter-syntax), so you can check the license at any granularity — a whole collection, or a single study or series by its UID: ```bash # license of one study curl -s https://api.imaging.datacommons.cancer.gov/v3/licenses \ -H 'content-type: application/json' \ -d '{"terms": {"StudyInstanceUID": ["1.3.6.1.4.1.14519.5.2.1.7695.4164.129908397467389975396031099306"]}}' # license of one series curl -s https://api.imaging.datacommons.cancer.gov/v3/licenses \ -H 'content-type: application/json' \ -d '{"terms": {"SeriesInstanceUID": ["1.3.6.1.4.1.14519.5.2.1.7695.4164.174071765480311650274095134055"]}}' ``` The same applies to `citations`, `cohort/counts`, and the manifest endpoints — `SeriesInstanceUID`, `StudyInstanceUID`, and `PatientID` are all filterable attributes. ### Licenses for a manifest The manifest itself (`cohort/manifest.txt`, or the `series` / `download` payload of `cohort/manifest`) does **not** carry license information — it's just download URLs. Because a manifest is defined by a cohort filter, the license breakdown for exactly the manifest's contents is the `licenses` response for the **same filter**. Build the manifest and check its licenses with one filter reused across both endpoints. If you want the license on **each row** of a manifest, build the manifest with SQL instead and select the `license_short_name` column alongside the series URL: ```bash curl -s https://api.imaging.datacommons.cancer.gov/v3/sql \ -H 'content-type: application/json' \ -d '{"sql": "SELECT SeriesInstanceUID, license_short_name, series_aws_url FROM index WHERE collection_id = '"'"'nlst'"'"'", "max_rows": 5}' ``` Every series URL you can `SELECT` this way *is* a manifest, so this gives you a per-series manifest with the license attached. ## Citations When you publish results using IDC data, include the per-dataset citations **and** acknowledge IDC itself by citing the IDC paper ([Fedorov et al., 10.1148/rg.230180](https://doi.org/10.1148/rg.230180)). `citations` returns both for your cohort, in your choice of format — `apa`, `bibtex`, `csl-json`, or `turtle`: ```bash curl -s https://api.imaging.datacommons.cancer.gov/v3/citations \ -H 'content-type: application/json' \ -d '{"filters": {"terms": {"collection_id": ["nlst"]}}, "citation_format": "bibtex"}' ``` The response carries the per-dataset citations (from the cohort's source DOIs) and the IDC paper as a separate acknowledgment, so you can drop both straight into your manuscript. Like `licenses`, the filter is an ordinary [cohort filter](/api/idc-api-concepts#filter-syntax) (wrapped in `filters` here), so you can get the citations for a single study or series just as easily — the endpoint resolves the correct source dataset(s) from whatever the selection covers: ```bash # citations for one study curl -s https://api.imaging.datacommons.cancer.gov/v3/citations \ -H 'content-type: application/json' \ -d '{"filters": {"terms": {"StudyInstanceUID": ["1.3.6.1.4.1.14519.5.2.1.7695.4164.129908397467389975396031099306"]}}, "citation_format": "apa"}' # citations for one series curl -s https://api.imaging.datacommons.cancer.gov/v3/citations \ -H 'content-type: application/json' \ -d '{"filters": {"terms": {"SeriesInstanceUID": ["1.3.6.1.4.1.14519.5.2.1.7695.4164.174071765480311650274095134055"]}}, "citation_format": "apa"}' ``` See [Publications](/publications) for more on citing and acknowledging IDC. # MCP server The IDC **MCP server** lets LLM agents query IDC directly. It exposes the same capabilities as the [REST API](/api/api) — discovery, cohort building, retrieval, SQL, clinical data, licenses, and citations — as [Model Context Protocol](https://modelcontextprotocol.io/) tools, so an agent such as Claude can find, describe, and download IDC data on your behalf, in plain conversation. The [Model Context Protocol](https://modelcontextprotocol.io/) is an open standard for connecting LLM applications to external tools and data sources. Any spec-conformant MCP client can use the IDC server. The server is live in production at: ``` https://api.imaging.datacommons.cancer.gov/mcp ``` It is public and **unauthenticated** — no API key, account, or config file needed. Point a client at the URL and ask it to find an imaging cohort. {% hint style="info" %} The MCP server and the REST API are two surfaces over one backend, released together as version 3, currently in **beta**. The contract may still change in response to feedback before the final `3.0.0` release. To see exactly which beta build is deployed, ask your agent to call the [`get_idc_version` tool](/mcp/tools) (or call the REST API's [`GET /v3/version`](/api/endpoint-details)) — the `api_version` field reports the running release. Feedback is welcome on the [IDC support forum](https://discourse.canceridc.dev). {% endhint %} ## One backend, two surfaces The MCP server and the REST API share a single core, so a capability is implemented once and exposed in both. The difference is *who is calling*: * Use the [**REST API**](/api/api) when *you* write the code — scripts, apps, notebooks (plain HTTP/JSON, with a Swagger UI). * Use **MCP** when an *LLM agent* does the querying — the same capabilities as tools, with prescriptive descriptions and an `idc://guide` resource so the agent follows the recommended "ground-first" workflow on its own. Because the two surfaces share a backend, the [data model](/api/idc-api-concepts#the-data-model) and [query surfaces](/api/idc-api-concepts#the-query-surfaces-and-how-they-relate) documented for the REST API apply unchanged to MCP. ## Relation to the Imaging Data Commons Skill The [Imaging Data Commons Skill](https://github.com/ImagingDataCommons/imaging-data-commons-skill) is a *different access path* to the same data. An [Agent Skill](https://agentskills.io/) that works with Claude and any other agent that supports the format, it has the agent write and run Python directly against [`idc-index`](https://github.com/ImagingDataCommons/idc-index) inside a code-execution sandbox (e.g. Claude Code, or Claude Desktop / claude.ai with code execution enabled) — no server involved, and no network round trip for metadata, but it only works where the client can execute Python locally. The MCP server (and the REST API) instead expose the same `idc-index` data as callable tools over the network, for clients that can't or don't want to run code: remote-MCP connectors, non-Python agent frameworks, or a curated tool surface instead of hand-written SQL. Both share the same data model and the same ground-first workflow. * Pick the **skill** when local Python execution is available. * Pick **MCP** for network-only clients or the hosted, zero-setup path. ## In this section * [Getting started](/mcp/getting-started) — connect a client to the IDC MCP server. * [Tools and resources](/mcp/tools) — the full list of capabilities the server exposes. The server is developed in the open at [ImagingDataCommons/IDC-REST-MCP](https://github.com/ImagingDataCommons/IDC-REST-MCP). # Getting started The IDC MCP server is public and unauthenticated at **`https://api.imaging.datacommons.cancer.gov/mcp`**. Point any spec-conformant MCP client at that URL — there is no API key, config file, or account to set up. ## Connect a client ### claude.ai / Claude Desktop (remote connector) Add the URL as a custom (remote) connector in Claude's connector settings: ``` https://api.imaging.datacommons.cancer.gov/mcp ``` Then ask, for example: *"Find breast MRI in IDC, show the counts and total size, and give me a download command."* The agent will discover valid filter values, size the cohort, and return the download commands on its own. ### Claude Desktop / Claude Code (local, for developers) If you are developing against the server, you can run it locally over stdio. Add to your MCP client config: ```json { "mcpServers": { "idc": { "command": "uv", "args": ["run", "--directory", "/absolute/path/to/IDC-REST-MCP", "idc-mcp"] } } } ``` See the [IDC-REST-MCP repository](https://github.com/ImagingDataCommons/IDC-REST-MCP) for install instructions. ### Other MCP clients Any spec-conformant remote-MCP client works — point it at `https://api.imaging.datacommons.cancer.gov/mcp`. Inspect or debug the tools with the [MCP Inspector](https://github.com/modelcontextprotocol/inspector): ```bash npx @modelcontextprotocol/inspector ``` ## How the hosted transport behaves The hosted service uses **streamable-HTTP, configured stateless with plain-JSON responses** — each request is self-contained. In practice: * **Any spec-conformant remote-MCP client works**, and the service autoscales behind a plain load balancer with no session affinity. * **No session handshake is needed to script it** — you can `POST` a `tools/list` or `tools/call` directly (set `Accept: application/json, text/event-stream`); you don't have to `initialize` first or carry an `Mcp-Session-Id` header. * **Session-bound MCP features are not available** (server→client sampling, elicitation, resource subscriptions, streamed progress) — the server exposes only client-initiated tools and static resources. Once connected, see [Tools and resources](/mcp/tools) for what the server can do, and the [API Concepts](/api/idc-api-concepts) page for the shared data model and recommended workflow. # Tools and resources The IDC MCP server exposes the same capabilities as the [REST API](/api/api), grouped by the [query surfaces](/api/idc-api-concepts#the-query-surfaces-and-how-they-relate) described for the API. Tool descriptions are prescriptive about *when* to call each one, and the server ships an `idc://guide` resource with the same conceptual model as the [API Concepts](/api/idc-api-concepts) page — so a capable agent can follow the recommended ground-first workflow without extra prompting. ## Tools, by surface * **Discovery:** `get_idc_version`, `get_stats`, `list_collections`, `get_collection`, `list_analysis_results`, `list_attributes`, `get_attribute_values` * **Schema (for SQL):** `list_tables`, `get_table_schema` * **Clinical data:** `list_clinical_tables`, `get_clinical_table_schema`, `get_clinical_table` * **Cohort / query:** `build_cohort`, `run_sql` * **Retrieval & side tools:** `get_cohort_urls`, `get_viewer_url`, `get_citations`, `get_licenses` Each tool maps to the REST endpoint of the same purpose (see the [Endpoint Details](/api/endpoint-details) reference). For example, `build_cohort` corresponds to `POST /v3/cohort/manifest`, `run_sql` to `POST /v3/sql`, and `get_cohort_urls` to `POST /v3/cohort/manifest.txt`. `run_sql` is safe to leave in an agent's hands: the data is public, the connection is read-only, and the same guardrails apply as for the REST endpoint (see [Limits of the SQL endpoint](/api/querying-with-sql#limits-of-the-sql-endpoint)). ## Resources * `idc://guide` — the data model and recommended workflow (the same conceptual model as the [API Concepts](/api/idc-api-concepts) page). * `idc://tables` — the tables available to SQL. * `idc://schema/{table}` — the column schema for a given table. ## Typical agent workflow Given a natural-language request, a well-behaved agent will: 1. **Ground values** with `list_attributes` and `get_attribute_values` (or **ground the schema** with `list_tables` / `get_table_schema` for a relational question). 2. **Build and size** the cohort with `build_cohort`, or query directly with `run_sql`. 3. **Retrieve** with `get_cohort_urls` — returning `idc` CLI commands and a manifest that pull files directly from public S3/GCS buckets (see [Getting the data](/api/getting-data)). 4. **Be a good citizen** — check `get_licenses` (CC BY vs CC BY-NC) and include `get_citations` when publishing. See [Querying with SQL](/api/querying-with-sql) for the tables `run_sql` can reach and the clinical-data layer. # Colab notebooks This section contains various pointers that may be helpful when working with [Google Colab](https://colab.research.google.com/notebooks/intro.ipynb). Google Colaboratory, or “Colab” for short, is a product from Google Research. Colab allows anybody to write and execute arbitrary python code through the browser, and is especially well suited to machine learning, data analysis and education. More technically, Colab is a hosted Jupyter notebook service that requires no setup to use, while providing free access to computing resources including GPUs. {% hint style="info" %} If you would like to share an interesting notebook that uses IDC/cloud for imaging research, please let us know and we would be happy to review and reference it here! {% endhint %} ## IDC maintained notebooks IDC Colab example notebooks are maintained in this repository: Notebook demonstrating deployment and application of abdominal structures segmentation tool to IDC data, developed for the [2021 RSNA Deep Learning Lab](https://github.com/RSNA/AI-Deep-Learning-Lab-2021) course: ## Community-contributed notebooks * [Deep-learning Nodule Segmentation using LIDC dataset on Google Cloud Platform](https://github.com/Mayo-Radiology-Informatics-Lab/IDC_GoogleCloud_Notebook), contributed by [Kuan (Kevin) Zhang](https://www.mayo.edu/research/labs/radiology-informatics/faculty-staff), Mayo Clinic * [3D Brain MRI Classification Using Imaging Data Commons, Google Cloud, and NIFTI files](https://github.com/Mayo-Radiology-Informatics-Lab/IDC_GoogleCloud_Notebook_3DClassification), contributed by [Pouria Rouzokh](https://www.mayo.edu/research/labs/radiology-informatics/faculty-staff), Mayo Clinic * Notebooks contributed by [Fabian Seidl](https://www.linkedin.com/in/fabian-seidl-390147a4/), ISB-CGC, demonstrating the utility of BigQuery in correlative analysis of radiomics and genomics data: * [How to compare tumor features with gene expression data](https://github.com/isb-cgc/Community-Notebooks/blob/master/Notebooks/How_to_compare_tumor_features_with_gene_expression_data.ipynb) * [How to compare tumor features with mutation data](https://github.com/isb-cgc/Community-Notebooks/blob/master/Notebooks/How_to_compare_tumor_features_with_mutation_data.ipynb) ## Relevant resources * Colab limitations: * Transferring data between Colab and Google Drive: * Potentially interesting sources of example notebooks: * [SimpleITK notebooks](https://github.com/InsightSoftwareConsortium/SimpleITK-Notebooks/tree/master/Python) * * * * Google Colab Tips for Power Users: * Mounting GCS bucket using `gcsfuse`: # BigQuery Get started with using BigQuery with IDC data following our [Getting Started tutorial](https://github.com/ImagingDataCommons/IDC-Examples/tree/master/notebooks/getting_started) [This document](https://docs.google.com/document/d/1qC5_qUFBQ2HmEjfYQa9WaH1Y-erMlfis00bbU8UPnRs/edit?usp=sharing) contains various examples of using BigQuery for searching DICOM metadata in IDC BQ tables. ## Learning resources * [Intro to SQL course on Kaggle](https://www.kaggle.com/learn/intro-to-sql) * Query performance considerations * * * Temporary tables: # Data Studio dashboards [Google Data Studio](https://support.google.com/datastudio/answer/6283323?hl=en) is a free tool that turns your data into informative, easy to read, easy to share, and fully customizable dashboards and reports. {% hint style="info" %} If you would like to share an interesting Data Studio dashboard that uses IDC/cloud for imaging research, please let us know and we would be happy to review and reference it from the IDC documentation! {% endhint %} In this section you can learn how to very quickly make a custom Data Studio dashboard to explore the content of your cohort, and find some additional examples of using Data Studio for analyzing content of IDC. # Dashboard for your cohort Use IDC-provided Data Studio template to build a custom dashboard for your cohort You can use [this Data Studio template](http://bit.ly/3jdCmON) to build a custom dashboard for your own cohort, which will look like the screenshot below in three relatively simple steps. ![Screenshot of the DataStudio dashboard template you can use to explore your cohort.](/files/-MS4kaUIT20nK1zTx9EU) **Step 1: Prepare the manifest BigQuery table** Export the cohort manifest as a BigQuery table, and take note of the location of the resulting table. ![Name of the BQ table you will need is highlighted with the red rectangle.](/files/-MS4mQkLYrPYHJRo8Dye) **Step 2: Duplicate the template** Open the dashboard template following this link: , and click "Use template" to make a copy of the dashboard. ![](/files/-MSdYLMANevNG3jLSMIT) When prompted, do not change the default options, and click "Copy Report". ![](/files/-MS4n7UW_xvwFX6KMR_g) **Step 3: Configure data source** Select "Resource > Manage added data sources" ![](/files/-MS4nTnPrIu0SCgvD7xX) Select "Edit" action: ![](/files/-MS4o4M9_mfPrZ2iVWO1) Update the custom query as instructed. This will select all of the DICOM metadata available for the instances in your cohort. ![](/files/-MS4oRns_nlNPaFFSjmd) For example, if the location of your manifest table is `canceridc-user-data.user_manifests.manifest_cohort_101_20210127_213746`, the custom query that will join your manifest with the DICOM metadata will be the following: ```sql SELECT all_of_idc.* FROM `canceridc-user-data.user_manifests.manifest_cohort_101_20210127_213746` AS my_cohort JOIN `bigquery-public-data.idc_current.dicom_all` AS all_of_idc ON all_of_idc.SOPInstanceUID = my_cohort.SOPInstanceUID ``` Once you updated the query, click "Reconnect" in the upper right corner. {% hint style="warning" %} Make sure you select a valid Billing Project that you can use to support the queries! {% endhint %} ![](/files/tq6GwiLgrNe1WK4qZzNg) Accept the below, if prompted (you may also be notified about changes to the schema of the table, so the message may be different). ![](/files/-MS4paGIZrs7yZVvMmIJ) Click "Done" on the next screen: ![](/files/-MS4pmmbY09judr1mwv0) Click "Close" on the next screen: ![](/files/-MS4pxczqsVMNo_17DoM) **You are Done!** The dashboard for your cohort is now live: you can "View" it to interact with the content, you can edit it to explore additional attributes in the cohort, and you can choose to keep it private or share with a link! ![](/files/-MS4qN-KfCufLpp3A7p-) # More dashboard examples Topic-specific dashboards * [Digital pathology / slide microscopy dashboard](https://datastudio.google.com/reporting/e4795618-80ad-4cfe-a668-89a8090d560b/page/NM1rD) * [Preclinical images dashboard](https://datastudio.google.com/reporting/053952fa-0290-4273-a0f2-9f7811270edd) * [CT contrast images exploration](https://datastudio.google.com/reporting/a5497bdc-54a5-4f48-b617-6dfc5a80a492) * [MRI dashboard](https://datastudio.google.com/reporting/e7507e51-8a97-41d9-9dac-fe87c6e3a482) Program- and Collection-specific dashboards * [HTAN program dashboard](https://datastudio.google.com/reporting/b20f4731-ddbb-4369-8d52-1d4652cd755a) * [RMS-Mutation-Prediction collection dashboard](https://datastudio.google.com/reporting/7f267400-8774-42e1-b5d1-ca11863c52a9) * [LIDC-IDRI collection dashboard](https://bit.ly/39EaVXT) (see details in [this paper](https://pubmed.ncbi.nlm.nih.gov/32772385/)) * [TotalSegmentator-CT-Segmentations](https://datastudio.google.com/reporting/c3e2965e-e615-4b4b-b523-1fc335dd9d43) # ACCESS allocations [Advanced Cyberinfrastructure Coordination Ecosystem (ACCESS)](https://access-ci.org/) is a program supported by the US National Science Foundation (NSF) to provide educators with **free** and convenient access to advanced computational resources. If you have a university email account, you can complete a relatively easy application process to receive an allocation of free credits that you can then use to create pre-configured GPU-enabled cloud-based linux virtual machines with desktop interface available via browser. You can use those machines, for example, to have a convenient access to an instance of [3D Slicer](https://www.slicer.org/) for experimenting with AI models, or for training DL networks.

Example of the configurations available for the Ubuntu 22.04 base image

GPU configurations available, from https://docs.jetstream-cloud.org/general/vmsizes/#jetstream2-gpu

## How to get started Follow these steps: 1. Create an account and request an ACCESS allocation at this page: . There are 4 different levels, with each giving you a different number of “credits” that you use to create your VM instances. Each of these levels requires you to submit a different application. For the Explore ACCESS allocation (lowest tier), you need to write a simple abstract to justify why you needed these resources. Other tiers require more lengthy descriptions of what you’ll do with the ACCESS resources. In our experience, applications can be approved in as soon as a few days after submitting the application. You can be a PI and have multiple Co-PIs with you on the project, so you can all access the Jetstream2 resources. 2. Once you get approved, your allocation is valid for a 12 month period, and you get half of the credits to start. To start using these credits you exchange them for Service Units (SUs) on different platforms. We experimented with the one called [JetStream2](https://jetstream-cloud.org/), which provides easy interface to cloud-based computing resources. If you want to use JetStream2, you will need to exchange your ACCESS credit allocation for JetStream2 SUs here: . Usually this exchange is approved within a few days if not less. 3. Once you get the SU’s you can access JetStream interface to configure and create VMs here: (you can lean more about available configurations from this documentation page: ). 4. Once you created a VM and your setup is complete, it’s very easy to connect to your VMs through ssh or Web desktop interface. ## Why we recommend ACCESS allocations * **It is free for academics!** * **Very easy to set up.** As of writing, there is no similar product available from Google Cloud, which would provide desktop access to a VM with a comparable ease of access. AWS provides [AppStream2](https://aws.amazon.com/appstream2/), but we have yet to experiment to evaluate it. * **You can do a lot with the basic credit allocation!** Entry-level allocations can be on the order of 100,000s, while the burn rate is, for example, 8 SUs/hour for a medium size VM (8 CPUs/30 GB RAM). As a reference: * it takes about 1 hour to build Slicer application from scratch on a medium-sized VM using 7 threads * it took \~7 days and \~5000 SUs to train the model in [this repository](https://github.com/pieper/nnmouse) (see summary in the slides [here](https://docs.google.com/presentation/d/1zlTCcGPwYRzuZnXhJ3RHJ7_AQM4Nr8aU8Me1A6TQJjM/edit?usp=sharing)) using `g3.large` VM configuration * **Geared to help you save!** Unlike the VMs you get from the commercial providers, JetStream VMs can be *shelved.* Once a VM is shelved, you spend zero SUs for keeping it around (in comparison, you will keep paying for the disk storage of your GCP VMs even when they are turned off). * **Customer support is excellent!** We received responses within 1-2 days. On some occasions, we observed glitches with Web Desktop, but those could often be resolved by restarting the VM. ## Acknowledging ACCESS and JetStream2 * JetStream2: * ACCESS: # Compute engine This section contains various recipes that might be useful in utilizing GCP Compute Engine (GCE). You are also encouraged to review the slides in the following presentation that provides an introduction into GCE, and shares some best practices the its usage. > W. Longabaugh. *Introduction to Google Cloud Platform*. Presented at MICCAI 2021. ([slides in Google Slides](https://docs.google.com/presentation/d/1HNZ34xkZCXt6WRDcEtmAUGNq5TM0xzPaK7sojKJfoBc/edit?usp=sharing)) # 3D Slicer desktop VM These instructions provide a reference example of how you can start up a traditional workstation desktop on a VM instance to run interactive applications like [3D Slicer](https://slicer.org) and access the desktop via a conventional web browser. Two options are shown, either with or without a GPU. Note that GPUs are significantly more expensive so only enable it if needed. For 3D Slicer the main benefit of a GPU is for rendering, so operations like dicom processing and image segmentation are quite usable without a GPU. Even volume rendering is fairly usable if you choose the CPU rendering option. Other operations such as training machine learning models may benefit from an appropriate GPU. A motivation for using desktop applications like 3D Slicer on a VM is that their computing power close to the data, so heavy network operations such as storage bucket or dicom store access may be significantly faster than accessing the same resources from a remote machine. They are also highly configurable, so that you can easily allocate the number of cores or memory needed for a given task. Note that can even change these configurations so that, for example, you can shut down the machine, add a GPU and more memory, and then boot the same instance and pick up where you left off. In addition, these desktops are persistent in the sense that you can start a task such as labeling data for a machine learning task, disconnect your ssh session, and reconnect later to pick up where you left off without needing to restart applications or reload data. This can be convenient when tending long-running computations, accessing your work from different computers, or working on a network that sometimes disconnects. The instructions here are just a starting point. There are many cloud options available to manage access scopes for the service accounts, allocate disks, and configure other options. {% hint style="info" %} In order to follow these instructions, you will need to have a project that has billing enabled. Please follow the instructions in [Getting started with GCP](/introduction/google-cloud-platform/getting-started-with-gcp) to set up billing. {% endhint %} ## With a GPU You can launch a VM with a GPU in your project with a command like this in your local terminal (replace `vm-name` with a name for your machine): ``` export VMNAME=vm-name gcloud compute instances create ${VMNAME} \ --machine-type=n1-standard-8 \ --accelerator=type=nvidia-tesla-k80,count=1 \ --image-family=slicer \ --image-project=idc-sandbox-000 \ --boot-disk-size=200GB \ --boot-disk-type=pd-balanced \ --maintenance-policy=TERMINATE ``` Once it boots in about 90 seconds you can type: ``` gcloud compute ssh ${VMNAME} -- -L 6080:localhost:6080 ``` Then you can open to get to your desktop. ## Without a GPU You can launch a VM without a GPU in your project with a command like this in your local terminal (replace `vm-name` with a name for your machine): ``` export VMNAME=vm-name gcloud compute instances create ${VMNAME} \ --machine-type=n1-standard-8 \ --image-family=slicer \ --image-project=idc-sandbox-000 \ --boot-disk-size=200GB \ --boot-disk-type=pd-balanced \ --maintenance-policy=TERMINATE ``` Once it boots in about 90 seconds you can type: ``` gcloud compute ssh ${VMNAME} -- -L 6080:localhost:6080 ``` On the remote machine run: ``` # these are on-time installs sudo systemctl stop novnc sudo apt-get update sudo apt-get -y install tigervnc-standalone-server websockify ``` Each time you reboot the machine, run this: ``` vncserver -xstartup xfce4-session # here you will be prompted for a password for vnc if you haven't already sudo systemctl stop novnc nohup websockify --web /opt/novnc/noVNC/ 6080 localhost:5901 & ``` Then you can open to get to your desktop. ## Note This effort is a work in progress with a minimal desktop environment. Further refinement is expected and community contributions would be welcome! A description of the background and possible evolution of this work is [in this document](https://docs.google.com/document/d/1jfHqjS7Fer7Lhqea5bjyown0b04AsqeOhIBY2gWUDO4/edit). # Using VS Code with GCP VMs [Visual Studio Code](https://code.visualstudio.com/) has a useful feature of allowing you to develop code on a remote VM from the convenience of your desktop. You can follow the steps below to configure your development environment for this task. ## Prerequisites * \`\`[`gcloud` SDK](https://cloud.google.com/sdk/docs/install) installed on your computer * [Visual Studio Code](https://code.visualstudio.com/) installed on your computer * A GCP VM you want to use for code development is up and running ## **Step 1: Install "Remote - SSH" extension** ![](/files/-MTlJdbOKuFBlkWzDGH2) ## **Step 2: Populate SSH config files** Run the following command to populate SSH config files with host entries for each VM instance you have running ```bash $ gcloud compute config-ssh ``` ## Step 3: Connect to host If the previous step completed successfully, you should see the running VMs in the Remote Explorer of VS Code, as in the screenshot below, and should be able to open a new session to those remove VMs. ![](/files/-MTlLoiD6dpJ_azKj-q1) {% hint style="warning" %} Note that the SSH configuration may/will change if you restart your VM. In this case you will need to re-configure (re-run step 2 above). {% endhint %} # Security considerations ### Computing on the Cloud Most of the same linux commands, scripts, pipelines/workflows, imaging software packages and docker containers that you run on your local machine can be executed on virtual machines on Google Cloud with experimentation and fine tuning. 1. The basics and best practices on how to launch virtual machines (VMs) are described [here](https://docs.google.com/document/d/1U3JalN711lNcuGd_In8T89uGmaJTj480OladQaOPdOk/edit) in our documentation. NOTE: When launching VMs, please maintain the default firewall settings. 2. Compute Engine instances can run the public images for Linux and Windows Server that Google provides as well as private custom images that you can [create](https://cloud.google.com/compute/docs/images/create-delete-deprecate-private-images) or [import from your existing systems](https://cloud.google.com/compute/docs/images/importing-virtual-disks).\ \ Be careful as you spin up a machine, as larger machines cost you more. If you are not using a machine, shut it down. You can always restart it easily when you need it.\ \ Example use-case: You would like to run Windows-only genomics software package on the TCGA data. You can create a Windows based VM instance. 3. More details on how to deploy docker containers on VMs are described here in Google’s documentation: [deploying containers](https://cloud.google.com/compute/docs/containers/deploying-containers) 4. A good way to estimate costs for running a workflow/pipeline on large data sets is to test them first on a small subset of data. 5. There are different VM types depending on the sort of jobs you wish to execute. By default, when you create a VM instance, it remains active until you either stop it or delete it. The costs associated with VM instances are detailed here: [compute pricing](https://cloud.google.com/compute/pricing) 6. If you plan on running many short compute-intensive jobs (for example indexing and sorting thousands of large bam files), you can execute your jobs on preemptible virtual machines. They are 80% cheaper than regular instances. [preemptible vms](https://cloud.google.com/preemptible-vms/) Example use-cases: * Using preemptible VMs, researchers were able to quantify transcript levels on over 11K TGCA RNAseq samples for a total cost of $1,065.49.\ Tatlow PJ, Piccolo SR. [A cloud-based workflow to quantify transcript-expression levels in public cancer compendia](https://www.nature.com/articles/srep39259). Scientific Reports 6, 39259 * Also Broad’s popular variant caller pipeline, GATK, was designed to be able to run on preemptible VMs. * Google cloud computing can be estimated [here](https://cloud.google.com/compute/all-pricing). ### Be Very Careful with Tokens containing passwords. They should NOT be moved to Github Because of the ability to see a [history](https://docs.github.com/en/pull-requests/committing-changes-to-your-project/viewing-and-comparing-commits/differences-between-commit-views) of Github postings, if a password or bearer token is part of software code (e.g. notebook or colaboratory) it will be permanently available on Github. This is a security risk!! Do not put bearer tokens or other passwords into workbooks, instead refer to them in the code and place those in a location not posted into Github (if you do post it to GitHub, it then immediately becomes public, usable, and able to be stolen and used maliciously by others). If you do accidentally post one to Github: 1) immediately change passwords on your systems to remove the exposure provided by the exposed password, 2) let those who involved in the security of your system and data know, and 3) remedy your code-base so future saves to Github do not include passwords or tokens in your codebase. ### Storage on the Cloud The Google Cloud Platform offers a number of different storage options for your virtual machine instances: [disks](https://cloud.google.com/compute/docs/disks/) 1. [Block Storage:](https://cloud.google.com/compute/docs/disks/#pdspecs) * By default, each virtual machine instance has a single boot persistent disk that contains the operating system. The default size is 10GB but can be adjusted up to 64TB in size. (Be careful! High costs here, spend wisely!) * Persistent disks are restricted to the zone where your instance is located. * Use persistent disks if you are running analyses that require low latency and high-throughput. 2. [Object Storage:](https://cloud.google.com/compute/docs/disks/#gcsbuckets) Google Cloud Storage (GCS) buckets are the most flexible and economical storage option. * Unlike persistent disks, Cloud Storage buckets are not restricted to the zone where your instance is located. * Additionally, you can read and write data to a bucket from multiple instances simultaneously. * You can mount a GCS bucket to your VM instance when latency is not a priority or when you need to share data easily between multiple instances or zones.\ An example use-case: You want to slice thousands of bam files and save the resulting slices to share with a collaborator who has instances in another zone to use for downstream statistical analyses. You can save objects to GCS buckets including images, videos, blobs and unstructured data.\ A comparison table detailing the current pricing of Google’s storage options can be found here: [storage features](https://cloud.google.com/storage/features/) # NCI Cloud Resources [NCI Cloud Resources](https://datacommons.cancer.gov/analytical-tools) are components of the [NCI Cancer Research Data Commons](https://datacommons.cancer.gov/) that bring data and computational power together to enable cancer research and discovery. Our current experience in using NCI Cloud Resources for cancer image analysis is summarized in the following preprint: > Thiriveedhi, V. K., Krishnaswamy, D., Clunie, D., Pieper, S., Kikinis, R. & Fedorov, A. Cloud-based large-scale curation of medical imaging data using AI segmentation. *Research Square* (2024). doi:[10.21203/rs.3.rs-4351526/v1](https://dx.doi.org/10.21203/rs.3.rs-4351526/v1) # Archive This page links to historical versions of documentation that has been removed from the main navigation. The content is preserved in the repository's git history and can be accessed via the links below. ## IDC V2 API Documentation for the IDC REST API version 2, superseded by the current [API (v3)](/api/api). The V2 endpoints remain operational for now on the legacy backend, but are no longer the recommended way to access IDC programmatically. * [Getting Started](https://github.com/ImagingDataCommons/IDC-Docs/blob/aa7ae7d/api/getting-started.md) * [IDC API Concepts](https://github.com/ImagingDataCommons/IDC-Docs/blob/aa7ae7d/api/idc-api-concepts.md) * [Manifests](https://github.com/ImagingDataCommons/IDC-Docs/blob/aa7ae7d/api/manifests.md) * [Accessing the API](https://github.com/ImagingDataCommons/IDC-Docs/blob/aa7ae7d/api/accessing-the-api.md) * [Endpoint Details](https://github.com/ImagingDataCommons/IDC-Docs/blob/aa7ae7d/api/endpoint-details.md) ## IDC V1 API Documentation for the IDC REST API version 1, superseded by the [V2 API](#idc-v2-api) and the current [API (v3)](/api/api). * [Getting Started](https://github.com/ImagingDataCommons/IDC-Docs/blob/570c98c/api/v1-api/getting-started.md) * [IDC Data Model Concepts](https://github.com/ImagingDataCommons/IDC-Docs/blob/570c98c/api/v1-api/idc-data-model-concepts.md) * [Accessing the API](https://github.com/ImagingDataCommons/IDC-Docs/blob/570c98c/api/v1-api/accessing-the-api.md) * [Endpoint Details](https://github.com/ImagingDataCommons/IDC-Docs/blob/570c98c/api/v1-api/endpoint-details.md) * [Release Notes](https://github.com/ImagingDataCommons/IDC-Docs/blob/570c98c/api/v1-api/release-notes.md) ## Organization of data — IDC data release v1 Documentation for IDC data release v1, superseded by [current organization of data](/data/organization-of-data). * [Organization of data in v1](https://github.com/ImagingDataCommons/IDC-Docs/blob/570c98c/data/organization-of-data/organization-of-data-v1.md) ## Organization of data — IDC data releases v2–v13 Documentation for IDC data releases v2 through v13, superseded by [current organization of data](/data/organization-of-data). * [Overview](https://github.com/ImagingDataCommons/IDC-Docs/blob/570c98c/data/organization-of-data/organization-of-data-v2-through-v13-deprecated/README.md) * [Files and metadata](https://github.com/ImagingDataCommons/IDC-Docs/blob/570c98c/data/organization-of-data/organization-of-data-v2-through-v13-deprecated/files-and-metadata.md) * [Resolving CRDC GUIDs](https://github.com/ImagingDataCommons/IDC-Docs/blob/570c98c/data/organization-of-data/organization-of-data-v2-through-v13-deprecated/guids-and-uuids.md) * [Clinical data](https://github.com/ImagingDataCommons/IDC-Docs/blob/570c98c/data/organization-of-data/organization-of-data-v2-through-v13-deprecated/clinical.md) ## Loading DICOM files onto a VM using a BQ manifest This approach has been superseded by [idc-index](https://github.com/ImagingDataCommons/idc-index), which provides a simpler interface for downloading IDC data. * [Using a BQ Manifest to Load DICOM Files onto a VM](https://github.com/ImagingDataCommons/IDC-Docs/blob/570c98c/cookbook/virtual-machines/using-a-bq-manifest-to-load-dicom-files-onto-a-vm.md)