diff --git a/README.md b/README.md index 7e7ca7aaf..0d3822f48 100644 --- a/README.md +++ b/README.md @@ -302,6 +302,10 @@ This tutorial shows several visualization approaches for 3D image during transfo #### [Auto3DSeg](./auto3dseg/) This folder shows how to run the comprehensive Auto3DSeg pipeline with minimal inputs and customize the Auto3Dseg modules to meet different user requirements. +#### **NVIDIA NIMs** +#### [Vista3D Remote NIM](nvidia_nims/vista_3d_remove_num.ipynb) +This tutorial illustrates sending an image to the Vista3D NIM running on NVIDIA's AI Enterprise / GPU Cloud servers. Results are saved to disk and viewed using pyvista. Registration to generate an API Key is required at [build.nvidia.com](https://build.nvidia.com). Registration and key generation is free and provides free credits for processing images. + #### **Self-Supervised Learning** ##### [self_supervised_pretraining](self_supervised_pretraining/vit_unetr_ssl/ssl_train.ipynb) This tutorial shows how to construct a training workflow of self-supervised learning where unlabeled data is utilized. The tutorial shows how to train a model on TCIA dataset of unlabeled Covid-19 cases. diff --git a/nvidia_nims/vista_3d_remote_nim.ipynb b/nvidia_nims/vista_3d_remote_nim.ipynb new file mode 100644 index 000000000..ded289084 --- /dev/null +++ b/nvidia_nims/vista_3d_remote_nim.ipynb @@ -0,0 +1,523 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7b50dffd", + "metadata": {}, + "source": [ + "\n", + "Copyright (c) MONAI Consortium \n", + "Licensed under the Apache License, Version 2.0 (the \"License\"); \n", + "you may not use this file except in compliance with the License. \n", + "You may obtain a copy of the License at \n", + "    http://www.apache.org/licenses/LICENSE-2.0 \n", + "Unless required by applicable law or agreed to in writing, software \n", + "distributed under the License is distributed on an \"AS IS\" BASIS, \n", + "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. \n", + "See the License for the specific language governing permissions and \n", + "limitations under the License. \n", + "\n", + "# Calling Vista-3D NIM on NVAIE\n", + "\n", + "In this tutorial, we will cover the following:\n", + "\n", + "- **Convert image to Nifty format:** The Vista-3D NIM requires files to be in compressed nifty (.nii.gz) format. We use ITK to convert DICOM and other file formats to nifty.\n", + "\n", + "- **Create a single-use URL to the image:** We use the website file.io to provide a single-use URL for our image. The Vista-3D NIM requires a URL to the image to be processed. File.io provides an API for receiving a file and returning a one-time use URL to that file.\n", + "\n", + "- **Communicate with the Vista-3D NIM on NVAIE:** The Vista-3D NIM can be downloaded and run locally, but herein we are running it on the NVIDIA AI Enterprise (NVAIE) / NVIDIA GPU Cloud (NGC) servers. You can get free NVAIE/NGC credits and an API Key for remotely accessing those servers by registering at https://build.nvidia.com/nvidia/vista-3d. Also, you can apply for unlimited access as a researcher by joining the (free) NVIDIA Developer Program: https://developer.nvidia.com/join-nvidia-developer-program\n", + "\n", + "- **Visualize and/or save the results:** We provide pyvista for visualization and use ITK to save the results in any of a variety of image formats.\n", + "\n", + "Learn more about the Vista-3D foundation model for 3D CT segmentation:
\n", + "https://arxiv.org/abs/2304.02643\n", + "\n", + "Learn more about NIMs:
\n", + "https://www.nvidia.com/en-us/ai/" + ] + }, + { + "cell_type": "markdown", + "id": "b1fc03f5", + "metadata": {}, + "source": [ + "\n", + "## Obtaining an NVIDIA AI Enterprise / NVIDIA GPU Cloud (NVAIE/NGC) API Key\n", + "\n", + "This demonstration runs the Vista-3D NIM (AI container) on the NVIDIA AI Enterprise servers, so an API key is required. The good news is that any individual can get an API key and 1,000 to 5,000 free credits (processing one image requires one credit) when they first sign-up at:\n", + "\n", + "- https://build.nvidia.com/nvidia/vista-3d\n", + "\n", + "Also, you can apply for unlimited local access as a researcher by joining the (free) NVIDIA Developer Program:\n", + "\n", + "- https://developer.nvidia.com/join-nvidia-developer-program\n", + "\n", + "This API Key is specific to your account at NVIDIA.\n", + "\n", + "We recommend storing and accessing your personal/private API key as an environment variable, rather than embedding it within your code where it may be accidentally shared with or viewed by others. For example, in bash you can run this command in your environment or embed it into your ~/.bashrc file:\n", + "\n", + " `export NGC_API_KEY=`\n", + "\n", + "and then launch jupyter lab / vscode to run this notebook." + ] + }, + { + "cell_type": "markdown", + "id": "1f78537f", + "metadata": {}, + "source": [ + "## Setup environment\n", + "\n", + "- `itk`: used to support a variety of image file formats and to convert images for viewing via pyvista.\n", + " - Learn more at https://github.com/InsightSoftwareConsortium/ITK\n", + "- `pyvista`: provides 2D and 3D scientific visualizations, compatible with VSCode and most jupyter notebook/lab systems\n", + " - The `trame` option is used to provide interactive rendering support.\n", + " - Learn more at https://docs.pyvista.org/ " + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "beab0aef", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Traceback (most recent call last):\n", + " File \"\", line 1, in \n", + "ModuleNotFoundError: No module named 'pyvista'\n", + "\n", + "[notice] A new release of pip is available: 23.0.1 -> 24.3.1\n", + "[notice] To update, run: python.exe -m pip install --upgrade pip\n" + ] + } + ], + "source": [ + "!python -c \"import itk\" || pip install -q \"itk\"\n", + "!python -c \"import pyvista\" || pip install -q \"pyvista[all]\"" + ] + }, + { + "cell_type": "markdown", + "id": "24ff5fb5", + "metadata": {}, + "source": [ + "## Setup imports" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "3ce61753-11ad-4ade-9afe-6ad1bc748e25", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "`TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "MONAI version: 1.5.dev2450\n", + "Numpy version: 1.26.4\n", + "Pytorch version: 2.5.1+cpu\n", + "MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False\n", + "MONAI rev id: 0726cee7ba6c1b31f34d54fc26fe6b8db5f5c79c\n", + "MONAI __file__: c:\\src\\MONAI\\tutorials\\nvidia_nims\\venv\\lib\\site-packages\\monai\\__init__.py\n", + "\n", + "Optional dependencies:\n", + "Pytorch Ignite version: 0.4.11\n", + "ITK version: 5.4.0\n", + "Nibabel version: 5.3.2\n", + "scikit-image version: 0.25.0\n", + "scipy version: 1.14.1\n", + "Pillow version: 11.0.0\n", + "Tensorboard version: 2.18.0\n", + "gdown version: 5.2.0\n", + "TorchVision version: 0.20.1+cpu\n", + "tqdm version: 4.67.1\n", + "lmdb version: 1.5.1\n", + "psutil version: 6.1.0\n", + "pandas version: 2.2.3\n", + "einops version: 0.8.0\n", + "transformers version: 4.40.2\n", + "mlflow version: 2.19.0\n", + "pynrrd version: 1.1.1\n", + "clearml version: 1.17.0rc0\n", + "\n", + "For details about installing the optional dependencies, please visit:\n", + " https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies\n", + "\n" + ] + } + ], + "source": [ + "import io\n", + "import os\n", + "import requests\n", + "import tempfile\n", + "import zipfile\n", + "\n", + "import itk\n", + "import pyvista as pv\n", + "\n", + "from monai.config import print_config\n", + "\n", + "print_config()" + ] + }, + { + "cell_type": "markdown", + "id": "c301afd1", + "metadata": {}, + "source": [ + "## Upload a file to file.io and return its URL\n", + "\n", + "https://file.io provides a single-use URL to access uploaded files.\n", + "- This service is free.\n", + "- The file is deleted once it is downloaded using the URL.\n", + "\n", + "__Note:__ The Vista-3D NIM on NVAIE has a firewall that limits it to receiving input images from AWS and https://file.io. You cannot upload images from any other service or from your own system. Once registered, you can download the Vista-3D NIM and use it to process data using local GPU resources and without input image source restrictions." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "adc99c7a-94f0-400a-afbf-70321362ded9", + "metadata": {}, + "outputs": [], + "source": [ + "def post_image_to_fileio(input_image: itk.Image) -> str:\n", + " \"\"\"Post an image to file.io and return the link.\"\"\"\n", + "\n", + " # Save the image to a temporary file\n", + " # Note: The Vista-3D NIM on NVAIE is limited to reading .nii.gz files\n", + " tmp_dir = tempfile.mkdtemp()\n", + " tmp_filename = os.path.join(tmp_dir, \"tmp.nii.gz\")\n", + " itk.imwrite(input_image, tmp_filename)\n", + "\n", + " # Upload the file\n", + " with open(os.path.join(tmp_filename), \"rb\") as f:\n", + " res = requests.post(\"https://file.io/\", files={\"file\": f})\n", + " if res.status_code != 200:\n", + " raise RuntimeError(f\"Cannot upload file. The response {res}.\")\n", + "\n", + " # Get the link\n", + " res = res.json()\n", + " link = res[\"link\"]\n", + "\n", + " return link" + ] + }, + { + "cell_type": "markdown", + "id": "3148eb4b", + "metadata": {}, + "source": [ + "## Communicate with the Vista-3D NIM on NVAIE\n", + "\n", + "Run the Vista-3D segmentation on given image.\n", + "\n", + "Vista-3D expects images to be 1.5mm isotropic CT scans of large sections of the human body. Only rudimentary checks are performed to verify these requirements." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "822b918b-382a-42e1-a174-080b90f14d24", + "metadata": {}, + "outputs": [], + "source": [ + "def nvaie_vista3d_nim(api_key: str, input_image: itk.Image) -> list:\n", + " \"\"\"Run the MONAI VISTA 3D model on the input image and return the result.\"\"\"\n", + "\n", + " # The API endpoint for the MONAI VISTA 3D model on NVIDIA AI Enterprise\n", + " invoke_url = \"https://health.api.nvidia.com/v1/medicalimaging/nvidia/vista-3d\"\n", + "\n", + " # Check the input image\n", + " assert len(input_image.GetSpacing()) == 3, \"The input image must be 3D.\"\n", + " isotropy = (\n", + " (input_image.GetSpacing()[1] / input_image.GetSpacing()[0])\n", + " + (input_image.GetSpacing()[2] / input_image.GetSpacing()[0])\n", + " ) / 2\n", + " if isotropy < 0.9 or isotropy > 1.1 or input_image.GetSpacing()[0] != 1.5:\n", + " print(\"WARNING: The input image should have 1.5 mm isotropic spacing. Performance will be degraded.\")\n", + " print(\" The input image has spacing:\", input_image.GetSpacing())\n", + " input_image_arr = itk.array_view_from_image(input_image)\n", + " minv = input_image_arr.min()\n", + " maxv = input_image_arr.max()\n", + " if minv < -1024 or minv > 0 or maxv < 1000 or maxv > 3071:\n", + " print(\"WARNING: Input image intensities should within [-1024, 3071]. Performance will be degraded.\")\n", + " print(\" The input image has Hounsfield Units in the range:\", [minv, maxv])\n", + "\n", + " # Post the image to file.io and get the link\n", + " input_image_url = post_image_to_fileio(input_image)\n", + "\n", + " # Define the header and payload for the API call\n", + " header = {\n", + " \"Authorization\": \"Bearer \" + api_key,\n", + " }\n", + "\n", + " payload = {\n", + " \"image\": input_image_url,\n", + " # Optionally limited processing to specific classes\n", + " # \"prompts\": {\n", + " # \"classes\": [\"liver\", \"spleen\"]\n", + " # }\n", + " }\n", + "\n", + " # Call the API\n", + " session = requests.Session()\n", + " response = session.post(invoke_url, headers=header, json=payload)\n", + "\n", + " # Check the response\n", + " response.raise_for_status()\n", + "\n", + " # Get the result\n", + " with tempfile.TemporaryDirectory() as temp_dir:\n", + " z = zipfile.ZipFile(io.BytesIO(response.content))\n", + " z.extractall(temp_dir)\n", + " file_list = os.listdir(temp_dir)\n", + " for filename in file_list:\n", + " filepath = os.path.join(temp_dir, filename)\n", + " if os.path.isfile(filepath) and filename.endswith(\".nrrd\"):\n", + " # SUCCESS: Return the results\n", + " return itk.imread(filepath, pixel_type=itk.SS)\n", + "\n", + " # FAILURE: Return None\n", + " return None" + ] + }, + { + "cell_type": "markdown", + "id": "7559d64f", + "metadata": {}, + "source": [ + "## Retrieve NVAIE / NGC API Key\n", + "\n", + "See documentation at the start of this notebook on how to obtain an API key and store it as an environment variable." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "41a4939f-71d3-4081-b383-40b4ed7c1be1", + "metadata": {}, + "outputs": [], + "source": [ + "ngc_api_key = os.environ.get(\"NGC_API_KEY\")\n", + "\n", + "if ngc_api_key is None:\n", + " raise ValueError(\"Please set the NGC_API_KEY environment variable.\")" + ] + }, + { + "cell_type": "markdown", + "id": "6a6adf55", + "metadata": {}, + "source": [ + "## Specify the filename of the image to process\n", + "\n", + "Provide the path to the file via the `input_image_filename` variable, or a default image (\\\"vista3d-example-1.nii.gz\\\") will be downloaded and cached in the MONAI_DATA_DIRECTORY (defined by an environment variable) or in a temporary directory." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "9d2e5d21", + "metadata": {}, + "outputs": [], + "source": [ + "input_image_filename = None\n", + "\n", + "if input_image_filename is None:\n", + " monai_data_directory = os.environ.get(\"MONAI_DATA_DIRECTORY\")\n", + " if monai_data_directory is not None:\n", + " os.makedirs(monai_data_directory, exist_ok=True)\n", + " else:\n", + " monai_data_directory = tempfile.mkdtemp()\n", + " input_image_filename = os.path.join(monai_data_directory, \"vista3d-example-1.nii.gz\")\n", + " if not os.path.exists(input_image_filename):\n", + " resp = requests.get(\"https://assets.ngc.nvidia.com/products/api-catalog/vista3d/example-1.nii.gz\")\n", + " with open(input_image_filename, \"wb\") as f:\n", + " f.write(resp.content)" + ] + }, + { + "cell_type": "markdown", + "id": "2eb2918f", + "metadata": {}, + "source": [ + "## Process the image and save results\n" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "aba10460-8920-4aa8-ab42-16b565c20d5f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "WARNING: The input image should have 1.5 mm isotropic spacing. Performance will be degraded.\n", + " The input image has spacing: itkVectorD3 ([0.902344, 0.902344, 5])\n" + ] + } + ], + "source": [ + "# read the image from disk - a wide variety of formats are supported\n", + "input_image = itk.imread(input_image_filename)\n", + "\n", + "# run the model using the api key and input image\n", + "output_image = nvaie_vista3d_nim(ngc_api_key, input_image)\n", + "\n", + "# save the results to \"output_image.mha\"\n", + "itk.imwrite(output_image, \"./output_image.mha\")" + ] + }, + { + "cell_type": "markdown", + "id": "fc099dab", + "metadata": {}, + "source": [ + "## The rest of this notebook visualizes the results" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "a4cf276b", + "metadata": {}, + "outputs": [], + "source": [ + "# Convert the input and output images to PyVista images\n", + "image = pv.wrap(itk.vtk_image_from_image(input_image))\n", + "labels = pv.wrap(itk.vtk_image_from_image(output_image))\n", + "\n", + "# Extract the contours from the label (output) image\n", + "contours = labels.contour_labeled(\n", + " smoothing=True,\n", + " smoothing_num_iterations=10,\n", + " output_mesh_type=\"triangles\",\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "5b1e97fc", + "metadata": {}, + "outputs": [], + "source": [ + "# It is critical to use the \"client\" mode for PyVista so that vtk.js is used for rendering.\n", + "# vtk.js respects image orientation information, whereas other backends do not.\n", + "pv.set_jupyter_backend(\"client\")" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "00d28b5c", + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "364c68a8ed2749a4af0b3f113b41d48a", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Widget(value='