From 8579ce1346539c92f5aba8abfe365d699287e1b6 Mon Sep 17 00:00:00 2001 From: Stephen Aylward Date: Sat, 14 Dec 2024 19:23:32 -0500 Subject: [PATCH 01/10] Call Vista3D NIM on NVIDIA AI enterprise / GPU cloud Illustrates functions for uploading images to file.io and using their URLs to launch Vista3D NIM on NVIDIA AI enterprise / GPU cloud servers. Enables researchers with limited GPU resources to take advantage of NVIDIA resources while evaluating MONAI Vista3D. --- nvidia_nims/vista_3d_remote_nim.ipynb | 464 ++++++++++++++++++++++++++ 1 file changed, 464 insertions(+) create mode 100644 nvidia_nims/vista_3d_remote_nim.ipynb diff --git a/nvidia_nims/vista_3d_remote_nim.ipynb b/nvidia_nims/vista_3d_remote_nim.ipynb new file mode 100644 index 000000000..2bb89cbab --- /dev/null +++ b/nvidia_nims/vista_3d_remote_nim.ipynb @@ -0,0 +1,464 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7b50dffd", + "metadata": {}, + "source": [ + "```\n", + "Copyright (c) MONAI Consortium\n", + "Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "you may not use this file except in compliance with the License.\n", + "You may obtain a copy of the License at\n", + " http://www.apache.org/licenses/LICENSE-2.0\n", + "Unless required by applicable law or agreed to in writing, software\n", + "distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "See the License for the specific language governing permissions and\n", + "limitations under the License.\n", + "```\n", + "\n", + "# Calling Vista-3D NIM on NVAIE\n", + "\n", + "In this tutorial, we will cover the following:\n", + "\n", + "- **Convert image to Nifty format:** The Vista-3D NIM requires files to be in compressed nifty (.nii.gz) format. We use ITK to convert DICOM and other file formats to nifty.\n", + "\n", + "- **Create a single-use URL to the image:** We use the website file.io to provide a single-use URL for our image. The Vista-3D NIM requires a URL to the image to be processed. File.io provides an API for receiving a file and returning a one-time use URL to that file.\n", + "\n", + "- **Communicate with the Vista-3D NIM on NVAIE:** The Vista-3D NIM can be downloaded and run locally, but herein we are running it on the NVIDIA AI Enterprise (NVAIE) / NVIDIA GPU Cloud (NGC) servers. You can get free NVAIE/NGC credits and an API Key for remotely accessing those servers by registering at https://build.nvidia.com/nvidia/vista-3d. Also, you can apply for unlimited access as a researcher by joining the (free) NVIDIA Developer Program: https://developer.nvidia.com/join-nvidia-developer-program\n", + "\n", + "- **Visualize and/or save the results:** We provide pyvista for visualization and use ITK to save the results in any of a variety of image formats.\n", + "\n", + "Learn more about the Vista-3D foundation model for 3D CT segmentation:
\n", + "https://arxiv.org/abs/2304.02643\n", + "\n", + "Learn more about NIMs:
\n", + "https://www.nvidia.com/en-us/ai/" + ] + }, + { + "cell_type": "markdown", + "id": "b1fc03f5", + "metadata": {}, + "source": [ + "\n", + "## Obtaining an NVIDIA AI Enterprise / NVIDIA GPU Cloud (NVAIE/NGC) API Key\n", + "\n", + "This demonstration runs the Vista-3D NIM (AI container) on the NVIDIA AI Enterprise servers, so an API key is required. The good news is that any individual can get an API key and 1,000 to 5,000 free credits (processing one image requires one credit) when they first sign-up at:\n", + "\n", + "- https://build.nvidia.com/nvidia/vista-3d\n", + "\n", + "Also, you can apply for unlimited local access as a researcher by joining the (free) NVIDIA Developer Program:\n", + "\n", + "- https://developer.nvidia.com/join-nvidia-developer-program\n", + "\n", + "This API Key is specific to your account at NVIDIA.\n", + "\n", + "We recommend storing and accessing your personal/private API key as an environment variable, rather than embedding it within your code where it may be accidentally shared with or viewed by others. For example, in bash you can run this command in your environment or embed it into your ~/.bashrc file:\n", + "\n", + " `export NGC_API_KEY=`\n", + "\n", + "and then launch jupyter lab / vscode to run this notebook." + ] + }, + { + "cell_type": "markdown", + "id": "1f78537f", + "metadata": {}, + "source": [ + "## Setup the python environment\n", + "\n", + "- `itk`: used to support a variety of image file formats and to convert images for viewing via pyvista.\n", + " - Learn more at https://github.com/InsightSoftwareConsortium/ITK\n", + "- `pyvista`: provides 2D and 3D scientific visualizations, compatible with VSCode and most jupyter notebook/lab systems\n", + " - The `trame` option is used to provide interactive rendering support.\n", + " - Learn more at https://docs.pyvista.org/ " + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "beab0aef", + "metadata": {}, + "outputs": [], + "source": [ + "!python -c \"import itk\" || pip install -q \"itk\"\n", + "!python -c \"import pyvista\" || ipywidgets 'pyvista[all,trame]'" + ] + }, + { + "cell_type": "markdown", + "id": "24ff5fb5", + "metadata": {}, + "source": [ + "## Setup the imports" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "3ce61753-11ad-4ade-9afe-6ad1bc748e25", + "metadata": {}, + "outputs": [], + "source": [ + "import io\n", + "import os\n", + "import requests\n", + "import tempfile\n", + "import zipfile\n", + "\n", + "import itk\n", + "import pyvista as pv\n" + ] + }, + { + "cell_type": "markdown", + "id": "c301afd1", + "metadata": {}, + "source": [ + "## Upload a file to file.io and return its URL\n", + "\n", + "https://file.io provides a single-use URL to access uploaded files.\n", + "- This service is free.\n", + "- The file is deleted once it is downloaded using the URL.\n", + "\n", + "__Note:__ The Vista-3D NIM on NVAIE has a firewall that limits it to receiving input images from AWS and https://file.io. You cannot upload images from any other service or from your own system. Once registered, you can download the Vista-3D NIM and use it to process data using local GPU resources and without input image source restrictions." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "adc99c7a-94f0-400a-afbf-70321362ded9", + "metadata": {}, + "outputs": [], + "source": [ + "def post_image_to_fileio(input_image:itk.Image) -> str:\n", + " '''Post an image to file.io and return the link.'''\n", + "\n", + " # Save the image to a temporary file\n", + " # Note: The Vista-3D NIM on NVAIE is limited to reading .nii.gz files\n", + " tmp_dir = tempfile.mkdtemp()\n", + " tmp_filename = os.path.join(tmp_dir, \"tmp.nii.gz\")\n", + " itk.imwrite(input_image, tmp_filename)\n", + " \n", + " # Upload the file\n", + " with open(os.path.join(tmp_filename), \"rb\") as f:\n", + " res = requests.post(\n", + " \"https://file.io/\",\n", + " files={\"file\": f}\n", + " )\n", + " if res.status_code != 200:\n", + " raise RuntimeError(f\"Cannot upload file. The response {res}.\")\n", + " \n", + " # Get the link\n", + " res = res.json()\n", + " link = res[\"link\"]\n", + " \n", + " return link" + ] + }, + { + "cell_type": "markdown", + "id": "3148eb4b", + "metadata": {}, + "source": [ + "## Communicate with the Vista-3D NIM on NVAIE\n", + "\n", + "Run the Vista-3D segmentation on given image.\n", + "\n", + "Vista-3D expects images to be 1.5mm isotropic CT scans of large sections of the human body. Only rudimentary checks are performed to verify these requirements." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "822b918b-382a-42e1-a174-080b90f14d24", + "metadata": {}, + "outputs": [], + "source": [ + "def nvaie_vista3d_nim(api_key:str, input_image:itk.Image) -> list:\n", + " '''Run the MONAI VISTA 3D model on the input image and return the result.'''\n", + "\n", + " # The API endpoint for the MONAI VISTA 3D model on NVIDIA AI Enterprise\n", + " invoke_url = \"https://health.api.nvidia.com/v1/medicalimaging/nvidia/vista-3d\"\n", + " \n", + " # Check the input image\n", + " assert len(input_image.GetSpacing()) == 3, \"The input image must be 3D.\"\n", + " isotropy = ((input_image.GetSpacing()[1] / input_image.GetSpacing()[0]) +\n", + " (input_image.GetSpacing()[2] / input_image.GetSpacing()[0])) / 2\n", + " if isotropy < 0.9 or isotropy > 1.1 or input_image.GetSpacing()[0] != 1.5:\n", + " print(\"WARNING: The input image should have 1.5 mm isotropic spacing. Performance will be degraded.\")\n", + " print(\" The input image has spacing:\", input_image.GetSpacing())\n", + " input_image_arr = itk.array_view_from_image(input_image)\n", + " minv = input_image_arr.min()\n", + " maxv = input_image_arr.max()\n", + " if minv < -1024 or maxv > 3071:\n", + " print(\"WARNING: The input image should have Hounsfield Units in the range [-1024, 3071]. Performance will be degraded.\")\n", + " print(\" The input image has Hounsfield Units in the range:\", [minv, maxv])\n", + "\n", + " # Post the image to file.io and get the link\n", + " input_image_url = post_image_to_fileio(input_image)\n", + "\n", + " # Define the header and payload for the API call\n", + " header = {\n", + " \"Authorization\": \"Bearer \" + api_key, \n", + " }\n", + " \n", + " payload = {\n", + " \"image\": input_image_url,\n", + " # Optionally limited processing to specific classes\n", + " #\"prompts\": {\n", + " # \"classes\": [\"liver\", \"spleen\"]\n", + " #}\n", + " }\n", + "\n", + " # Call the API\n", + " session = requests.Session()\n", + " response = session.post(invoke_url, headers=header, json=payload)\n", + " \n", + " # Check the response\n", + " response.raise_for_status()\n", + " \n", + " # Get the result\n", + " with tempfile.TemporaryDirectory() as temp_dir:\n", + " z = zipfile.ZipFile(io.BytesIO(response.content))\n", + " z.extractall(temp_dir)\n", + " file_list = os.listdir(temp_dir)\n", + " for filename in file_list:\n", + " filepath = os.path.join(temp_dir, filename)\n", + " if os.path.isfile(filepath) and filename.endswith(\".nrrd\"):\n", + " # SUCCESS: Return the results\n", + " return itk.imread(filepath, pixel_type=itk.SS)\n", + " \n", + " # FAILURE: Return None\n", + " return None" + ] + }, + { + "cell_type": "markdown", + "id": "7559d64f", + "metadata": {}, + "source": [ + "## Retrieve NVAIE / NGC API Key\n", + "\n", + "See documentation at the start of this notebook on how to obtain an API key and store it as an environment variable." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "41a4939f-71d3-4081-b383-40b4ed7c1be1", + "metadata": {}, + "outputs": [], + "source": [ + "ngc_api_key = os.environ['NGC_API_KEY']" + ] + }, + { + "cell_type": "markdown", + "id": "6a6adf55", + "metadata": {}, + "source": [ + "## Specify the filename of the image to process\n", + "\n", + "Provide the path to the file via the `input_image_filename` variable, or a default image (\\\"vista3d-example-1.nii.gz\\\") will be downloaded and cached in the MONAI_DATA_DIRECTORY (defined by an environment variable) or in a temporary directory." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "9d2e5d21", + "metadata": {}, + "outputs": [], + "source": [ + "input_image_filename = None\n", + "\n", + "if input_image_filename == None:\n", + " monai_data_directory = os.environ.get(\"MONAI_DATA_DIRECTORY\")\n", + " if monai_data_directory is not None:\n", + " os.makedirs(monai_data_directory, exist_ok=True)\n", + " else:\n", + " monai_data_directory = tempfile.mkdtemp()\n", + " input_image_filename = os.path.join(\n", + " monai_data_directory,\n", + " \"vista3d-example-1.nii.gz\"\n", + " )\n", + " if not os.path.exists(input_image_filename):\n", + " resp = requests.get(\"https://assets.ngc.nvidia.com/products/api-catalog/vista3d/example-1.nii.gz\")\n", + " with open(input_image_filename, \"wb\") as f: \n", + " f.write(resp.content)" + ] + }, + { + "cell_type": "markdown", + "id": "2eb2918f", + "metadata": {}, + "source": [ + "## Process the image and save results\n" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "aba10460-8920-4aa8-ab42-16b565c20d5f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "WARNING: The input image should have 1.5 mm isotropic spacing. Performance will be degraded.\n", + " The input image has spacing: itkVectorD3 ([0.902344, 0.902344, 5])\n" + ] + } + ], + "source": [ + "# read the image from disk - a wide variety of formats are supported\n", + "input_image = itk.imread(input_image_filename)\n", + "\n", + "# run the model using the api key and input image\n", + "output_image = nvaie_vista3d_nim(ngc_api_key, input_image)\n", + "\n", + "# save the results to \"output_image.mha\"\n", + "itk.imwrite(output_image, \"./output_image.mha\")" + ] + }, + { + "cell_type": "markdown", + "id": "fc099dab", + "metadata": {}, + "source": [ + "## The rest of this notebook visulizes the results" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "a4cf276b", + "metadata": {}, + "outputs": [], + "source": [ + "# Convert the input and output images to PyVista images\n", + "image = pv.wrap(itk.vtk_image_from_image(input_image))\n", + "labels = pv.wrap(itk.vtk_image_from_image(output_image))\n", + "\n", + "# Extract the contours from the label (output) image\n", + "contours = labels.contour_labeled(\n", + " smoothing=True,\n", + " smoothing_num_iterations=10,\n", + " output_mesh_type=\"triangles\",\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "5b1e97fc", + "metadata": {}, + "outputs": [], + "source": [ + "# It is critical to use the \"client\" mode for PyVista so that vtk.js is used for rendering.\n", + "# vtk.js respects image orientation information, whereas other backends do not.\n", + "pv.set_jupyter_backend(\"client\")" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "00d28b5c", + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "5bea8bd791b34f96a2b72f9ccf2a5756", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Widget(value='