notebook_examples/deploy-langchain-as-oci-data-science-model-deployment.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "@notebook{deploy-langchain-as-oci-data-science-model-deployment.ipynb,\n",
    "    title: Deploy LangChain Application as OCI Data Science Model Deployment,\n",
    "    summary: Deploy LangChain applications as OCI data science model deployment,\n",
    "    developed_on: pytorch21_p39_gpu_v1,\n",
    "    keywords: langchain, deploy model, register model, LLM,\n",
    "    license: Universal Permissive License v 1.0\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=gray>Oracle Data Science service sample notebook.\n",
    "\n",
    "Copyright (c) 2023 Oracle, Inc.  All rights reserved.\n",
    "Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.\n",
    "</font>\n",
    "\n",
    "***\n",
    "# <font color=red>Deploy LangChain Application as OCI Data Science Model Deployment</font>\n",
    "<p style=\"margin-left:10%; margin-right:10%;\">by the <font color=teal> Oracle Cloud Infrastructure Data Science Service Team </font></p>\n",
    "\n",
    "***\n",
    "\n",
    "## Overview\n",
    "\n",
    "The notebook demonstrates how to deploy LangChain application as OCI Data Science Model Deployment using Oracle Accelerated Data Science (ADS) SDK.\n",
    "\n",
    "The `ChainDeployment` class in ADS allows you to rapidly get a LangChain application into production. The `.prepare()` method serializes the LangChain application as `chain.yaml` file and generates `score.py` file which can further be uploaded to OCI model catalog. The uploaded model can be subsequently deployed into production.\n",
    "\n",
    "Compatible conda pack: [pytorch21_p39_gpu_v1](https://docs.oracle.com/en-us/iaas/data-science/using/conda-gml-fam.htm) for CPU on Python 3.9 (version 1.0)\n",
    "\n",
    "### Prequisites\n",
    "\n",
    "This notebook requires authorization to work with the OCI Data Science Service. Details can be found [here](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/authentication.html#). For the purposes of this notebook what is important to to know is that resource principals will be used absent api_key authentication.\n",
    "\n",
    "---\n",
    "\n",
    "## Contents\n",
    "\n",
    "* <a href='#intro'>Introduction</a>\n",
    "* <a href='#create'>Create a LangChain Application</a>\n",
    "* <a href='#deploy'>Deploy LangChain Application as OCI Model Deployment</a>\n",
    "    * <a href='#deploy_chaindeployment'>Create a ChainDeployment</a>\n",
    "    * <a href='#deploy_prepare'>Prepare</a>\n",
    "    * <a href='#deploy_verify'>Verify</a>\n",
    "    * <a href='#deploy_save'>Save</a>\n",
    "    * <a href='#deploy_deploy'>Deploy</a>\n",
    "    * <a href='#deploy_predict'>Predict</a>\n",
    "* <a href='#clean_up'>Clean Up</a>\n",
    "* <a href='#ref'>References</a>    \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ads\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "from ads.llm.deploy import ChainDeployment\n",
    "from langchain.llms import Cohere\n",
    "from langchain.chains import LLMChain\n",
    "from langchain.prompts import PromptTemplate\n",
    "from shutil import rmtree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='intro'></a>\n",
    "# Introduction\n",
    "\n",
    "In this notebook, you will create a custom LangChain application that links prompt and Cohere model and deploy it on OCI model deployment. It is designed to demonstrate how to use the `ChainDeployment` class in Oracle ADS SDK.\n",
    "\n",
    "The `.prepare()` method will serialize the LangChain application as `chain.yaml` file. It will also generate a `score.py` file that will load the LangChain yaml and call the `predict()` method. The `.save()` and `.deploy()` methods will upload the artifacts to OCI model catalog and deploy the uploaded model, respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authenticate\n",
    "\n",
    "Authentication to the OCI Data Science service is required. Here we default to resource principals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ads.set_auth(auth=\"resource_principal\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='create'></a>\n",
    "# Create a LangChain Application\n",
    "\n",
    "The next cell creates a LangChain application that links prompt and Cohere model. The LangChain application will utilize Cohere model to generate a joke based on the subject that user provides. Remember to replace the `<cohere_api_key>` with the actual api key as Cohere model needs it. You can acquire this key by registering on [Cohere](https://dashboard.cohere.com/welcome/register)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"COHERE_API_KEY\"] = \"<cohere_api_key>\"\n",
    "\n",
    "cohere = Cohere() \n",
    "prompt = PromptTemplate.from_template(\"Tell me a joke about {subject}\")\n",
    "llm_chain = LLMChain(prompt=prompt, llm=cohere, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have a LangChain object `llm_chain`. Try running it with the prompt `{\"subject\": \"animals\"}` and it should give you the corresponding answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm_chain.run({\"subject\": \"animals\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='deploy'></a>\n",
    "# Deploy LangChain Application as OCI Model Deployment\n",
    "\n",
    "<a id='deploy_chaindeployment'></a>\n",
    "## Create a ChainDeployment\n",
    "\n",
    "The next cell creates a model artifact directory. This directory is used to store the artifacts that are needed to deploy the model. It also creates the `ChainDeployment` object. The `ChainDeployment` requires the LangChain object `llm_chain` as `chain` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "artifact_dir = tempfile.mkdtemp()\n",
    "\n",
    "chain_deployment = ChainDeployment(\n",
    "    chain=llm_chain,\n",
    "    artifact_dir=artifact_dir\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='deploy_prepare'></a>\n",
    "## Prepare\n",
    "\n",
    "The prepare step is performed by the `.prepare()` method of the `ChainDeployment` class. It creates a number of customized files that are used to run the model once it is deployed. These include:\n",
    "\n",
    "* `chain.yaml`: A YAML file that is serialized from the LangChain application and can be deserialized in `load_model` in `score.py`.\n",
    "* `runtime.yaml`: This file contains information that is needed to set up the runtime environment on the deployment server.\n",
    "* `score.py`: This script contains the `load_model()` and `predict()` functions. The `load_model()` function understands the format the model file was saved in, and loads it into memory. The `.predict()` method is used to make inferences in a deployed model.\n",
    "\n",
    "To create the model artifacts, you use the `.prepare()` method\n",
    "\n",
    "* `inference_conda_env` variable defines the slug of the conda environment that is used to train the model\n",
    "\n",
    "Note that you can only pass in slug for service conda environment. For custom conda environment, you have to pass in the full path along with the `inference_python_version`. \n",
    "\n",
    "Here, replace `<custom_conda_environment_uri>` with your conda environment uri that has the latest ADS 2.9.1 and replace `<python_version>` with your conda environment python version. For how to customize and publish conda environment, take reference to [Publishing a Conda Environment to an Object Storage Bucket](https://docs.oracle.com/en-us/iaas/data-science/using/conda_publishs_object.htm)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chain_deployment.prepare(\n",
    "    inference_conda_env=\"<custom_conda_environment_uri>\",\n",
    "    inference_python_version=\"<python_version>\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='deploy_verify'></a>\n",
    "## Verify\n",
    "\n",
    "The `.verify()` method takes a set of test parameters and performs the prediction by calling the `predict` function in `score.py`. It also runs the `load_model` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chain_deployment.verify({\"subject\": \"animals\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='deploy_save'></a>\n",
    "## Save\n",
    "\n",
    "Call `.save()` to pack and upload the artifacts under `artifact_dir` to OCI data science model catalog. Once the artifacts are successfully uploaded, you should be able to see the id of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chain_deployment.save(display_name=\"LangChain Model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='deploy_deploy'></a>\n",
    "## Deploy\n",
    "\n",
    "Deploy the LangChain model from previous step by calling `.deploy()`. For more information regarding the allowed parameters, see [here](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/model_serialization/genericmodel.html#deploy). Remember to replace the `<cohere_api_key>` with the actual cohere api key in the `environment_variables`. It usually takes a couple of minutes to deploy the model and you should see the model deployment in the output once the process completes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chain_deployment.deploy(\n",
    "    display_name=\"LangChain Model Deployment\",\n",
    "    environment_variables={\"COHERE_API_KEY\":\"<cohere_api_key>\"}, \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='deploy_predict'></a>\n",
    "## Predict\n",
    "\n",
    "After the deployment is active, you can call the `predict()` on the `ChainDeployment` object to send request to the deployed endpoint. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chain_deployment.predict(data={\"subject\": \"animals\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='clean_up'></a>\n",
    "# Clean Up\n",
    "\n",
    "This notebook created a model deployment and a model. This section deletes those resources. \n",
    "\n",
    "The model deployment must be deleted before the model can be deleted. You can use the `.delete_deployment()` method on the `ChainDeployment` object to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "delete = chain_deployment.delete_deployment(wait_for_completion=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `.delete()` method to delete the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chain_deployment.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell removes the model artifacts that were stored on your local drive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rmtree(artifact_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ref'></a>\n",
    "# References\n",
    "- [ADS Library Documentation](https://accelerated-data-science.readthedocs.io/en/latest/index.html)\n",
    "- [Data Science YouTube Videos](https://www.youtube.com/playlist?list=PLKCk3OyNwIzv6CWMhvqSB_8MLJIZdO80L)\n",
    "- [OCI Data Science Documentation](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/data-science.htm)\n",
    "- [Oracle Data & AI Blog](https://blogs.oracle.com/datascience/)\n",
    "- [Understanding Conda Environments](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/use-notebook-sessions.htm#conda_understand_environments)\n",
    "- [Use Resource Manager to Configure Your Tenancy for Data Science](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/orm-configure-tenancy.htm)\n",
    "- [runtime.yaml](https://docs.content.oci.oracleiaas.com/en-us/iaas/data-science/using/model_runtime_yaml.htm#model_runtime_yaml)\n",
    "- [score.py](https://docs.content.oci.oracleiaas.com/en-us/iaas/data-science/using/model_score_py.htm#model_score_py)\n",
    "- [Model artifact](https://docs.content.oci.oracleiaas.com/en-us/iaas/data-science/using/models_saving_catalog.htm#create-models)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pytorch110_p38_cpu_v1]",
   "language": "python",
   "name": "conda-env-pytorch110_p38_cpu_v1-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}