docs/derivapy-datapath-example-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# DataPath Example 2\n",
    "This notebook gives a very basic example of how to access data. \n",
    "It assumes that you understand the concepts presented in the \n",
    "example 1 notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import deriva modules\n",
    "from deriva.core import ErmrestCatalog, get_credential"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect with the deriva catalog\n",
    "protocol = 'https'\n",
    "hostname = 'www.facebase.org'\n",
    "catalog_number = 1\n",
    "credential = get_credential(hostname)\n",
    "catalog = ErmrestCatalog(protocol, hostname, catalog_number, credential)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the path builder interface for this catalog\n",
    "pb = catalog.getPathBuilder()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataPaths\n",
    "The `PathBuilder` object allows you to begin `DataPath`s from the base `Table`s. A `DataPath` begins with a `Table` (or an `TableAlias` to be discussed later) as its \"root\" from which one can \"`link`\", \"`filter`\", and fetch its \"`entities`\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start a path rooted at a table from the catalog\n",
    "We will reference a table from the PathBuilder `pb` variable from above. Using the PathBuilder, we will reference the \"isa\" schema, then the \"dataset\" table, and from that table start a path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = pb.schemas['isa'].tables['dataset'].path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could have used the more compact dot-notation to start the same path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = pb.isa.dataset.path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting the URI of the current path\n",
    "All DataPaths have URIs for the referenced resources in ERMrest. The URI identifies the resources which are available through \"RESTful\" Web protocols supported by ERMrest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.facebase.org/ermrest/catalog/1/entity/dataset:=isa:dataset\n"
     ]
    }
   ],
   "source": [
    "print(path.uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ResultSets\n",
    "The data from a DataPath are accessed through a pythonic container object, the `ResultSet`. The `ResultSet` is returned by the DataPath's `entities()` and other methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = path.entities()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fetch entities from the catalog\n",
    "Now we can get entities from the server using the ResultSet's `fetch()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<deriva.core.datapath.ResultSet at 0x113547fd0>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.fetch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ResultSet`s behave like python containers. For example, we can check the count of rows in this ResultSet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "815"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: If we had not explicitly called the `fetch()` method, then it would have been called implicitly on the first container operation such as `len(...)`, `list(...)`, `iter(...)` or get item `[...]`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get an entity\n",
    "To get one entity from the set, use the usual container operator to get an item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 14197,\n",
       " 'accession': 'FB00000953',\n",
       " 'title': 'FB0109_Male with Robin sequence, cleft palate, midface hypoplasia, round almond-shaped eyes, negative CMA, positive family Hx_Candidate Gene: HOXB2',\n",
       " 'project': 309,\n",
       " 'funding': None,\n",
       " 'summary': None,\n",
       " 'description': \"The purpose of this study is to collect, process, and study samples from individuals with known or possible genetic disease, and their family members. The study’s broad goals are to better understand the genetic causes of disease in order to improve the ability to diagnose, treat, and even prevent illness. Our goal is to obtain a genetic diagnosis for health problem(s) the proband has, so the information can be used, when appropriate, to guide medical decisions made by the affected individuals doctor.\\n\\n**This is restricted-access human data.**  To gain access to this data, you must first go through the [process outlined here](/odocs/data-guidelines/).\\n\\nThis case was brought to the attention of FaceBase from Dr. Pedro Sanchez of Children's Hospital Lost Angeles.\",\n",
       " 'mouse_genetic': None,\n",
       " 'human_anatomic': None,\n",
       " 'study_design': '1. Interesting cases are seen by the clinicians on our protocol and they are presented at a monthly meeting.\\n2. The cases are looked at based on the solvability of the case, if we can obtain the correct family members for sequencing and if the family is willing to participate.\\n3. Samples are obtained, usually for WES first, but sometimes WES has not led to an answer, so WGS is done.\\n4. The data is sent to Brigham and Women’s Hospital analysis program, Brigham Genomic Medicine, where computational biologists look at the sequences and find the variant(s) that explains the phenotype seen in the proband.\\n5. Functional analysis is done to mimic the phenotype in a mouse, which confirms what was seen with the variant.\\n6. Findings are shared with the clinicians, so they may share with their patient.\\n\\n',\n",
       " 'release_date': '2018-02-27',\n",
       " 'show_in_jbrowse': None,\n",
       " '_keywords': 'Rapid Identification and Validation of Human Craniofacial Development Genes Pending',\n",
       " 'RID': '2BA6',\n",
       " 'RCB': 'https://auth.globus.org/de244c2a-618a-4f51-9497-4910a200e99a',\n",
       " 'RMB': 'https://auth.globus.org/de244c2a-618a-4f51-9497-4910a200e99a',\n",
       " 'RCT': '2018-02-27T14:29:49.493621-08:00',\n",
       " 'RMT': '2018-10-03T05:44:41.57982-07:00',\n",
       " 'released': True}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results[9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get a specific attribute value from an entity\n",
    "To get one attribute value from an entity get the item using its `Column`'s `name` property."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FB00000953\n"
     ]
    }
   ],
   "source": [
    "dataset = pb.schemas['isa'].tables['dataset']\n",
    "print (results[9][dataset.accession.name])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fetch a Limited Number of Results\n",
    "To set a limit on the number of results to be fetched from the catalog, use the explicit `fetch(limit=...)` method with the desired upper limit to fetch from the catalog."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.fetch(limit=3)\n",
    "len(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Iterate over the ResultSet\n",
    "`ResultSet`s are iterable like a typical container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FB00000970\n",
      "FB00000998\n",
      "FB00000936\n"
     ]
    }
   ],
   "source": [
    "for entity in results:\n",
    "    print(entity[dataset.accession.name])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert to Pandas DataFrame\n",
    "ResultSets can be transformed into the popular Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RCB</th>\n",
       "      <th>RCT</th>\n",
       "      <th>RID</th>\n",
       "      <th>RMB</th>\n",
       "      <th>RMT</th>\n",
       "      <th>_keywords</th>\n",
       "      <th>accession</th>\n",
       "      <th>description</th>\n",
       "      <th>funding</th>\n",
       "      <th>human_anatomic</th>\n",
       "      <th>id</th>\n",
       "      <th>mouse_genetic</th>\n",
       "      <th>project</th>\n",
       "      <th>release_date</th>\n",
       "      <th>released</th>\n",
       "      <th>show_in_jbrowse</th>\n",
       "      <th>study_design</th>\n",
       "      <th>summary</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://auth.globus.org/f8ae714f-6015-48da-971...</td>\n",
       "      <td>2018-05-02T10:59:18.781231-07:00</td>\n",
       "      <td>3TYP</td>\n",
       "      <td>https://auth.globus.org/b153e992-d274-11e5-8df...</td>\n",
       "      <td>2018-09-05T10:44:44.083799-07:00</td>\n",
       "      <td>None</td>\n",
       "      <td>FB00000970</td>\n",
       "      <td>Single Cell RNA-Seq libraries are from C57BL/6...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>14214</td>\n",
       "      <td>None</td>\n",
       "      <td>302</td>\n",
       "      <td>2018-05-02</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Single Cell Sequencing - Coronal Suture, Wild ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://auth.globus.org/f8ae714f-6015-48da-971...</td>\n",
       "      <td>2018-08-09T08:59:44.686625-07:00</td>\n",
       "      <td>1-3X0M</td>\n",
       "      <td>https://auth.globus.org/b506963e-d274-11e5-99f...</td>\n",
       "      <td>2018-10-29T18:59:40.882334-07:00</td>\n",
       "      <td>None</td>\n",
       "      <td>FB00000998</td>\n",
       "      <td>RNA-Seq libraries are from laser capture micro...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>14242</td>\n",
       "      <td>None</td>\n",
       "      <td>302</td>\n",
       "      <td>2018-08-29</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lambdoid Suture, WT and Fgfr2+/S252W (Apert sy...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://auth.globus.org/93416595-2bc4-42a8-b5b...</td>\n",
       "      <td>2018-02-21T15:30:25.884648-08:00</td>\n",
       "      <td>2AC6</td>\n",
       "      <td>https://auth.globus.org/bb256144-d274-11e5-adb...</td>\n",
       "      <td>2018-11-28T14:21:46.302023-08:00</td>\n",
       "      <td>Integrated research of functional genomics and...</td>\n",
       "      <td>FB00000936</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>14180</td>\n",
       "      <td>None</td>\n",
       "      <td>300</td>\n",
       "      <td>2018-03-05</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>E10.5 mandible of mutants and controls were di...</td>\n",
       "      <td>RNAseq of Wnt1cre;Alk5fl/fl mutants and contro...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 RCB  \\\n",
       "0  https://auth.globus.org/f8ae714f-6015-48da-971...   \n",
       "1  https://auth.globus.org/f8ae714f-6015-48da-971...   \n",
       "2  https://auth.globus.org/93416595-2bc4-42a8-b5b...   \n",
       "\n",
       "                                RCT     RID  \\\n",
       "0  2018-05-02T10:59:18.781231-07:00    3TYP   \n",
       "1  2018-08-09T08:59:44.686625-07:00  1-3X0M   \n",
       "2  2018-02-21T15:30:25.884648-08:00    2AC6   \n",
       "\n",
       "                                                 RMB  \\\n",
       "0  https://auth.globus.org/b153e992-d274-11e5-8df...   \n",
       "1  https://auth.globus.org/b506963e-d274-11e5-99f...   \n",
       "2  https://auth.globus.org/bb256144-d274-11e5-adb...   \n",
       "\n",
       "                                RMT  \\\n",
       "0  2018-09-05T10:44:44.083799-07:00   \n",
       "1  2018-10-29T18:59:40.882334-07:00   \n",
       "2  2018-11-28T14:21:46.302023-08:00   \n",
       "\n",
       "                                           _keywords   accession  \\\n",
       "0                                               None  FB00000970   \n",
       "1                                               None  FB00000998   \n",
       "2  Integrated research of functional genomics and...  FB00000936   \n",
       "\n",
       "                                         description funding human_anatomic  \\\n",
       "0  Single Cell RNA-Seq libraries are from C57BL/6...    None           None   \n",
       "1  RNA-Seq libraries are from laser capture micro...    None           None   \n",
       "2                                               None    None           None   \n",
       "\n",
       "      id mouse_genetic  project release_date  released  show_in_jbrowse  \\\n",
       "0  14214          None      302   2018-05-02      True            False   \n",
       "1  14242          None      302   2018-08-29      True             True   \n",
       "2  14180          None      300   2018-03-05      True             True   \n",
       "\n",
       "  study_design                                            summary  \\\n",
       "0         None                                               None   \n",
       "1         None                                               None   \n",
       "2         None  E10.5 mandible of mutants and controls were di...   \n",
       "\n",
       "                                               title  \n",
       "0  Single Cell Sequencing - Coronal Suture, Wild ...  \n",
       "1  Lambdoid Suture, WT and Fgfr2+/S252W (Apert sy...  \n",
       "2  RNAseq of Wnt1cre;Alk5fl/fl mutants and contro...  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Selecting Attributes\n",
    "It is also possible to fetch only a subset of attributes from the catalog. The `attributes(...)` method accepts a variable argument list followed by keyword arguments. Each argument must be a `Column` object from the table's `columns` container.\n",
    "\n",
    "### Renaming selected attributes\n",
    "To rename the selected attributes, use \"named\" (a.k.a., \"keyword\") arguments in the method. For example, `attributes(..., new_name=table.column)` will rename `table.column` with `new_name` in the entities returned from the server. (It will not change anything in the stored catalog data.) Note that in python, the named arguments _must come after_ positional arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = path.attributes(dataset.accession, dataset.title, is_released=dataset.released).fetch(limit=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert to list\n",
    "Now we can look at the results from the above fetch. To demonstrate a different access mode, we can convert the entities to a standard python list and dump to the console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'accession': 'FB00000970',\n",
       "  'title': 'Single Cell Sequencing - Coronal Suture, Wild Type, E18.5 and P10',\n",
       "  'is_released': True},\n",
       " {'accession': 'FB00000998',\n",
       "  'title': 'Lambdoid Suture, WT and Fgfr2+/S252W (Apert syndrome mouse model), E16.5 and E18.5',\n",
       "  'is_released': True},\n",
       " {'accession': 'FB00000936',\n",
       "  'title': 'RNAseq of Wnt1cre;Alk5fl/fl mutants and controls at E10.5',\n",
       "  'is_released': True},\n",
       " {'accession': 'FB00000982',\n",
       "  'title': 'FB0064_Male with Congenital craniosynostosis_Candidate Gene: FREM2',\n",
       "  'is_released': True},\n",
       " {'accession': 'FB00000858',\n",
       "  'title': 'Wild Type E18.5 Frontal Suture Images',\n",
       "  'is_released': True}]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}