docs/derivapy-datapath-example-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# DataPath Example 1\n",
    "This notebook gives an example of how to access the model elements (e.g.,\n",
    "schemas, tables, columns) that are used when building data paths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import deriva modules\n",
    "from deriva.core import ErmrestCatalog, get_credential"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect with the deriva catalog\n",
    "protocol = 'https'\n",
    "hostname = 'www.facebase.org'\n",
    "catalog_number = 1\n",
    "credential = get_credential(hostname)\n",
    "catalog = ErmrestCatalog(protocol, hostname, catalog_number, credential)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The PathBuilder Interface\n",
    "The path builder interface gives you access to a representation of the catalog's data model beginning with the catalog's schemas. The path builder does *not* record any state about your paths, it is just an entry point to begin building paths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the path builder interface for this catalog\n",
    "pb = catalog.getPathBuilder()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Access to Schemas\n",
    "The `.schemas` property acts like a python dictionary or Map object. Use its `keys()` method to\n",
    "get a listing of the schema names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'vocab', u'viz', u'isa', u'_acl_admin']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pb.schemas.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Here we will get a handle to the `isa` schmea."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "isa = pb.schemas['isa']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**PROTIP**: Jupyter Notebook supports `<tab>` completion. Press the `<tab>` key\n",
    "after typing the brackets of a dictionary to see the keys. Typing\n",
    "`pb.schemas[<tab>]` will give you a dropdown of schema names. Note that the notebook\n",
    "interpretter will not know anything about your objects until you have executed a \n",
    "step with them. So instantiate an object in one step, and then you can use tab-completion\n",
    "in the following steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alternative access method for schemas and other model objects\n",
    "\n",
    "An alternative way to get a handle to the same schema object is \n",
    "directly as a property of the path builder object itself. However,\n",
    "this only works for schema names that are _valid python identifiers_.\n",
    "\n",
    "A valid python identifier may start with '`_`' or a letter as its first\n",
    "character and have '`_`', letters, or numbers for the rest of its characters.\n",
    "- **Valid Python identifiers**: `'dataset'`, `'assay'`, `'Molecule_Type'`, etc.\n",
    "- **Invalid Python identifiers**: `'Sample 1 Type'`, `'Control?'`, `'# of reads'`, etc.\n",
    "\n",
    "**IMPORTANT**\n",
    "Similar access methods will be demonstrated for tables and columns \n",
    "below. Since not all catalog model names are valid python identifiers\n",
    "when you use this method, you may not see your catalog's complete\n",
    "data model. However, the notation is more compact and ideal for\n",
    "cases where your model uses (all or mostly) valid python identifiers\n",
    "in its model element names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "isa = pb.isa  # same schema object we got from the previous step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Access to Tables\n",
    "Similarly, a schema object has a `tables` property that gives you access\n",
    "to a representation of the catalog schema's tables. Again, use its `keys()`\n",
    "method to list the table names in the schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'enhancer',\n",
       " u'protocol',\n",
       " u'dataset_mutation',\n",
       " u'dataset_gene',\n",
       " u'dataset_somite_count',\n",
       " u'dataset',\n",
       " u'sample',\n",
       " u'dataset_instrument',\n",
       " u'mesh_data',\n",
       " u'file',\n",
       " u'project_publication',\n",
       " u'dataset_genotype',\n",
       " u'dataset_data_type',\n",
       " u'clinical_assay',\n",
       " u'alignment',\n",
       " u'biosample_cell_source',\n",
       " u'project_investigator',\n",
       " u'publication',\n",
       " u'library',\n",
       " u'imaging_data',\n",
       " u'sequencing_data',\n",
       " u'experiment',\n",
       " u'replicate',\n",
       " u'processed_data',\n",
       " u'project_member',\n",
       " u'dataset_cell_characterization',\n",
       " u'thumbnail',\n",
       " u'biosample',\n",
       " u'dataset_experiment_type',\n",
       " u'dataset_mouse_genetic_background',\n",
       " u'array_data',\n",
       " u'pipeline',\n",
       " u'previews',\n",
       " u'tracks',\n",
       " u'dataset_gender',\n",
       " u'track_data',\n",
       " u'dataset_phenotype',\n",
       " u'dataset_anatomy',\n",
       " u'icon',\n",
       " u'dataset_cell_source',\n",
       " u'dataset_human_age',\n",
       " u'dataset_chromosome',\n",
       " u'biosample_cell_characterization',\n",
       " u'dataset_enhancer',\n",
       " u'sample_replicate_group',\n",
       " u'dataset_stage',\n",
       " u'dataset_geo',\n",
       " u'project',\n",
       " u'person',\n",
       " u'external_reference',\n",
       " u'dataset_organism']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "isa.tables.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Similarly we can get a table from the schema's `tables` property in\n",
    "both of the demonstrated methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = isa.tables['dataset']\n",
    "# or\n",
    "dataset = isa.dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Access to Columns\n",
    "A table has a `columns` dictionary. We can get the column names as usual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'RMB',\n",
       " u'funding',\n",
       " u'description',\n",
       " u'_keywords',\n",
       " u'title',\n",
       " u'RCT',\n",
       " u'release_date',\n",
       " u'show_in_jbrowse',\n",
       " u'accession',\n",
       " u'human_anatomic',\n",
       " u'summary',\n",
       " u'project',\n",
       " u'study_design',\n",
       " u'RMT',\n",
       " u'mouse_genetic',\n",
       " u'RID',\n",
       " u'released',\n",
       " u'id',\n",
       " u'RCB']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.column_definitions.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Again, we have the following methods to get handles to the table's\n",
    "column objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "accession = dataset.column_definitions['accession']\n",
    "# or\n",
    "accession = dataset.accession"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Final Thought\n",
    "\n",
    "The model introspection provided in the datapath\n",
    "module (i.e., the PathBuilder) is intended for the narrowly scoped\n",
    "usage required for building paths and accessing data from ERMrest\n",
    "catalogs. It is _not_ intended for general introspection of catalogs\n",
    "and therefore _does not_ include details such as constraints, \n",
    "annotations, ACLs, column data types, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}