UnixAndPythonCheatSheet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use Unix and/or Python to view, sort and parse (split or separarate) large data sets such as those from genome-wide gene expression studies. <a name='UnixPython'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### UNIX COMMANDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pwd  #lists the name of the current directory \n",
    "!ls #displays the contents of the current directory \n",
    "!wc #counts the number of characters in a file \n",
    "!wc -l #counts the number of lines in a file \n",
    "!cat #prints the contents of a file\n",
    "!head -n10 # prints the first 10 lines of a file. Note: if your file is large, use the head or tail command rather than the cat\n",
    "           # command to examine its contents \n",
    "!tail -n10 #prints the last 10 lines of a file. Note: if your file is large, use the head or tail command rather than the cat\n",
    "           # command to examine its contents \n",
    "!grep \"target\" source_file_name  #searches for the string \"target\" inside of the source file\n",
    "!cut -f1,2,3  #cuts the specified columns (in this case 1,2,3) from a file\n",
    "!echo text  #writes \"text\" to the screen or to a file\n",
    "!zcat #view a zipped file such as a file with a .gz extension\n",
    "!sort #sorts a file, use the -k flag to sort by a specific column, the -r flag to sort in reverse order"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### UNIX OPERATORS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "> write to a file\n",
    "| send the output of one command to the input of the next"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get help with a command you can type the command and use the --h or -help flags. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PYTHON COMMANDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#gets the current working directory\n",
    "import os\n",
    "os.getcwd()\n",
    "#changes the current working directory\n",
    "os.chdir('..')\n",
    "#lists files in the current directory, a single period (.)  stands for the current directory \n",
    "os.listdir('.')\n",
    "#make a subdirectory in the working directory \n",
    "os.mkdir('YOUR DIRECTORY')\n",
    "\n",
    "\n",
    "#Open a file and create a file object \n",
    "#The 'r' means that the file is readable.'w' would mean the file is writable.  \n",
    "sequence=open('sequence.txt','r')\n",
    "#open a file removing white space and splitting at line breaks\n",
    "data=open(\"bedfile.bed\",'r').read().strip().split('\\n')\n",
    "   \n",
    "\n",
    "#Read the sequence file contents and print them \n",
    "print(sequence.read())\n",
    "\n",
    "#calculate the length of a string\n",
    "len(sequence)\n",
    "\n",
    "#convert a number i to a string\n",
    "str(i)\n",
    "\n",
    "\n",
    "#make a for loop \n",
    "for i in range(0,len(sequence)):\n",
    "\n",
    "#make an if statement:     \n",
    "if sequence[i]=='G':\n",
    "    \n",
    "#Creating a dictionary\n",
    "geneticcode3let={'UUU':'Phe','UUC':'Phe','UAA':'Leu','UUG':'Leu'}\n",
    "\n",
    "#Defining a function that returns x\n",
    "def my_function(a,b):\n",
    "    '''this function adds two numbers '''\n",
    "    x=a+b\n",
    "    return(x)\n",
    "\n",
    "#get help on a function or module \n",
    "help (my_function)\n",
    "\n",
    " \n",
    "#write contents of a window\n",
    "%%writefile ../helpers/central_dogma_helpers_updated.py\n",
    "\n",
    "#write a string to an output file \n",
    "outf=open(\"outputfilename.txt\",'w') #opens the file for writing\n",
    "mydata=\"data\"\n",
    "outf.write(mydata)# writes data to the file\n",
    "outf.close() # closes the file \n",
    "\n",
    "\n",
    "#append data to an existing file, without overwriting it\n",
    "outf=open(\"outputfilename.txt\",'a') #opens the file for apending \n",
    "       \n",
    "\n",
    "#importing .tsv file into Python using pandas\n",
    "import pandas\n",
    "df = pd.read_table(\n",
    "     filepath_or_buffer='filename.tsv', \n",
    "     header=0,\n",
    "     index_col=0\n",
    ")\n",
    "\n",
    "\n",
    "#importing .csv file into Python using pandas\n",
    "df = pd.read_csv(\n",
    "     filepath_or_buffer='filename.csv', \n",
    "     header=0,\n",
    "     index_col=0\n",
    ")\n",
    "\n",
    "#specifying rows or columns in a dataframe\n",
    "x=df['columnname']\n",
    "x=df.loc[rowname]\n",
    "x=df.iloc[rownumber(s),columnnumber(s)]\n",
    "\n",
    "#sorting pandas dataframe\n",
    "df_sorted=pd.DataFrame.sort_values(df,by=\"columnname\",ascending=False)\n",
    " \n",
    "#importing packages\n",
    "import Bio \n",
    "print(Bio.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BIOLOGY TOOLS "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#SeqIO package provides many functions for sequence operations \n",
    "seq1=SeqIO.read('sequence1.txt',\"fasta\")\n",
    "\n",
    "\n",
    "#Bio package contains alignment functions \n",
    "\n",
    "#pairwise alignment \n",
    "from Bio import pairwise2 \n",
    "alignments = pairwise2.align.globalxx(seq1.seq, seq2.seq)\n",
    "\n",
    "#multiple sequence alignment with MUSCLE algorithm \n",
    "from Bio.Align.Applications import MuscleCommandline\n",
    "\n",
    "#alignment to reference genome with Bowtie2 algorithm "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TOOLS FOR STATISTICAL ANALYSIS AND MACHINE LEARNING"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#PCA \n",
    "from sklearn.decomposition import PCA as sklearnPCA\n",
    "#We decompose the data in dataframe df into 10 principal components \n",
    "sklearn_pca = sklearnPCA(n_components=10)\n",
    "pca_results = sklearn_pca.fit_transform(df)\n",
    "\n",
    "#KMeans clustering \n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "#GO Term enrichment analysis \n",
    "http://cbl-gorilla.cs.technion.ac.il/\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tools for investigating if a variant is in a UTR, coding region (CDS), promotor or enhancer\n",
    "\n",
    "[Bedtools](http://bedtools.readthedocs.io/en/latest/)\n",
    "* `bedtools sort -i myFile.bed` sorts myFile.bed by chromosome, start coordinate, end coordinate \n",
    "* `bedtools intersect -wa -a fileA -b fileB` -- finds all entries in fileB that intersect entries in fileA \n",
    "* `bedtools closest -wa -a fileA -b fileB` -- finds the closest entry in fileB to each entry in fileA. \n",
    "* `bedtools getfasta -fi hg19.genome.fa -bed human_insulin_exon_boundaries.bed -fo human_insulin_exons.fa.out` -- extracts the fasta sequence from the hg19 reference genome corresponding to the positions in the input bed file 'human_insulin_exon_boundaries.bed'   \n",
    "\n",
    "\n",
    "[WashU Epigenome Roadmap Browser](http://epigenomegateway.wustl.edu/browser/) with Chromatin State Tracks from the Public Track Roadmap Epigenomics Integrative Analysis Hub. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyze datasets from cases and controls to identify sites in the genome that are likely to be relevant to a disease. <a name='CasesandControls'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=#GWAS> See GWAS Review above </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The Biobank Engine https://biobankengine.stanford.edu/ is a databse of GWAS associations from the UK Biobank dataset. \n",
    "\n",
    "\n",
    "* The GWAS Catalog https://www.ebi.ac.uk/gwas/ is often regarded as a \"gold standard\" for querying known GWAS hits. \n",
    "\n",
    "\n",
    "* Gene Cards http://www.genecards.org/ provides information about the function of a gene as well as known associations with diseases, and known pathogenic variants inside the gene. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Query a large data set and visualize the data by making or interpreting a scatter plot, barplot, histogram or heatmap. <a name='Plot'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting packages \n",
    "\n",
    "Plotly: https://plot.ly/python/ <br>\n",
    "Matplotlib: https://matplotlib.org/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#load the necessary modules for plotly\n",
    "import plotly \n",
    "plotly.offline.init_notebook_mode()\n",
    "import plotly.plotly as py\n",
    "from plotly.graph_objs import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#Make a Scatter Plot using plotly\n",
    "trace=Scatter(\n",
    "    #selects the x-values for the scatter plot\n",
    "    x,       \n",
    "    #selects the y-values for the scatter plot \n",
    "    y, \n",
    "    #adds in text that will be displayed on the points\n",
    "    text=text,\n",
    "    #defines the mode of the plot, in this case markers (as opposed to lines or text)\n",
    "    mode=\"markers\")\n",
    "\n",
    "#Label the axes \n",
    "layout=Layout(xaxis=dict(title='x-axis title'),yaxis=dict(title='y-axis title'),showlegend=True)\n",
    "\n",
    "#Draw the figure \n",
    "fig=Figure(data=[trace],layout=layout)\n",
    "plotly.offline.iplot(fig)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#Make a Bar plot using plotly\n",
    "trace=Bar(\n",
    "    #selects the x-values for the scatter plot\n",
    "    x,   \n",
    "    #selects the y-values for the scatter plot\n",
    "    y      \n",
    "    )\n",
    "#Label the axes \n",
    "layout=Layout(xaxis=dict(title='x-axis title'),yaxis=dict(title='y-axis title'),showlegend=True)\n",
    "\n",
    "#Draw the figure \n",
    "fig=Figure(data=[trace],layout=layout)\n",
    "plotly.offline.iplot(fig) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#Make a histogram using plotly\n",
    "#Define the values for the histogram\n",
    "data = [Histogram(x=x)]\n",
    "\n",
    "#Label the axes\n",
    "layout=Layout(xaxis=dict(title='x-axis title'),yaxis=dict(title='y-axis title'))\n",
    "\n",
    "#Draw the figure  \n",
    "fig=Figure(data=data,layout=layout)\n",
    "plotly.offline.iplot(fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Anaconda to perform computational analysis on your computer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have been running code on the Google Compute Cloud, but you can also run all of this analysis on your own computer. To do this, you can install the Anaconda Python distribution from the Anaconda website: https://www.anaconda.com/download/\n",
    "\n",
    "Once you have installed Anaconda following the instructions on the website, you can add Python packages and libraries with the following command: \n",
    "\n",
    "**`conda install packagename`**\n",
    "\n",
    "for example: \n",
    "**`conda install pandas`** will install the pandas library.\n",
    "\n",
    "If you want to check whether a given package is installed, you can use the command: \n",
    "\n",
    "**`conda list packagename`** \n",
    "\n",
    "for example: \n",
    "\n",
    "**`conda list pandas`** \n",
    "\n",
    "If you want to upgrade a package to a more recent version: \n",
    "\n",
    "**`conda upgrade packagename`** \n",
    "\n",
    "And finally, if you want to uninstall a package: \n",
    "\n",
    "** `conda uninstall packagename`**\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}