examples/Pyspark Kernel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Features available with the new kernels\n",
    "\n",
    "By default Jupyter notebook comes with a `Python` kernel. sparkmagic provides two additional kernels that you can use with Jupyter. These are:\n",
    "\n",
    "1. **PySpark** (for applications written in Python). PySpark kernel exposes the Spark programming model to Python\n",
    "2. **Spark** (for applications written in Scala)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------\n",
    "## How do I use the new kernels? \n",
    "\n",
    "1. Create a notebook with the new kernels. Click **New**, and then click **PySpark** or **Spark**. \n",
    "![Create notebooks with new kernels](https://mysstorage.blob.core.windows.net/notebookimages/overview/jupyter-kernels.png \"Create notebooks with new kernels\") \n",
    "2. This should open a new notebook with the kernel you selected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------\n",
    "## Notebook setup\n",
    "\n",
    "When using PySpark kernel notebooks, there is no need to create a SparkContext or a HiveContext; those are all created for you automatically when you run the first code cell, and you'll be able to see the progress printed. The contexts are created with the following variable names:\n",
    "- SparkContext (sc)\n",
    "- HiveContext (sqlContext)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everytime you run a cell, your web browser window title will show a **(Busy)** status along with the notebook title. You will also see a solid circle next to the **PySpark** text in the top-right corner. After the job completes, this will change to a hollow circle.\n",
    "\n",
    "![Status of a Jupyter notebook job](https://mysstorage.blob.core.windows.net/notebookimages/overview/HDI.Spark.Jupyter.Job.Status.png \"Status of a Jupyter notebook job\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------\n",
    "## PySpark magics \n",
    "\n",
    "The PySpark kernel provides some predefined “magics”, which are special commands that you can call with `%%` (e.g. `%%MAGIC` <args>). The magic command must be the first word in a code cell and allow for multiple lines of content. You can’t put comments before a cell magic.\n",
    "\n",
    "For more information on magics, see [here](http://ipython.readthedocs.org/en/stable/interactive/magics.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Help magic (%%help)\n",
    "\n",
    "This magic gives you information about the different supported magics in PySpark kernel and the usage for each. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<table>\n",
       "  <tr>\n",
       "    <th>Magic</th>\n",
       "    <th>Example</th>\n",
       "    <th>Explanation</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <td>info</td>\n",
       "    <td>%%info</td>\n",
       "    <td>Outputs session information for the current Livy endpoint.</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <td>cleanup</td>\n",
       "    <td>%%cleanup -f</td>\n",
       "    <td>Deletes all sessions for the current Livy endpoint, including this notebook's session. The force flag is mandatory.</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <td>delete</td>\n",
       "    <td>%%delete -f -s 0</td>\n",
       "    <td>Deletes a session by number for the current Livy endpoint. Cannot delete this kernel's session.</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <td>logs</td>\n",
       "    <td>%%logs</td>\n",
       "    <td>Outputs the current session's Livy logs.</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <td>configure</td>\n",
       "    <td>%%configure -f<br/>{\"executorMemory\": \"1000M\", \"executorCores\": 4}</td>\n",
       "    <td>Configure the session creation parameters. The force flag is mandatory if a session has already been\n",
       "    created and the session will be dropped and recreated.<br/>Look at <a href=\"https://github.com/cloudera/livy#request-body\">\n",
       "    Livy's POST /sessions Request Body</a> for a list of valid parameters. Parameters must be passed in as a JSON string.</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <td>sql</td>\n",
       "    <td>%%sql -o tables -q<br/>SHOW TABLES</td>\n",
       "    <td>Executes a SQL query against the sqlContext.\n",
       "    Parameters:\n",
       "      <ul>\n",
       "        <li>-o VAR_NAME: The result of the query will be available in the %%local Python context as a\n",
       "          <a href=\"http://pandas.pydata.org/\">Pandas</a> dataframe.</li>\n",
       "        <li>-q: The magic will return None instead of the dataframe (no visualization).</li>\n",
       "        <li>-m METHOD: Sample method, either <tt>take</tt> or <tt>sample</tt>.</li>\n",
       "        <li>-n MAXROWS: The maximum number of rows of a SQL query that will be pulled from Livy to Jupyter.\n",
       "            If this number is negative, then the number of rows will be unlimited.</li>\n",
       "        <li>-r FRACTION: Fraction used for sampling.</li>\n",
       "      </ul>\n",
       "    </td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <td>local</td>\n",
       "    <td>%%local<br/>a = 1</td>\n",
       "    <td>All the code in subsequent lines will be executed locally. Code must be valid Python code.</td>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "execute_result",
     "execution_count": 1
    }
   ],
   "source": [
    "%%help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Session information (%%info)\n",
    "\n",
    "Livy is an open source REST server for Spark. When you execute a code cell in a PySpark notebook, it creates a Livy session to execute your code. You can use the `%%info` magic to display the current Livy session information. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Current session configs: <tt>{u'kind': 'pyspark'}</tt><br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "execute_result",
     "execution_count": 1
    },
    {
     "data": {
      "text/html": [
       "No active sessions."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "execute_result",
     "execution_count": 1
    }
   ],
   "source": [
    "%%info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Session configuration (%%configure)\n",
    " \n",
    "Use the `%%configure` magic to configure new or existing Livy sessions.\n",
    "* If a session is already running, you can change the configuration by using the `-f` argument with `%%configure` magic. This will delete the current session and recreate it with the applied configurations. If you don't provide the `-f` argument, an error will be displayed and no configuration changes will be applied.\n",
    "* If you haven't already started the session, then the `-f` argument is not mandatory. Even if you use it with a session that you are just creating, it will not delete any currently running sessions.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "These are some session attributes that can be used for configuration \n",
    "- **\"name\"**: Name of the application\n",
    "- **\"driverMemory\"**: Memory for driver (e.g. 1000M, 2G) \n",
    "- **\"executorMemory\"**: Memory for executor (e.g. 1000M, 2G) \n",
    "\n",
    "For more attributes for session configuration see <a href=\"https://github.com/cloudera/livy/tree/6fe1e80cfc72327c28107e0de20c818c1f13e027#post-sessions\" target=\"_blank\"> the Livy documentation</a>.\n",
    "\n",
    "> **TIP**: The application name should start with `remotesparkmagics` to allow sessions to get automatically cleaned up if an error happened. If you provide a name that does not start with `remotesparkmagics` it will not result in an error but the cleanup won't occur."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Current session configs: <tt>{u'executorCores': 4, u'kind': 'pyspark', u'executorMemory': u'4G', u'name': u'remotesparkmagics-sample'}</tt><br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "execute_result",
     "execution_count": 1
    },
    {
     "data": {
      "text/html": [
       "No active sessions."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "execute_result",
     "execution_count": 1
    }
   ],
   "source": [
    "%%configure -f \n",
    "{\"name\":\"remotesparkmagics-sample\", \"executorMemory\": \"4G\", \"executorCores\":4}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------\n",
    "## Automatic visualization of queries \n",
    "\n",
    "The Pyspark kernel automatically visualizes the output of SQL (HiveQL) queries. You are given the option to choose between several different types of visualizations:\n",
    "- Table\n",
    "- Pie\n",
    "- Line \n",
    "- Area\n",
    "- Bar\n",
    "\n",
    ">**TIP**: When you perform a SQL query, the number of rows of data that will be included in the result data set will be limited by default to 2500 rows. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SQL magic (%%sql)\n",
    "\n",
    "The PySpark kernel supports easy inline HiveQL queries against the `sqlContext`. The (`-o VARIABLE_NAME`) argument persists the output of the SQL query as a <a href=\"http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html\" target=\"_blank\">Pandas dataframe</a> on the Jupyter server (e.g. `-o query1` in the example below). This means it'll be available in the local mode which will be explained later. The output will be automatically visualized after you run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting Spark application\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>Current session?</th></tr><tr><td>4</td><td>None</td><td>pyspark</td><td>idle</td><td></td><td></td><td>✔</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SparkSession available as 'spark'.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/itamarst/Devel/sparkmagic/autovizwidget/autovizwidget/widget/utils.py:50: FutureWarning:\n",
      "\n",
      "A future version of pandas will default to `skipna=True`. To silence this warning, pass `skipna=True|False` explicitly.\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b9e21546ff44b808f6770ae9587cdce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(HTML(value='Type:'), Button(description='Table', layout=Layout(width='70px'), st…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c32aad77d89941f388d8e819bb91e2b4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql -o query1\n",
    "SELECT 1, 2, 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![widget](images/widget.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to `-o`, a number of other configuration parameters for SQL queries are available (as described in the `%%help` output above). These include:\n",
    "\n",
    "* `-q`. This causes the magic to return `None`, turning off visualizations for that cell. If you don't want to auto-visualize the content of a cell and just want to capture it as a dataframe, then use `-q -o VARIABLE`. If you want to turn off visualizations without capturing the results (e.g. for running a SQL query with side effects, like a `CREATE TABLE` statement), just use `-q` without specifying a `-o` argument.\n",
    "\n",
    "> Remember that the kernel by default limits the output of a SQL query to 2500 rows. If you want to adjust this default behavior, use these options.\n",
    "\n",
    "* `-m METHOD`, where `METHOD` is either `take` or `sample` (defaults to `take`). If the method is `take`, the kernel will take elements from the top of the result data set; if the method is `sample`, the kernel will randomly sample elements of the data set according to `-r`, described below.\n",
    "\n",
    "* `-r FRACTION`, where `FRACTION` is some floating-point number between 0.0 and 1.0. If the sample method for the SQL query is `sample`, then the kernel will randomly sample this fraction of the elements of the result set for you; e.g. if you run a SQL query with the arguments `-m sample -r 0.01`, then 1% of the result rows will be randomly sampled.\n",
    "\n",
    "* `-n MAXROWS`, where `MAXROWS` is some integer. The kernel will limit the number of output rows to `MAXROWS`. If `MAXROWS` is a negative number such as `-1`, then the number of rows in the result set will not be limited.\n",
    "\n",
    "> **WARNING**: Be careful with the `-n` option, as it is possible to cause the Jupyter server to run out of memory if you try to collect too many result rows. We recommend only using `-n -1` if you are certain that the result dataset will not be too large.\n",
    "\n",
    "As a final example, this SQL query randomly samples 10% of the rows in the **hivesampletable**, limits the size of the result set to 500, and saves the output into a dataframe called `query2` without doing any visualizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%sql -q -m sample -r 0.1 -n 500 -o query2 \n",
    "SELECT * FROM hivesampletable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Server-side rendering\n",
    "\n",
    "You can render images on the server and then display them locally. First, we create the image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<matplotlib.axes._subplots.AxesSubplot object at 0x7f2831efbe10>"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "df = spark.read.json(\"/apps/spark-2.3.3/examples/src/main/resources/people.json\")\n",
    "df.createOrReplaceTempView(\"people\")\n",
    "df.toPandas().plot.bar(x='name',y='age')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then we render it locally using the `%matplot` magic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAHgCAYAAAA10dzkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XuUlnW9///XcAZhGFGHGWLkYJ5RM01lechTKhbb09a0tolarm1YW6mlm1VZ6Daq3TJrbbNdIZhllqWWttNVpLBR8YCBWIlIkJgcdracEdwONDO/P/o2vybxsNOZC+bzeKx1rTX3dV337ZtZ94Kn1/2577umo6OjIwAAFKNP1QMAANCzBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGH6VT3Atqy9vT3PPvtshg0blpqamqrHAQBeh46OjrzwwgsZNWpU+vQp81qYAHwDnn322TQ1NVU9BgDwd1i9enVGjx5d9RiVEIBvwLBhw5L8+QlUW1tb8TQAwOvR0tKSpqamzn/HSyQA34C/vOxbW1srAAFgG1Py8q0yX/gGACiYAAQAKIwABAAojDWAALAN6ejoyJ/+9Ke0tbVVPcpWq2/fvunXr1/Ra/xeiwAEgG3Epk2bsmbNmrz44otVj7LVGzJkSBobGzNgwICqR9kqCUAA2Aa0t7dn5cqV6du3b0aNGpUBAwa4wrUFHR0d2bRpU/7nf/4nK1euzK677lrshz2/GgEIANuATZs2pb29PU1NTRkyZEjV42zVBg8enP79++d3v/tdNm3alEGDBlU90lZHEgPANsTVrNfH7+nV+e0AABRGAAIAFKbXrgG87rrrct1112XVqlVJkr333juXX355Jk2alCR56aWX8rGPfSw333xzWltbc/zxx+erX/1qRo4cWeHUAPB/N/Zff9Jj/61Vn3t3j/236D699grg6NGj87nPfS6LFi3KI488kqOPPjonnXRSfvWrXyVJLrnkktxxxx255ZZbMm/evDz77LM59dRTK54aAKD79doAnDx5ck488cTsuuuu2W233XLVVVdl6NChWbhwYZqbmzNr1qxcffXVOfroo3PAAQdk9uzZuf/++7Nw4cKqRweAXuWuu+7KYYcdlrq6uuywww55z3vekxUrVnQev//++/O2t70tgwYNyoEHHpjbb789NTU1Wbx4cec5jz/+eCZNmpShQ4dm5MiROfvss/OHP/yhij9Or9BrA/CvtbW15eabb87GjRszceLELFq0KJs3b86xxx7bec4ee+yRnXfeOQ888MArPk5ra2taWlq6bADAq9u4cWOmTZuWRx55JHPnzk2fPn1yyimnpL29PS0tLZk8eXL22WefPProo7nyyitz2WWXdbn/888/n6OPPjr7779/Hnnkkdx1111Zt25dzjjjjIr+RNu+XrsGMEmWLl2aiRMn5qWXXsrQoUNz2223Za+99srixYszYMCA1NXVdTl/5MiRWbt27Ss+3syZMzNjxozuHhuK0JNrlnoz67HYFpx22mldbl9//fXZaaed8utf/zoLFixITU1NvvGNb2TQoEHZa6+98vvf/z4f+tCHOs//j//4j+y///757Gc/2+Uxmpqa8uSTT2a33XbrsT9Lb9GrrwDuvvvuWbx4cR588MFceOGFOeecc/LrX//673686dOnp7m5uXNbvXr1mzgtAPROy5cvz1lnnZXx48entrY2Y8eOTZI8/fTTWbZsWfbdd98uH9Z80EEHdbn/kiVLcs8992To0KGd2x577JEkXV5K5vXr1VcABwwYkLe+9a1JkgMOOCAPP/xwvvzlL+e9731vNm3alOeff77LVcB169aloaHhFR9v4MCBGThwYLfPDQC9yeTJkzNmzJh84xvfyKhRo9Le3p4JEyZk06ZNr+v+GzZsyOTJk/P5z3/+ZccaGxvf7HGL0KsD8G+1t7entbU1BxxwQPr375+5c+d2XpZetmxZnn766UycOLHiKQGg93juueeybNmyfOMb38jhhx+eJFmwYEHn8d133z3f/va309ra2nmR5eGHH+7yGG9/+9vzwx/+MGPHjk2/fkWlS7fptS8BT58+PfPnz8+qVauydOnSTJ8+Pffee2/e//73Z/jw4Tn//PMzbdq03HPPPVm0aFHOPffcTJw4MYccckjVowNAr7H99ttnhx12yNe//vU89dRT+cUvfpFp06Z1Hn/f+96X9vb2XHDBBfnNb36Tu+++O1/84heTJDU1NUmSqVOn5o9//GPOOuusPPzww1mxYkXuvvvunHvuuWlra6vkz7Wt67UBuH79+nzgAx/I7rvvnmOOOSYPP/xw7r777rzrXe9KknzpS1/Ke97znpx22mk54ogj0tDQkFtvvbXiqQGgd+nTp09uvvnmLFq0KBMmTMgll1ySf//3f+88XltbmzvuuCOLFy/O2972tnziE5/I5ZdfniSd6wJHjRqV++67L21tbTnuuOOyzz775OKLL05dXZ3v/P071XR0dHRUPcS2qqWlJcOHD09zc3Nqa2urHge2Kd4F/ObwLuByvPTSS1m5cmXGjRvX5Q0TvdF3vvOdnHvuuWlubs7gwYP/rsd4td+Xf78LWwMIAGx9vvWtb2X8+PF5y1vekiVLluSyyy7LGWec8XfHH69NAAIAlVq7dm0uv/zyrF27No2NjTn99NNz1VVXVT1WryYAAYBKXXrppbn00kurHqMoVk4CABRGAAIAFEYAAsA2xId3vD5+T69OAALANqB///5JkhdffLHiSbYNf/k9/eX3RlfeBAIA24C+ffumrq4u69evT5IMGTKk85sy+P91dHTkxRdfzPr161NXV5e+fftWPdJWSQACwDaioaEhSTojkFdWV1fX+fvi5QQgAGwjampq0tjYmPr6+mzevLnqcbZa/fv3d+XvNQhAANjG9O3bV+DwhngTCABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGF6bQDOnDkz73jHOzJs2LDU19fn5JNPzrJly7qcc+SRR6ampqbL9s///M8VTQwA0DN6bQDOmzcvU6dOzcKFC/Ozn/0smzdvznHHHZeNGzd2Oe9DH/pQ1qxZ07l94QtfqGhiAICe0a/qAbrLXXfd1eX2nDlzUl9fn0WLFuWII47o3D9kyJA0NDT09HgAAJXptVcA/1Zzc3OSZMSIEV32f+c738mOO+6YCRMmZPr06XnxxRerGA8AoMf02iuAf629vT0XX3xxDj300EyYMKFz//ve976MGTMmo0aNymOPPZbLLrssy5Yty6233rrFx2ltbU1ra2vn7ZaWlm6fHQDgzVZEAE6dOjWPP/54FixY0GX/BRdc0PnzPvvsk8bGxhxzzDFZsWJFdtlll5c9zsyZMzNjxoxunxcAoDv1+peAL7rootx555255557Mnr06Fc99+CDD06SPPXUU1s8Pn369DQ3N3duq1evftPnBQDobr32CmBHR0c+8pGP5Lbbbsu9996bcePGveZ9Fi9enCRpbGzc4vGBAwdm4MCBb+qcAAA9rdcG4NSpU3PTTTflRz/6UYYNG5a1a9cmSYYPH57BgwdnxYoVuemmm3LiiSdmhx12yGOPPZZLLrkkRxxxRPbdd9+KpwcA6D69NgCvu+66JH/+sOe/Nnv27EyZMiUDBgzIz3/+81xzzTXZuHFjmpqactppp+WTn/xkBdMCAPScXhuAHR0dr3q8qakp8+bN66FpAAC2Hr3+TSAAAHQlAAEACiMAAQAKIwABAAojAAEACiMAAQAKIwABAAojAAEACiMAAQAKIwABAAojAAEACiMAAQAKIwABAAojAAEACiMAAQAKIwABAAojAAEACiMAAQAKIwABAAojAAEACiMAAQAKIwABAAojAAEACiMAAQAKIwABAAojAAEACiMAAQAKIwABAAojAAEACiMAAQAKIwABAAojAAEACiMAAQAKIwABAAojAAEACiMAAQAKIwABAAojAAEACiMAAQAKIwABAAojAAEACiMAAQAKIwABAAojAAEACiMAAQAKIwABAAojAAEACiMAAQAK02sDcObMmXnHO96RYcOGpb6+PieffHKWLVvW5ZyXXnopU6dOzQ477JChQ4fmtNNOy7p16yqaGACgZ/TaAJw3b16mTp2ahQsX5mc/+1k2b96c4447Lhs3buw855JLLskdd9yRW265JfPmzcuzzz6bU089tcKpAQC6X7+qB+gud911V5fbc+bMSX19fRYtWpQjjjgizc3NmTVrVm666aYcffTRSZLZs2dnzz33zMKFC3PIIYdUMTYAQLfrtVcA/1Zzc3OSZMSIEUmSRYsWZfPmzTn22GM7z9ljjz2y884754EHHqhkRgCAntBrrwD+tfb29lx88cU59NBDM2HChCTJ2rVrM2DAgNTV1XU5d+TIkVm7du0WH6e1tTWtra2dt1taWrpvaACAblLEFcCpU6fm8ccfz8033/yGHmfmzJkZPnx459bU1PQmTQgA0HN6fQBedNFFufPOO3PPPfdk9OjRnfsbGhqyadOmPP/8813OX7duXRoaGrb4WNOnT09zc3Pntnr16m6dHQCgO/TaAOzo6MhFF12U2267Lb/4xS8ybty4LscPOOCA9O/fP3Pnzu3ct2zZsjz99NOZOHHiFh9z4MCBqa2t7bIBAGxreu0awKlTp+amm27Kj370owwbNqxzXd/w4cMzePDgDB8+POeff36mTZuWESNGpLa2Nh/5yEcyceJE7wAGAHq1XhuA1113XZLkyCOP7LJ/9uzZmTJlSpLkS1/6Uvr06ZPTTjstra2tOf744/PVr361hycFAOhZvTYAOzo6XvOcQYMG5dprr821117bAxMBAGwdeu0aQAAAtkwAAgAURgACABRGAAIAFEYAAgAUpte+CxgA/i/G/utPqh6h11j1uXdXPQKvwRVAAIDCCEAAgMIIQACAwghAAIDCCEAAgMIIQACAwghAAIDCCEAAgMIIQACAwghAAIDCCEAAgMIIQACAwghAAIDCCEAAgMIIQACAwghAAIDCCEAAgMIIQACAwghAAIDCCEAAgMIIQACAwghAAIDCCEAAgMIIQACAwghAAIDCCEAAgMIIQACAwghAAIDCCEAAgMIIQACAwghAAIDCCEAAgMIIQACAwghAAIDCCEAAgMIIQACAwghAAIDCCEAAgMIIQACAwghAAIDCCEAAgMIIQACAwvTaAJw/f34mT56cUaNGpaamJrfffnuX41OmTElNTU2X7YQTTqhoWgCAntNrA3Djxo3Zb7/9cu21177iOSeccELWrFnTuX33u9/twQkBAKrRr+oBusukSZMyadKkVz1n4MCBaWho6KGJAAC2Dr32CuDrce+996a+vj677757Lrzwwjz33HOven5ra2taWlq6bAAA25piA/CEE07It771rcydOzef//znM2/evEyaNCltbW2veJ+ZM2dm+PDhnVtTU1MPTgwA8ObotS8Bv5Yzzzyz8+d99tkn++67b3bZZZfce++9OeaYY7Z4n+nTp2fatGmdt1taWkQgALDNKfYK4N8aP358dtxxxzz11FOveM7AgQNTW1vbZQMA2NYIwP/nmWeeyXPPPZfGxsaqRwEA6Fa99iXgDRs2dLmat3LlyixevDgjRozIiBEjMmPGjJx22mlpaGjIihUrcumll+atb31rjj/++AqnBgDofr02AB955JEcddRRnbf/snbvnHPOyXXXXZfHHnssN9xwQ55//vmMGjUqxx13XK688soMHDiwqpEBAHpErw3AI488Mh0dHa94/O677+7BaQAAth7WAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAUptcG4Pz58zN58uSMGjUqNTU1uf3227sc7+joyOWXX57GxsYMHjw4xx57bJYvX17RtAAAPafXBuDGjRuz33775dprr93i8S984Qv5yle+kq997Wt58MEHs9122+X444/PSy+91MOTAgD0rH5VD9BdJk2alEmTJm3xWEdHR6655pp88pOfzEknnZQk+da3vpWRI0fm9ttvz5lnntmTowIA9KheewXw1axcuTJr167Nscce27lv+PDhOfjgg/PAAw9UOBkAQPfrtVcAX83atWuTJCNHjuyyf+TIkZ3HtqS1tTWtra2dt1taWrpnQACAblTkFcC/18yZMzN8+PDOrampqeqRAAD+z4oMwIaGhiTJunXruuxft25d57EtmT59epqbmzu31atXd+ucAADdocgAHDduXBoaGjJ37tzOfS0tLXnwwQczceLEV7zfwIEDU1tb22UDANjW9No1gBs2bMhTTz3VeXvlypVZvHhxRowYkZ133jkXX3xx/u3f/i277rprxo0bl0996lMZNWpUTj755AqnBgDofr02AB955JEcddRRnbenTZuWJDnnnHMyZ86cXHrppdm4cWMuuOCCPP/88znssMNy1113ZdCgQVWNDADQI3ptAB555JHp6Oh4xeM1NTW54oorcsUVV/TgVAAA1StyDSAAQMkEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhBCAAQGEEIABAYQQgAEBhig7Az3zmM6mpqemy7bHHHlWPBQDQrfpVPUDV9t577/z85z/vvN2vX/G/EgCglyu+dvr165eGhoaqxwAA6DFFvwScJMuXL8+oUaMyfvz4vP/978/TTz9d9UgAAN2q6CuABx98cObMmZPdd989a9asyYwZM3L44Yfn8ccfz7Bhw152fmtra1pbWztvt7S09OS4AABviqIDcNKkSZ0/77vvvjn44IMzZsyYfP/738/555//svNnzpyZGTNm9OSIAABvuuJfAv5rdXV12W233fLUU09t8fj06dPT3Nzcua1evbqHJwQAeOME4F/ZsGFDVqxYkcbGxi0eHzhwYGpra7tsAADbmqID8OMf/3jmzZuXVatW5f77788pp5ySvn375qyzzqp6NACAblP0GsBnnnkmZ511Vp577rnstNNOOeyww7Jw4cLstNNOVY8GANBtig7Am2++ueoRAAB6XNEvAQMAlEgAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABRGAAIAFEYAAgAURgACABSm+AC89tprM3bs2AwaNCgHH3xwHnrooapHAgDoVkUH4Pe+971MmzYtn/70p/Poo49mv/32y/HHH5/169dXPRoAQLcpOgCvvvrqfOhDH8q5556bvfbaK1/72tcyZMiQXH/99VWPBgDQbfpVPUBVNm3alEWLFmX69Omd+/r06ZNjjz02DzzwwBbv09ramtbW1s7bzc3NSZKWlpbuHRZ6ofbWF6seoVfw98+bx3PyzbO1Py//Ml9HR0fFk1Sn2AD8wx/+kLa2towcObLL/pEjR+aJJ57Y4n1mzpyZGTNmvGx/U1NTt8wI8FqGX1P1BPBy28rz8oUXXsjw4cOrHqMSxQbg32P69OmZNm1a5+329vb88Y9/zA477JCampoKJ9v2tbS0pKmpKatXr05tbW3V44DnJFsdz8k3T0dHR1544YWMGjWq6lEqU2wA7rjjjunbt2/WrVvXZf+6devS0NCwxfsMHDgwAwcO7LKvrq6u22YsUW1trb/Y2Kp4TrK18Zx8c5R65e8vin0TyIABA3LAAQdk7ty5nfva29szd+7cTJw4scLJAAC6V7FXAJNk2rRpOeecc3LggQfmoIMOyjXXXJONGzfm3HPPrXo0AIBu0/czn/nMZ6oeoioTJkxIXV1drrrqqnzxi19MknznO9/J7rvvXvFkZerbt2+OPPLI9OtX9P+XsBXxnGRr4znJm6Wmo+T3QAMAFKjYNYAAAKUSgAAAhRGAAACFEYAAAIURgAAAhfE+coAk73znO3P++efn9NNPz+DBg6seB7Jx48Z87nOfy9y5c7N+/fq0t7d3Of7b3/62osnoDQQgQJL9998/H//4x/ORj3wkZ5xxRs4///wccsghVY9FwT74wQ9m3rx5Ofvss9PY2Og753lT+RxAetT222//uv8S++Mf/9jN00BXf/rTn/LjH/84N9xwQ37605/mrW99a84777ycffbZGTlyZNXjUZi6urr85Cc/yaGHHlr1KPRCApAedcMNN7zuc88555xunARe3fr16/P1r389V111Vdra2nLiiSfmox/9aI4++uiqR6MQ48aNy3/9139lzz33rHoUeiEBCPA3HnroocyePTs333xzamtrM2XKlPz+97/PTTfdlA9/+MOdXx0J3enb3/52fvSjH+WGG27IkCFDqh6HXkYAUqkVK1Zk9uzZWbFiRb785S+nvr4+P/3pT7Pzzjtn7733rno8CrJ+/frceOONmT17dpYvX57Jkyfngx/8YI4//vjOZQsLFizICSeckA0bNlQ8LSXYf//9s2LFinR0dGTs2LHp379/l+OPPvpoRZPRG3gTCJWZN29eJk2alEMPPTTz58/PVVddlfr6+ixZsiSzZs3KD37wg6pHpCCjR4/OLrvskvPOOy9TpkzJTjvt9LJz9t1337zjHe+oYDpKdPLJJ1c9Ar2YK4BUZuLEiTn99NMzbdq0DBs2LEuWLMn48ePz0EMP5dRTT80zzzxT9YgU5L//+79z+OGHVz0GQI/wQdBUZunSpTnllFNetr++vj5/+MMfKpiIkok/oCReAqYydXV1WbNmTcaNG9dl/y9/+cu85S1vqWgqSrL//vu/7o8lst6KnjBixIg8+eST2XHHHV/zY7N8VBZvhACkMmeeeWYuu+yy3HLLLampqUl7e3vuu+++fPzjH88HPvCBqsejAH+9xuqll17KV7/61ey1116ZOHFikmThwoX51a9+lQ9/+MNVjUhhvvSlL2XYsGGdP/vwZ7qLNYBUZtOmTZk6dWrmzJmTtra29OvXL21tbXnf+96XOXPmpG/fvlWPSEE++MEPprGxMVdeeWWX/Z/+9KezevXqXH/99RVNBvDmE4BUbvXq1Vm6dGk2bNiQ/fffP7vuumvVI1Gg4cOH55FHHnnZ82/58uU58MAD09zcXNFklKpv375Zs2ZN6uvru+x/7rnnUl9fn7a2toomozfwEjCVa2pqSlNTU9VjULjBgwfnvvvue1kA3nfffRk0aFBFU1GyV7o+09ramgEDBvTwNPQ2ApBKPfPMM/nxj3+cp59+Ops2bepy7Oqrr65oKkp08cUX58ILL8yjjz6agw46KEny4IMP5vrrr8+nPvWpiqejJF/5yleSJDU1NfnmN7+ZoUOHdh5ra2vL/Pnzs8cee1Q1Hr2El4CpzNy5c/MP//APGT9+fJ544olMmDAhq1atSkdHR97+9rfnF7/4RdUjUpjvf//7+fKXv5zf/OY3SZI999wz//Iv/5Izzjij4skoyV8+GeF3v/tdRo8e3WU99IABAzJ27NhcccUVOfjgg6sakV5AAFKZgw46KJMmTcqMGTM6Pwi6vr4+73//+3PCCSfkwgsvrHpEgMocddRRufXWW7P99ttXPQq9kACkMsOGDcvixYuzyy67ZPvtt8+CBQuy9957Z8mSJTnppJOyatWqqkekQJs2bcr69evT3t7eZf/OO+9c0UTwZ21tbVm6dGnGjBkjCnnDfBMIldluu+061/01NjZmxYoVncd8Ewg9bfny5Tn88MMzePDgjBkzJuPGjcu4ceMyduzYl31YOfSEiy++OLNmzUry5/g74ogj8va3vz1NTU259957qx2ObZ43gVCZQw45JAsWLMiee+6ZE088MR/72MeydOnS3HrrrTnkkEOqHo/CTJkyJf369cudd96ZxsZGH8BL5W655Zb80z/9U5LkjjvuyKpVq/LEE09tuJg/AAAEp0lEQVTkxhtvzCc+8Yncd999FU/ItsxLwFTmt7/9bTZs2JB99903GzduzMc+9rHcf//92XXXXXP11VdnzJgxVY9IQbbbbrssWrTIuyvZagwaNChPPfVURo8enQsuuCBDhgzJNddck5UrV2a//fZLS0tL1SOyDXMFkMqMHz++8+ftttsuX/va1yqchtLttddelh6wVRk5cmR+/etfp7GxMXfddVeuu+66JMmLL77om5J4wwQglbPonq3B5z//+Vx66aX57Gc/m3322Sf9+/fvcry2traiySjVueeemzPOOKNzScKxxx6b5M+fT+lKNW+Ul4CpzJNPPpnzzz8/999/f5f9HR0dqamp8TVH9Kg+ff78nri/Xfvn+UiVfvCDH2T16tU5/fTTM3r06CTJDTfckLq6upx00kkVT8e2TABSmUMPPTT9+vXLv/7rv25x0f1+++1X0WSUaN68ea94bOnSpbnooot6cBqA7iUAqYxF92zNXnjhhXz3u9/NN7/5zSxatMgVQHrcFVdc8arHL7/88h6ahN7IGkAqY9E9W6P58+dn1qxZ+eEPf5hRo0bl1FNPzbXXXlv1WBTotttu63J78+bNWblyZfr165dddtlFAPKGCEB61F9/bIFF92wt1q5dmzlz5mTWrFlpaWnJGWeckdbW1tx+++3Za6+9qh6PQv3yl7982b6WlpZMmTIlp5xySgUT0Zt4CZge1adPny5r/f6ywP6vWXRPT5o8eXLmz5+fd7/73Z3fQ923b9/0798/S5YsEYBsdZYuXZrJkyf7ukzeEFcA6VH33HNP1SNAFz/96U/z0Y9+NBdeeGF23XXXqseB19Tc3Jzm5uaqx2AbJwDpUe985zurHgG6WLBgQWbNmpUDDjgge+65Z84+++yceeaZVY8F+cpXvtLldkdHR9asWZMbb7wxkyZNqmgqegsvAVOZ2bNnZ+jQoTn99NO77L/lllvy4osv5pxzzqloMkq0cePGfO9738v111+fhx56KG1tbbn66qtz3nnnZdiwYVWPR4HGjRvX5XafPn2y00475eijj8706dM9L3lDBCCV2W233fKf//mfOeqoo7rsnzdvXi644IIsW7asosko3bJlyzJr1qzceOONef755/Oud70rP/7xj6seC+BNIwCpzKBBg/LEE09k7NixXfavWrUqe+65Z/73f/+3msHg/2lra8sdd9yR66+/XgDSY0499dTXPKdfv35paGjIu971rkyePLkHpqK36VP1AJSrvr4+jz322Mv2L1myJDvssEMFE0FXffv2zcknnyz+6FHDhw9/zW3w4MFZvnx53vve9/o8QP4urgBSmcsuuyzf+973Mnv27BxxxBFJ/vzy73nnnZd//Md/zBe/+MWKJwTYut1555358Ic/nKeffrrqUdjGeBcwlbnyyiuzatWqHHPMMenX789Pxfb29nzgAx/IZz/72YqnA9j6HXbYYTnwwAOrHoNtkCuAVO7JJ5/MkiVLMnjw4Oyzzz4ZM2ZM1SMBQK8mAAEACuMlYHrUtGnTcuWVV2a77bbLtGnTXvXcq6++uoemAoCyCEB61C9/+cts3ry582cAoOf9f9IuBJb35xNqAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplot plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running locally (%%local)\n",
    "\n",
    "You can use the `%%local` magic to run your code locally on the Jupyter server without going to Spark. Here's a typical use case for this scenario. \n",
    "\n",
    "By default, the output of any code snippet that you run from a Jupyter notebook is available within the context of the session that is persisted on the worker nodes. However, if you want to save a trip to the worker nodes for every computation, and all the data that you need for your computation is available locally on the Jupyter server node (which is the headnode), you can use the `%%local` magic to run the code snippet on the Jupyter server. Typically, you would use `%%local` magic in conjunction with the `%%sql` magic with `-o` parameter. The `-o` parameter would persist the output of the SQL query locally and then `%%local` magic would trigger the next set of code snippet to run locally against the output of the SQL queries that is persisted locally.\n",
    "\n",
    "> **TIPS**: \n",
    "> * Working against locally persisted data is especially useful when you want the flexibility of using visualization libraries such as **matplotlib**. If you work directly against the data on the remote worker nodes, the output received through Livy is always text that limits the options of visual representation.\n",
    "\n",
    "\n",
    "> * Remember that SQL queries, by default, limit the number of result rows to 2500 -- and that it is possible to get Out of Memory errors if you accidentally collect too many results from your SQL query. Therefore, if your dataset is large, it is considered a best practice to do your computation or number-crunching on the cluster or in the SQL query rather than in local mode.\n",
    "\n",
    "When you use `%%local` all subsequent lines in the cell will be executed locally. The code in the cell must be valid Python code.\n",
    "\n",
    "This code block prints the length of the result set of the dataframe `query2` -- remember, we used the `-n 500` option to limit the size of the dataframe to 500."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "500\n"
     ]
    }
   ],
   "source": [
    "%%local\n",
    "print(len(query2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Send to spark (%%send_to_spark)\n",
    "\n",
    "This command gives you the ability to send local data to remote cluster. Note that passing even a simple string allows you to retain pretrained, saved ML model from local filesystem.\n",
    "\n",
    "#### Pass string to a remote cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%local\n",
    "variable = 'this string will be available in spark cluster'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%send_to_spark -i variable -n custom_name -t str"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print custom_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Send pretrained scikit-learn model to a remote cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%local\n",
    "## todo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Session logs (%%logs)\n",
    "\n",
    "You can get the logs of your current Livy session to debug any issues you encounter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16/06/09 23:43:28 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.\n",
      "16/06/09 23:43:28 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.\n",
      "16/06/09 23:43:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "16/06/09 23:43:30 INFO TimelineClientImpl: Timeline service address: http://localhost:8188/ws/v1/timeline/\n",
      "16/06/09 23:43:30 INFO Client: Requesting a new application from cluster with 10 NodeManagers\n",
      "16/06/09 23:43:30 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (25600 MB per container)\n",
      "16/06/09 23:43:30 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead\n",
      "16/06/09 23:43:30 INFO Client: Setting up container launch context for our AM\n",
      "16/06/09 23:43:30 INFO Client: Setting up the launch environment for our AM container\n",
      "16/06/09 23:43:30 INFO Client: Preparing resources for our AM container\n",
      "16/06/09 23:43:31 INFO MetricsConfig: loaded properties from hadoop-metrics2-azure-file-system.properties\n",
      "16/06/09 23:43:31 INFO WasbAzureIaasSink: Init starting.\n",
      "16/06/09 23:43:31 INFO AzureIaasSink: Init starting. Initializing MdsLogger.\n",
      "16/06/09 23:43:31 INFO AzureIaasSink: Init completed.\n",
      "16/06/09 23:43:31 INFO WasbAzureIaasSink: Init completed.\n",
      "16/06/09 23:43:31 INFO MetricsSinkAdapter: Sink azurefs2 started\n",
      "16/06/09 23:43:31 INFO MetricsSystemImpl: Scheduled snapshot period at 60 second(s).\n",
      "16/06/09 23:43:31 INFO MetricsSystemImpl: azure-file-system metrics system started\n",
      "16/06/09 23:43:31 INFO Client: Uploading resource file:/usr/hdp/current/spark-client/python/lib/pyspark.zip -> wasb://MYSTORAGE@store.blob.core.windows.net/user/spark/.sparkStaging/application_1464100251524_0001/pyspark.zip\n",
      "16/06/09 23:43:32 INFO Client: Uploading resource file:/usr/hdp/current/spark-client/python/lib/py4j-0.8.2.1-src.zip -> wasb://MYSTORAGE@store.blob.core.windows.net/user/spark/.sparkStaging/application_1464100251524_0001/py4j-0.8.2.1-src.zip\n",
      "16/06/09 23:43:32 INFO Client: Uploading resource file:/usr/hdp/current/spark-client/conf/hive-site.xml -> wasb://MYSTORAGE@store.blob.core.windows.net/user/spark/.sparkStaging/application_1464100251524_0001/hive-site.xml\n",
      "16/06/09 23:43:33 INFO Client: Uploading resource file:/usr/hdp/2.3.3.1-7/spark/python/lib/pyspark.zip -> wasb://MYSTORAGE@store.blob.core.windows.net/user/spark/.sparkStaging/application_1464100251524_0001/pyspark.zip\n",
      "16/06/09 23:43:33 INFO Client: Uploading resource file:/usr/hdp/2.3.3.1-7/spark/python/lib/py4j-0.8.2.1-src.zip -> wasb://MYSTORAGE@store.blob.core.windows.net/user/spark/.sparkStaging/application_1464100251524_0001/py4j-0.8.2.1-src.zip\n",
      "16/06/09 23:43:34 INFO Client: Uploading resource file:/tmp/spark-1c0a5560-2256-4f63-ba1f-c0dfd25dc24b/__spark_conf__324896021270563710.zip -> wasb://MYSTORAGE@store.blob.core.windows.net/user/spark/.sparkStaging/application_1464100251524_0001/__spark_conf__324896021270563710.zip\n",
      "16/06/09 23:43:34 WARN Client: spark.yarn.am.extraJavaOptions will not take effect in cluster mode\n",
      "16/06/09 23:43:34 INFO SecurityManager: Changing view acls to: spark\n",
      "16/06/09 23:43:34 INFO SecurityManager: Changing modify acls to: spark\n",
      "16/06/09 23:43:34 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)\n",
      "16/06/09 23:43:34 INFO Client: Submitting application 1 to ResourceManager\n",
      "16/06/09 23:43:35 INFO YarnClientImpl: Submitted application application_1464100251524_0001\n",
      "16/06/09 23:43:36 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:36 INFO Client: \n",
      "\t client token: N/A\n",
      "\t diagnostics: N/A\n",
      "\t ApplicationMaster host: N/A\n",
      "\t ApplicationMaster RPC port: -1\n",
      "\t queue: default\n",
      "\t start time: 1465515814875\n",
      "\t final status: UNDEFINED\n",
      "\t tracking URL: http://localhost:8088/proxy/application_1464100251524_0001/\n",
      "\t user: spark\n",
      "16/06/09 23:43:37 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:38 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:39 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:40 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:41 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:42 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:43 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:44 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:45 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:46 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:47 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:47 INFO Client: \n",
      "\t client token: N/A\n",
      "\t diagnostics: N/A\n",
      "\t ApplicationMaster host: 10.0.0.4\n",
      "\t ApplicationMaster RPC port: 0\n",
      "\t queue: default\n",
      "\t start time: 1465515814875\n",
      "\t final status: UNDEFINED\n",
      "\t tracking URL: http://localhost:8088/proxy/application_1464100251524_0001/\n",
      "\t user: spark\n",
      "16/06/09 23:43:48 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:49 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:50 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:51 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:52 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:53 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:54 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:55 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:56 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:57 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:58 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:59 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:00 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:01 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:02 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:03 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:04 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:05 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:06 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:07 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:08 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:09 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:10 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:11 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:12 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:13 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:14 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:15 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:16 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:17 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:18 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:19 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:20 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:21 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:22 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:23 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:24 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:25 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)"
     ]
    }
   ],
   "source": [
    "%%logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete session (%%delete)\n",
    "\n",
    "Use '`%%delete -f -s <session #>`' to delete a session given its session ID. Note that you cannot delete the session that is initiated for the kernel itself. Another notebook might go into an error state, since notebooks are supposed to manage sessions by themselves, and all work will be lost on that session."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sessions cleanup (%%cleanup)\n",
    "\n",
    "Use '`%%cleanup -f`' magic to delete all of the sessions for this cluster, including this notebook's session.\n",
    "The force flag `-f` is mandatory."
   ]
  },
  {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "### Pretty Print Spark Dataframes (%%pretty)\n",
     "\n",
     "Use '`%%pretty`' magic to display a Spark dataframe as a HTML formatted table"
    ]
   },
  {
    "cell_type": "code",
    "execution_count": 3,
    "metadata": {},
    "outputs": [
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
        "model_id": "",
        "version_major": 2,
        "version_minor": 0
       },
       "text/plain": [
       ]
      },
      "metadata": {},
      "output_type": "display_data"
     },
     {
      "data": {
       "text/html": [
        "<table><tr><th>age</th><th>name</th></tr><tr><td>null</td><td>Michael</td></tr><tr><td>30</td><td>Andy</td></tr><tr><td>19</td><td>Justin</td></tr></table><br /><pre></pre>"
       ],
       "text/plain": [
        "<IPython.core.display.HTML object>"
       ]
      },
      "metadata": {},
      "output_type": "display_data"
     }
    ],
    "source": [
     "%%pretty\n",
     "df = spark.read.json(\"/apps/spark-2.3.3/examples/src/main/resources/people.json\")\n",
     "df.show()"
    ]
   }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PySpark",
   "language": "",
   "name": "pysparkkernel"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "python",
    "version": 2
   },
   "mimetype": "text/x-python",
   "name": "pyspark",
   "pygments_lexer": "python2"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}