notebooks/overview_netl.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from constants import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2011\n",
    "------------\n",
    "\n",
    "<br>\n",
    "\n",
    "### DATASETS\n",
    "\n",
    "BLOGS: 120,000 articles\n",
    "\n",
    "BOOKS: 1,000 books\n",
    "\n",
    "NEWS: 29,000 articles\n",
    "\n",
    "PUBMED: 77,000 articles\n",
    "\n",
    "<br>\n",
    "\n",
    "### TOPIC MODELING pipeline:\n",
    "\n",
    "- token -> lemma -> stop -> filter out token w/ count < 10 in vocabulary \n",
    "- BOW for each document\n",
    "- 100 topic models\n",
    "- average PMI score for each topic (Newman 2010b) -> filter out topics w/ score < 0.4\n",
    "- filter out topics w/ count(token: token default nominal in Wikipedia) < 5\n",
    "\n",
    "<br>\n",
    "\n",
    "### RESULTS\n",
    "\n",
    "| **topics**        | **dataset**       |\n",
    "| ------------- |:-------------:|\n",
    "| 45            | BLOGS         |\n",
    "| 38            | BOOKS         |\n",
    "| 60            | NEWS          |\n",
    "| 85            | PUBMED        |\n",
    "| *228*            | *sum*         |\n",
    "\n",
    "\n",
    "=> 6000 labels (~27 per topic)\n",
    "\n",
    "<br> \n",
    "\n",
    "#### annotation task:\n",
    "\n",
    "per topic:\n",
    "- 10 top topic terms\n",
    "- 10 label suggestions\n",
    "- 4 answer options\n",
    "\n",
    "10 annotations (at least) per label candidate\n",
    "\n",
    "-> filtered malitious annotators\n",
    "\n",
    "<br>\n",
    "\n",
    "45,533 label ratings => ~4,500 successfull HITs (Amazon Mechanical Turk)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "45533\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>topic_id</th>\n",
       "      <th>label</th>\n",
       "      <th>rate0</th>\n",
       "      <th>rate1</th>\n",
       "      <th>rate2</th>\n",
       "      <th>rate3</th>\n",
       "      <th>rate4</th>\n",
       "      <th>rate5</th>\n",
       "      <th>rate6</th>\n",
       "      <th>rate7</th>\n",
       "      <th>rate8</th>\n",
       "      <th>rate9</th>\n",
       "      <th>rate10</th>\n",
       "      <th>rate11</th>\n",
       "      <th>rate12</th>\n",
       "      <th>rate13</th>\n",
       "      <th>mean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>244</th>\n",
       "      <td>6</td>\n",
       "      <td>immune system</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.142857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>323</th>\n",
       "      <td>9</td>\n",
       "      <td>2008 summer olympics</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1551</th>\n",
       "      <td>45</td>\n",
       "      <td>gothic architecture</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1948</th>\n",
       "      <td>61</td>\n",
       "      <td>gothic architecture</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2334</th>\n",
       "      <td>75</td>\n",
       "      <td>gothic architecture</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2592</th>\n",
       "      <td>84</td>\n",
       "      <td>israeli–palestinian conflict</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.625000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2662</th>\n",
       "      <td>86</td>\n",
       "      <td>immune system</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.166667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4615</th>\n",
       "      <td>145</td>\n",
       "      <td>immune system</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4645</th>\n",
       "      <td>146</td>\n",
       "      <td>immune system</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.363636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6000</th>\n",
       "      <td>195</td>\n",
       "      <td>immune system</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6899</th>\n",
       "      <td>225</td>\n",
       "      <td>immune system</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.285714</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      topic_id                         label  rate0  rate1  rate2  rate3  \\\n",
       "244          6                 immune system      0      1      1    1.0   \n",
       "323          9          2008 summer olympics      2      2      3    3.0   \n",
       "1551        45           gothic architecture      2      2      2    3.0   \n",
       "1948        61           gothic architecture      1      1      2    2.0   \n",
       "2334        75           gothic architecture      1      1      2    2.0   \n",
       "2592        84  israeli–palestinian conflict      2      2      2    3.0   \n",
       "2662        86                 immune system      1      1      1    1.0   \n",
       "4615       145                 immune system      0      1      2    2.0   \n",
       "4645       146                 immune system      1      2      2    2.0   \n",
       "6000       195                 immune system      1      1      1    2.0   \n",
       "6899       225                 immune system      1      1      1    1.0   \n",
       "\n",
       "      rate4  rate5  rate6  rate7          rate8  rate9  rate10  rate11  \\\n",
       "244     1.0    2.0    2.0    NaN            NaN    NaN     NaN     NaN   \n",
       "323     3.0    NaN    NaN    NaN            NaN    NaN     NaN     NaN   \n",
       "1551    3.0    NaN    NaN    NaN            NaN    NaN     NaN     NaN   \n",
       "1948    2.0    NaN    NaN    NaN            NaN    NaN     NaN     NaN   \n",
       "2334    2.0    2.0    NaN    NaN            NaN    NaN     NaN     NaN   \n",
       "2592    3.0    3.0    3.0    3.0            NaN    NaN     NaN     NaN   \n",
       "2662    1.0    2.0    NaN    NaN            NaN    NaN     NaN     NaN   \n",
       "4615    2.0    3.0    NaN    NaN            NaN    NaN     NaN     NaN   \n",
       "4645    2.0    2.0    3.0    3.0            3.0    3.0     3.0     NaN   \n",
       "6000    2.0    NaN    NaN    NaN            NaN    NaN     NaN     NaN   \n",
       "6899    1.0    2.0    2.0    NaN            NaN    NaN     NaN     NaN   \n",
       "\n",
       "      rate12  rate13      mean  \n",
       "244      NaN     NaN  1.142857  \n",
       "323      NaN     NaN  2.600000  \n",
       "1551     NaN     NaN  2.400000  \n",
       "1948     NaN     NaN  1.600000  \n",
       "2334     NaN     NaN  1.666667  \n",
       "2592     NaN     NaN  2.625000  \n",
       "2662     NaN     NaN  1.166667  \n",
       "4615     NaN     NaN  1.666667  \n",
       "4645     NaN     NaN  2.363636  \n",
       "6000     NaN     NaN  1.400000  \n",
       "6899     NaN     NaN  1.285714  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_2011 = '../data/topiclabel/2011'\n",
    "f_topics2011 = join(data_2011, 'topics.csv')\n",
    "f_annotation2011 = join(data_2011, 'topiclabels.csv')\n",
    "readme = join(data_2011, 'README.txt')\n",
    "topics2011 = pd.read_csv(f_topics2011)\n",
    "annotation2011 = pd.read_csv(f_annotation2011)\n",
    "\n",
    "print(annotation2011.iloc[:, 2:].count().sum())\n",
    "examples = ['2008 summer olympics', 'gothic architecture', 'israeli–palestinian conflict', 'immune system']\n",
    "mask11 = annotation2011.label.isin(examples)\n",
    "annotation2011['mean'] = annotation2011.loc[mask11, 'rate0':].mean(axis=1)\n",
    "annotation2011.loc[mask11]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>topic_id</th>\n",
       "      <th>label</th>\n",
       "      <th>rate0</th>\n",
       "      <th>rate1</th>\n",
       "      <th>rate2</th>\n",
       "      <th>rate3</th>\n",
       "      <th>rate4</th>\n",
       "      <th>rate5</th>\n",
       "      <th>rate6</th>\n",
       "      <th>rate7</th>\n",
       "      <th>rate8</th>\n",
       "      <th>rate9</th>\n",
       "      <th>rate10</th>\n",
       "      <th>rate11</th>\n",
       "      <th>rate12</th>\n",
       "      <th>rate13</th>\n",
       "      <th>mean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>258</th>\n",
       "      <td>7</td>\n",
       "      <td>1/2 cup</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>381</th>\n",
       "      <td>11</td>\n",
       "      <td>1/2 cup</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2740</th>\n",
       "      <td>89</td>\n",
       "      <td>1/2 cup</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3321</th>\n",
       "      <td>105</td>\n",
       "      <td>1/2 cup</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>259</th>\n",
       "      <td>7</td>\n",
       "      <td>1/4 cup</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>382</th>\n",
       "      <td>11</td>\n",
       "      <td>1/4 cup</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2741</th>\n",
       "      <td>89</td>\n",
       "      <td>1/4 cup</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3323</th>\n",
       "      <td>105</td>\n",
       "      <td>1/4 cup</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3898</th>\n",
       "      <td>122</td>\n",
       "      <td>1960s in fashion</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1138</th>\n",
       "      <td>34</td>\n",
       "      <td>1960s in fashion</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3133</th>\n",
       "      <td>100</td>\n",
       "      <td>1988</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>5</td>\n",
       "      <td>1988</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>962</th>\n",
       "      <td>29</td>\n",
       "      <td>2003 invasion</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2259</th>\n",
       "      <td>73</td>\n",
       "      <td>2003 invasion</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>963</th>\n",
       "      <td>29</td>\n",
       "      <td>2003 invasion of iraq</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2260</th>\n",
       "      <td>73</td>\n",
       "      <td>2003 invasion of iraq</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2543</th>\n",
       "      <td>83</td>\n",
       "      <td>2008</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>1</td>\n",
       "      <td>2008</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2608</th>\n",
       "      <td>85</td>\n",
       "      <td>2009 in baseball</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3431</th>\n",
       "      <td>109</td>\n",
       "      <td>2009 in baseball</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1102</th>\n",
       "      <td>33</td>\n",
       "      <td>25 article</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1072</th>\n",
       "      <td>32</td>\n",
       "      <td>25 article</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1850</th>\n",
       "      <td>57</td>\n",
       "      <td>742 evergreen terrace</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3988</th>\n",
       "      <td>125</td>\n",
       "      <td>742 evergreen terrace</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1531</th>\n",
       "      <td>45</td>\n",
       "      <td>abbey</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2316</th>\n",
       "      <td>75</td>\n",
       "      <td>abbey</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1006</th>\n",
       "      <td>30</td>\n",
       "      <td>abbey</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5254</th>\n",
       "      <td>168</td>\n",
       "      <td>acid</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6262</th>\n",
       "      <td>204</td>\n",
       "      <td>acid</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6769</th>\n",
       "      <td>222</td>\n",
       "      <td>acid</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1648</th>\n",
       "      <td>48</td>\n",
       "      <td>word</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2542</th>\n",
       "      <td>82</td>\n",
       "      <td>word</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6585</th>\n",
       "      <td>215</td>\n",
       "      <td>working memory</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6666</th>\n",
       "      <td>218</td>\n",
       "      <td>working memory</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1004</th>\n",
       "      <td>29</td>\n",
       "      <td>world war</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2292</th>\n",
       "      <td>73</td>\n",
       "      <td>world war</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2488</th>\n",
       "      <td>80</td>\n",
       "      <td>world war</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2293</th>\n",
       "      <td>73</td>\n",
       "      <td>world war i</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1005</th>\n",
       "      <td>29</td>\n",
       "      <td>world war i</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2489</th>\n",
       "      <td>80</td>\n",
       "      <td>world war i</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>539</th>\n",
       "      <td>15</td>\n",
       "      <td>world wide web</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3641</th>\n",
       "      <td>114</td>\n",
       "      <td>world wide web</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5304</th>\n",
       "      <td>169</td>\n",
       "      <td>wound healing</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4967</th>\n",
       "      <td>157</td>\n",
       "      <td>wound healing</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1355</th>\n",
       "      <td>39</td>\n",
       "      <td>write</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1264</th>\n",
       "      <td>36</td>\n",
       "      <td>write</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3642</th>\n",
       "      <td>114</td>\n",
       "      <td>write</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4225</th>\n",
       "      <td>132</td>\n",
       "      <td>wyandanch</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2212</th>\n",
       "      <td>70</td>\n",
       "      <td>wyandanch</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2213</th>\n",
       "      <td>70</td>\n",
       "      <td>wyandanch, new york</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4226</th>\n",
       "      <td>132</td>\n",
       "      <td>wyandanch, new york</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5747</th>\n",
       "      <td>186</td>\n",
       "      <td>x-ray diffraction</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6488</th>\n",
       "      <td>211</td>\n",
       "      <td>x-ray diffraction</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2258</th>\n",
       "      <td>72</td>\n",
       "      <td>young</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3941</th>\n",
       "      <td>123</td>\n",
       "      <td>young</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6088</th>\n",
       "      <td>197</td>\n",
       "      <td>young</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3942</th>\n",
       "      <td>123</td>\n",
       "      <td>young child</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1219</th>\n",
       "      <td>35</td>\n",
       "      <td>young child</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>736</th>\n",
       "      <td>21</td>\n",
       "      <td>young girl</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1220</th>\n",
       "      <td>35</td>\n",
       "      <td>young girl</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2690 rows × 17 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      topic_id                  label  rate0  rate1  rate2  rate3  rate4  \\\n",
       "258          7                1/2 cup      1      1      1    1.0    2.0   \n",
       "381         11                1/2 cup      0      0      0    0.0    0.0   \n",
       "2740        89                1/2 cup      0      0      0    0.0    1.0   \n",
       "3321       105                1/2 cup      0      0      0    1.0    1.0   \n",
       "259          7                1/4 cup      1      1      1    1.0    2.0   \n",
       "382         11                1/4 cup      0      0      0    0.0    0.0   \n",
       "2741        89                1/4 cup      0      0      0    0.0    0.0   \n",
       "3323       105                1/4 cup      0      0      1    1.0    1.0   \n",
       "3898       122       1960s in fashion      0      1      1    1.0    1.0   \n",
       "1138        34       1960s in fashion      1      1      1    1.0    1.0   \n",
       "3133       100                   1988      0      0      0    0.0    2.0   \n",
       "196          5                   1988      0      0      0    1.0    1.0   \n",
       "962         29          2003 invasion      1      1      1    1.0    2.0   \n",
       "2259        73          2003 invasion      0      1      1    1.0    2.0   \n",
       "963         29  2003 invasion of iraq      1      2      2    2.0    2.0   \n",
       "2260        73  2003 invasion of iraq      0      1      1    1.0    2.0   \n",
       "2543        83                   2008      0      0      0    1.0    1.0   \n",
       "34           1                   2008      1      1      1    1.0    1.0   \n",
       "2608        85       2009 in baseball      1      1      1    3.0    3.0   \n",
       "3431       109       2009 in baseball      1      2      2    2.0    2.0   \n",
       "1102        33             25 article      0      0      0    0.0    0.0   \n",
       "1072        32             25 article      0      0      0    0.0    0.0   \n",
       "1850        57  742 evergreen terrace      0      1      1    1.0    1.0   \n",
       "3988       125  742 evergreen terrace      0      0      0    1.0    3.0   \n",
       "1531        45                  abbey      1      1      1    1.0    2.0   \n",
       "2316        75                  abbey      1      1      2    2.0    2.0   \n",
       "1006        30                  abbey      0      0      0    0.0    0.0   \n",
       "5254       168                   acid      0      1      1    1.0    1.0   \n",
       "6262       204                   acid      0      0      0    1.0    1.0   \n",
       "6769       222                   acid      0      1      1    2.0    3.0   \n",
       "...        ...                    ...    ...    ...    ...    ...    ...   \n",
       "1648        48                   word      0      0      0    0.0    0.0   \n",
       "2542        82                   word      1      1      1    1.0    2.0   \n",
       "6585       215         working memory      0      0      0    1.0    1.0   \n",
       "6666       218         working memory      1      2      2    2.0    2.0   \n",
       "1004        29              world war      1      1      2    2.0    3.0   \n",
       "2292        73              world war      1      2      2    2.0    2.0   \n",
       "2488        80              world war      0      0      1    1.0    2.0   \n",
       "2293        73            world war i      1      2      2    2.0    2.0   \n",
       "1005        29            world war i      1      2      2    2.0    3.0   \n",
       "2489        80            world war i      0      1      1    2.0    2.0   \n",
       "539         15         world wide web      2      2      3    3.0    3.0   \n",
       "3641       114         world wide web      0      0      0    0.0    0.0   \n",
       "5304       169          wound healing      0      0      1    1.0    1.0   \n",
       "4967       157          wound healing      0      1      1    1.0    3.0   \n",
       "1355        39                  write      0      1      1    2.0    2.0   \n",
       "1264        36                  write      1      1      2    2.0    2.0   \n",
       "3642       114                  write      1      1      1    2.0    2.0   \n",
       "4225       132              wyandanch      0      0      0    0.0    0.0   \n",
       "2212        70              wyandanch      0      0      0    0.0    1.0   \n",
       "2213        70    wyandanch, new york      0      0      0    0.0    0.0   \n",
       "4226       132    wyandanch, new york      0      0      0    0.0    0.0   \n",
       "5747       186      x-ray diffraction      0      0      1    1.0    2.0   \n",
       "6488       211      x-ray diffraction      0      0      0    0.0    1.0   \n",
       "2258        72                  young      0      1      1    2.0    2.0   \n",
       "3941       123                  young      0      1      1    2.0    2.0   \n",
       "6088       197                  young      0      1      2    2.0    2.0   \n",
       "3942       123            young child      0      1      2    2.0    3.0   \n",
       "1219        35            young child      1      1      2    2.0    NaN   \n",
       "736         21             young girl      0      0      0    0.0    0.0   \n",
       "1220        35             young girl      0      1      1    1.0    1.0   \n",
       "\n",
       "      rate5  rate6  rate7          rate8  rate9  rate10  rate11  rate12  \\\n",
       "258     NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "381     0.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "2740    1.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "3321    1.0    1.0    2.0            2.0    3.0     NaN     NaN     NaN   \n",
       "259     NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "382     0.0    0.0    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "2741    1.0    1.0    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "3323    NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "3898    2.0    2.0    2.0            2.0    2.0     2.0     3.0     NaN   \n",
       "1138    2.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "3133    NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "196     1.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "962     2.0    2.0    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "2259    2.0    2.0    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "963     2.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "2260    2.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "2543    1.0    2.0    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "34      NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "2608    NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "3431    2.0    2.0    3.0            3.0    NaN     NaN     NaN     NaN   \n",
       "1102    0.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "1072    0.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "1850    2.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "3988    NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "1531    2.0    2.0    3.0            NaN    NaN     NaN     NaN     NaN   \n",
       "2316    3.0    3.0    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "1006    0.0    1.0    1.0            1.0    1.0     NaN     NaN     NaN   \n",
       "5254    2.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "6262    NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "6769    3.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "...     ...    ...    ...            ...    ...     ...     ...     ...   \n",
       "1648    0.0    0.0    0.0            1.0    1.0     1.0     NaN     NaN   \n",
       "2542    NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "6585    1.0    2.0    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "6666    2.0    3.0    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "1004    3.0    3.0    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "2292    2.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "2488    2.0    3.0    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "2293    2.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "1005    3.0    3.0    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "2489    3.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "539     NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "3641    1.0    1.0    2.0            NaN    NaN     NaN     NaN     NaN   \n",
       "5304    1.0    1.0    1.0            2.0    2.0     2.0     3.0     NaN   \n",
       "4967    NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "1355    NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "1264    NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "3642    2.0    2.0    2.0            2.0    3.0     NaN     NaN     NaN   \n",
       "4225    1.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "2212    1.0    2.0    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "2213    0.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "4226    1.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "5747    NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "6488    1.0    1.0    1.0            1.0    1.0     NaN     NaN     NaN   \n",
       "2258    2.0    2.0    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "3941    2.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "6088    3.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "3942    3.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "1219    NaN    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "736     1.0    NaN    NaN            NaN    NaN     NaN     NaN     NaN   \n",
       "1220    1.0    1.0    2.0            2.0    NaN     NaN     NaN     NaN   \n",
       "\n",
       "      rate13  mean  \n",
       "258      NaN   NaN  \n",
       "381      NaN   NaN  \n",
       "2740     NaN   NaN  \n",
       "3321     NaN   NaN  \n",
       "259      NaN   NaN  \n",
       "382      NaN   NaN  \n",
       "2741     NaN   NaN  \n",
       "3323     NaN   NaN  \n",
       "3898     NaN   NaN  \n",
       "1138     NaN   NaN  \n",
       "3133     NaN   NaN  \n",
       "196      NaN   NaN  \n",
       "962      NaN   NaN  \n",
       "2259     NaN   NaN  \n",
       "963      NaN   NaN  \n",
       "2260     NaN   NaN  \n",
       "2543     NaN   NaN  \n",
       "34       NaN   NaN  \n",
       "2608     NaN   NaN  \n",
       "3431     NaN   NaN  \n",
       "1102     NaN   NaN  \n",
       "1072     NaN   NaN  \n",
       "1850     NaN   NaN  \n",
       "3988     NaN   NaN  \n",
       "1531     NaN   NaN  \n",
       "2316     NaN   NaN  \n",
       "1006     NaN   NaN  \n",
       "5254     NaN   NaN  \n",
       "6262     NaN   NaN  \n",
       "6769     NaN   NaN  \n",
       "...      ...   ...  \n",
       "1648     NaN   NaN  \n",
       "2542     NaN   NaN  \n",
       "6585     NaN   NaN  \n",
       "6666     NaN   NaN  \n",
       "1004     NaN   NaN  \n",
       "2292     NaN   NaN  \n",
       "2488     NaN   NaN  \n",
       "2293     NaN   NaN  \n",
       "1005     NaN   NaN  \n",
       "2489     NaN   NaN  \n",
       "539      NaN   NaN  \n",
       "3641     NaN   NaN  \n",
       "5304     NaN   NaN  \n",
       "4967     NaN   NaN  \n",
       "1355     NaN   NaN  \n",
       "1264     NaN   NaN  \n",
       "3642     NaN   NaN  \n",
       "4225     NaN   NaN  \n",
       "2212     NaN   NaN  \n",
       "2213     NaN   NaN  \n",
       "4226     NaN   NaN  \n",
       "5747     NaN   NaN  \n",
       "6488     NaN   NaN  \n",
       "2258     NaN   NaN  \n",
       "3941     NaN   NaN  \n",
       "6088     NaN   NaN  \n",
       "3942     NaN   NaN  \n",
       "1219     NaN   NaN  \n",
       "736      NaN   NaN  \n",
       "1220     NaN   NaN  \n",
       "\n",
       "[2690 rows x 17 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "annotation2011[annotation2011.duplicated('label', keep=False)].sort_values('label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "228\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>topic_id</th>\n",
       "      <th>domain</th>\n",
       "      <th>term0</th>\n",
       "      <th>term1</th>\n",
       "      <th>term2</th>\n",
       "      <th>term3</th>\n",
       "      <th>term4</th>\n",
       "      <th>term5</th>\n",
       "      <th>term6</th>\n",
       "      <th>term7</th>\n",
       "      <th>term8</th>\n",
       "      <th>term9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>blogs</td>\n",
       "      <td>food</td>\n",
       "      <td>eat</td>\n",
       "      <td>cook</td>\n",
       "      <td>chicken</td>\n",
       "      <td>recipe</td>\n",
       "      <td>cup</td>\n",
       "      <td>cheese</td>\n",
       "      <td>add</td>\n",
       "      <td>taste</td>\n",
       "      <td>tomato</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>11</td>\n",
       "      <td>blogs</td>\n",
       "      <td>look</td>\n",
       "      <td>white</td>\n",
       "      <td>color</td>\n",
       "      <td>black</td>\n",
       "      <td>red</td>\n",
       "      <td>hair</td>\n",
       "      <td>blue</td>\n",
       "      <td>light</td>\n",
       "      <td>green</td>\n",
       "      <td>eye</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>89</td>\n",
       "      <td>news</td>\n",
       "      <td>food</td>\n",
       "      <td>wine</td>\n",
       "      <td>restaurant</td>\n",
       "      <td>eat</td>\n",
       "      <td>drink</td>\n",
       "      <td>bottle</td>\n",
       "      <td>beer</td>\n",
       "      <td>coffee</td>\n",
       "      <td>bar</td>\n",
       "      <td>ice</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>105</td>\n",
       "      <td>news</td>\n",
       "      <td>cup</td>\n",
       "      <td>cook</td>\n",
       "      <td>minute</td>\n",
       "      <td>add</td>\n",
       "      <td>pepper</td>\n",
       "      <td>salt</td>\n",
       "      <td>serve</td>\n",
       "      <td>tablespoon</td>\n",
       "      <td>oil</td>\n",
       "      <td>sauce</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     topic_id domain term0  term1       term2    term3   term4         term5  \\\n",
       "7           7  blogs  food    eat        cook  chicken  recipe           cup   \n",
       "11         11  blogs  look  white       color    black     red          hair   \n",
       "89         89   news  food   wine  restaurant      eat   drink        bottle   \n",
       "105       105   news   cup   cook      minute      add  pepper          salt   \n",
       "\n",
       "      term6       term7  term8   term9  \n",
       "7    cheese         add  taste  tomato  \n",
       "11     blue       light  green     eye  \n",
       "89     beer      coffee    bar     ice  \n",
       "105   serve  tablespoon    oil   sauce  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(topics2011.shape[0])\n",
    "topics2011.iloc[[7, 11, 89, 105]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2016\n",
    "------\n",
    "\n",
    "<br>\n",
    "\n",
    "### DATASETS and annotation\n",
    "\n",
    "- same datasets and topics as 2011\n",
    "- very similar annotation method\n",
    "- annotation quality controlled by 2011 dataset mean ratings\n",
    "\n",
    "<br>\n",
    "\n",
    "- 19 top candidates from unsupervised relevance ranking\n",
    "- 10 annotations per label candidate (pre-filtering)\n",
    "- 6.4 annotations per label candidate on average (post-filtering)\n",
    "\n",
    "\n",
    "<br> \n",
    "\n",
    "#### annotation task:\n",
    "\n",
    "per topic:\n",
    "- 10 top topic terms\n",
    "- 10 label suggestions\n",
    "- 4 answer options\n",
    "\n",
    "<br>\n",
    "\n",
    "27,788 label ratings => ~2,800 successfull HITs (CrowdFlower)\n",
    "\n",
    "<br>\n",
    "\n",
    "compute mean of ratings and rank accordingly\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27788\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>topic_id</th>\n",
       "      <th>annotator1</th>\n",
       "      <th>annotator2</th>\n",
       "      <th>annotator3</th>\n",
       "      <th>annotator4</th>\n",
       "      <th>annotator5</th>\n",
       "      <th>annotator6</th>\n",
       "      <th>annotator7</th>\n",
       "      <th>annotator8</th>\n",
       "      <th>annotator9</th>\n",
       "      <th>annotator10</th>\n",
       "      <th>mean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>175</th>\n",
       "      <td>2008 summer olympics</td>\n",
       "      <td>9.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>855</th>\n",
       "      <td>gothic architecture</td>\n",
       "      <td>45.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.857143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1177</th>\n",
       "      <td>gothic architecture</td>\n",
       "      <td>61.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1430</th>\n",
       "      <td>gothic architecture</td>\n",
       "      <td>75.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2027</th>\n",
       "      <td>immune system</td>\n",
       "      <td>106.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2783</th>\n",
       "      <td>immune system</td>\n",
       "      <td>146.0</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.285714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2833</th>\n",
       "      <td>immune system</td>\n",
       "      <td>149.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2899</th>\n",
       "      <td>immune system</td>\n",
       "      <td>152.0</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2960</th>\n",
       "      <td>immune system</td>\n",
       "      <td>155.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2989</th>\n",
       "      <td>immune system</td>\n",
       "      <td>157.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.833333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3224</th>\n",
       "      <td>immune system</td>\n",
       "      <td>169.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3718</th>\n",
       "      <td>immune system</td>\n",
       "      <td>195.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.200000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     label  topic_id  annotator1  annotator2  annotator3  \\\n",
       "175   2008 summer olympics       9.0           3           3         2.0   \n",
       "855    gothic architecture      45.0           3           1         2.0   \n",
       "1177   gothic architecture      61.0           1           0         2.0   \n",
       "1430   gothic architecture      75.0           3           2         3.0   \n",
       "2027         immune system     106.0           0           0         0.0   \n",
       "2783         immune system     146.0           2           3         3.0   \n",
       "2833         immune system     149.0           1           1         2.0   \n",
       "2899         immune system     152.0           2           3         2.0   \n",
       "2960         immune system     155.0           1           2         1.0   \n",
       "2989         immune system     157.0           2           2         2.0   \n",
       "3224         immune system     169.0           2           0         0.0   \n",
       "3718         immune system     195.0           1           3         2.0   \n",
       "\n",
       "      annotator4  annotator5  annotator6  annotator7  annotator8  annotator9  \\\n",
       "175          2.0         NaN         NaN         NaN         NaN         NaN   \n",
       "855          1.0         3.0         1.0         2.0         NaN         NaN   \n",
       "1177         2.0         2.0         3.0         NaN         NaN         NaN   \n",
       "1430         NaN         NaN         NaN         NaN         NaN         NaN   \n",
       "2027         2.0         1.0         NaN         NaN         NaN         NaN   \n",
       "2783         2.0         1.0         3.0         2.0         NaN         NaN   \n",
       "2833         2.0         1.0         1.0         NaN         NaN         NaN   \n",
       "2899         1.0         1.0         NaN         NaN         NaN         NaN   \n",
       "2960         2.0         1.0         NaN         NaN         NaN         NaN   \n",
       "2989         2.0         1.0         2.0         NaN         NaN         NaN   \n",
       "3224         1.0         3.0         NaN         NaN         NaN         NaN   \n",
       "3718         3.0         2.0         NaN         NaN         NaN         NaN   \n",
       "\n",
       "      annotator10      mean  \n",
       "175           NaN  2.500000  \n",
       "855           NaN  1.857143  \n",
       "1177          NaN  1.666667  \n",
       "1430          NaN  2.666667  \n",
       "2027          NaN  0.600000  \n",
       "2783          NaN  2.285714  \n",
       "2833          NaN  1.333333  \n",
       "2899          NaN  1.800000  \n",
       "2960          NaN  1.400000  \n",
       "2989          NaN  1.833333  \n",
       "3224          NaN  1.200000  \n",
       "3718          NaN  2.200000  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_2016 = '../data/topiclabel/2016'\n",
    "f_topics2016 = join(data_2016, 'topics.csv')\n",
    "f_annotation2016 = join(data_2016, 'annotated_dataset.csv')\n",
    "topics2016 = pd.read_csv(f_topics2016)\n",
    "annotation2016 = pd.read_csv(f_annotation2016, sep='\\t')\n",
    "\n",
    "print(annotation2016.iloc[:, 2:].count().sum())\n",
    "mask16 = annotation2016.label.isin(examples)\n",
    "annotation2016['mean'] = annotation2016.loc[mask16, 'annotator1':].mean(axis=1)\n",
    "annotation2016.loc[mask16]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>topic_id</th>\n",
       "      <th>annotator1</th>\n",
       "      <th>annotator2</th>\n",
       "      <th>annotator3</th>\n",
       "      <th>annotator4</th>\n",
       "      <th>annotator5</th>\n",
       "      <th>annotator6</th>\n",
       "      <th>annotator7</th>\n",
       "      <th>annotator8</th>\n",
       "      <th>annotator9</th>\n",
       "      <th>annotator10</th>\n",
       "      <th>mean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>683</th>\n",
       "      <td>a woman</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1383</th>\n",
       "      <td>a woman</td>\n",
       "      <td>72.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2680</th>\n",
       "      <td>a woman</td>\n",
       "      <td>141.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>656</th>\n",
       "      <td>a woman</td>\n",
       "      <td>34.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3404</th>\n",
       "      <td>abdominal surgery</td>\n",
       "      <td>179.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1018</th>\n",
       "      <td>abdominal surgery</td>\n",
       "      <td>53.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3692</th>\n",
       "      <td>acetylcholine</td>\n",
       "      <td>194.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2883</th>\n",
       "      <td>acetylcholine</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3512</th>\n",
       "      <td>acetylcholine</td>\n",
       "      <td>184.0</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3853</th>\n",
       "      <td>acetylcysteine</td>\n",
       "      <td>202.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4136</th>\n",
       "      <td>acetylcysteine</td>\n",
       "      <td>217.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3933</th>\n",
       "      <td>acetylcysteine</td>\n",
       "      <td>207.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>428</th>\n",
       "      <td>acquittal</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2266</th>\n",
       "      <td>acquittal</td>\n",
       "      <td>119.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>325</th>\n",
       "      <td>actin</td>\n",
       "      <td>17.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3612</th>\n",
       "      <td>actin</td>\n",
       "      <td>190.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3686</th>\n",
       "      <td>actin</td>\n",
       "      <td>194.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2795</th>\n",
       "      <td>actin</td>\n",
       "      <td>147.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3350</th>\n",
       "      <td>actin</td>\n",
       "      <td>176.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3688</th>\n",
       "      <td>action potential</td>\n",
       "      <td>194.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3508</th>\n",
       "      <td>action potential</td>\n",
       "      <td>184.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4231</th>\n",
       "      <td>activated carbon</td>\n",
       "      <td>222.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3548</th>\n",
       "      <td>activated carbon</td>\n",
       "      <td>186.0</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4322</th>\n",
       "      <td>activated carbon</td>\n",
       "      <td>227.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4289</th>\n",
       "      <td>activation</td>\n",
       "      <td>225.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3784</th>\n",
       "      <td>activation</td>\n",
       "      <td>199.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3826</th>\n",
       "      <td>addiction</td>\n",
       "      <td>201.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3740</th>\n",
       "      <td>addiction</td>\n",
       "      <td>196.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2752</th>\n",
       "      <td>adenocarcinoma</td>\n",
       "      <td>144.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3082</th>\n",
       "      <td>adenocarcinoma</td>\n",
       "      <td>162.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>windows mobile</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>347</th>\n",
       "      <td>windows mobile</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>723</th>\n",
       "      <td>windows mobile</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>739</th>\n",
       "      <td>windows phone</td>\n",
       "      <td>38.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>348</th>\n",
       "      <td>windows phone</td>\n",
       "      <td>18.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>734</th>\n",
       "      <td>windows phone 7</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>350</th>\n",
       "      <td>windows phone 7</td>\n",
       "      <td>18.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>windows server 2003</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>windows server 2003</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>windows server 2008</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>windows server 2008</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2037</th>\n",
       "      <td>winning streak (sports)</td>\n",
       "      <td>107.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2457</th>\n",
       "      <td>winning streak (sports)</td>\n",
       "      <td>129.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2064</th>\n",
       "      <td>winning streak (sports)</td>\n",
       "      <td>108.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>490</th>\n",
       "      <td>winning streak (sports)</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1925</th>\n",
       "      <td>winning streak (sports)</td>\n",
       "      <td>101.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4145</th>\n",
       "      <td>working memory</td>\n",
       "      <td>218.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4089</th>\n",
       "      <td>working memory</td>\n",
       "      <td>215.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>382</th>\n",
       "      <td>world series</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2035</th>\n",
       "      <td>world series</td>\n",
       "      <td>107.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3216</th>\n",
       "      <td>wound healing</td>\n",
       "      <td>169.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4133</th>\n",
       "      <td>wound healing</td>\n",
       "      <td>217.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1582</th>\n",
       "      <td>write-in candidate</td>\n",
       "      <td>83.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110</th>\n",
       "      <td>write-in candidate</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1915</th>\n",
       "      <td>write-in candidate</td>\n",
       "      <td>100.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1495</th>\n",
       "      <td>writer</td>\n",
       "      <td>78.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>694</th>\n",
       "      <td>writer</td>\n",
       "      <td>36.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2174</th>\n",
       "      <td>writer</td>\n",
       "      <td>114.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1289</th>\n",
       "      <td>you</td>\n",
       "      <td>67.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>717</th>\n",
       "      <td>you</td>\n",
       "      <td>37.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2039 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                        label  topic_id  annotator1  annotator2  annotator3  \\\n",
       "683                   a woman      35.0           1           2         1.0   \n",
       "1383                  a woman      72.0           2           2         2.0   \n",
       "2680                  a woman     141.0           3           1         2.0   \n",
       "656                   a woman      34.0           3           2         1.0   \n",
       "3404        abdominal surgery     179.0           2           2         1.0   \n",
       "1018        abdominal surgery      53.0           2           1         1.0   \n",
       "3692            acetylcholine     194.0           2           1         2.0   \n",
       "2883            acetylcholine     151.0           2           1         3.0   \n",
       "3512            acetylcholine     184.0           2           3         2.0   \n",
       "3853           acetylcysteine     202.0           2           0         1.0   \n",
       "4136           acetylcysteine     217.0           1           0         1.0   \n",
       "3933           acetylcysteine     207.0           1           1         0.0   \n",
       "428                 acquittal      22.0           1           2         3.0   \n",
       "2266                acquittal     119.0           1           2         0.0   \n",
       "325                     actin      17.0           1           1         1.0   \n",
       "3612                    actin     190.0           1           1         0.0   \n",
       "3686                    actin     194.0           1           3         1.0   \n",
       "2795                    actin     147.0           1           1         2.0   \n",
       "3350                    actin     176.0           0           3         3.0   \n",
       "3688         action potential     194.0           1           1         1.0   \n",
       "3508         action potential     184.0           2           0         1.0   \n",
       "4231         activated carbon     222.0           0           1         1.0   \n",
       "3548         activated carbon     186.0           2           3         0.0   \n",
       "4322         activated carbon     227.0           2           0         0.0   \n",
       "4289               activation     225.0           1           1         1.0   \n",
       "3784               activation     199.0           1           1         2.0   \n",
       "3826                addiction     201.0           1           2         1.0   \n",
       "3740                addiction     196.0           3           2         0.0   \n",
       "2752           adenocarcinoma     144.0           3           2         1.0   \n",
       "3082           adenocarcinoma     162.0           3           3         2.0   \n",
       "...                       ...       ...         ...         ...         ...   \n",
       "39             windows mobile       2.0           1           2         1.0   \n",
       "347            windows mobile      18.0           1           2         0.0   \n",
       "723            windows mobile      38.0           1           0         1.0   \n",
       "739             windows phone      38.0           0           0         2.0   \n",
       "348             windows phone      18.0           0           0         3.0   \n",
       "734           windows phone 7      38.0           1           0         1.0   \n",
       "350           windows phone 7      18.0           0           3         3.0   \n",
       "47        windows server 2003       2.0           2           3         1.0   \n",
       "14        windows server 2003       0.0           1           1         2.0   \n",
       "43        windows server 2008       2.0           2           2         1.0   \n",
       "13        windows server 2008       0.0           0           2         1.0   \n",
       "2037  winning streak (sports)     107.0           0           0         2.0   \n",
       "2457  winning streak (sports)     129.0           2           2         1.0   \n",
       "2064  winning streak (sports)     108.0           1           1         2.0   \n",
       "490   winning streak (sports)      25.0           3           2         1.0   \n",
       "1925  winning streak (sports)     101.0           1           1         1.0   \n",
       "4145           working memory     218.0           1           2         2.0   \n",
       "4089           working memory     215.0           1           2         2.0   \n",
       "382              world series      20.0           0           3         2.0   \n",
       "2035             world series     107.0           0           1         0.0   \n",
       "3216            wound healing     169.0           0           2         1.0   \n",
       "4133            wound healing     217.0           2           2         3.0   \n",
       "1582       write-in candidate      83.0           1           2         1.0   \n",
       "110        write-in candidate       5.0           2           2         2.0   \n",
       "1915       write-in candidate     100.0           3           3         1.0   \n",
       "1495                   writer      78.0           3           2         3.0   \n",
       "694                    writer      36.0           1           1         3.0   \n",
       "2174                   writer     114.0           3           1         2.0   \n",
       "1289                      you      67.0           1           0         0.0   \n",
       "717                       you      37.0           1           0         0.0   \n",
       "\n",
       "      annotator4  annotator5  annotator6  annotator7  annotator8  annotator9  \\\n",
       "683          1.0         1.0         2.0         NaN         NaN         NaN   \n",
       "1383         0.0         1.0         0.0         0.0         NaN         NaN   \n",
       "2680         NaN         NaN         NaN         NaN         NaN         NaN   \n",
       "656          2.0         2.0         0.0         0.0         NaN         NaN   \n",
       "3404         2.0         0.0         1.0         NaN         NaN         NaN   \n",
       "1018         0.0         3.0         3.0         2.0         2.0         NaN   \n",
       "3692         2.0         1.0         0.0         2.0         2.0         NaN   \n",
       "2883         0.0         2.0         NaN         NaN         NaN         NaN   \n",
       "3512         2.0         2.0         1.0         1.0         1.0         NaN   \n",
       "3853         1.0         1.0         1.0         1.0         NaN         NaN   \n",
       "4136         1.0         2.0         3.0         1.0         NaN         NaN   \n",
       "3933         1.0         2.0         0.0         0.0         NaN         NaN   \n",
       "428          3.0         3.0         2.0         2.0         2.0         2.0   \n",
       "2266         NaN         NaN         NaN         NaN         NaN         NaN   \n",
       "325          2.0         1.0         2.0         0.0         NaN         NaN   \n",
       "3612         3.0         1.0         1.0         2.0         1.0         NaN   \n",
       "3686         2.0         1.0         1.0         1.0         NaN         NaN   \n",
       "2795         0.0         1.0         2.0         NaN         NaN         NaN   \n",
       "3350         1.0         NaN         NaN         NaN         NaN         NaN   \n",
       "3688         0.0         2.0         2.0         2.0         2.0         3.0   \n",
       "3508         1.0         1.0         2.0         NaN         NaN         NaN   \n",
       "4231         1.0         1.0         1.0         1.0         2.0         1.0   \n",
       "3548         NaN         NaN         NaN         NaN         NaN         NaN   \n",
       "4322         1.0         1.0         NaN         NaN         NaN         NaN   \n",
       "4289         1.0         1.0         1.0         NaN         NaN         NaN   \n",
       "3784         0.0         1.0         1.0         1.0         1.0         0.0   \n",
       "3826         2.0         3.0         1.0         2.0         NaN         NaN   \n",
       "3740         2.0         2.0         2.0         3.0         2.0         1.0   \n",
       "2752         2.0         1.0         0.0         1.0         1.0         NaN   \n",
       "3082         NaN         NaN         NaN         NaN         NaN         NaN   \n",
       "...          ...         ...         ...         ...         ...         ...   \n",
       "39           1.0         NaN         NaN         NaN         NaN         NaN   \n",
       "347          2.0         2.0         3.0         2.0         1.0         3.0   \n",
       "723          0.0         2.0         0.0         NaN         NaN         NaN   \n",
       "739          0.0         1.0         2.0         0.0         NaN         NaN   \n",
       "348          1.0         1.0         2.0         0.0         NaN         NaN   \n",
       "734          1.0         2.0         NaN         NaN         NaN         NaN   \n",
       "350          1.0         1.0         3.0         NaN         NaN         NaN   \n",
       "47           2.0         2.0         3.0         NaN         NaN         NaN   \n",
       "14           1.0         2.0         1.0         NaN         NaN         NaN   \n",
       "43           2.0         0.0         0.0         NaN         NaN         NaN   \n",
       "13           0.0         3.0         NaN         NaN         NaN         NaN   \n",
       "2037         3.0         NaN         NaN         NaN         NaN         NaN   \n",
       "2457         2.0         2.0         2.0         1.0         0.0         2.0   \n",
       "2064         2.0         2.0         2.0         1.0         NaN         NaN   \n",
       "490          1.0         2.0         0.0         1.0         NaN         NaN   \n",
       "1925         2.0         2.0         2.0         1.0         1.0         1.0   \n",
       "4145         2.0         3.0         3.0         NaN         NaN         NaN   \n",
       "4089         3.0         0.0         1.0         2.0         1.0         1.0   \n",
       "382          NaN         NaN         NaN         NaN         NaN         NaN   \n",
       "2035         2.0         3.0         1.0         NaN         NaN         NaN   \n",
       "3216         0.0         NaN         NaN         NaN         NaN         NaN   \n",
       "4133         2.0         3.0         1.0         2.0         1.0         1.0   \n",
       "1582         1.0         NaN         NaN         NaN         NaN         NaN   \n",
       "110          1.0         1.0         2.0         2.0         NaN         NaN   \n",
       "1915         2.0         0.0         NaN         NaN         NaN         NaN   \n",
       "1495         1.0         3.0         2.0         3.0         2.0         2.0   \n",
       "694          2.0         3.0         1.0         3.0         2.0         NaN   \n",
       "2174         2.0         2.0         3.0         2.0         2.0         NaN   \n",
       "1289         1.0         0.0         0.0         0.0         1.0         NaN   \n",
       "717          2.0         1.0         1.0         1.0         NaN         NaN   \n",
       "\n",
       "      annotator10  mean  \n",
       "683           NaN   NaN  \n",
       "1383          NaN   NaN  \n",
       "2680          NaN   NaN  \n",
       "656           NaN   NaN  \n",
       "3404          NaN   NaN  \n",
       "1018          NaN   NaN  \n",
       "3692          NaN   NaN  \n",
       "2883          NaN   NaN  \n",
       "3512          NaN   NaN  \n",
       "3853          NaN   NaN  \n",
       "4136          NaN   NaN  \n",
       "3933          NaN   NaN  \n",
       "428           NaN   NaN  \n",
       "2266          NaN   NaN  \n",
       "325           NaN   NaN  \n",
       "3612          NaN   NaN  \n",
       "3686          NaN   NaN  \n",
       "2795          NaN   NaN  \n",
       "3350          NaN   NaN  \n",
       "3688          NaN   NaN  \n",
       "3508          NaN   NaN  \n",
       "4231          NaN   NaN  \n",
       "3548          NaN   NaN  \n",
       "4322          NaN   NaN  \n",
       "4289          NaN   NaN  \n",
       "3784          1.0   NaN  \n",
       "3826          NaN   NaN  \n",
       "3740          2.0   NaN  \n",
       "2752          NaN   NaN  \n",
       "3082          NaN   NaN  \n",
       "...           ...   ...  \n",
       "39            NaN   NaN  \n",
       "347           1.0   NaN  \n",
       "723           NaN   NaN  \n",
       "739           NaN   NaN  \n",
       "348           NaN   NaN  \n",
       "734           NaN   NaN  \n",
       "350           NaN   NaN  \n",
       "47            NaN   NaN  \n",
       "14            NaN   NaN  \n",
       "43            NaN   NaN  \n",
       "13            NaN   NaN  \n",
       "2037          NaN   NaN  \n",
       "2457          NaN   NaN  \n",
       "2064          NaN   NaN  \n",
       "490           NaN   NaN  \n",
       "1925          NaN   NaN  \n",
       "4145          NaN   NaN  \n",
       "4089          NaN   NaN  \n",
       "382           NaN   NaN  \n",
       "2035          NaN   NaN  \n",
       "3216          NaN   NaN  \n",
       "4133          NaN   NaN  \n",
       "1582          NaN   NaN  \n",
       "110           NaN   NaN  \n",
       "1915          NaN   NaN  \n",
       "1495          NaN   NaN  \n",
       "694           NaN   NaN  \n",
       "2174          NaN   NaN  \n",
       "1289          NaN   NaN  \n",
       "717           NaN   NaN  \n",
       "\n",
       "[2039 rows x 13 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "annotation2016[annotation2016.duplicated('label', keep=False)].sort_values('label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "228\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>topic_id</th>\n",
       "      <th>domain</th>\n",
       "      <th>term0</th>\n",
       "      <th>term1</th>\n",
       "      <th>term2</th>\n",
       "      <th>term3</th>\n",
       "      <th>term4</th>\n",
       "      <th>term5</th>\n",
       "      <th>term6</th>\n",
       "      <th>term7</th>\n",
       "      <th>term8</th>\n",
       "      <th>term9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>35</td>\n",
       "      <td>blogs</td>\n",
       "      <td>child</td>\n",
       "      <td>woman</td>\n",
       "      <td>family</td>\n",
       "      <td>kid</td>\n",
       "      <td>parent</td>\n",
       "      <td>baby</td>\n",
       "      <td>mother</td>\n",
       "      <td>home</td>\n",
       "      <td>husband</td>\n",
       "      <td>young</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>72</td>\n",
       "      <td>iabooks</td>\n",
       "      <td>mr.</td>\n",
       "      <td>mrs.</td>\n",
       "      <td>young</td>\n",
       "      <td>lady</td>\n",
       "      <td>look</td>\n",
       "      <td>friend</td>\n",
       "      <td>tell</td>\n",
       "      <td>mother</td>\n",
       "      <td>miss</td>\n",
       "      <td>father</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>141</th>\n",
       "      <td>141</td>\n",
       "      <td>news</td>\n",
       "      <td>woman</td>\n",
       "      <td>ms.</td>\n",
       "      <td>sex</td>\n",
       "      <td>sexual</td>\n",
       "      <td>male</td>\n",
       "      <td>female</td>\n",
       "      <td>age</td>\n",
       "      <td>bill</td>\n",
       "      <td>begin</td>\n",
       "      <td>husband</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>34</td>\n",
       "      <td>blogs</td>\n",
       "      <td>wear</td>\n",
       "      <td>baby</td>\n",
       "      <td>dress</td>\n",
       "      <td>wedding</td>\n",
       "      <td>look</td>\n",
       "      <td>love</td>\n",
       "      <td>clothes</td>\n",
       "      <td>girl</td>\n",
       "      <td>kid</td>\n",
       "      <td>need</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     topic_id   domain  term0  term1   term2    term3   term4         term5  \\\n",
       "35         35    blogs  child  woman  family      kid  parent          baby   \n",
       "72         72  iabooks    mr.   mrs.   young     lady    look        friend   \n",
       "141       141     news  woman    ms.     sex   sexual    male        female   \n",
       "34         34    blogs   wear   baby   dress  wedding    look          love   \n",
       "\n",
       "       term6   term7    term8    term9  \n",
       "35    mother    home  husband    young  \n",
       "72      tell  mother     miss   father  \n",
       "141      age    bill    begin  husband  \n",
       "34   clothes    girl      kid     need  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(topics2016.shape[0])\n",
    "topics2016.iloc[[35, 72, 141, 34]]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}