2. Embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Embeddings\n",
    "\n",
    "## Word2Vec\n",
    "\n",
    "Векторные модели, которые мы рассматривали до этого, условно называются *счётными*. Они основываются на том, что так или иначе \"считают\" слова и их соседей, и на основе этого строят вектора для слов. \n",
    "\n",
    "Другой класс моделей, который более повсевмёстно распространён на сегодняшний день, называется *предсказательными* (или *нейронными*) моделями. Идея этих моделей заключается в использовании нейросетевых архитектур, которые \"предсказывают\" (а не считают) соседей слов. Одной из самых известных таких моделей является word2vec. Технология основана на нейронной сети, предсказывающей вероятность встретить слово в заданном контексте. Этот инструмент был разработан группой исследователей Google в 2013 году, руководителем проекта был Томаш Миколов (сейчас работает в Facebook). Вот две самые главные статьи:\n",
    "\n",
    "* [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/pdf/1301.3781.pdf)\n",
    "* [Distributed Representations of Words and Phrases and their Compositionality](https://arxiv.org/abs/1310.4546)\n",
    "\n",
    "$$\\hat{P}(w_1^T) = \\prod_{t=1}^T \\hat{P}(w_t \\mid w_1^{t-1})$$\n",
    "\n",
    "Полученные таким образом вектора называются *распределенными представлениями слов*, или **эмбеддингами**.\n",
    "\n",
    "### Зачем это нужно?\n",
    "\n",
    "* Решать лингвистические задачи (в основном это про семантику и сочетаемость)\n",
    "* Подавать на вход нейронным сетям\n",
    "\n",
    "### Как это обучается?\n",
    "Мы задаём вектор для каждого слова с помощью матрицы $w$ и вектор контекста с помощью матрицы $W$. По сути, word2vec является обобщающим названием для двух архитектур Skip-Gram и Continuous Bag-Of-Words (CBOW).  \n",
    "\n",
    "**CBOW** предсказывает текущее слово, исходя из окружающего его контекста. \n",
    "\n",
    "**Skip-gram**, наоборот, использует текущее слово, чтобы предугадывать окружающие его слова. \n",
    "\n",
    "### Как это работает?\n",
    "Word2vec принимает большой текстовый корпус в качестве входных данных и сопоставляет каждому слову вектор, выдавая координаты слов на выходе. Сначала он создает словарь, «обучаясь» на входных текстовых данных, а затем вычисляет векторное представление слов. Векторное представление основывается на контекстной близости: слова, встречающиеся в тексте рядом с одинаковыми словами (а следовательно, согласно дистрибутивной гипотезе, имеющие схожий смысл), в векторном представлении будут иметь близкие координаты векторов-слов. Для вычисления близости слов используется косинусное расстояние между их векторами.\n",
    "\n",
    "\n",
    "<img src=\"https://nycdatascience.com/blog/wp-content/uploads/2017/06/cossim.png\" width=\"500\">\n",
    "\n",
    "\n",
    "С помощью дистрибутивных векторных моделей можно строить семантические пропорции (они же аналогии) и решать примеры:\n",
    "\n",
    "* *король: мужчина = королева: женщина* \n",
    " $\\Rightarrow$ \n",
    "* *король - мужчина + женщина = королева*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![w2v](https://cdn-images-1.medium.com/max/2600/1*sXNXYfAqfLUeiDXPCo130w.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Проблемы\n",
    "1. Матрицы слишком разреженные (→ очень большие)\n",
    "2. Невозможно установить тип семантических отношений между словами: синонимы, антонимы и т.д. будут одинаково близки, потому что обычно употребляются в схожих контекстахю Поэтому близкие в векторном пространстве слова называют *семантическими ассоциатами*. Это значит, что они семантически связаны, но как именно — непонятно.\n",
    "\n",
    "\n",
    "## RusVectōrēs\n",
    "\n",
    "\n",
    "На сайте [RusVectōrēs](https://rusvectores.org/ru/) собраны предобученные на различных данных модели для русского языка, а также можно поискать наиболее близкие слова к заданному, посчитать семантическую близость нескольких слов и порешать примеры с помощью «калькулятором семантической близости».\n",
    "\n",
    "<img src=\"./img/rusvectores2.png\" width=\"500\">\n",
    "\n",
    "Для других языков также можно найти предобученные модели — например, модели [fastText](https://fasttext.cc/docs/en/english-vectors.html) и [GloVe](https://nlp.stanford.edu/projects/glove/) (о них чуть дальше)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gensim\n",
    "\n",
    "Использовать предобученную модель эмбеддингов или обучить свою можно с помощью библиотеки `gensim`. Вот [ее документация](https://radimrehurek.com/gensim/models/word2vec.html).\n",
    "\n",
    "### Как использовать готовую модель\n",
    "\n",
    "Модели word2vec бывают разных форматов:\n",
    "\n",
    "* .vec.gz — обычный файл\n",
    "* .bin.gz — бинарник\n",
    "\n",
    "Загружаются они с помощью одного и того же гласса `KeyedVectors`, меняется только параметр `binary` у функции `load_word2vec_format`. \n",
    "\n",
    "Если же эмбеддинги обучены **не** с помощью word2vec, то для загрузки нужно использовать функцию `load`. Т.е. для загрузки предобученных эмбеддингов *glove, fasttext, bpe* и любых других нужна именно она.\n",
    "\n",
    "Скачаем с RusVectōrēs модель для русского языка, обученную на НКРЯ образца 2015 г. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import gensim\n",
    "import logging\n",
    "import nltk.data \n",
    "import pandas as pd\n",
    "import urllib.request\n",
    "from bs4 import BeautifulSoup\n",
    "from nltk.corpus import stopwords\n",
    "from gensim.models import word2vec\n",
    "from nltk.tokenize import sent_tokenize, RegexpTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('ruscorpora_mystem_cbow_300_2_2015.bin.gz',\n",
       " <http.client.HTTPMessage at 0x7fcb7efb06a0>)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urllib.request.urlretrieve(\"http://rusvectores.org/static/models/rusvectores2/ruscorpora_mystem_cbow_300_2_2015.bin.gz\", \"ruscorpora_mystem_cbow_300_2_2015.bin.gz\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 15:45:53,489 : INFO : loading projection weights from ruscorpora_mystem_cbow_300_2_2015.bin.gz\n",
      "2019-04-12 15:46:05,229 : INFO : loaded (281776, 300) matrix from ruscorpora_mystem_cbow_300_2_2015.bin.gz\n"
     ]
    }
   ],
   "source": [
    "model_path = 'ruscorpora_mystem_cbow_300_2_2015.bin.gz'\n",
    "\n",
    "model_ru = gensim.models.KeyedVectors.load_word2vec_format(model_path, binary=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = ['день_S', 'ночь_S', 'человек_S', 'семантика_S', 'биткоин_S']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Частеречные тэги нужны, поскольку это специфика скачанной модели - она была натренирована на словах, аннотированных их частями речи (и лемматизированных). **NB!** В названиях моделей на `rusvectores` указано, какой тегсет они используют (mystem, upos и т.д.)\n",
    "\n",
    "Попросим у модели 10 ближайших соседей для каждого слова и коэффициент косинусной близости для каждого:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 15:46:12,221 : INFO : precomputing L2-norms of word weight vectors\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "день_S\n",
      "[-0.02580778  0.00970898  0.01941961 -0.02332282  0.02017624  0.07275085\n",
      " -0.01444375  0.03316632  0.01242602  0.02833412]\n",
      "неделя_S 0.7165195941925049\n",
      "месяц_S 0.6310489177703857\n",
      "вечер_S 0.5828738808631897\n",
      "утро_S 0.5676206946372986\n",
      "час_S 0.5605547428131104\n",
      "минута_S 0.5297019481658936\n",
      "гекатомбеон_S 0.4897990822792053\n",
      "денек_S 0.48224717378616333\n",
      "полчаса_S 0.48217129707336426\n",
      "ночь_S 0.478074848651886\n",
      "\n",
      "\n",
      "ночь_S\n",
      "[-0.00688948  0.00408364  0.06975466 -0.00959525  0.0194835   0.04057068\n",
      " -0.00994112  0.06064967 -0.00522624  0.00520327]\n",
      "вечер_S 0.6946247816085815\n",
      "утро_S 0.5730193853378296\n",
      "ноченька_S 0.5582467317581177\n",
      "рассвет_S 0.5553582906723022\n",
      "ночка_S 0.5351512432098389\n",
      "полдень_S 0.5334426760673523\n",
      "полночь_S 0.478694349527359\n",
      "день_S 0.4780748784542084\n",
      "сумерки_S 0.43902185559272766\n",
      "фундерфун_S 0.4340824782848358\n",
      "\n",
      "\n",
      "человек_S\n",
      "[ 0.02013756 -0.02670703 -0.02039861 -0.05477146  0.00086402 -0.01636335\n",
      "  0.04240306 -0.00025525 -0.14045681  0.04785006]\n",
      "женщина_S 0.5979775190353394\n",
      "парень_S 0.4991787374019623\n",
      "мужчина_S 0.4767409563064575\n",
      "мужик_S 0.47383999824523926\n",
      "россиянин_S 0.4719043970108032\n",
      "народ_S 0.4654741883277893\n",
      "согражданин_S 0.45378509163856506\n",
      "горожанин_S 0.44368088245391846\n",
      "девушка_S 0.44314485788345337\n",
      "иностранец_S 0.43849870562553406\n",
      "\n",
      "\n",
      "семантика_S\n",
      "[-0.03066749  0.0053851   0.1110732   0.0152335   0.00440643  0.00384104\n",
      "  0.00096944 -0.03538784 -0.00079585  0.03220548]\n",
      "семантический_A 0.5334584712982178\n",
      "понятие_S 0.5030269026756287\n",
      "сочетаемость_S 0.4817051887512207\n",
      "актант_S 0.47596412897109985\n",
      "хронотоп_S 0.46330296993255615\n",
      "метафора_S 0.46158891916275024\n",
      "мышление_S 0.4610119163990021\n",
      "парадигма_S 0.4579665958881378\n",
      "лексема_S 0.45688074827194214\n",
      "смысловой_A 0.4543077349662781\n",
      "\n",
      "\n",
      "Увы, слова \"биткоин_S\" нет в модели!\n"
     ]
    }
   ],
   "source": [
    "for word in words:\n",
    "    # есть ли слово в модели? \n",
    "    if word in model_ru:\n",
    "        print(word)\n",
    "        # смотрим на вектор слова (его размерность 300, смотрим на первые 10 чисел)\n",
    "        print(model_ru[word][:10])\n",
    "        # выдаем 10 ближайших соседей слова:\n",
    "        for i in model_ru.most_similar(positive=[word], topn=10):\n",
    "            # слово + коэффициент косинусной близости\n",
    "            print(i[0], i[1])\n",
    "        print('\\n')\n",
    "    else:\n",
    "        # Увы!\n",
    "        print('Увы, слова \"%s\" нет в модели!' % word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Находим косинусную близость пары слов:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.23895609343220622\n"
     ]
    }
   ],
   "source": [
    "print(model_ru.similarity('человек_S', 'обезьяна_S'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Что получится, если вычесть из пиццы Италию и прибавить Сибирь?\n",
    "\n",
    "* positive — вектора, которые мы складываем\n",
    "* negative — вектора, которые вычитаем"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "пельмень_S\n"
     ]
    }
   ],
   "source": [
    "print(model_ru.most_similar(positive=['пицца_S', 'сибирь_S'], negative=['италия_S'])[0][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Как обучить свою модель\n",
    "\n",
    "В качестве обучающих данных возьмем размеченные и неразмеченные отзывы о фильмах (датасет взят с Kaggle)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read 25000 labeled train reviews, 25000 labeled test reviews, and 50000 unlabeled reviews\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train = pd.read_csv(\"./data/w2v/train/labeledTrainData.tsv\", header=0, delimiter=\"\\t\", quoting=3)\n",
    "test = pd.read_csv(\"./data/w2v/train/testData.tsv\", header=0, delimiter=\"\\t\", quoting=3)\n",
    "unlabeled_train = pd.read_csv(\"./data/w2v/train/unlabeledTrainData.tsv\", header=0, delimiter=\"\\t\", quoting=3)\n",
    "\n",
    "print(\"Read %d labeled train reviews, %d labeled test reviews, and %d unlabeled reviews\\n\" \\\n",
    "      % (train[\"review\"].size, test[\"review\"].size, unlabeled_train[\"review\"].size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Убираем из данных ссылки, html-разметку и небуквенные символы, а затем приводим все к нижнему регистру и токенизируем. На выходе получается массив из предложений, каждое из которых представляет собой массив слов. Здесь используется токенизатор из библиотеки `nltk`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "\n",
    "def review_to_wordlist(review, remove_stopwords=False ):\n",
    "    # убираем ссылки вне тегов\n",
    "    review = re.sub(r\"http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+\", \" \", review)\n",
    "    review_text = BeautifulSoup(review, \"lxml\").get_text()\n",
    "    review_text = re.sub(\"[^a-zA-Z]\",\" \", review_text)\n",
    "    words = review_text.lower().split()\n",
    "    if remove_stopwords:\n",
    "        stops = stopwords.words(\"english\")\n",
    "        words = [w for w in words if not w in stops]\n",
    "    return(words)\n",
    "\n",
    "def review_to_sentences(review, tokenizer, remove_stopwords=False):\n",
    "    raw_sentences = tokenizer.tokenize(review.strip())\n",
    "    sentences = []\n",
    "    for raw_sentence in raw_sentences:\n",
    "        if len(raw_sentence) > 0:\n",
    "            sentences.append(review_to_wordlist(raw_sentence, remove_stopwords))\n",
    "    return sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parsing sentences from training set...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ancatmara/anaconda3/lib/python3.6/site-packages/bs4/__init__.py:273: UserWarning: \"b'.'\" looks like a filename, not markup. You should probably open this file and pass the filehandle into Beautiful Soup.\n",
      "  ' Beautiful Soup.' % markup)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parsing sentences from unlabeled set...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ancatmara/anaconda3/lib/python3.6/site-packages/bs4/__init__.py:273: UserWarning: \"b'..'\" looks like a filename, not markup. You should probably open this file and pass the filehandle into Beautiful Soup.\n",
      "  ' Beautiful Soup.' % markup)\n"
     ]
    }
   ],
   "source": [
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)\n",
    "\n",
    "sentences = []  \n",
    "\n",
    "print(\"Parsing sentences from training set...\")\n",
    "for review in train[\"review\"]:\n",
    "    sentences += review_to_sentences(review, tokenizer)\n",
    "\n",
    "print(\"Parsing sentences from unlabeled set...\")\n",
    "for review in unlabeled_train[\"review\"]:\n",
    "    sentences += review_to_sentences(review, tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "795538\n",
      "['with', 'all', 'this', 'stuff', 'going', 'down', 'at', 'the', 'moment', 'with', 'mj', 'i', 've', 'started', 'listening', 'to', 'his', 'music', 'watching', 'the', 'odd', 'documentary', 'here', 'and', 'there', 'watched', 'the', 'wiz', 'and', 'watched', 'moonwalker', 'again']\n"
     ]
    }
   ],
   "source": [
    "print(len(sentences))\n",
    "print(sentences[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обучаем и сохраняем модель. \n",
    "\n",
    "\n",
    "Основные параметры:\n",
    "* данные должны быть итерируемым объектом \n",
    "* size — размер вектора, \n",
    "* window — размер окна наблюдения,\n",
    "* min_count — мин. частотность слова в корпусе,\n",
    "* sg — используемый алгоритм обучения (0 — CBOW, 1 — Skip-gram),\n",
    "* sample — порог для downsampling'a высокочастотных слов,\n",
    "* workers — количество потоков,\n",
    "* alpha — learning rate,\n",
    "* iter — количество итераций,\n",
    "* max_vocab_size — позволяет выставить ограничение по памяти при создании словаря (т.е. если ограничение привышается, то низкочастотные слова будут выбрасываться). Для сравнения: 10 млн слов = 1Гб RAM.\n",
    "\n",
    "**NB!** Обратите внимание, что тренировка модели не включает препроцессинг! Это значит, что избавляться от пунктуации, приводить слова к нижнему регистру, лемматизировать их, проставлять частеречные теги придется до тренировки модели (если, конечно, это необходимо для вашей задачи). Т.е. в каком виде слова будут в исходном тексте, в таком они будут и в модели."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 17:21:25,258 : INFO : collecting all words and their counts\n",
      "2019-04-12 17:21:25,260 : INFO : PROGRESS: at sentence #0, processed 0 words, keeping 0 word types\n",
      "2019-04-12 17:21:25,314 : INFO : PROGRESS: at sentence #10000, processed 225800 words, keeping 17774 word types\n",
      "2019-04-12 17:21:25,368 : INFO : PROGRESS: at sentence #20000, processed 451863 words, keeping 24941 word types\n",
      "2019-04-12 17:21:25,408 : INFO : PROGRESS: at sentence #30000, processed 671259 words, keeping 30023 word types\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 17:21:25,465 : INFO : PROGRESS: at sentence #40000, processed 897753 words, keeping 34338 word types\n",
      "2019-04-12 17:21:25,508 : INFO : PROGRESS: at sentence #50000, processed 1116873 words, keeping 37749 word types\n",
      "2019-04-12 17:21:25,558 : INFO : PROGRESS: at sentence #60000, processed 1338294 words, keeping 40708 word types\n",
      "2019-04-12 17:21:25,606 : INFO : PROGRESS: at sentence #70000, processed 1561470 words, keeping 43318 word types\n",
      "2019-04-12 17:21:25,652 : INFO : PROGRESS: at sentence #80000, processed 1780749 words, keeping 45696 word types\n",
      "2019-04-12 17:21:25,698 : INFO : PROGRESS: at sentence #90000, processed 2004848 words, keeping 48113 word types\n",
      "2019-04-12 17:21:25,752 : INFO : PROGRESS: at sentence #100000, processed 2226808 words, keeping 50184 word types\n",
      "2019-04-12 17:21:25,803 : INFO : PROGRESS: at sentence #110000, processed 2446405 words, keeping 52057 word types\n",
      "2019-04-12 17:21:25,849 : INFO : PROGRESS: at sentence #120000, processed 2668571 words, keeping 54091 word types\n",
      "2019-04-12 17:21:25,901 : INFO : PROGRESS: at sentence #130000, processed 2894095 words, keeping 55819 word types\n",
      "2019-04-12 17:21:25,948 : INFO : PROGRESS: at sentence #140000, processed 3106769 words, keeping 57315 word types\n",
      "2019-04-12 17:21:25,996 : INFO : PROGRESS: at sentence #150000, processed 3332383 words, keeping 59024 word types\n",
      "2019-04-12 17:21:26,055 : INFO : PROGRESS: at sentence #160000, processed 3555071 words, keeping 60586 word types\n",
      "2019-04-12 17:21:26,111 : INFO : PROGRESS: at sentence #170000, processed 3778402 words, keeping 62045 word types\n",
      "2019-04-12 17:21:26,164 : INFO : PROGRESS: at sentence #180000, processed 3998928 words, keeping 63458 word types\n",
      "2019-04-12 17:21:26,219 : INFO : PROGRESS: at sentence #190000, processed 4224124 words, keeping 64754 word types\n",
      "2019-04-12 17:21:26,283 : INFO : PROGRESS: at sentence #200000, processed 4448267 words, keeping 66045 word types\n",
      "2019-04-12 17:21:26,346 : INFO : PROGRESS: at sentence #210000, processed 4669611 words, keeping 67344 word types\n",
      "2019-04-12 17:21:26,417 : INFO : PROGRESS: at sentence #220000, processed 4894593 words, keeping 68652 word types\n",
      "2019-04-12 17:21:26,504 : INFO : PROGRESS: at sentence #230000, processed 5117163 words, keeping 69912 word types\n",
      "2019-04-12 17:21:26,584 : INFO : PROGRESS: at sentence #240000, processed 5344645 words, keeping 71119 word types\n",
      "2019-04-12 17:21:26,642 : INFO : PROGRESS: at sentence #250000, processed 5558760 words, keeping 72303 word types\n",
      "2019-04-12 17:21:26,719 : INFO : PROGRESS: at sentence #260000, processed 5778737 words, keeping 73429 word types\n",
      "2019-04-12 17:21:26,808 : INFO : PROGRESS: at sentence #270000, processed 6000010 words, keeping 74717 word types\n",
      "2019-04-12 17:21:26,860 : INFO : PROGRESS: at sentence #280000, processed 6225883 words, keeping 76319 word types\n",
      "2019-04-12 17:21:26,929 : INFO : PROGRESS: at sentence #290000, processed 6449035 words, keeping 77787 word types\n",
      "2019-04-12 17:21:27,019 : INFO : PROGRESS: at sentence #300000, processed 6673613 words, keeping 79115 word types\n",
      "2019-04-12 17:21:27,098 : INFO : PROGRESS: at sentence #310000, processed 6898916 words, keeping 80424 word types\n",
      "2019-04-12 17:21:27,159 : INFO : PROGRESS: at sentence #320000, processed 7123789 words, keeping 81752 word types\n",
      "2019-04-12 17:21:27,220 : INFO : PROGRESS: at sentence #330000, processed 7345526 words, keeping 82974 word types\n",
      "2019-04-12 17:21:27,281 : INFO : PROGRESS: at sentence #340000, processed 7575023 words, keeping 84219 word types\n",
      "2019-04-12 17:21:27,343 : INFO : PROGRESS: at sentence #350000, processed 7798275 words, keeping 85364 word types\n",
      "2019-04-12 17:21:27,398 : INFO : PROGRESS: at sentence #360000, processed 8018874 words, keeping 86531 word types\n",
      "2019-04-12 17:21:27,464 : INFO : PROGRESS: at sentence #370000, processed 8246066 words, keeping 87643 word types\n",
      "2019-04-12 17:21:27,523 : INFO : PROGRESS: at sentence #380000, processed 8471186 words, keeping 88811 word types\n",
      "2019-04-12 17:21:27,599 : INFO : PROGRESS: at sentence #390000, processed 8700909 words, keeping 89838 word types\n",
      "2019-04-12 17:21:27,655 : INFO : PROGRESS: at sentence #400000, processed 8923833 words, keeping 90847 word types\n",
      "2019-04-12 17:21:27,712 : INFO : PROGRESS: at sentence #410000, processed 9145166 words, keeping 91810 word types\n",
      "2019-04-12 17:21:27,777 : INFO : PROGRESS: at sentence #420000, processed 9366231 words, keeping 92840 word types\n",
      "2019-04-12 17:21:27,831 : INFO : PROGRESS: at sentence #430000, processed 9593763 words, keeping 93858 word types\n",
      "2019-04-12 17:21:27,880 : INFO : PROGRESS: at sentence #440000, processed 9820504 words, keeping 94832 word types\n",
      "2019-04-12 17:21:27,931 : INFO : PROGRESS: at sentence #450000, processed 10044254 words, keeping 95960 word types\n",
      "2019-04-12 17:21:27,982 : INFO : PROGRESS: at sentence #460000, processed 10276995 words, keeping 97012 word types\n",
      "2019-04-12 17:21:28,035 : INFO : PROGRESS: at sentence #470000, processed 10504886 words, keeping 97854 word types\n",
      "2019-04-12 17:21:28,090 : INFO : PROGRESS: at sentence #480000, processed 10725265 words, keeping 98783 word types\n",
      "2019-04-12 17:21:28,138 : INFO : PROGRESS: at sentence #490000, processed 10951923 words, keeping 99787 word types\n",
      "2019-04-12 17:21:28,189 : INFO : PROGRESS: at sentence #500000, processed 11173555 words, keeping 100679 word types\n",
      "2019-04-12 17:21:28,237 : INFO : PROGRESS: at sentence #510000, processed 11398824 words, keeping 101612 word types\n",
      "2019-04-12 17:21:28,293 : INFO : PROGRESS: at sentence #520000, processed 11622159 words, keeping 102511 word types\n",
      "2019-04-12 17:21:28,346 : INFO : PROGRESS: at sentence #530000, processed 11846542 words, keeping 103312 word types\n",
      "2019-04-12 17:21:28,402 : INFO : PROGRESS: at sentence #540000, processed 12071124 words, keeping 104171 word types\n",
      "2019-04-12 17:21:28,462 : INFO : PROGRESS: at sentence #550000, processed 12296653 words, keeping 105037 word types\n",
      "2019-04-12 17:21:28,514 : INFO : PROGRESS: at sentence #560000, processed 12517927 words, keeping 105901 word types\n",
      "2019-04-12 17:21:28,575 : INFO : PROGRESS: at sentence #570000, processed 12746971 words, keeping 106689 word types\n",
      "2019-04-12 17:21:28,626 : INFO : PROGRESS: at sentence #580000, processed 12968451 words, keeping 107565 word types\n",
      "2019-04-12 17:21:28,681 : INFO : PROGRESS: at sentence #590000, processed 13193962 words, keeping 108401 word types\n",
      "2019-04-12 17:21:28,726 : INFO : PROGRESS: at sentence #600000, processed 13416148 words, keeping 109117 word types\n",
      "2019-04-12 17:21:28,771 : INFO : PROGRESS: at sentence #610000, processed 13637150 words, keeping 109989 word types\n",
      "2019-04-12 17:21:28,819 : INFO : PROGRESS: at sentence #620000, processed 13863432 words, keeping 110731 word types\n",
      "2019-04-12 17:21:28,863 : INFO : PROGRESS: at sentence #630000, processed 14087718 words, keeping 111505 word types\n",
      "2019-04-12 17:21:28,911 : INFO : PROGRESS: at sentence #640000, processed 14308493 words, keeping 112308 word types\n",
      "2019-04-12 17:21:28,958 : INFO : PROGRESS: at sentence #650000, processed 14534231 words, keeping 113088 word types\n",
      "2019-04-12 17:21:29,007 : INFO : PROGRESS: at sentence #660000, processed 14757002 words, keeping 113837 word types\n",
      "2019-04-12 17:21:29,056 : INFO : PROGRESS: at sentence #670000, processed 14980380 words, keeping 114533 word types\n",
      "2019-04-12 17:21:29,104 : INFO : PROGRESS: at sentence #680000, processed 15205199 words, keeping 115244 word types\n",
      "2019-04-12 17:21:29,157 : INFO : PROGRESS: at sentence #690000, processed 15427386 words, keeping 116021 word types\n",
      "2019-04-12 17:21:29,212 : INFO : PROGRESS: at sentence #700000, processed 15656076 words, keeping 116833 word types\n",
      "2019-04-12 17:21:29,261 : INFO : PROGRESS: at sentence #710000, processed 15879053 words, keeping 117485 word types\n",
      "2019-04-12 17:21:29,317 : INFO : PROGRESS: at sentence #720000, processed 16104322 words, keeping 118108 word types\n",
      "2019-04-12 17:21:29,373 : INFO : PROGRESS: at sentence #730000, processed 16330689 words, keeping 118838 word types\n",
      "2019-04-12 17:21:29,424 : INFO : PROGRESS: at sentence #740000, processed 16551699 words, keeping 119547 word types\n",
      "2019-04-12 17:21:29,469 : INFO : PROGRESS: at sentence #750000, processed 16770015 words, keeping 120171 word types\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 17:21:29,515 : INFO : PROGRESS: at sentence #760000, processed 16989388 words, keeping 120805 word types\n",
      "2019-04-12 17:21:29,565 : INFO : PROGRESS: at sentence #770000, processed 17216519 words, keeping 121578 word types\n",
      "2019-04-12 17:21:29,612 : INFO : PROGRESS: at sentence #780000, processed 17446646 words, keeping 122275 word types\n",
      "2019-04-12 17:21:29,659 : INFO : PROGRESS: at sentence #790000, processed 17673722 words, keeping 122939 word types\n",
      "2019-04-12 17:21:29,686 : INFO : collected 123376 word types from a corpus of 17796809 raw words and 795538 sentences\n",
      "2019-04-12 17:21:29,687 : INFO : Loading a fresh vocabulary\n",
      "2019-04-12 17:21:29,773 : INFO : min_count=10 retains 34112 unique words (27% of original 123376, drops 89264)\n",
      "2019-04-12 17:21:29,774 : INFO : min_count=10 leaves 17588810 word corpus (98% of original 17796809, drops 207999)\n",
      "2019-04-12 17:21:29,888 : INFO : deleting the raw counts dictionary of 123376 items\n",
      "2019-04-12 17:21:29,891 : INFO : sample=0.001 downsamples 48 most-common words\n",
      "2019-04-12 17:21:29,892 : INFO : downsampling leaves estimated 13139433 word corpus (74.7% of prior 17588810)\n",
      "2019-04-12 17:21:30,045 : INFO : estimated required memory for 34112 words and 300 dimensions: 98924800 bytes\n",
      "2019-04-12 17:21:30,046 : INFO : resetting layer weights\n",
      "2019-04-12 17:21:30,448 : INFO : training model with 4 workers on 34112 vocabulary and 300 features, using sg=0 hs=0 sample=0.001 negative=5 window=10\n",
      "2019-04-12 17:21:31,457 : INFO : EPOCH 1 - PROGRESS: at 4.64% examples, 610027 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:32,468 : INFO : EPOCH 1 - PROGRESS: at 9.51% examples, 618128 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:33,469 : INFO : EPOCH 1 - PROGRESS: at 15.01% examples, 650219 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:34,472 : INFO : EPOCH 1 - PROGRESS: at 19.94% examples, 647584 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:35,479 : INFO : EPOCH 1 - PROGRESS: at 25.05% examples, 651223 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:36,481 : INFO : EPOCH 1 - PROGRESS: at 30.15% examples, 654215 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:37,512 : INFO : EPOCH 1 - PROGRESS: at 35.28% examples, 652557 words/s, in_qsize 8, out_qsize 1\n",
      "2019-04-12 17:21:38,514 : INFO : EPOCH 1 - PROGRESS: at 40.75% examples, 660921 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:39,524 : INFO : EPOCH 1 - PROGRESS: at 45.66% examples, 658806 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:40,533 : INFO : EPOCH 1 - PROGRESS: at 50.80% examples, 660126 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:41,534 : INFO : EPOCH 1 - PROGRESS: at 55.83% examples, 660351 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:42,555 : INFO : EPOCH 1 - PROGRESS: at 61.36% examples, 665441 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:43,567 : INFO : EPOCH 1 - PROGRESS: at 66.63% examples, 667037 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:44,574 : INFO : EPOCH 1 - PROGRESS: at 72.20% examples, 671684 words/s, in_qsize 6, out_qsize 1\n",
      "2019-04-12 17:21:45,577 : INFO : EPOCH 1 - PROGRESS: at 76.72% examples, 666111 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:46,585 : INFO : EPOCH 1 - PROGRESS: at 82.21% examples, 669225 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:47,589 : INFO : EPOCH 1 - PROGRESS: at 87.14% examples, 667805 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:48,590 : INFO : EPOCH 1 - PROGRESS: at 92.39% examples, 669212 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:49,606 : INFO : EPOCH 1 - PROGRESS: at 97.07% examples, 665622 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:50,189 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
      "2019-04-12 17:21:50,205 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2019-04-12 17:21:50,215 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2019-04-12 17:21:50,216 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2019-04-12 17:21:50,217 : INFO : EPOCH - 1 : training on 17796809 raw words (13138610 effective words) took 19.8s, 664793 effective words/s\n",
      "2019-04-12 17:21:51,242 : INFO : EPOCH 2 - PROGRESS: at 4.21% examples, 544295 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:52,261 : INFO : EPOCH 2 - PROGRESS: at 8.95% examples, 575519 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:53,266 : INFO : EPOCH 2 - PROGRESS: at 13.77% examples, 591827 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:54,286 : INFO : EPOCH 2 - PROGRESS: at 18.47% examples, 593679 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:55,296 : INFO : EPOCH 2 - PROGRESS: at 23.76% examples, 612037 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:56,300 : INFO : EPOCH 2 - PROGRESS: at 29.37% examples, 632133 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:57,307 : INFO : EPOCH 2 - PROGRESS: at 34.34% examples, 632740 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:58,332 : INFO : EPOCH 2 - PROGRESS: at 39.20% examples, 631812 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:21:59,345 : INFO : EPOCH 2 - PROGRESS: at 43.42% examples, 623005 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:00,347 : INFO : EPOCH 2 - PROGRESS: at 47.40% examples, 613035 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:01,364 : INFO : EPOCH 2 - PROGRESS: at 51.71% examples, 607969 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:02,378 : INFO : EPOCH 2 - PROGRESS: at 56.06% examples, 604554 words/s, in_qsize 8, out_qsize 0\n",
      "2019-04-12 17:22:03,387 : INFO : EPOCH 2 - PROGRESS: at 61.24% examples, 610851 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:04,393 : INFO : EPOCH 2 - PROGRESS: at 66.63% examples, 617410 words/s, in_qsize 6, out_qsize 0\n",
      "2019-04-12 17:22:05,405 : INFO : EPOCH 2 - PROGRESS: at 71.49% examples, 618491 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:06,415 : INFO : EPOCH 2 - PROGRESS: at 76.44% examples, 619994 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:07,423 : INFO : EPOCH 2 - PROGRESS: at 81.37% examples, 621381 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:08,434 : INFO : EPOCH 2 - PROGRESS: at 84.78% examples, 611577 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:09,479 : INFO : EPOCH 2 - PROGRESS: at 88.02% examples, 600569 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:10,492 : INFO : EPOCH 2 - PROGRESS: at 90.87% examples, 589090 words/s, in_qsize 8, out_qsize 0\n",
      "2019-04-12 17:22:11,492 : INFO : EPOCH 2 - PROGRESS: at 95.76% examples, 591191 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:12,233 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
      "2019-04-12 17:22:12,242 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2019-04-12 17:22:12,252 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2019-04-12 17:22:12,259 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2019-04-12 17:22:12,259 : INFO : EPOCH - 2 : training on 17796809 raw words (13138885 effective words) took 22.0s, 596319 effective words/s\n",
      "2019-04-12 17:22:13,293 : INFO : EPOCH 3 - PROGRESS: at 3.31% examples, 425379 words/s, in_qsize 8, out_qsize 1\n",
      "2019-04-12 17:22:14,303 : INFO : EPOCH 3 - PROGRESS: at 6.80% examples, 438565 words/s, in_qsize 6, out_qsize 1\n",
      "2019-04-12 17:22:15,305 : INFO : EPOCH 3 - PROGRESS: at 10.15% examples, 437047 words/s, in_qsize 6, out_qsize 1\n",
      "2019-04-12 17:22:16,305 : INFO : EPOCH 3 - PROGRESS: at 13.43% examples, 434787 words/s, in_qsize 6, out_qsize 1\n",
      "2019-04-12 17:22:17,321 : INFO : EPOCH 3 - PROGRESS: at 17.46% examples, 450977 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:18,322 : INFO : EPOCH 3 - PROGRESS: at 21.87% examples, 471405 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:19,344 : INFO : EPOCH 3 - PROGRESS: at 26.46% examples, 488727 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:20,360 : INFO : EPOCH 3 - PROGRESS: at 29.99% examples, 484683 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:21,365 : INFO : EPOCH 3 - PROGRESS: at 34.88% examples, 500799 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:22,368 : INFO : EPOCH 3 - PROGRESS: at 39.43% examples, 510182 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:23,379 : INFO : EPOCH 3 - PROGRESS: at 43.70% examples, 514777 words/s, in_qsize 7, out_qsize 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 17:22:24,389 : INFO : EPOCH 3 - PROGRESS: at 48.82% examples, 527831 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:25,393 : INFO : EPOCH 3 - PROGRESS: at 54.27% examples, 541868 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:26,397 : INFO : EPOCH 3 - PROGRESS: at 59.40% examples, 551878 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:27,400 : INFO : EPOCH 3 - PROGRESS: at 64.90% examples, 563022 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:28,415 : INFO : EPOCH 3 - PROGRESS: at 69.37% examples, 564163 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:29,418 : INFO : EPOCH 3 - PROGRESS: at 74.40% examples, 569875 words/s, in_qsize 6, out_qsize 1\n",
      "2019-04-12 17:22:30,431 : INFO : EPOCH 3 - PROGRESS: at 79.28% examples, 573404 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:31,436 : INFO : EPOCH 3 - PROGRESS: at 84.06% examples, 576002 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:32,448 : INFO : EPOCH 3 - PROGRESS: at 88.18% examples, 574143 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:33,461 : INFO : EPOCH 3 - PROGRESS: at 92.79% examples, 575214 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:34,465 : INFO : EPOCH 3 - PROGRESS: at 98.00% examples, 580049 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:34,929 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
      "2019-04-12 17:22:34,930 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2019-04-12 17:22:34,942 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2019-04-12 17:22:34,958 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2019-04-12 17:22:34,959 : INFO : EPOCH - 3 : training on 17796809 raw words (13139831 effective words) took 22.7s, 579097 effective words/s\n",
      "2019-04-12 17:22:35,979 : INFO : EPOCH 4 - PROGRESS: at 4.98% examples, 649048 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:36,987 : INFO : EPOCH 4 - PROGRESS: at 10.26% examples, 664353 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:37,994 : INFO : EPOCH 4 - PROGRESS: at 14.78% examples, 638346 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:39,002 : INFO : EPOCH 4 - PROGRESS: at 19.49% examples, 630406 words/s, in_qsize 6, out_qsize 1\n",
      "2019-04-12 17:22:40,014 : INFO : EPOCH 4 - PROGRESS: at 24.61% examples, 636826 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:41,020 : INFO : EPOCH 4 - PROGRESS: at 29.26% examples, 632059 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:42,022 : INFO : EPOCH 4 - PROGRESS: at 33.78% examples, 624696 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:43,032 : INFO : EPOCH 4 - PROGRESS: at 38.52% examples, 624205 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:44,040 : INFO : EPOCH 4 - PROGRESS: at 43.75% examples, 631173 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:45,040 : INFO : EPOCH 4 - PROGRESS: at 48.50% examples, 630592 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:46,048 : INFO : EPOCH 4 - PROGRESS: at 53.16% examples, 628468 words/s, in_qsize 6, out_qsize 1\n",
      "2019-04-12 17:22:47,052 : INFO : EPOCH 4 - PROGRESS: at 57.74% examples, 626817 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:48,058 : INFO : EPOCH 4 - PROGRESS: at 62.83% examples, 629894 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:49,060 : INFO : EPOCH 4 - PROGRESS: at 68.10% examples, 634288 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:50,061 : INFO : EPOCH 4 - PROGRESS: at 73.18% examples, 636671 words/s, in_qsize 6, out_qsize 1\n",
      "2019-04-12 17:22:51,082 : INFO : EPOCH 4 - PROGRESS: at 78.32% examples, 638445 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:52,087 : INFO : EPOCH 4 - PROGRESS: at 82.99% examples, 636640 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:53,087 : INFO : EPOCH 4 - PROGRESS: at 87.48% examples, 634027 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:54,093 : INFO : EPOCH 4 - PROGRESS: at 92.56% examples, 635774 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:55,096 : INFO : EPOCH 4 - PROGRESS: at 97.23% examples, 634495 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:55,705 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
      "2019-04-12 17:22:55,709 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2019-04-12 17:22:55,724 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2019-04-12 17:22:55,730 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2019-04-12 17:22:55,730 : INFO : EPOCH - 4 : training on 17796809 raw words (13139271 effective words) took 20.8s, 632826 effective words/s\n",
      "2019-04-12 17:22:56,754 : INFO : EPOCH 5 - PROGRESS: at 5.14% examples, 667251 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:57,755 : INFO : EPOCH 5 - PROGRESS: at 10.37% examples, 672364 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:22:58,780 : INFO : EPOCH 5 - PROGRESS: at 15.07% examples, 647143 words/s, in_qsize 6, out_qsize 1\n",
      "2019-04-12 17:22:59,785 : INFO : EPOCH 5 - PROGRESS: at 20.28% examples, 653803 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:23:00,806 : INFO : EPOCH 5 - PROGRESS: at 24.94% examples, 642877 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:23:01,811 : INFO : EPOCH 5 - PROGRESS: at 30.10% examples, 648030 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:23:02,816 : INFO : EPOCH 5 - PROGRESS: at 35.45% examples, 653761 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:23:03,816 : INFO : EPOCH 5 - PROGRESS: at 40.58% examples, 656729 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:23:04,817 : INFO : EPOCH 5 - PROGRESS: at 45.41% examples, 654122 words/s, in_qsize 6, out_qsize 1\n",
      "2019-04-12 17:23:05,817 : INFO : EPOCH 5 - PROGRESS: at 50.11% examples, 651320 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:23:06,826 : INFO : EPOCH 5 - PROGRESS: at 55.21% examples, 652471 words/s, in_qsize 6, out_qsize 1\n",
      "2019-04-12 17:23:07,843 : INFO : EPOCH 5 - PROGRESS: at 59.96% examples, 650022 words/s, in_qsize 7, out_qsize 1\n",
      "2019-04-12 17:23:08,846 : INFO : EPOCH 5 - PROGRESS: at 65.18% examples, 652665 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:23:09,865 : INFO : EPOCH 5 - PROGRESS: at 70.05% examples, 650940 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:23:10,874 : INFO : EPOCH 5 - PROGRESS: at 74.68% examples, 648040 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:23:11,890 : INFO : EPOCH 5 - PROGRESS: at 79.75% examples, 648326 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:23:12,904 : INFO : EPOCH 5 - PROGRESS: at 84.45% examples, 646046 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:23:13,912 : INFO : EPOCH 5 - PROGRESS: at 89.58% examples, 647540 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:23:14,920 : INFO : EPOCH 5 - PROGRESS: at 94.04% examples, 643859 words/s, in_qsize 7, out_qsize 0\n",
      "2019-04-12 17:23:15,921 : INFO : EPOCH 5 - PROGRESS: at 99.05% examples, 644781 words/s, in_qsize 7, out_qsize 2\n",
      "2019-04-12 17:23:16,083 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
      "2019-04-12 17:23:16,108 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2019-04-12 17:23:16,115 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2019-04-12 17:23:16,118 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2019-04-12 17:23:16,119 : INFO : EPOCH - 5 : training on 17796809 raw words (13138364 effective words) took 20.4s, 644629 effective words/s\n",
      "2019-04-12 17:23:16,121 : INFO : training on a 88984045 raw words (65694961 effective words) took 105.7s, 621692 effective words/s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5min 54s, sys: 1.11 s, total: 5min 55s\n",
      "Wall time: 1min 50s\n"
     ]
    }
   ],
   "source": [
    "print(\"Training model...\")\n",
    "\n",
    "%time model_en = word2vec.Word2Vec(sentences, workers=4, size=300, min_count=10, window=10, sample=1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Смотрим, сколько в модели слов."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "34112\n"
     ]
    }
   ],
   "source": [
    "print(len(model_en.wv.vocab))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Попробуем оценить модель вручную, порешав примеры. Несколько дано ниже, попробуйте придумать свои."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 17:23:47,048 : INFO : precomputing L2-norms of word weight vectors\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('actress', 0.7272005677223206)]\n",
      "[('men', 0.6438334584236145)]\n",
      "[('europe', 0.7301366329193115), ('germany', 0.7224695682525635), ('australia', 0.7180286645889282)]\n",
      "novel\n"
     ]
    }
   ],
   "source": [
    "print(model_en.wv.most_similar(positive=[\"woman\", \"actor\"], negative=[\"man\"], topn=1))\n",
    "print(model_en.wv.most_similar(positive=[\"dogs\", \"man\"], negative=[\"dog\"], topn=1))\n",
    "\n",
    "print(model_en.wv.most_similar(\"usa\", topn=3))\n",
    "\n",
    "print(model_en.wv.doesnt_match(\"comedy thriller western novel\".split()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Как дообучить существующую модель\n",
    "\n",
    "При тренировке модели \"с нуля\" веса инициализируются случайно, однако можно использовать для инициализации векторов веса из предобученной модели, таким образом как бы дообучая ее.\n",
    "\n",
    "Сначала посмотрим близость какой-нибудь пары слов в имеющейс модели, чтобы потом сравнить результат с дообученной."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3478375567627835"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_en.wv.similarity('lion', 'unicorn')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В качестве дополнительных данных для обучения возьмем английский текст «Алисы в Зазеркалье»."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['through', 'the', 'looking-glass', 'by', 'lewis', 'carroll', 'chapter', 'i', 'looking-glass', 'house', 'one', 'thing', 'was', 'certain', 'that', 'the', 'white', 'kitten', 'had', 'had', 'nothing', 'to', 'do', 'with', 'it', '', 'it', 'was', 'the', 'black', 'kitten’s', 'fault', 'entirely'], ['for', 'the', 'white', 'kitten', 'had', 'been', 'having', 'its', 'face', 'washed', 'by', 'the', 'old', 'cat', 'for', 'the', 'last', 'quarter', 'of', 'an', 'hour', 'and', 'bearing', 'it', 'pretty', 'well', 'considering', 'so', 'you', 'see', 'that', 'it', 'couldn’t', 'have', 'had', 'any', 'hand', 'in', 'the', 'mischief']]\n"
     ]
    }
   ],
   "source": [
    "with open(\"./data/w2v/train/alice.txt\", 'r', encoding='utf-8') as f:\n",
    "    text = f.read()\n",
    "\n",
    "text = re.sub('\\n', ' ', text)\n",
    "sents = sent_tokenize(text)\n",
    "\n",
    "punct = '!\"#$%&()*+,-./:;<=>?@[\\]^_`{|}~„“«»†*—/\\-‘’'\n",
    "clean_sents = []\n",
    "\n",
    "for sent in sents:\n",
    "    s = [w.lower().strip(punct) for w in sent.split()]\n",
    "    clean_sents.append(s)\n",
    "    \n",
    "print(clean_sents[:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Чтобы дообучить модель, надо сначала ее сохранить, а потом загрузить. Все параметры тренировки (размер вектора, мин. частота слова и т.п.) будут взяты из загруженной модели, т.е. задать их заново нельзя.\n",
    "\n",
    "**NB!** Дообучить можно только полную модель, а `KeyedVectors` — нельзя. Поэтому сохранять модель нужно в соотвествующем формате. Подробнее о разнице [вот тут](https://radimrehurek.com/gensim/models/keyedvectors.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 17:24:04,239 : INFO : saving Word2Vec object under ./data/w2v/movie_reviews/movie_reviews.model, separately None\n",
      "2019-04-12 17:24:04,241 : INFO : not storing attribute vectors_norm\n",
      "2019-04-12 17:24:04,242 : INFO : not storing attribute cum_table\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving model...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 17:24:05,390 : INFO : saved ./data/w2v/movie_reviews/movie_reviews.model\n"
     ]
    }
   ],
   "source": [
    "model_path = \"./data/w2v/movie_reviews/movie_reviews.model\"\n",
    "\n",
    "print(\"Saving model...\")\n",
    "model_en.save(model_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 17:24:18,864 : INFO : loading Word2Vec object from ./data/w2v/movie_reviews/movie_reviews.model\n",
      "2019-04-12 17:24:19,527 : INFO : loading wv recursively from ./data/w2v/movie_reviews/movie_reviews.model.wv.* with mmap=None\n",
      "2019-04-12 17:24:19,527 : INFO : setting ignored attribute vectors_norm to None\n",
      "2019-04-12 17:24:19,528 : INFO : loading vocabulary recursively from ./data/w2v/movie_reviews/movie_reviews.model.vocabulary.* with mmap=None\n",
      "2019-04-12 17:24:19,528 : INFO : loading trainables recursively from ./data/w2v/movie_reviews/movie_reviews.model.trainables.* with mmap=None\n",
      "2019-04-12 17:24:19,529 : INFO : setting ignored attribute cum_table to None\n",
      "2019-04-12 17:24:19,529 : INFO : loaded ./data/w2v/movie_reviews/movie_reviews.model\n",
      "2019-04-12 17:24:19,615 : INFO : collecting all words and their counts\n",
      "2019-04-12 17:24:19,616 : INFO : PROGRESS: at sentence #0, processed 0 words, keeping 0 word types\n",
      "2019-04-12 17:24:19,623 : INFO : collected 2956 word types from a corpus of 30045 raw words and 1222 sentences\n",
      "2019-04-12 17:24:19,623 : INFO : Updating model with new vocabulary\n",
      "2019-04-12 17:24:19,625 : INFO : New added 410 unique words (12% of original 3366) and increased the count of 410 pre-existing words (12% of original 3366)\n",
      "2019-04-12 17:24:19,631 : INFO : deleting the raw counts dictionary of 2956 items\n",
      "2019-04-12 17:24:19,632 : INFO : sample=0.001 downsamples 158 most-common words\n",
      "2019-04-12 17:24:19,633 : INFO : downsampling leaves estimated 29258 word corpus (120.6% of prior 24256)\n",
      "2019-04-12 17:24:19,713 : INFO : estimated required memory for 820 words and 300 dimensions: 2378000 bytes\n",
      "2019-04-12 17:24:19,714 : INFO : updating layer weights\n",
      "2019-04-12 17:24:19,749 : WARNING : Effective 'alpha' higher than previous training cycles\n",
      "2019-04-12 17:24:19,750 : INFO : training model with 4 workers on 34152 vocabulary and 300 features, using sg=0 hs=0 sample=0.001 negative=5 window=10\n",
      "2019-04-12 17:24:19,780 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
      "2019-04-12 17:24:19,799 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2019-04-12 17:24:19,812 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2019-04-12 17:24:19,818 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2019-04-12 17:24:19,820 : INFO : EPOCH - 1 : training on 30045 raw words (19542 effective words) took 0.1s, 308966 effective words/s\n",
      "2019-04-12 17:24:19,864 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
      "2019-04-12 17:24:19,880 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2019-04-12 17:24:19,900 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2019-04-12 17:24:19,901 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2019-04-12 17:24:19,902 : INFO : EPOCH - 2 : training on 30045 raw words (19573 effective words) took 0.1s, 286688 effective words/s\n",
      "2019-04-12 17:24:19,926 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
      "2019-04-12 17:24:19,963 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2019-04-12 17:24:19,966 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2019-04-12 17:24:19,967 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2019-04-12 17:24:19,967 : INFO : EPOCH - 3 : training on 30045 raw words (19441 effective words) took 0.1s, 319685 effective words/s\n",
      "2019-04-12 17:24:19,991 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
      "2019-04-12 17:24:20,022 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2019-04-12 17:24:20,032 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2019-04-12 17:24:20,043 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2019-04-12 17:24:20,047 : INFO : EPOCH - 4 : training on 30045 raw words (19549 effective words) took 0.1s, 298508 effective words/s\n",
      "2019-04-12 17:24:20,080 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
      "2019-04-12 17:24:20,100 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2019-04-12 17:24:20,128 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2019-04-12 17:24:20,133 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2019-04-12 17:24:20,141 : INFO : EPOCH - 5 : training on 30045 raw words (19512 effective words) took 0.1s, 230073 effective words/s\n",
      "2019-04-12 17:24:20,144 : INFO : training on a 150225 raw words (97617 effective words) took 0.4s, 247742 effective words/s\n",
      "2019-04-12 17:24:20,147 : WARNING : under 10 jobs per worker: consider setting a smaller `batch_words' for smoother alpha decay\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(97617, 150225)"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = word2vec.Word2Vec.load(model_path)\n",
    "\n",
    "model.build_vocab(clean_sents, update=True)\n",
    "model.train(clean_sents, total_examples=model.corpus_count, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Лев и единорог стали гораздо ближе друг к другу!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.48919319531985217"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.similarity('lion', 'unicorn')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Можно нормализовать вектора, тогда модель будет занимать меньше RAM. Однако после этого её нельзя дотренировывать. Здесь используется L2-нормализация: вектора нормализуются так, что если сложить квадраты всех элементов вектора, в сумме получится 1. \n",
    "\n",
    "Кроме того, сохраним не полные вектора, а `KeyedVectors`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 17:27:58,544 : INFO : precomputing L2-norms of word weight vectors\n",
      "2019-04-12 17:27:58,886 : INFO : storing 34112x300 projection weights into ./data/w2v/movie_reviews/movies_alice.bin\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving model...\n"
     ]
    }
   ],
   "source": [
    "model.init_sims(replace=True)\n",
    "model_path = \"./data/w2v/movie_reviews/movies_alice.bin\"\n",
    "\n",
    "print(\"Saving model...\")\n",
    "model_en.wv.save_word2vec_format(model_path, binary=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Оценка\n",
    "\n",
    "Это, конечно, хорошо, но как понять, какая модель лучше? Или вот, например, я сделал свою модель, а как понять, насколько она хорошая?\n",
    "\n",
    "Для этого существуют специальные датасеты для оценки качества дистрибутивных моделей. Основных два: один измеряет точность решения задач на аналогии (про Россию и пельмени), а второй используется для оценки коэффициента семантической близости. \n",
    "\n",
    "### Word Similarity\n",
    "\n",
    "Этот метод заключается в том, чтобы оценить, насколько представления о семантической близости слов в модели соотносятся с \"представлениями\" людей.\n",
    "\n",
    "| слово 1    | слово 2    | близость | \n",
    "|------------|------------|----------|\n",
    "| кошка      | собака     | 0.7      |  \n",
    "| чашка      | кружка     | 0.9      |       \n",
    "\n",
    "Для каждой пары слов из заранее заданного датасета мы можем посчитать косинусное расстояние, и получить список таких значений близости. При этом у нас уже есть список значений близостей, сделанный людьми. Мы можем сравнить эти два списка и понять, насколько они похожи (например, посчитав корреляцию). Эта мера схожести должна говорить о том, насколько модель хорошо моделирует расстояния о слова.\n",
    "\n",
    "### Аналогии\n",
    "\n",
    "Другая популярная задача для \"внутренней\" оценки называется задачей поиска аналогий. Как мы уже разбирали выше, с помощью простых арифметических операций мы можем модифицировать значение слова. Если заранее собрать набор слов-модификаторов, а также слов, которые мы хотим получить в результаты модификации, то на основе подсчёта количества \"попаданий\" в желаемое слово мы можем оценить, насколько хорошо работает модель.\n",
    "\n",
    "В качестве слов-модификатор мы можем использовать семантические аналогии. Скажем, если у нас есть некоторое отношение \"страна-столица\", то для оценки модели мы можем использовать пары наподобие \"Россия-Москва\", \"Норвегия-Осло\", и т.д. Датасет будет выглядеть следующм образом:\n",
    "\n",
    "| слово 1    | слово 2    | отношение     | \n",
    "|------------|------------|---------------|\n",
    "| Россия     | Москва     | страна-столица|  \n",
    "| Норвегия   | Осло       | страна-столица|\n",
    "\n",
    "Рассматривая случайные две пары из этого набора, мы хотим, имея триплет (Россия, Москва, Норвегия) хотим получить слово \"Осло\", т.е. найти такое слово, которое будет находиться в том же отношении со словом \"Норвегия\", как \"Россия\" находится с Москвой. \n",
    "\n",
    "Датасеты для русского языка можно скачать на странице с моделями на RusVectores. Посчитаем качество нашей модели НКРЯ на датасете про аналогии:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 17:56:40,962 : INFO : capital-common-countries: 19.0% (58/306)\n",
      "2019-04-12 17:56:43,182 : INFO : capital-world: 10.1% (52/515)\n",
      "2019-04-12 17:56:43,727 : INFO : currency: 4.6% (6/130)\n",
      "2019-04-12 17:56:45,008 : INFO : family: 71.2% (218/306)\n",
      "2019-04-12 17:56:48,419 : INFO : gram1-Aective-to-adverb: 18.7% (152/812)\n",
      "2019-04-12 17:56:49,954 : INFO : gram2-opposite: 32.1% (122/380)\n",
      "2019-04-12 17:56:53,728 : INFO : gram6-nationality-Aective: 32.3% (293/907)\n",
      "2019-04-12 17:56:53,729 : INFO : total: 26.8% (901/3356)\n"
     ]
    }
   ],
   "source": [
    "res = model_ru.accuracy('./data/w2v/evaluation/ru_analogy_tagged.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('МАЛЬЧИК_S', 'ДЕВОЧКА_S', 'ДЕД_S', 'БАБКА_S'), ('МАЛЬЧИК_S', 'ДЕВОЧКА_S', 'КОРОЛЬ_S', 'КОРОЛЕВА_S'), ('МАЛЬЧИК_S', 'ДЕВОЧКА_S', 'ПРИНЦ_S', 'ПРИНЦЕССА_S'), ('МАЛЬЧИК_S', 'ДЕВОЧКА_S', 'ОТЧИМ_S', 'МАЧЕХА_S'), ('МАЛЬЧИК_S', 'ДЕВОЧКА_S', 'ПАСЫНОК_S', 'ПАДЧЕРИЦА_S'), ('БРАТ_S', 'СЕСТРА_S', 'ДЕД_S', 'БАБКА_S'), ('БРАТ_S', 'СЕСТРА_S', 'ОТЧИМ_S', 'МАЧЕХА_S'), ('БРАТ_S', 'СЕСТРА_S', 'ПАСЫНОК_S', 'ПАДЧЕРИЦА_S'), ('ПАПА_S', 'МАМА_S', 'ДЕД_S', 'БАБКА_S'), ('ПАПА_S', 'МАМА_S', 'ОТЧИМ_S', 'МАЧЕХА_S')]\n"
     ]
    }
   ],
   "source": [
    "print(res[4]['incorrect'][:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vecto\n",
    "\n",
    "Также для \"внутренней\" (intrinsic) оценки, т.е. оценки в отрыве от конкретной задачи, можно использовать фреймворк [vecto](https://vecto.readthedocs.io/en/latest/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 15:54:10,679 : INFO : font search path ['/home/ancatmara/anaconda3/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf', '/home/ancatmara/anaconda3/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/afm', '/home/ancatmara/anaconda3/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/pdfcorefonts']\n",
      "2019-04-12 15:54:25,215 : INFO : generated new fontManager\n"
     ]
    }
   ],
   "source": [
    "from vecto.benchmarks.analogy import Analogy\n",
    "from vecto.benchmarks.similarity import Similarity\n",
    "import vecto.embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vecto использует свои собственные структуры для работы с векторными моделями, поэтому нужно загрузить модель в память ещё раз. Для простоты загрузим только одну из рассмотренных моделей, Word2Vec."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 17:28:10,799 : INFO : Detected VSM in the w2v original binary format\n"
     ]
    }
   ],
   "source": [
    "embeddings = vecto.embeddings.load_from_dir(\"./data/w2v/movie_reviews/\")\n",
    "embeddings.cache_normalized_copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В Vecto можно также как и в Gensim получать вектора для слов или искать наиболее похожие слова из словаря модели:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['movie', 1.0],\n",
       " ['film', 0.9272411],\n",
       " ['flick', 0.85202676],\n",
       " ['movies', 0.7854172],\n",
       " ['it', 0.7843939],\n",
       " ['picture', 0.7730175],\n",
       " ['sequel', 0.76599324],\n",
       " ['documentary', 0.75504],\n",
       " ['storyline', 0.7395749],\n",
       " ['show', 0.73571754]]"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embeddings.get_most_similar_words('movie')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'experiment_setup': {'cnt_found_pairs_total': 2,\n",
       "   'cnt_pairs_total': 2,\n",
       "   'embeddings': {'_class': 'vecto.embeddings.legacy_w2v.ModelW2V',\n",
       "    'normalized': True},\n",
       "   'category': 'default',\n",
       "   'dataset': 'ws',\n",
       "   'method': 'cosine_distance',\n",
       "   'language': 'en',\n",
       "   'description': 'TEST FILE',\n",
       "   'version': '-',\n",
       "   'measurement': 'spearman',\n",
       "   'task': 'similarity',\n",
       "   'timestamp': '2019-04-12T17:46:45.086542',\n",
       "   'default_measurement': 'spearman'},\n",
       "  'result': {'spearman': -1},\n",
       "  'details': []}]"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity = Similarity()\n",
    "similarity.get_result(embeddings, path_dataset='./data/w2v/evaluation/similarity')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 17:45:29,938 : INFO : processing ./data/w2v/evaluation/analogy/category1/subcategory_a.txt\n",
      "N/A% (0 of 2) |                          | Elapsed Time: 0:00:00 ETA:  --:--:--2019-04-12 17:45:30,003 : INFO : processing ./data/w2v/evaluation/analogy/category2/subcategory_b.txt\n",
      "N/A% (0 of 4) |                          | Elapsed Time: 0:00:00 ETA:  --:--:--"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'details': [{'question verbose': \"What is to apple as ['quick'] is to fast\",\n",
       "    'b': 'apple',\n",
       "    'expected answer': ['banana'],\n",
       "    'predictions': [{'score': 0.728181004524231,\n",
       "      'answer': 'lebowski',\n",
       "      'hit': False},\n",
       "     {'score': 0.726725697517395, 'answer': 'midget', 'hit': False},\n",
       "     {'score': 0.7247903347015381, 'answer': 'booty', 'hit': False},\n",
       "     {'score': 0.7216457724571228, 'answer': 'coat', 'hit': False},\n",
       "     {'score': 0.7199649214744568, 'answer': 'sauce', 'hit': False},\n",
       "     {'score': 0.7191541194915771, 'answer': 'horn', 'hit': False}],\n",
       "    'set_exclude': ['fast', 'apple', 'quick'],\n",
       "    'rank': 816,\n",
       "    'landing_b': True,\n",
       "    'landing_b_prime': False,\n",
       "    'landing_a': False,\n",
       "    'landing_a_prime': False,\n",
       "    'similarity predicted to b_prime cosine': 0.6400811225175858,\n",
       "    'similarity a to a_prime cosine': 0.7612074613571167,\n",
       "    'similarity a_prime to b_prime cosine': 0.5334567055106163,\n",
       "    'similarity b to b_prime cosine': 0.670642763376236,\n",
       "    'similarity a to b_prime cosine': 0.5013529043644667,\n",
       "    'distance a to a_prime euclidean': 0.9773280620574951,\n",
       "    'distance a_prime to b_prime euclidean': 1.366079568862915,\n",
       "    'distance b to b_prime euclidean': 1.147793173789978,\n",
       "    'distance a to b_prime euclidean': 1.4122990369796753,\n",
       "    'crowdedness of b_prime': [0.8023819923400879,\n",
       "     0.8016062378883362,\n",
       "     0.7981831431388855,\n",
       "     0.7968431711196899,\n",
       "     0.7955393195152283,\n",
       "     0.7927921414375305,\n",
       "     0.7924220561981201,\n",
       "     0.7920680046081543,\n",
       "     0.7914565801620483,\n",
       "     0.7905268669128418],\n",
       "    'b in neighbourhood of b_prime': 8154,\n",
       "    'b_prime in neighbourhood of b': 486},\n",
       "   {'question verbose': \"What is to fast as ['banana'] is to apple\",\n",
       "    'b': 'fast',\n",
       "    'expected answer': ['quick'],\n",
       "    'predictions': [{'score': 0.7522187829017639,\n",
       "      'answer': 'leisurely',\n",
       "      'hit': False},\n",
       "     {'score': 0.7285716533660889, 'answer': 'brisk', 'hit': False},\n",
       "     {'score': 0.7226551175117493, 'answer': 'slow', 'hit': False},\n",
       "     {'score': 0.7142457962036133, 'answer': 'snail', 'hit': False},\n",
       "     {'score': 0.706622302532196, 'answer': 'snails', 'hit': False},\n",
       "     {'score': 0.7058804035186768, 'answer': 'frenetic', 'hit': False}],\n",
       "    'set_exclude': ['apple', 'fast', 'banana'],\n",
       "    'rank': 23,\n",
       "    'landing_b': True,\n",
       "    'landing_b_prime': False,\n",
       "    'landing_a': False,\n",
       "    'landing_a_prime': False,\n",
       "    'similarity predicted to b_prime cosine': 0.6776228547096252,\n",
       "    'similarity a to a_prime cosine': 0.670642763376236,\n",
       "    'similarity a_prime to b_prime cosine': 0.5334567055106163,\n",
       "    'similarity b to b_prime cosine': 0.7612074613571167,\n",
       "    'similarity a to b_prime cosine': 0.5206461418420076,\n",
       "    'distance a to a_prime euclidean': 1.147793173789978,\n",
       "    'distance a_prime to b_prime euclidean': 1.366079568862915,\n",
       "    'distance b to b_prime euclidean': 0.9773280620574951,\n",
       "    'distance a to b_prime euclidean': 1.3847076892852783,\n",
       "    'crowdedness of b_prime': [0.7612074613571167,\n",
       "     0.7461464405059814,\n",
       "     0.7251861095428467,\n",
       "     0.7244290113449097,\n",
       "     0.7233452796936035,\n",
       "     0.721161961555481,\n",
       "     0.7205275893211365,\n",
       "     0.7205029129981995,\n",
       "     0.720267653465271,\n",
       "     0.7182656526565552],\n",
       "    'b in neighbourhood of b_prime': 1,\n",
       "    'b_prime in neighbourhood of b': 4}],\n",
       "  'result': {'cnt_questions_correct': 0,\n",
       "   'cnt_questions_total': 2,\n",
       "   'accuracy': 0.0},\n",
       "  'experiment_setup': {'dataset': {'_base_path': './data/w2v/evaluation/analogy',\n",
       "    'class': 'dataset',\n",
       "    'task': 'analogy',\n",
       "    'language': ['english'],\n",
       "    'name': 'TestAnalogy',\n",
       "    'description': 'Test Analogy Set',\n",
       "    'domain': 'general',\n",
       "    'date': '2016',\n",
       "    'source': 'original',\n",
       "    'project_page': 'http://vecto.space/',\n",
       "    'version': '3.0',\n",
       "    'size': 'small',\n",
       "    'cite': {'title': \"Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn't\",\n",
       "     'author': 'Gladkova, Anna and Drozd, Aleksandr and Matsuoka, Satoshi',\n",
       "     'doi': '10.18653/v1/N16-2002',\n",
       "     'url': 'https://www.aclweb.org/anthology/N/N16/N16-2002.pdf',\n",
       "     'booktitle': 'Proceedings of the NAACL-HLT SRW',\n",
       "     'publisher': 'ACL',\n",
       "     'year': 2016,\n",
       "     'pages': '47-54',\n",
       "     'type': 'inproceedings',\n",
       "     'id': 'GladkovaDrozdEtAl_2016'},\n",
       "    '_class': 'vecto.data.base.Dataset'},\n",
       "   'embeddings': {'_class': 'vecto.embeddings.legacy_w2v.ModelW2V',\n",
       "    'normalized': True},\n",
       "   'category': 'category1',\n",
       "   'subcategory': 'subcategory_a.txt',\n",
       "   'task': 'word_analogy',\n",
       "   'default_measurement': 'accuracy',\n",
       "   'method': '3CosAdd',\n",
       "   'uuid': '3fb95144-6d37-43f0-bf86-7dbd1970ed7f',\n",
       "   'timestamp': '2019-04-12T17:45:29.939314'}},\n",
       " {'details': [{'question verbose': \"What is to apple as ['quick'] is to fast\",\n",
       "    'b': 'apple',\n",
       "    'expected answer': ['banana'],\n",
       "    'predictions': [{'score': 0.728181004524231,\n",
       "      'answer': 'lebowski',\n",
       "      'hit': False},\n",
       "     {'score': 0.726725697517395, 'answer': 'midget', 'hit': False},\n",
       "     {'score': 0.7247903347015381, 'answer': 'booty', 'hit': False},\n",
       "     {'score': 0.7216457724571228, 'answer': 'coat', 'hit': False},\n",
       "     {'score': 0.7199649214744568, 'answer': 'sauce', 'hit': False},\n",
       "     {'score': 0.7191541194915771, 'answer': 'horn', 'hit': False}],\n",
       "    'set_exclude': ['fast', 'apple', 'quick'],\n",
       "    'rank': 816,\n",
       "    'landing_b': True,\n",
       "    'landing_b_prime': False,\n",
       "    'landing_a': False,\n",
       "    'landing_a_prime': False,\n",
       "    'similarity predicted to b_prime cosine': 0.6400811225175858,\n",
       "    'similarity a to a_prime cosine': 0.7612074613571167,\n",
       "    'similarity a_prime to b_prime cosine': 0.5334567055106163,\n",
       "    'similarity b to b_prime cosine': 0.670642763376236,\n",
       "    'similarity a to b_prime cosine': 0.5013529043644667,\n",
       "    'distance a to a_prime euclidean': 0.9773280620574951,\n",
       "    'distance a_prime to b_prime euclidean': 1.366079568862915,\n",
       "    'distance b to b_prime euclidean': 1.147793173789978,\n",
       "    'distance a to b_prime euclidean': 1.4122990369796753,\n",
       "    'crowdedness of b_prime': [0.8023819923400879,\n",
       "     0.8016062378883362,\n",
       "     0.7981831431388855,\n",
       "     0.7968431711196899,\n",
       "     0.7955393195152283,\n",
       "     0.7927921414375305,\n",
       "     0.7924220561981201,\n",
       "     0.7920680046081543,\n",
       "     0.7914565801620483,\n",
       "     0.7905268669128418],\n",
       "    'b in neighbourhood of b_prime': 8154,\n",
       "    'b_prime in neighbourhood of b': 486},\n",
       "   {'rank': -1,\n",
       "    'question verbose': \"What is to apple as ['banana_missing'] is to apple\"},\n",
       "   {'rank': -1,\n",
       "    'question verbose': \"What is to apple as ['banana'] is to apple_missing\"},\n",
       "   {'question verbose': \"What is to fast as ['banana'] is to apple\",\n",
       "    'b': 'fast',\n",
       "    'expected answer': ['quick'],\n",
       "    'predictions': [{'score': 0.7522187829017639,\n",
       "      'answer': 'leisurely',\n",
       "      'hit': False},\n",
       "     {'score': 0.7285716533660889, 'answer': 'brisk', 'hit': False},\n",
       "     {'score': 0.7226551175117493, 'answer': 'slow', 'hit': False},\n",
       "     {'score': 0.7142457962036133, 'answer': 'snail', 'hit': False},\n",
       "     {'score': 0.706622302532196, 'answer': 'snails', 'hit': False},\n",
       "     {'score': 0.7058804035186768, 'answer': 'frenetic', 'hit': False}],\n",
       "    'set_exclude': ['apple', 'fast', 'banana'],\n",
       "    'rank': 23,\n",
       "    'landing_b': True,\n",
       "    'landing_b_prime': False,\n",
       "    'landing_a': False,\n",
       "    'landing_a_prime': False,\n",
       "    'similarity predicted to b_prime cosine': 0.6776228547096252,\n",
       "    'similarity a to a_prime cosine': 0.670642763376236,\n",
       "    'similarity a_prime to b_prime cosine': 0.5334567055106163,\n",
       "    'similarity b to b_prime cosine': 0.7612074613571167,\n",
       "    'similarity a to b_prime cosine': 0.5206461418420076,\n",
       "    'distance a to a_prime euclidean': 1.147793173789978,\n",
       "    'distance a_prime to b_prime euclidean': 1.366079568862915,\n",
       "    'distance b to b_prime euclidean': 0.9773280620574951,\n",
       "    'distance a to b_prime euclidean': 1.3847076892852783,\n",
       "    'crowdedness of b_prime': [0.7612074613571167,\n",
       "     0.7461464405059814,\n",
       "     0.7251861095428467,\n",
       "     0.7244290113449097,\n",
       "     0.7233452796936035,\n",
       "     0.721161961555481,\n",
       "     0.7205275893211365,\n",
       "     0.7205029129981995,\n",
       "     0.720267653465271,\n",
       "     0.7182656526565552],\n",
       "    'b in neighbourhood of b_prime': 1,\n",
       "    'b_prime in neighbourhood of b': 4},\n",
       "   {'rank': -1,\n",
       "    'question verbose': \"What is to fast as ['banana_missing'] is to apple\"},\n",
       "   {'rank': -1,\n",
       "    'question verbose': \"What is to fast as ['banana'] is to apple_missing\"},\n",
       "   {'rank': -1,\n",
       "    'question verbose': \"What is to apple as ['banana'] is to apple\"},\n",
       "   {'rank': -1,\n",
       "    'question verbose': \"What is to apple as ['quick'] is to fast\"},\n",
       "   {'rank': -1,\n",
       "    'question verbose': \"What is to apple as ['banana'] is to apple_missing\"},\n",
       "   {'rank': -1,\n",
       "    'question verbose': \"What is to apple_missing as ['banana'] is to apple\"},\n",
       "   {'rank': -1,\n",
       "    'question verbose': \"What is to apple_missing as ['quick'] is to fast\"},\n",
       "   {'rank': -1,\n",
       "    'question verbose': \"What is to apple_missing as ['banana_missing'] is to apple\"}],\n",
       "  'result': {'cnt_questions_correct': 0,\n",
       "   'cnt_questions_total': 14,\n",
       "   'accuracy': 0.0},\n",
       "  'experiment_setup': {'dataset': {'_base_path': './data/w2v/evaluation/analogy',\n",
       "    'class': 'dataset',\n",
       "    'task': 'analogy',\n",
       "    'language': ['english'],\n",
       "    'name': 'TestAnalogy',\n",
       "    'description': 'Test Analogy Set',\n",
       "    'domain': 'general',\n",
       "    'date': '2016',\n",
       "    'source': 'original',\n",
       "    'project_page': 'http://vecto.space/',\n",
       "    'version': '3.0',\n",
       "    'size': 'small',\n",
       "    'cite': {'title': \"Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn't\",\n",
       "     'author': 'Gladkova, Anna and Drozd, Aleksandr and Matsuoka, Satoshi',\n",
       "     'doi': '10.18653/v1/N16-2002',\n",
       "     'url': 'https://www.aclweb.org/anthology/N/N16/N16-2002.pdf',\n",
       "     'booktitle': 'Proceedings of the NAACL-HLT SRW',\n",
       "     'publisher': 'ACL',\n",
       "     'year': 2016,\n",
       "     'pages': '47-54',\n",
       "     'type': 'inproceedings',\n",
       "     'id': 'GladkovaDrozdEtAl_2016'},\n",
       "    '_class': 'vecto.data.base.Dataset'},\n",
       "   'embeddings': {'_class': 'vecto.embeddings.legacy_w2v.ModelW2V',\n",
       "    'normalized': True},\n",
       "   'category': 'category2',\n",
       "   'subcategory': 'subcategory_b.txt',\n",
       "   'task': 'word_analogy',\n",
       "   'default_measurement': 'accuracy',\n",
       "   'method': '3CosAdd',\n",
       "   'uuid': '550d5ae0-62a5-41c3-b83a-0e0ed4a0bff3',\n",
       "   'timestamp': '2019-04-12T17:45:30.005169'}}]"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy = Analogy()\n",
    "analogy.get_result(embeddings, path_dataset='./data/w2v/evaluation/analogy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Кроме этих двух задач, существует множество других. Бенчмарки для них можно найти на сайте http://vecto.space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Что еще бывает?\n",
    "\n",
    "## doc2vec\n",
    "\n",
    "Это word2vec с дополнительной меткой id документа, которая представляет собой отдельный вектор. Вектора id документа и всех слов в нем конкатенируются / усредняются, и таким образом получается вектор документа.\n",
    "\n",
    "Также реализован в `gensim`: `gensim.models.doc2vec`.\n",
    "\n",
    "\n",
    "![img](img/w2v_4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FastText\n",
    "\n",
    "FastText использует не только эмбеддинги слов, но и эмбеддинги n-грамов. В корпусе каждое слово автоматически представляется в виде набора символьных n-грамм. Скажем, если мы установим n=3, то вектор для слова \"where\" будет представлен суммой векторов следующих триграм: \"<wh\", \"whe\", \"her\", \"ere\", \"re>\" (где \"<\" и \">\" символы, обозначающие начало и конец слова). Благодаря этому мы можем также получать вектора для слов, отсутствуюших в словаре, а также эффективно работать с текстами, содержащими ошибки и опечатки.\n",
    "\n",
    "* [Статья](https://aclweb.org/anthology/Q17-1010)\n",
    "* [Сайт](https://fasttext.cc/)\n",
    "* [Тьюториал](https://fasttext.cc/docs/en/support.html)\n",
    "* [Вектора для 157 языков](https://fasttext.cc/docs/en/crawl-vectors.html)\n",
    "* [Вектора, обученные на википедии](https://fasttext.cc/docs/en/pretrained-vectors.html) (отдельно для 294 разных языков)\n",
    "* [Репозиторий](https://github.com/facebookresearch/fasttext)\n",
    "\n",
    "Есть библиотека `fasttext` для питона (с готовыми моделями можно работать и через `gensim`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GloVe\n",
    "\n",
    "Эмбеддинги от стэнфордской NLP-лаборатории. Похожи на word2vec, но в отличие от него GloVe строит матрицу совместной встречаемости слов и учится на ней, т.е. это гибридный метод. [Вот тут](https://www.quora.com/How-is-GloVe-different-from-word2vec) очень хорошо написано об их отличиях.\n",
    "\n",
    "* [Статья](https://nlp.stanford.edu/pubs/glove.pdf)\n",
    "* [Сайт](https://nlp.stanford.edu/projects/glove/) (ссылки на модели прямо на главной)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BPE\n",
    "\n",
    "Модель, основанная на кодировании BPE (Byte Pair Encoding). Это один из способов представления текста, в котором мы используем в качестве токена не слова или символы, а наборы символов в зависимости от глубины кодирования. Предобученные BPE-эмбеддинги для разных языков можно свободно скачивать.\n",
    "\n",
    "* [Статья №1](https://arxiv.org/pdf/1508.07909.pdf) \n",
    "* [Статья №2](https://aclweb.org/anthology/L18-1473)\n",
    "* [Сайт](https://nlp.h-its.org/bpemb/)\n",
    "* [Репозиторий](https://github.com/bheinzerling/bpemb) библиотеки `bpemb` от авторов второй статьи.\n",
    "\n",
    "Скажем, мы хотим закодировать aaabdaaabac. В этой последовательности сочетание \"aa\" встречается наиболее часто. Мы можем \"слить\" эти буквы вместе и рассматривать их как отдельный символ. И так далее итеративно.  Чем больше размер словаря, тем длиннее будут входящие в него единицы: условно, при объеме словаря = 100 текст разобьется на элементы из 2 символов, а при объеме = 500 — на элементы из 5 символов. Выглядит примерно вот так:\n",
    "\n",
    "![bpe1](img/bpe1.jpg)\n",
    "\n",
    "![bpe2](img/bpe2.jpg)\n",
    "\n",
    "А вот 2D-визуализация для BPE-эмбеддингов для русского.\n",
    "\n",
    "![bpe](https://nlp.h-its.org/bpemb/ru/ru.wiki.bpe.emb.vs100000.d100.png)\n",
    "\n",
    "\n",
    "А еще можно посмотреть ее [интерактивную визуализацию](http://projector.tensorflow.org/?config=https://nlp.h-its.org/bpemb/ru/projector.config.json) в TensorFlow Projector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-12 18:32:22,383 : INFO : loading projection weights from /home/ancatmara/.cache/bpemb/en/en.wiki.bpe.vs10000.d50.w2v.bin\n",
      "2019-04-12 18:32:22,500 : INFO : loaded (10000, 50) matrix from /home/ancatmara/.cache/bpemb/en/en.wiki.bpe.vs10000.d50.w2v.bin\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['▁through', '▁the', '▁looking', '-', 'gl', 'ass', '▁by', '▁lewis', '▁car', 'roll', '▁chapter', '▁i', '▁looking', '-', 'gl', 'ass', '▁house', '▁one', '▁thing', '▁was', '▁certain', '▁that', '▁the', '▁white', '▁kit', 'ten', '▁had', '▁had', '▁nothing', '▁to', '▁do', '▁with', '▁it', '▁it', '▁was', '▁the', '▁black', '▁kit', 'ten', '’', 's', '▁fault', '▁entirely']\n"
     ]
    }
   ],
   "source": [
    "from bpemb import BPEmb\n",
    "\n",
    "bpemb_en = BPEmb(lang='en', dim=50)\n",
    "encoded_text = bpemb_en.encode(' '.join(clean_sents[0]))\n",
    "print(encoded_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Задание\n",
    "\n",
    "Скачайте [газетный корпус](https://www.dropbox.com/s/7mzom3urqgzyfwa/major_papers.rar?dl=0) (новости из крупных газет на русском языке в xml) или [корпус региональной прессы](https://www.dropbox.com/s/d2gl5atnp7nmgfv/regional_papers.rar?dl=0) (тоже на русском языке, в plain text) и обучите на нем модель word2vec. Можете использовать как оригинальные тексты, так и лемматизированный корпус. \n",
    "\n",
    "**NB!** Нужны данные из папок, метаданные в csv-таблицах не понадобятся.\n",
    "\n",
    "1. Найдите по 5 ближайших слов к словам \"город\", \"деревня\", \"спорт\", \"бизнес\", \"Россия\", \"происшествие\", \"река\", \"озеро\", \"море\", \"горы\", \"депутат\", \"север\", \"юг\", \"кавказ\", \"сибирь\", \"газпром\". Учтите, что слова может не быть в модели.\n",
    "2. Посчитайте семантическую близкость слов \"театр\" и \"кино\", \"Владивосток\" и \"Москва\", \"церковь\" и \"государство\", \"культура\" и \"отдых\", \"преступление\" и \"наказание\".\n",
    "3. Решите примеры:\n",
    "        * москва + екатеринбург - собянин\n",
    "        * спартак - москва + санкт-петербург\n",
    "        * иркутск - байкал + сочи\n",
    "        * татарстан - татарский + бурятия\n",
    "        * чай - лимон + кофе\n",
    "        * авиакомпания - аэрофлот + ржд\n",
    "4. Найдите лишнее, попробуйте интерпретировать результаты\n",
    "        * магазин, супермаркет, рынок, тц\n",
    "        * теннис, хоккей, футбол, дзюдо\n",
    "        * кошка, собака, попугай, кролик\n",
    "        * коми, дагестан, башкирия, камчатка\n",
    "\n",
    "Сохраните модель, она вам еще понадобится!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}