toloka_flow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align=\"center\">Side-by-Side</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# План\n",
    "1. Пререквизиты\n",
    "2. Pairwise vs Pointwise vs Listwise подход\n",
    "3. Простой SBS: сравнение двух экранов\n",
    "4. SBS с таргетированием\n",
    "5. SBS-ранжирование нескольких картинок\n",
    "6. SBS с поиском.\n",
    "7. (*) Вебвизор"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Пререквизиты:\n",
    "\n",
    "1. Необходимо [зарегистрироваться](https://toloka.yandex.ru/for-requesters/) в Толоке как заказчик\n",
    "2. [Получить OAuth-токен](https://toloka.yandex.ru/requester/profile/integration) по ссылке. Подробнее в [документации](https://yandex.ru/dev/toloka/doc/concepts/access-docpage/) \n",
    "3. Разобраться со одним из способов хранения файлов для заданий\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Куда складывать файлы?\n",
    "1. Использовать произвольное постоянное S3–хранилище файлов. Например, [Yandex.Cloud.Object.Storage](https://cloud.yandex.ru/docs/storage/quickstart?utm_source=console&utm_medium=empty-page&utm_campaign=storage)<br>\n",
    "<span style=\"color:green\">Подходит, если планируете ставить разметки на большой поток</span>\n",
    "\n",
    "\n",
    "2. Использовать встроенный в Толоку механизм [Яндекс.Диска](https://yandex.ru/support/toloka-requester/concepts/prepare-data.html)<br>\n",
    "<span style=\"color:green\">Подходит для разовых запусков</span>\n",
    "\n",
    "\n",
    "3. Использовать произвольное хранилище, которое позволяет получить URL на картинку (графический объект) вида: https://sbs.s3.yandex.net/39a307e3f4859c96f37161b3ab00aa5daa99858fbd6df1b70f53fa9a649ea467.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Замечание: весь написанный код должен воспроизвестись, если указать свой токен заказчика. Тем не менее, что-то может устареть и потребовать доработок.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import scipy.stats\n",
    "import time\n",
    "import pandas as pd\n",
    "import json\n",
    "from IPython.core.display import display, HTML\n",
    "from toloka_api import *\n",
    "\n",
    "from pathlib import Path \n",
    "home = str(Path.home())\n",
    "\n",
    "#Класс для работы с Толокой и функции для агрегации результатов\n",
    "from toloka_api import TolokaProcessingApi\n",
    "from toloka_results_aggregation import prepare_toloka_task_for_2_screens, prepare_toloka_task_for_many_screens\n",
    "\n",
    "#Считываем токен Толоки из файлика.\n",
    "toloka_token = open(home + \"/.tokens/toloka\").read().strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Класс для работы с API толоки"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "toloka_api = TolokaProcessingApi(toloka_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pointwise vs Pairwise vs Listwise?\n",
    "\n",
    "* Поточечный подход (pointwise)\n",
    "* Попарный подход (pairwise) \n",
    "* Списочный подход (listwise)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Подробнее можно прочитать в источниках:\n",
    "* https://medium.com/@nikhilbd/pointwise-vs-pairwise-vs-listwise-learning-to-rank-80a8fe8fadfd\n",
    "* http://www.machinelearning.ru/wiki/images/8/89/Voron-ML-Ranking-slides.pdf\n",
    "\n",
    "\n",
    "<img src=\"https://sbs.s3.yandex.net/d7e825321372f31fd8761224fa572d379e1f218d3ffd512a17e588257a293478.jpg\" width=\"400\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Примеры решаемых задач с помощью Side-by-Side\n",
    "\n",
    "1. Сравнение поисковых выдач\n",
    "2. Сравнение интерфейсных решений\n",
    "3. Выбор иконок\n",
    "4. Ранжирование картинок (например, фонов)\n",
    "5. Сравнение видео"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Настраиваем шаблон (проект и пул) для сравнения картинок\n",
    "\n",
    "В Толоке есть стандартный шаблон для парных сравнений:\n",
    "\n",
    "**Шаг 1.** Создаем новый проект.  https://toloka.yandex.ru/requester/templates\n",
    "\n",
    "**Шаг 2.** Выбираем шаблон *парное сравнение изображений*\n",
    "![](https://sbs.s3.yandex.net/16546bf0f93cfb664c4ceb2da7048fecfe734ed2833deb1986949cdc591d0322.png)\n",
    "\n",
    "\n",
    "**Шаг 3.** Заполняем название, описание, инструкцию\n",
    "![](https://sbs.s3.yandex.net/c9689c2359f2990bb432cc9ff4d4aa186ddd83addb605c8d509a299a7d424c61.png)\n",
    "\n",
    "**Шаг 4.** Смотрим предпросмотр – как это все будет выглядеть для __респондентов__\n",
    "![](https://sbs.s3.yandex.net/579e97f314fc945c323d0a9bee9784feea45b6d439992bed44e0da6f2b20834b.png)\n",
    "\n",
    "\n",
    "**Шаг 5.** Чего не хватает?\n",
    "1. У нас не \"дубли/не дубли\", а выбор Left/Right\n",
    "2. Описания комментария/задания вверху страницы\n",
    "3. Текстового ответа: чтобы респондент смог ответить на вопрос комментарием\n",
    "\n",
    "**Шаг 6.** Создадим проект и добавим того, чего не хватает\n",
    "![](https://sbs.s3.yandex.net/81441b92db3d1216881dcfbca2f17249cdb244cf29c34570df659088cdc3627a.png)\n",
    "\n",
    "**Шаг 7.** Получается вот такой опросник для респондента\n",
    "![](https://sbs.s3.yandex.net/45cf35af758525beff86d3370f5967aa95a298f22553b016d269386b90171ee8.png)\n",
    "\n",
    "**Шаг 8.** Заводим \"эталонный пул\" с настройками\n",
    "![](https://sbs.s3.yandex.net/aeb5998ba0eaca6d087ff1e7b9eb7ff3ad1fdb9fde11b8102e4a5aa54c6c5088.png)\n",
    "\n",
    "__Замечания:__ \n",
    "1. Точно такой проект можно создать с помощью API, что и будет сделано далее\n",
    "2. Шаблон можно совершенствовать и полировать до бесконечности"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Заведем такой проект через [API Толоки](https://yandex.ru/dev/toloka/doc/concepts/create-prj-docpage/).\n",
    "\n",
    "Параметры проекта из примера сложены в файл `project_templates/pictures_comparison_project_template.json`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New project was created. New project id:  35047\n",
      "https://toloka.yandex.ru/requester/project/35047\n"
     ]
    }
   ],
   "source": [
    "project_template = json.loads(open(\"project_templates/pictures_comparison_project_template.json\").read())\n",
    "project_id = toloka_api.create_toloka_project(project_template)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Заведем шаблонный пул, с которого впоследствии будем клонировать с помощью [API Толоки](https://yandex.ru/dev/toloka/doc/concepts/create-pool-docpage/) \n",
    "\n",
    "\n",
    "Парамеры шаблонного пула лежат в файле `project_templates/pool_params_template.json`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New pool was created. New pool id:  12965550\n"
     ]
    }
   ],
   "source": [
    "pool_template = json.loads(open(\"project_templates/pool_params_template.json\").read())\n",
    "template_pool_id = toloka_api.create_toloka_pool(project_id, pool_template)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Хотим решить, какая плитка больше нравится людям"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "  <tr>\n",
    "    <td><img src=\"https://sbs.s3.yandex.net/44aebb8c77a29efa25e8a5d1ac4eba9877b84efa1147b4b003b2a8380273cc73.png\" width=\"250\"></td>\n",
    "      <td>  </td>\n",
    "     <td><img src=\"https://sbs.s3.yandex.net/93656faf648029e3bf4bf18421b3dc08d85fe4bdd2683498cbf497f0487e89db.png\" width=\"250\"></td>\n",
    "  </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Подготовим задания для сравнения двух картинок"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://sbs.s3.yandex.net/44aebb8c77a29efa25e8a5d1ac4eba9877b84efa1147b4b003b2a8380273cc73.png\n",
      "https://sbs.s3.yandex.net/93656faf648029e3bf4bf18421b3dc08d85fe4bdd2683498cbf497f0487e89db.png\n"
     ]
    }
   ],
   "source": [
    "first_url = \"https://sbs.s3.yandex.net/44aebb8c77a29efa25e8a5d1ac4eba9877b84efa1147b4b003b2a8380273cc73.png\"\n",
    "second_url = \"https://sbs.s3.yandex.net/93656faf648029e3bf4bf18421b3dc08d85fe4bdd2683498cbf497f0487e89db.png\"\n",
    "\n",
    "print(first_url)\n",
    "print(second_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pool was cloned. New pool was created. New pool id:  12965570\n",
      "https://toloka.yandex.ru/requester/pool/12965570\n"
     ]
    }
   ],
   "source": [
    "two_pictures_pool_id = toloka_api.clone_toloka_pool(template_pool_id)\n",
    "tasks = prepare_toloka_task_for_2_screens(first_url, second_url, \n",
    "                                         50, two_pictures_pool_id)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Далее добавим их в наш пул. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "toloka_api.upload_toloka_tasks(tasks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Получили новый пул с залитыми заданиями. Теперь запустим его"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://toloka.yandex.ru/requester/pool/12965570\n",
      "Pool 12965570 was started\n"
     ]
    }
   ],
   "source": [
    "toloka_api.print_toloka_pool_link(two_pictures_pool_id)\n",
    "toloka_api.run_toloka_pool(two_pictures_pool_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Пул уехал в обработку респонентами. \n",
    "Можно пойти пить кофе, пока пул обрабатывается\n",
    "\n",
    "Когда пул разметился, нужно выгрузить результаты и обработать их"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = toloka_api.get_pool_completed_tasks(two_pictures_pool_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>image_left</th>\n",
       "      <th>image_right</th>\n",
       "      <th>result</th>\n",
       "      <th>comment</th>\n",
       "      <th>winner_url</th>\n",
       "      <th>loser_url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>LEFT</td>\n",
       "      <td>на правом много красного\\nне прикольно</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>RIGHT</td>\n",
       "      <td>белый текст на красном фоне более читаемый\\n</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>RIGHT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>RIGHT</td>\n",
       "      <td>Нечеткое изображение</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>LEFT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             question  \\\n",
       "0  Какой вариант вам больше нравится?   \n",
       "1  Какой вариант вам больше нравится?   \n",
       "2  Какой вариант вам больше нравится?   \n",
       "3  Какой вариант вам больше нравится?   \n",
       "4  Какой вариант вам больше нравится?   \n",
       "\n",
       "                                          image_left  \\\n",
       "0  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "1  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "2  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "3  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "4  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "\n",
       "                                         image_right result  \\\n",
       "0  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   LEFT   \n",
       "1  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  RIGHT   \n",
       "2  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  RIGHT   \n",
       "3  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  RIGHT   \n",
       "4  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   LEFT   \n",
       "\n",
       "                                        comment  \\\n",
       "0        на правом много красного\\nне прикольно   \n",
       "1  белый текст на красном фоне более читаемый\\n   \n",
       "2                                           NaN   \n",
       "3                          Нечеткое изображение   \n",
       "4                                           NaN   \n",
       "\n",
       "                                          winner_url  \\\n",
       "0  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "1  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "2  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "3  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "4  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "\n",
       "                                           loser_url  \n",
       "0  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  \n",
       "1  https://sbs.s3.yandex.net/93656faf648029e3bf4b...  \n",
       "2  https://sbs.s3.yandex.net/93656faf648029e3bf4b...  \n",
       "3  https://sbs.s3.yandex.net/93656faf648029e3bf4b...  \n",
       "4  https://sbs.s3.yandex.net/93656faf648029e3bf4b...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Число людей, у которых:\n",
    "* слева first_url\n",
    "* выбрали first_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d[(d['image_left'] == first_url) & (d['result'] == \"LEFT\")].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Число людей, у которых:\n",
    "* слева second_url\n",
    "* выбрали second_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d[(d['image_left'] == second_url) & (d['result'] == \"LEFT\")].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Число людей, у которых:\n",
    "* слева first_url\n",
    "* выбрали second_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d[(d['image_left'] == first_url) & (d['result'] == \"RIGHT\")].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Число людей, у которых:\n",
    "* слева second_url\n",
    "* выбрали first_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d[(d['image_left'] == second_url) & (d['result'] == \"RIGHT\")].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для удобства сделаем колоночки winner_url (выбранный URL) и loser_url (проигравший URL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>image_left</th>\n",
       "      <th>image_right</th>\n",
       "      <th>result</th>\n",
       "      <th>comment</th>\n",
       "      <th>winner_url</th>\n",
       "      <th>loser_url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>LEFT</td>\n",
       "      <td>на правом много красного\\nне прикольно</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>RIGHT</td>\n",
       "      <td>белый текст на красном фоне более читаемый\\n</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>RIGHT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>RIGHT</td>\n",
       "      <td>Нечеткое изображение</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>LEFT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             question  \\\n",
       "0  Какой вариант вам больше нравится?   \n",
       "1  Какой вариант вам больше нравится?   \n",
       "2  Какой вариант вам больше нравится?   \n",
       "3  Какой вариант вам больше нравится?   \n",
       "4  Какой вариант вам больше нравится?   \n",
       "\n",
       "                                          image_left  \\\n",
       "0  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "1  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "2  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "3  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "4  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "\n",
       "                                         image_right result  \\\n",
       "0  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   LEFT   \n",
       "1  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  RIGHT   \n",
       "2  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  RIGHT   \n",
       "3  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  RIGHT   \n",
       "4  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   LEFT   \n",
       "\n",
       "                                        comment  \\\n",
       "0        на правом много красного\\nне прикольно   \n",
       "1  белый текст на красном фоне более читаемый\\n   \n",
       "2                                           NaN   \n",
       "3                          Нечеткое изображение   \n",
       "4                                           NaN   \n",
       "\n",
       "                                          winner_url  \\\n",
       "0  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "1  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "2  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "3  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "4  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "\n",
       "                                           loser_url  \n",
       "0  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  \n",
       "1  https://sbs.s3.yandex.net/93656faf648029e3bf4b...  \n",
       "2  https://sbs.s3.yandex.net/93656faf648029e3bf4b...  \n",
       "3  https://sbs.s3.yandex.net/93656faf648029e3bf4b...  \n",
       "4  https://sbs.s3.yandex.net/93656faf648029e3bf4b...  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "https://sbs.s3.yandex.net/44aebb8c77a29efa25e8a5d1ac4eba9877b84efa1147b4b003b2a8380273cc73.png    35\n",
       "https://sbs.s3.yandex.net/93656faf648029e3bf4bf18421b3dc08d85fe4bdd2683498cbf497f0487e89db.png    15\n",
       "Name: winner_url, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d['winner_url'].value_counts() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Надо посчитать стат.значимость. Сделать это можно с помощью биномиального теста.\n",
    "\n",
    "https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.stats.binom_test.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.006600447966810915"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scipy.stats.binom_test(35, 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Можно сделать вывод о том, что различие значимое.\n",
    "\n",
    "Еще могут быть полезными комментарии"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>image_left</th>\n",
       "      <th>image_right</th>\n",
       "      <th>result</th>\n",
       "      <th>comment</th>\n",
       "      <th>winner_url</th>\n",
       "      <th>loser_url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>RIGHT</td>\n",
       "      <td>белый текст на красном фоне более читаемый\\n</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>RIGHT</td>\n",
       "      <td>Нечеткое изображение</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>LEFT</td>\n",
       "      <td>Нравиться больше так как красный фон</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>RIGHT</td>\n",
       "      <td>единый стиль</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>RIGHT</td>\n",
       "      <td>более контрасно</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              question  \\\n",
       "1   Какой вариант вам больше нравится?   \n",
       "3   Какой вариант вам больше нравится?   \n",
       "6   Какой вариант вам больше нравится?   \n",
       "9   Какой вариант вам больше нравится?   \n",
       "10  Какой вариант вам больше нравится?   \n",
       "\n",
       "                                           image_left  \\\n",
       "1   https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "3   https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "6   https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "9   https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "10  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "\n",
       "                                          image_right result  \\\n",
       "1   https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  RIGHT   \n",
       "3   https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  RIGHT   \n",
       "6   https://sbs.s3.yandex.net/93656faf648029e3bf4b...   LEFT   \n",
       "9   https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  RIGHT   \n",
       "10  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  RIGHT   \n",
       "\n",
       "                                         comment  \\\n",
       "1   белый текст на красном фоне более читаемый\\n   \n",
       "3                           Нечеткое изображение   \n",
       "6           Нравиться больше так как красный фон   \n",
       "9                                   единый стиль   \n",
       "10                              более контрасно    \n",
       "\n",
       "                                           winner_url  \\\n",
       "1   https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "3   https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "6   https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "9   https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "10  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "\n",
       "                                            loser_url  \n",
       "1   https://sbs.s3.yandex.net/93656faf648029e3bf4b...  \n",
       "3   https://sbs.s3.yandex.net/93656faf648029e3bf4b...  \n",
       "6   https://sbs.s3.yandex.net/93656faf648029e3bf4b...  \n",
       "9   https://sbs.s3.yandex.net/93656faf648029e3bf4b...  \n",
       "10  https://sbs.s3.yandex.net/93656faf648029e3bf4b...  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d[(d['winner_url'] == first_url)].dropna().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>image_left</th>\n",
       "      <th>image_right</th>\n",
       "      <th>result</th>\n",
       "      <th>comment</th>\n",
       "      <th>winner_url</th>\n",
       "      <th>loser_url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>LEFT</td>\n",
       "      <td>на правом много красного\\nне прикольно</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>RIGHT</td>\n",
       "      <td>не так ярко красно</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>LEFT</td>\n",
       "      <td>Более привлекательно выглядит</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>LEFT</td>\n",
       "      <td>красный цвет збирает на ебя все внимание а бел...</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Какой вариант вам больше нравится?</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "      <td>LEFT</td>\n",
       "      <td>Не такая яркая, не режет глаза.</td>\n",
       "      <td>https://sbs.s3.yandex.net/93656faf648029e3bf4b...</td>\n",
       "      <td>https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              question  \\\n",
       "0   Какой вариант вам больше нравится?   \n",
       "24  Какой вариант вам больше нравится?   \n",
       "25  Какой вариант вам больше нравится?   \n",
       "30  Какой вариант вам больше нравится?   \n",
       "31  Какой вариант вам больше нравится?   \n",
       "\n",
       "                                           image_left  \\\n",
       "0   https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "24  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   \n",
       "25  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "30  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "31  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "\n",
       "                                          image_right result  \\\n",
       "0   https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   LEFT   \n",
       "24  https://sbs.s3.yandex.net/93656faf648029e3bf4b...  RIGHT   \n",
       "25  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   LEFT   \n",
       "30  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   LEFT   \n",
       "31  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...   LEFT   \n",
       "\n",
       "                                              comment  \\\n",
       "0              на правом много красного\\nне прикольно   \n",
       "24                                 не так ярко красно   \n",
       "25                      Более привлекательно выглядит   \n",
       "30  красный цвет збирает на ебя все внимание а бел...   \n",
       "31                    Не такая яркая, не режет глаза.   \n",
       "\n",
       "                                           winner_url  \\\n",
       "0   https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "24  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "25  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "30  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "31  https://sbs.s3.yandex.net/93656faf648029e3bf4b...   \n",
       "\n",
       "                                            loser_url  \n",
       "0   https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  \n",
       "24  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  \n",
       "25  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  \n",
       "30  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  \n",
       "31  https://sbs.s3.yandex.net/44aebb8c77a29efa25e8...  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d[(d['winner_url'] == second_url)].dropna().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Задача: отсортировать множество объектов: например, фонов"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Есть несколько картинок:\n",
    "\n",
    "\n",
    "<table>\n",
    "  <tr>\n",
    "    <td><img src=\"https://sbs.s3.yandex.net/86b63f8f20396ec95cbbd0b1b9c5516dc294e6142e31b96c849101f7de36966b.png\" width=\"450\"></td>\n",
    "      <td>  </td>\n",
    "     <td><img src=\"https://sbs.s3.yandex.net/d6c07660358fa23737c6b8845f7e1320904bbdb665cda19550d898848378a96f.png\" width=\"450\"></td>\n",
    "  </tr>\n",
    "    \n",
    "  <tr>\n",
    "    <td><img src=\"https://sbs.s3.yandex.net/002fbd80aeebb07dbb68a363a986c06b4f3fbb4bbe6644822bb038856d686aaf.png\" width=\"450\"></td>\n",
    "      <td>  </td>\n",
    "     <td><img src=\"https://sbs.s3.yandex.net/2c9936faee69f50a7303ac5a75e856a6c99b52dd5b854c9d0c1cd09cd3c73a8e.png\" width=\"450\"></td>\n",
    "  </tr>\n",
    "    \n",
    "  <tr>\n",
    "     <td><img src=\"https://sbs.s3.yandex.net/5826d04de9a5f4cb01722bf8cf63b4d35e2142cf9d071e21f0604897f6ff61ae.png\" width=\"450\"></td>\n",
    "      <td>  </td>\n",
    "     <td><img src=\"https://sbs.s3.yandex.net/be1bffed70943f42c2f607292e76dfb96361a33f37824bca22223d29092402b6.png\" width=\"450\"></td>\n",
    "  </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "И не можем решить, что лучше использовать в качестве фона рабочего стола.\n",
    "\n",
    "Проведем множество парных сравнений и опросим...респондентов!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 466,
   "metadata": {},
   "outputs": [],
   "source": [
    "pictures_urls = [\"https://sbs.s3.yandex.net/86b63f8f20396ec95cbbd0b1b9c5516dc294e6142e31b96c849101f7de36966b.png\",\n",
    "                 \"https://sbs.s3.yandex.net/d6c07660358fa23737c6b8845f7e1320904bbdb665cda19550d898848378a96f.png\",\n",
    "                 \"https://sbs.s3.yandex.net/002fbd80aeebb07dbb68a363a986c06b4f3fbb4bbe6644822bb038856d686aaf.png\",\n",
    "                 \"https://sbs.s3.yandex.net/2c9936faee69f50a7303ac5a75e856a6c99b52dd5b854c9d0c1cd09cd3c73a8e.png\",\n",
    "                 \"https://sbs.s3.yandex.net/5826d04de9a5f4cb01722bf8cf63b4d35e2142cf9d071e21f0604897f6ff61ae.png\" ,\n",
    "                 \"https://sbs.s3.yandex.net/be1bffed70943f42c2f607292e76dfb96361a33f37824bca22223d29092402b6.png\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Заводим очередной пул"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 476,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pool was cloned. New pool was created. New pool id:  8809616\n",
      "https://toloka.yandex.ru/requester/pool/8809616\n"
     ]
    }
   ],
   "source": [
    "pictures_pool = clone_toloka_pool(template_pool_id)\n",
    "tasks = prepare_toloka_task_for_many_screens(pictures_urls, 5, \n",
    "                                          pictures_pool, \"Какой фон для рабочего стола вам больше нравится?\")\n",
    "\n",
    "upload_toloka_tasks(tasks)\n",
    "\n",
    "print_toloka_pool_link(pictures_pool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Проверяем, запускаем и снова идем пить кофе!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 477,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pool 8809616 was started\n"
     ]
    }
   ],
   "source": [
    "run_toloka_pool(pictures_pool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 536,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/5826d04de9a5f4cb01722bf8cf63b4d35e2142cf9d071e21f0604897f6ff61ae.png\"><img src=\"https://sbs.s3.yandex.net/5826d04de9a5f4cb01722bf8cf63b4d35e2142cf9d071e21f0604897f6ff61ae.png\" width=300px></a></td>\n",
       "      <td>0.280000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/002fbd80aeebb07dbb68a363a986c06b4f3fbb4bbe6644822bb038856d686aaf.png\"><img src=\"https://sbs.s3.yandex.net/002fbd80aeebb07dbb68a363a986c06b4f3fbb4bbe6644822bb038856d686aaf.png\" width=300px></a></td>\n",
       "      <td>0.226667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/d6c07660358fa23737c6b8845f7e1320904bbdb665cda19550d898848378a96f.png\"><img src=\"https://sbs.s3.yandex.net/d6c07660358fa23737c6b8845f7e1320904bbdb665cda19550d898848378a96f.png\" width=300px></a></td>\n",
       "      <td>0.226667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/2c9936faee69f50a7303ac5a75e856a6c99b52dd5b854c9d0c1cd09cd3c73a8e.png\"><img src=\"https://sbs.s3.yandex.net/2c9936faee69f50a7303ac5a75e856a6c99b52dd5b854c9d0c1cd09cd3c73a8e.png\" width=300px></a></td>\n",
       "      <td>0.133333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/be1bffed70943f42c2f607292e76dfb96361a33f37824bca22223d29092402b6.png\"><img src=\"https://sbs.s3.yandex.net/be1bffed70943f42c2f607292e76dfb96361a33f37824bca22223d29092402b6.png\" width=300px></a></td>\n",
       "      <td>0.106667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/86b63f8f20396ec95cbbd0b1b9c5516dc294e6142e31b96c849101f7de36966b.png\"><img src=\"https://sbs.s3.yandex.net/86b63f8f20396ec95cbbd0b1b9c5516dc294e6142e31b96c849101f7de36966b.png\" width=300px></a></td>\n",
       "      <td>0.026667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pictures_results = pd.DataFrame(get_pool_completed_tasks(pictures_pool)[\"winner_url\"].value_counts()).reset_index()\n",
    "pictures_results[\"winner_url\"] = pictures_results[\"winner_url\"] / sum(pictures_results[\"winner_url\"])\n",
    "pictures_results.columns = [\"url\", \"rate\"]\n",
    "pictures_results[\"url\"] = pictures_results[\"url\"].apply(\n",
    "                        lambda x: '<a href=\"{}\"><img src=\"{}\" width=300px></a>'.format(x, x))\n",
    "\n",
    "pd.options.display.max_colwidth = -1\n",
    "html_all = pictures_results.to_html(escape=False, )\n",
    "display(HTML(html_all))\n",
    "pd.options.display.max_colwidth = 50\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Пример с таргетированием"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Часто нужно как-то отсечь аудиторию. \n",
    "\n",
    "Рассмотрим пример: запустим эксперимент сначала на аудитории из Москвы, а потом на аудитории из Санкт-Петербурга"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "  <tr>\n",
    "    <td><img src=\"https://sbs.s3.yandex.net/af24437833aef6d80f05db322f441cd0573f3c886ccc692d68d410180de481b6.png\" width=\"450\"></td>\n",
    "      <td>  </td>\n",
    "     <td><img src=\"https://sbs.s3.yandex.net/35358c7261be07d066c4eb06451a83ed3049721b8a8ae8e50d1b4a0d909e7aec.png\" width=\"450\"></td>\n",
    "  </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Готовим 2 пула: \"Московский\" и \"Питерский\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 422,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_url = \"https://sbs.s3.yandex.net/af24437833aef6d80f05db322f441cd0573f3c886ccc692d68d410180de481b6.png\" \n",
    "second_url = \"https://sbs.s3.yandex.net/35358c7261be07d066c4eb06451a83ed3049721b8a8ae8e50d1b4a0d909e7aec.png\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 441,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pool was cloned. New pool was created. New pool id:  8808652\n"
     ]
    }
   ],
   "source": [
    "msk_pool = clone_toloka_pool(template_pool_id)\n",
    "tasks = prepare_toloka_task_for_2_screens(first_url, second_url, \n",
    "                                         50, msk_pool)\n",
    "\n",
    "upload_toloka_tasks(tasks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 442,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pool was cloned. New pool was created. New pool id:  8808656\n"
     ]
    }
   ],
   "source": [
    "spb_pool = clone_toloka_pool(template_pool_id)\n",
    "tasks = prepare_toloka_task_for_2_screens(first_url, second_url, \n",
    "                                         50, spb_pool)\n",
    "\n",
    "upload_toloka_tasks(tasks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 443,
   "metadata": {},
   "outputs": [],
   "source": [
    "msk_pool_params=get_pool_params(msk_pool)\n",
    "msk_target_filter = {'or': [{'category': 'profile',\n",
    "      'key': 'city',\n",
    "      'operator': 'IN',\n",
    "      'value': 213}]}\n",
    "\n",
    "user_filter = msk_pool_params['filter']['and']\n",
    "user_filter.append(msk_target_filter)\n",
    "msk_pool_params['filter']['and'] = user_filter\n",
    "\n",
    "update_pool_params(msk_pool, msk_pool_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 451,
   "metadata": {},
   "outputs": [],
   "source": [
    "spb_pool_params=get_pool_params(spb_pool)\n",
    "spb_target_filter = {'or': [{'category': 'profile',\n",
    "      'key': 'city',\n",
    "      'operator': 'IN',\n",
    "      'value': 2}]}\n",
    "\n",
    "user_filter = spb_pool_params['filter']['and']\n",
    "user_filter.append(spb_target_filter)\n",
    "spb_pool_params['filter']['and'] = user_filter\n",
    "\n",
    "update_pool_params(spb_pool, spb_pool_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 449,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Проверим в интерфейсе, что настройки правильно доехали"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 447,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://toloka.yandex.ru/requester/pool/8808652\n",
      "https://toloka.yandex.ru/requester/pool/8808656\n"
     ]
    }
   ],
   "source": [
    "print_toloka_pool_link(msk_pool)\n",
    "print_toloka_pool_link(spb_pool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "  <tr>\n",
    "    <td><img src=\"https://sbs.s3.yandex.net/c1cf9018843395a9bb6330116b34ca57d56831b0b17a6302c8e0d12a94061843.png\" width=\"450\"></td>\n",
    "      <td>  </td>\n",
    "     <td><img src=\"https://sbs.s3.yandex.net/ec79411d628a9380ff961eb94389326eee5433925cd387314f160a7d128cfed9.png\" width=\"450\"></td>\n",
    "  </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Стартуем оба пула и идем пить кофе"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 452,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pool 8808652 was started\n",
      "Pool 8808656 was started\n"
     ]
    }
   ],
   "source": [
    "run_toloka_pool(msk_pool)\n",
    "run_toloka_pool(spb_pool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Выводы:\n",
    "\n",
    "1. Аудитория из Санкт-Петербурга чаще выбираем вариант с \"Шаверма\". \n",
    "\n",
    "2. Аудитория из Москвы чаще выбираем вариант с \"Шаурма\". "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 548,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/af24437833aef6d80f05db322f441cd0573f3c886ccc692d68d410180de481b6.png\"><img src=\"https://sbs.s3.yandex.net/af24437833aef6d80f05db322f441cd0573f3c886ccc692d68d410180de481b6.png\" width=300px></a></td>\n",
       "      <td>0.78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/35358c7261be07d066c4eb06451a83ed3049721b8a8ae8e50d1b4a0d909e7aec.png\"><img src=\"https://sbs.s3.yandex.net/35358c7261be07d066c4eb06451a83ed3049721b8a8ae8e50d1b4a0d909e7aec.png\" width=300px></a></td>\n",
       "      <td>0.22</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "msk_results = pd.DataFrame(get_pool_completed_tasks(msk_pool)[\"winner_url\"].value_counts()).reset_index()\n",
    "msk_results[\"winner_url\"] = msk_results[\"winner_url\"] / sum(msk_results[\"winner_url\"])\n",
    "\n",
    "msk_results.columns = [\"url\", \"rate\"]\n",
    "msk_results[\"url\"] = msk_results[\"url\"].apply(\n",
    "                        lambda x: '<a href=\"{}\"><img src=\"{}\" width=300px></a>'.format(x, x))\n",
    "\n",
    "pd.options.display.max_colwidth = -1\n",
    "html_all = msk_results.to_html(escape=False, )\n",
    "display(HTML(html_all))\n",
    "pd.options.display.max_colwidth = 50\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 550,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/35358c7261be07d066c4eb06451a83ed3049721b8a8ae8e50d1b4a0d909e7aec.png\"><img src=\"https://sbs.s3.yandex.net/35358c7261be07d066c4eb06451a83ed3049721b8a8ae8e50d1b4a0d909e7aec.png\" width=300px></a></td>\n",
       "      <td>0.54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/af24437833aef6d80f05db322f441cd0573f3c886ccc692d68d410180de481b6.png\"><img src=\"https://sbs.s3.yandex.net/af24437833aef6d80f05db322f441cd0573f3c886ccc692d68d410180de481b6.png\" width=300px></a></td>\n",
       "      <td>0.46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "spb_results = pd.DataFrame(get_pool_completed_tasks(spb_pool)[\"winner_url\"].value_counts()).reset_index()\n",
    "spb_results[\"winner_url\"] = spb_results[\"winner_url\"] / sum(spb_results[\"winner_url\"])\n",
    "\n",
    "spb_results.columns = [\"url\", \"rate\"]\n",
    "spb_results[\"url\"] = spb_results[\"url\"].apply(\n",
    "                        lambda x: '<a href=\"{}\"><img src=\"{}\" width=300px></a>'.format(x, x))\n",
    "\n",
    "pd.options.display.max_colwidth = -1\n",
    "html_all = spb_results.to_html(escape=False, )\n",
    "display(HTML(html_all))\n",
    "pd.options.display.max_colwidth = 50\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Сравнение поисковых выдач\n",
    "\n",
    "\n",
    "\n",
    "Есть несколько картинок:\n",
    "\n",
    "\n",
    "<table>\n",
    "  <tr>\n",
    "<td>Запрос</td>\n",
    "      <td>  </td>\n",
    "     <td>Первый скриншот</td>\n",
    "      <td>  </td>\n",
    "     <td>Второй скриншот</td>\n",
    "  </tr>\n",
    "    <td>Дрель</td>\n",
    "      <td>  </td>\n",
    "     <td><img src=\"https://sbs.s3.yandex.net/059f49c93fce807412d023bb55c4844e36eb6db8ae587b9a34c11c3890762a7e.png\" width=\"450\"></td>\n",
    "      <td>  </td>\n",
    "     <td><img src=\"https://sbs.s3.yandex.net/1af370c9ad02557b5172b52176f37d1dbd9577359966b6daf0c6cfdc96dee955.png\" width=\"450\"></td>\n",
    "  </tr>\n",
    "    \n",
    "  <tr>\n",
    "    <td>Пассатижи</td>\n",
    "      <td>  </td>\n",
    "     <td><img src=\"https://sbs.s3.yandex.net/4d85b1a25fe8ec7872ae0b4265dc8299c74e72a2bbf5eb54c6300922f91f523e.png\" width=\"450\"></td>\n",
    "      <td>  </td>\n",
    "     <td><img src=\"https://sbs.s3.yandex.net/0998ab88cbe8113f3cf8d2f59c9b9730fd1e0ed0c0bfbcc7b0c739474d026de6.png\" width=\"450\"></td>\n",
    "  </tr>\n",
    "    \n",
    "  <tr>    \n",
    "     <td>Обои</td>\n",
    "      <td>  </td>\n",
    "     <td><img src=\"https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a6d324d89a0a3b2bee0aa54f414148f62a2bc10a44bce.png\" width=\"450\"></td>\n",
    "      <td>  </td>\n",
    "     <td><img src=\"https://sbs.s3.yandex.net/cfb6cf74a996059e09a54b05c3bd3eedc6db5e160dcd9f8fac77552d3047b7b1.png\" width=\"450\"></td>\n",
    "  </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 551,
   "metadata": {},
   "outputs": [],
   "source": [
    "queries = [\n",
    "    {\n",
    "        \"query\": \"Дрель\",\n",
    "        \"first_url\": \"https://sbs.s3.yandex.net/059f49c93fce807412d023bb55c4844e36eb6db8ae587b9a34c11c3890762a7e.png\", \n",
    "        \"second_url\": \"https://sbs.s3.yandex.net/1af370c9ad02557b5172b52176f37d1dbd9577359966b6daf0c6cfdc96dee955.png\"\n",
    "    }, \n",
    "    {\n",
    "        \"query\": \"Пассатижи\",\n",
    "        \"first_url\": \"https://sbs.s3.yandex.net/4d85b1a25fe8ec7872ae0b4265dc8299c74e72a2bbf5eb54c6300922f91f523e.png\", \n",
    "        \"second_url\": \"https://sbs.s3.yandex.net/0998ab88cbe8113f3cf8d2f59c9b9730fd1e0ed0c0bfbcc7b0c739474d026de6.png\"\n",
    "    }, \n",
    "    {\n",
    "        \"query\": \"Обои\",\n",
    "        \"first_url\": \"https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a6d324d89a0a3b2bee0aa54f414148f62a2bc10a44bce.png\", \n",
    "        \"second_url\": \"https://sbs.s3.yandex.net/cfb6cf74a996059e09a54b05c3bd3eedc6db5e160dcd9f8fac77552d3047b7b1.png\"\n",
    "    }, \n",
    "    \n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 553,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pool was cloned. New pool was created. New pool id:  8811095\n",
      "https://toloka.yandex.ru/requester/pool/8811095\n"
     ]
    }
   ],
   "source": [
    "leroy_pool = clone_toloka_pool(template_pool_id)\n",
    "print_toloka_pool_link(leroy_pool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 558,
   "metadata": {},
   "outputs": [],
   "source": [
    "leroy_tasks = []\n",
    "for query in queries:\n",
    "    leroy_tasks += prepare_toloka_task_for_2_screens(query[\"first_url\"], query[\"second_url\"], \n",
    "                                      default_question=\"Какая выдача больше подходит по запросу {}\".format(query[\"query\"]), \n",
    "                                     overlap=10, pool_id=leroy_pool)\n",
    "    \n",
    "upload_toloka_tasks(leroy_tasks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Проверяем, что залилось и запускаем"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 559,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pool 8811095 was started\n"
     ]
    }
   ],
   "source": [
    "run_toloka_pool(leroy_pool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 606,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image_left</th>\n",
       "      <th>image_right</th>\n",
       "      <th>question</th>\n",
       "      <th>comment</th>\n",
       "      <th>result</th>\n",
       "      <th>winner_url</th>\n",
       "      <th>loser_url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://sbs.s3.yandex.net/cfb6cf74a996059e09a5...</td>\n",
       "      <td>https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a...</td>\n",
       "      <td>Какая выдача больше подходит по запросу Обои</td>\n",
       "      <td>NaN</td>\n",
       "      <td>RIGHT</td>\n",
       "      <td>https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a...</td>\n",
       "      <td>https://sbs.s3.yandex.net/cfb6cf74a996059e09a5...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://sbs.s3.yandex.net/059f49c93fce807412d0...</td>\n",
       "      <td>https://sbs.s3.yandex.net/1af370c9ad02557b5172...</td>\n",
       "      <td>Какая выдача больше подходит по запросу Дрель</td>\n",
       "      <td>Вариант справа выглядит лучше по дизайну</td>\n",
       "      <td>RIGHT</td>\n",
       "      <td>https://sbs.s3.yandex.net/1af370c9ad02557b5172...</td>\n",
       "      <td>https://sbs.s3.yandex.net/059f49c93fce807412d0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://sbs.s3.yandex.net/4d85b1a25fe8ec7872ae...</td>\n",
       "      <td>https://sbs.s3.yandex.net/0998ab88cbe8113f3cf8...</td>\n",
       "      <td>Какая выдача больше подходит по запросу Пассатижи</td>\n",
       "      <td>шире выбор товара который запрашивают</td>\n",
       "      <td>LEFT</td>\n",
       "      <td>https://sbs.s3.yandex.net/4d85b1a25fe8ec7872ae...</td>\n",
       "      <td>https://sbs.s3.yandex.net/0998ab88cbe8113f3cf8...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a...</td>\n",
       "      <td>https://sbs.s3.yandex.net/cfb6cf74a996059e09a5...</td>\n",
       "      <td>Какая выдача больше подходит по запросу Обои</td>\n",
       "      <td>NaN</td>\n",
       "      <td>RIGHT</td>\n",
       "      <td>https://sbs.s3.yandex.net/cfb6cf74a996059e09a5...</td>\n",
       "      <td>https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://sbs.s3.yandex.net/cfb6cf74a996059e09a5...</td>\n",
       "      <td>https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a...</td>\n",
       "      <td>Какая выдача больше подходит по запросу Обои</td>\n",
       "      <td>NaN</td>\n",
       "      <td>LEFT</td>\n",
       "      <td>https://sbs.s3.yandex.net/cfb6cf74a996059e09a5...</td>\n",
       "      <td>https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          image_left  \\\n",
       "0  https://sbs.s3.yandex.net/cfb6cf74a996059e09a5...   \n",
       "1  https://sbs.s3.yandex.net/059f49c93fce807412d0...   \n",
       "2  https://sbs.s3.yandex.net/4d85b1a25fe8ec7872ae...   \n",
       "3  https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a...   \n",
       "4  https://sbs.s3.yandex.net/cfb6cf74a996059e09a5...   \n",
       "\n",
       "                                         image_right  \\\n",
       "0  https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a...   \n",
       "1  https://sbs.s3.yandex.net/1af370c9ad02557b5172...   \n",
       "2  https://sbs.s3.yandex.net/0998ab88cbe8113f3cf8...   \n",
       "3  https://sbs.s3.yandex.net/cfb6cf74a996059e09a5...   \n",
       "4  https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a...   \n",
       "\n",
       "                                            question  \\\n",
       "0       Какая выдача больше подходит по запросу Обои   \n",
       "1      Какая выдача больше подходит по запросу Дрель   \n",
       "2  Какая выдача больше подходит по запросу Пассатижи   \n",
       "3       Какая выдача больше подходит по запросу Обои   \n",
       "4       Какая выдача больше подходит по запросу Обои   \n",
       "\n",
       "                                     comment result  \\\n",
       "0                                        NaN  RIGHT   \n",
       "1  Вариант справа выглядит лучше по дизайну   RIGHT   \n",
       "2      шире выбор товара который запрашивают   LEFT   \n",
       "3                                        NaN  RIGHT   \n",
       "4                                        NaN   LEFT   \n",
       "\n",
       "                                          winner_url  \\\n",
       "0  https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a...   \n",
       "1  https://sbs.s3.yandex.net/1af370c9ad02557b5172...   \n",
       "2  https://sbs.s3.yandex.net/4d85b1a25fe8ec7872ae...   \n",
       "3  https://sbs.s3.yandex.net/cfb6cf74a996059e09a5...   \n",
       "4  https://sbs.s3.yandex.net/cfb6cf74a996059e09a5...   \n",
       "\n",
       "                                           loser_url  \n",
       "0  https://sbs.s3.yandex.net/cfb6cf74a996059e09a5...  \n",
       "1  https://sbs.s3.yandex.net/059f49c93fce807412d0...  \n",
       "2  https://sbs.s3.yandex.net/0998ab88cbe8113f3cf8...  \n",
       "3  https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a...  \n",
       "4  https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a...  "
      ]
     },
     "execution_count": 606,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "leroy_results = get_pool_completed_tasks(leroy_pool)\n",
    "leroy_results.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 607,
   "metadata": {},
   "outputs": [],
   "source": [
    "leroy_results = leroy_results.groupby([\"question\", \"winner_url\"])[[\"winner_url\"]].count()\n",
    "leroy_results.columns = [\"wins_number\"]\n",
    "leroy_results = leroy_results.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 608,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>winner_url</th>\n",
       "      <th>wins_number</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Какая выдача больше подходит по запросу Дрель</td>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/059f49c93fce807412d023bb55c4844e36eb6db8ae587b9a34c11c3890762a7e.png\"><img src=\"https://sbs.s3.yandex.net/059f49c93fce807412d023bb55c4844e36eb6db8ae587b9a34c11c3890762a7e.png\" width=300px></a></td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Какая выдача больше подходит по запросу Дрель</td>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/1af370c9ad02557b5172b52176f37d1dbd9577359966b6daf0c6cfdc96dee955.png\"><img src=\"https://sbs.s3.yandex.net/1af370c9ad02557b5172b52176f37d1dbd9577359966b6daf0c6cfdc96dee955.png\" width=300px></a></td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Какая выдача больше подходит по запросу Обои</td>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/cfb6cf74a996059e09a54b05c3bd3eedc6db5e160dcd9f8fac77552d3047b7b1.png\"><img src=\"https://sbs.s3.yandex.net/cfb6cf74a996059e09a54b05c3bd3eedc6db5e160dcd9f8fac77552d3047b7b1.png\" width=300px></a></td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Какая выдача больше подходит по запросу Обои</td>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a6d324d89a0a3b2bee0aa54f414148f62a2bc10a44bce.png\"><img src=\"https://sbs.s3.yandex.net/d5749d7cb9b3e3fdb13a6d324d89a0a3b2bee0aa54f414148f62a2bc10a44bce.png\" width=300px></a></td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Какая выдача больше подходит по запросу Пассатижи</td>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/0998ab88cbe8113f3cf8d2f59c9b9730fd1e0ed0c0bfbcc7b0c739474d026de6.png\"><img src=\"https://sbs.s3.yandex.net/0998ab88cbe8113f3cf8d2f59c9b9730fd1e0ed0c0bfbcc7b0c739474d026de6.png\" width=300px></a></td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Какая выдача больше подходит по запросу Пассатижи</td>\n",
       "      <td><a href=\"https://sbs.s3.yandex.net/4d85b1a25fe8ec7872ae0b4265dc8299c74e72a2bbf5eb54c6300922f91f523e.png\"><img src=\"https://sbs.s3.yandex.net/4d85b1a25fe8ec7872ae0b4265dc8299c74e72a2bbf5eb54c6300922f91f523e.png\" width=300px></a></td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "leroy_results\n",
    "\n",
    "leroy_results[\"winner_url\"] = leroy_results[\"winner_url\"].apply(\n",
    "                        lambda x: '<a href=\"{}\"><img src=\"{}\" width=300px></a>'.format(x, x))\n",
    "\n",
    "pd.options.display.max_colwidth = -1\n",
    "html_all = leroy_results.to_html(escape=False, )\n",
    "display(HTML(html_all))\n",
    "pd.options.display.max_colwidth = 50\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SBSить можно не только картинки, но и видео."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td><iframe width=\"500\" height=\"315\" src=\"https://www.youtube.com/embed/2tSHYKbKAX8\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe></td>\n",
       "      <td><iframe width=\"500\" height=\"315\" src=\"https://www.youtube.com/embed/cbq3BoEZc9s\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "first_video = \"\"\"<iframe width=\"500\" height=\"315\" src=\"https://www.youtube.com/embed/2tSHYKbKAX8\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\"\"\"\n",
    "second_video = \"\"\"<iframe width=\"500\" height=\"315\" src=\"https://www.youtube.com/embed/cbq3BoEZc9s\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\"\"\"\n",
    "\n",
    "videos_to_sbs = pd.DataFrame([first_video, second_video]).T\n",
    "\n",
    "pd.options.display.max_colwidth = 20\n",
    "html_all = videos_to_sbs.to_html(escape=False, )\n",
    "display(HTML(html_all))\n",
    "pd.options.display.max_colwidth = 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 622,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New project was created. New project id:  29702\n",
      "New pool was created. New pool id:  8812324\n"
     ]
    }
   ],
   "source": [
    "project_template = json.loads(open(\"project_templates/youtube_video_comparison_project_template.json\").read())\n",
    "project_id = create_toloka_project(project_template)\n",
    "\n",
    "\n",
    "pool_template = json.loads(open(\"project_templates/pool_params_template.json\").read())\n",
    "template_video_pool_id = create_toloka_pool(project_id, pool_template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 623,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pool was cloned. New pool was created. New pool id:  8812327\n"
     ]
    }
   ],
   "source": [
    "two_videos_pool_id = clone_toloka_pool(template_video_pool_id)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 625,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pool was cloned. New pool was created. New pool id:  8812342\n",
      "https://toloka.yandex.ru/requester/pool/8812342\n"
     ]
    }
   ],
   "source": [
    "two_videos_pool_id = clone_toloka_pool(template_video_pool_id)\n",
    "\n",
    "\n",
    "video_tasks = prepare_toloka_task_for_2_screens(first_video, second_video, \n",
    "                                      default_question=\"Какое видео с Яндекс.Алисой вам больше нравится?\", \n",
    "                                     overlap=100, pool_id=two_videos_pool_id)\n",
    "    \n",
    "upload_toloka_tasks(video_tasks)\n",
    "print_toloka_pool_link(two_videos_pool_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Снова проверяем и запускаем"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 626,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pool 8812342 was started\n"
     ]
    }
   ],
   "source": [
    "run_toloka_pool(two_videos_pool_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 635,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'8812342'"
      ]
     },
     "execution_count": 635,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "two_videos_pool_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 638,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td><iframe width=\"500\" height=\"315\" src=\"https://www.youtube.com/embed/2tSHYKbKAX8\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe></td>\n",
       "      <td>0.74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td><iframe width=\"500\" height=\"315\" src=\"https://www.youtube.com/embed/cbq3BoEZc9s\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe></td>\n",
       "      <td>0.26</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "video_results = pd.DataFrame(get_pool_completed_tasks(two_videos_pool_id)[\"winner_url\"].value_counts()).reset_index()\n",
    "video_results[\"winner_url\"] = video_results[\"winner_url\"] / sum(video_results[\"winner_url\"])\n",
    "\n",
    "video_results.columns = [\"url\", \"rate\"]\n",
    "\n",
    "pd.options.display.max_colwidth = -1\n",
    "html_all = video_results.to_html(escape=False, )\n",
    "display(HTML(html_all))\n",
    "pd.options.display.max_colwidth = 50\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Добавление вебвизора в проект"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Можно записывать действия толокера с помощью вебвизора. Для этого нужно разобраться с счетчиком яндекс.метрики. \n",
    "https://yandex.ru/support/metrica/ и добавить его к себе в проект"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New project was created. New project id:  29735\n"
     ]
    }
   ],
   "source": [
    "project_for_webvisor = json.loads(open(\"project_templates/pictures_comparison_project_template.json\").read())\n",
    "project_for_webvisor = create_toloka_project(project_for_webvisor)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New pool was created. New pool id:  8835749\n"
     ]
    }
   ],
   "source": [
    "pool_template = json.loads(open(\"project_templates/pool_params_template.json\").read())\n",
    "template_webvisor_pool_id = create_toloka_pool(project_for_webvisor, pool_template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://sbs.s3.yandex.net/44aebb8c77a29efa25e8a5d1ac4eba9877b84efa1147b4b003b2a8380273cc73.png\n",
      "https://sbs.s3.yandex.net/93656faf648029e3bf4bf18421b3dc08d85fe4bdd2683498cbf497f0487e89db.png\n"
     ]
    }
   ],
   "source": [
    "first_url = \"https://sbs.s3.yandex.net/44aebb8c77a29efa25e8a5d1ac4eba9877b84efa1147b4b003b2a8380273cc73.png\"\n",
    "second_url = \"https://sbs.s3.yandex.net/93656faf648029e3bf4bf18421b3dc08d85fe4bdd2683498cbf497f0487e89db.png\"\n",
    "\n",
    "print(first_url)\n",
    "print(second_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pool was cloned. New pool was created. New pool id:  8835752\n"
     ]
    }
   ],
   "source": [
    "two_pictures_pool_id = clone_toloka_pool(template_webvisor_pool_id)\n",
    "tasks = prepare_toloka_task_for_2_screens(first_url, second_url, \n",
    "                                         50, two_pictures_pool_id)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Далее зальем их в наш пул. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "upload_toloka_tasks(tasks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://toloka.yandex.ru/requester/pool/8835752\n"
     ]
    }
   ],
   "source": [
    "print_toloka_pool_link(two_pictures_pool_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pool 8835752 was started\n"
     ]
    }
   ],
   "source": [
    "run_toloka_pool(two_pictures_pool_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Через некоторое время  в Яндекс.Метрике станут доступны вебвизорные записи!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<video width=\"700\" controls=\"controls\"  src=\"https://sbs.s3.yandex.net/9171f87e99eb0a5b581631b987d6dfc74fdb0a222d1479e9b774f6c0021f79b5/webvisor_example.mp4\">\n",
    "\n",
    "</video>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}