Disambiguition.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__PyScopus__: An example for author disambiguity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes Scopus would mix up people with similar names. I recently come up with a not that difficult method to clean author publication profiles, which needs some manual work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you can think of a better way, please do let me know!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-09T13:45:20.253554Z",
     "start_time": "2018-10-09T13:45:17.592702Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.0.1'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pyscopus\n",
    "pyscopus.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-09T13:45:30.921152Z",
     "start_time": "2018-10-09T13:45:20.257628Z"
    }
   },
   "outputs": [],
   "source": [
    "from pyscopus import Scopus\n",
    "key = 'YOUR_OWN_APIKEY'\n",
    "scopus = Scopus(key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-09-29T23:12:58.418891Z",
     "start_time": "2018-09-29T23:12:58.177531Z"
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wrapper functions for disambiguity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-09T13:46:07.258839Z",
     "start_time": "2018-10-09T13:46:06.244118Z"
    }
   },
   "outputs": [],
   "source": [
    "import requests, time\n",
    "import pandas as pd\n",
    "from bs4 import BeautifulSoup as Soup\n",
    "BASE_URL = \"https://api.elsevier.com/content/abstract/scopus_id/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-09T13:46:24.823895Z",
     "start_time": "2018-10-09T13:46:24.813798Z"
    }
   },
   "outputs": [],
   "source": [
    "def _check_pub_validity(sid, author_id, author_affil_id_list, apikey):\n",
    "    global BASE_URL\n",
    "    r = requests.get(BASE_URL+sid, params={'apikey': apikey})\n",
    "    soup = Soup(r.content, 'lxml')\n",
    "    author_list = soup.find('authors').find_all('author')\n",
    "\n",
    "    ## go through the author list to find the author first, by matching author id\n",
    "    for au in author_list:\n",
    "        if au['auid'] == author_id:\n",
    "            ## find it and break\n",
    "            break\n",
    "\n",
    "    ## check the affiliation id: note that an author may have a list of affiliations\n",
    "    this_affil_id_list = [affil_tag['id'] for affil_tag in au.find_all('affiliation')]\n",
    "    ## get the affiliation id and check if there are any overlap\n",
    "    if len(set(author_affil_id_list).intersection(set(this_affil_id_list))) > 0:\n",
    "        return True\n",
    "    return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-09T13:46:28.273865Z",
     "start_time": "2018-10-09T13:46:24.827978Z"
    }
   },
   "outputs": [],
   "source": [
    "def check_pub_validity(scopus_obj, author_id, author_affil_id_list, apikey):\n",
    "    ## first find out all pub\n",
    "    pub_df = scopus_obj.search_author_publication(author_id)\n",
    "    ## do this for all non-null scopus ids\n",
    "    pub_df = pub_df[~pub_df.scopus_id.isnull()]\n",
    "    ## list to save all eligble scopus ids\n",
    "    eligible_scopus_id_list = list()\n",
    "    for i, sid in enumerate(pub_df.scopus_id.values):\n",
    "        if (i+1)%5==0:\n",
    "            time.sleep(pd.np.random.random()+.3)\n",
    "        if _check_pub_validity(sid, author_id, author_affil_id_list, apikey):\n",
    "            ## if true, save it\n",
    "            eligible_scopus_id_list.append(sid)\n",
    "    ## finally, get a subset of the original pub_df\n",
    "    filtered_pub_df = pub_df.query(\"scopus_id in @eligible_scopus_id_list\")\n",
    "    return filtered_pub_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When I was collecting data for my own research, I found that [Dr. Vivek K. Singh](http://web.media.mit.edu/~singhv/) has a very [noisy profile in Scopus](https://www.scopus.com/authid/detail.uri?authorId=7404651152). Let's use this as an example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The basic idea is to match author-affiliation pair:\n",
    "- For all the paper found in the _mixed profile_\n",
    "    - Find the focal author (in this case, Dr. Singh)\n",
    "    - Look at his/her affiliation\n",
    "        - Keep this paper if the affiliation is indeed where he/she is\n",
    "        - If not, discard the paper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Dr. Singh, I manually obtained his affiliation ids by searching through [Scopus affiliation search](https://www.scopus.com/search/form.uri?display=affiliationLookup). Upon obtaining that, create a dictionary containing _name (first/last)_, _affiliation name_, and _a list of affiliation ids_. Author and affiliation names would be used to search for this author. The list of affiliation ids would be used for cleaning papers: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- UC Irvine `60007278`\n",
    "- MIT `60022195`\n",
    "- Rutgers `60030623`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-09T13:46:42.602896Z",
     "start_time": "2018-10-09T13:46:42.593841Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'authfirst': 'Vivek',\n",
       " 'authlastname': 'Singh',\n",
       " 'affiliation': 'Rutgers',\n",
       " 'affil_id_list': ['60030623', '60022195', '60007278']}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = {'authfirst': 'Vivek', 'authlastname': 'Singh', 'affiliation': 'Rutgers',\n",
    "     'affil_id_list': ['60030623', '60022195', '60007278']\n",
    "    }\n",
    "d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-09T13:46:48.040998Z",
     "start_time": "2018-10-09T13:46:44.327019Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The history saving thread hit an unexpected error (OperationalError('database is locked',)).History will not be written to the database.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>author_id</th>\n",
       "      <th>name</th>\n",
       "      <th>document_count</th>\n",
       "      <th>affiliation</th>\n",
       "      <th>affiliation_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7404651152</td>\n",
       "      <td>Vivek Kumar N. Singh</td>\n",
       "      <td>491</td>\n",
       "      <td>Shri Mata Vaishno Devi University</td>\n",
       "      <td>60017187</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    author_id                  name  document_count  \\\n",
       "0  7404651152  Vivek Kumar N. Singh             491   \n",
       "\n",
       "                         affiliation affiliation_id  \n",
       "0  Shri Mata Vaishno Devi University       60017187  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = \"AUTHLASTNAME({}) and AUTHFIRST({}) and AFFIL({})\".format(d['authlastname'], d['authfirst'], d['affiliation'])\n",
    "author_search_df = scopus.search_author(query)\n",
    "author_search_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes we would obtain a list of author profiles for each author. In this case, we only have one and it is clear that the author profile is highly noisy.\n",
    "\n",
    "In the following step, I would use the helper functions in `utils` to screen each paper by this `author_id`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-09T13:47:10.682479Z",
     "start_time": "2018-10-09T13:47:10.674464Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('7404651152', ['60030623', '60022195', '60007278'])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "author_id = '7404651152'\n",
    "author_id, d['affil_id_list']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The filtering process may take a while, depending on how many documents are mixd up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-09T13:58:49.253297Z",
     "start_time": "2018-10-09T13:47:25.005514Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(134, 134, 0)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filterd_pub_df = check_pub_validity(scopus, author_id, d['affil_id_list'], key)\n",
    "filterd_pub_df.shape[0], filterd_pub_df.scopus_id.unique().size, filterd_pub_df.scopus_id.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obviously, the number of papers is highly reduced. We can now check a random subset to see if the filtered papers make sense for this author."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-09T13:59:13.491653Z",
     "start_time": "2018-10-09T13:59:13.466452Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>publication_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>Effects of high-energy irradiation on silicon ...</td>\n",
       "      <td>Optics InfoBase Conference Papers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>Physical-Cyber-Social Computing: Looking Back,...</td>\n",
       "      <td>IEEE Internet Computing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>141</th>\n",
       "      <td>Low-stress silicon nitride for mid-infrared mi...</td>\n",
       "      <td>Optics InfoBase Conference Papers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>194</th>\n",
       "      <td>Mid-infrared silicon waveguide resonators with...</td>\n",
       "      <td>Materials Research Society Symposium Proceedings</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>Effects of high-energy irradiation on silicon ...</td>\n",
       "      <td>Optics InfoBase Conference Papers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>194</th>\n",
       "      <td>Mid-infrared silicon waveguide resonators with...</td>\n",
       "      <td>Materials Research Society Symposium Proceedings</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>Preface</td>\n",
       "      <td>Geo-Intelligence and Visualization through Big...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>179</th>\n",
       "      <td>Demonstration of high-Q mid-infrared chalcogen...</td>\n",
       "      <td>Optics Letters</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>Low-Stress silicon nitride platform for broadb...</td>\n",
       "      <td>Optics InfoBase Conference Papers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>309</th>\n",
       "      <td>Situation based control for cyber-physical env...</td>\n",
       "      <td>Proceedings - IEEE Military Communications Con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Towards measuring fine-grained diversity using...</td>\n",
       "      <td>Proceedings of the 11th International Conferen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>306</th>\n",
       "      <td>Motivating contributors in social media networks</td>\n",
       "      <td>1st ACM SIGMM International Workshop on Social...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>Mid-infrared opto-nanofluidics for label-free ...</td>\n",
       "      <td>Optics InfoBase Conference Papers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>Gradient Polymer Nanofoams for Encrypted Recor...</td>\n",
       "      <td>ACS Nano</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>Physical-Cyber-Social Computing: Looking Back,...</td>\n",
       "      <td>IEEE Internet Computing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>336</th>\n",
       "      <td>Towards environment-to-environment (E2E) multi...</td>\n",
       "      <td>MM'08 - Proceedings of the 2008 ACM Internatio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>Low-stress silicon nitride platform for broadb...</td>\n",
       "      <td>Conference on Lasers and Electro-Optics Europe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209</th>\n",
       "      <td>Anisotropic photoluminescence from Er-TeO&lt;inf&gt;...</td>\n",
       "      <td>CLEO: Science and Innovations, CLEO_SI 2012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>Physical-Cyber-Social Computing: Looking Back,...</td>\n",
       "      <td>IEEE Internet Computing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>On-chip mid-infrared gas detection using chalc...</td>\n",
       "      <td>Applied Physics Letters</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 title  \\\n",
       "118  Effects of high-energy irradiation on silicon ...   \n",
       "95   Physical-Cyber-Social Computing: Looking Back,...   \n",
       "141  Low-stress silicon nitride for mid-infrared mi...   \n",
       "194  Mid-infrared silicon waveguide resonators with...   \n",
       "67   Effects of high-energy irradiation on silicon ...   \n",
       "194  Mid-infrared silicon waveguide resonators with...   \n",
       "89                                             Preface   \n",
       "179  Demonstration of high-Q mid-infrared chalcogen...   \n",
       "147  Low-Stress silicon nitride platform for broadb...   \n",
       "309  Situation based control for cyber-physical env...   \n",
       "42   Towards measuring fine-grained diversity using...   \n",
       "306   Motivating contributors in social media networks   \n",
       "146  Mid-infrared opto-nanofluidics for label-free ...   \n",
       "54   Gradient Polymer Nanofoams for Encrypted Recor...   \n",
       "95   Physical-Cyber-Social Computing: Looking Back,...   \n",
       "336  Towards environment-to-environment (E2E) multi...   \n",
       "144  Low-stress silicon nitride platform for broadb...   \n",
       "209  Anisotropic photoluminescence from Er-TeO<inf>...   \n",
       "95   Physical-Cyber-Social Computing: Looking Back,...   \n",
       "71   On-chip mid-infrared gas detection using chalc...   \n",
       "\n",
       "                                      publication_name  \n",
       "118                  Optics InfoBase Conference Papers  \n",
       "95                             IEEE Internet Computing  \n",
       "141                  Optics InfoBase Conference Papers  \n",
       "194   Materials Research Society Symposium Proceedings  \n",
       "67                   Optics InfoBase Conference Papers  \n",
       "194   Materials Research Society Symposium Proceedings  \n",
       "89   Geo-Intelligence and Visualization through Big...  \n",
       "179                                     Optics Letters  \n",
       "147                  Optics InfoBase Conference Papers  \n",
       "309  Proceedings - IEEE Military Communications Con...  \n",
       "42   Proceedings of the 11th International Conferen...  \n",
       "306  1st ACM SIGMM International Workshop on Social...  \n",
       "146                  Optics InfoBase Conference Papers  \n",
       "54                                            ACS Nano  \n",
       "95                             IEEE Internet Computing  \n",
       "336  MM'08 - Proceedings of the 2008 ACM Internatio...  \n",
       "144  Conference on Lasers and Electro-Optics Europe...  \n",
       "209        CLEO: Science and Innovations, CLEO_SI 2012  \n",
       "95                             IEEE Internet Computing  \n",
       "71                             Applied Physics Letters  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filterd_pub_df.iloc[pd.np.random.randint(0, high=134, size=20)][['title', 'publication_name']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, there may still be noise in it (e.g., papers published in optics/photonics venues). We can manually exclude those as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-09T13:59:16.200976Z",
     "start_time": "2018-10-09T13:59:16.054046Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(59, 16)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filterd_pub_df = filterd_pub_df.query(\"not publication_name.str.lower().str.contains('optic')\")\n",
    "filterd_pub_df = filterd_pub_df.query(\"not publication_name.str.lower().str.contains('photonic')\")\n",
    "filterd_pub_df = filterd_pub_df.query(\"not publication_name.str.lower().str.contains('nano')\")\n",
    "filterd_pub_df = filterd_pub_df.query(\"not publication_name.str.lower().str.contains('quantum')\")\n",
    "filterd_pub_df = filterd_pub_df.query(\"not publication_name.str.lower().str.contains('sensor')\")\n",
    "filterd_pub_df = filterd_pub_df.query(\"not publication_name.str.lower().str.contains('cleo')\")\n",
    "filterd_pub_df = filterd_pub_df.query(\"not publication_name.str.lower().str.contains('materials')\")\n",
    "filterd_pub_df = filterd_pub_df.query(\"not publication_name.str.lower().str.contains('physics')\")\n",
    "filterd_pub_df = filterd_pub_df.query(\"not publication_name.str.lower().str.contains('chip')\")\n",
    "filterd_pub_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's check again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-09T13:59:17.096150Z",
     "start_time": "2018-10-09T13:59:17.077181Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>publication_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>Assessing personality using demographic inform...</td>\n",
       "      <td>ACM International Conference Proceeding Series</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Examining information search behaviors in smal...</td>\n",
       "      <td>Proceedings of the Association for Information...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Examining information search behaviors in smal...</td>\n",
       "      <td>Proceedings of the Association for Information...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Social bridges in urban purchase behavior</td>\n",
       "      <td>ACM Transactions on Intelligent Systems and Te...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>If it looks like a spammer and behaves like a ...</td>\n",
       "      <td>International Journal of Information Security</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>EventShop: Recognizing situations in web data ...</td>\n",
       "      <td>WWW 2013 Companion - Proceedings of the 22nd I...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Are you altruistic? Your mobile phone could tell</td>\n",
       "      <td>2017 IEEE SmartWorld Ubiquitous Intelligence a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>Assessing personality using demographic inform...</td>\n",
       "      <td>ACM International Conference Proceeding Series</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>247</th>\n",
       "      <td>EventShop: From heterogeneous web streams to p...</td>\n",
       "      <td>Proceedings of the 4th Annual ACM Web Science ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>LTA 2016 - The first workshop on lifelogging t...</td>\n",
       "      <td>MM 2016 - Proceedings of the 2016 ACM Multimed...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Effect of gamma exposure on chalcogenide glass...</td>\n",
       "      <td>IEEE Radiation Effects Data Workshop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>Physical-Cyber-Social Computing: Looking Back,...</td>\n",
       "      <td>IEEE Internet Computing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>298</th>\n",
       "      <td>Structural analysis of the emerging event-web</td>\n",
       "      <td>Proceedings of the 19th International Conferen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>New Signals in Multimedia Systems and Applicat...</td>\n",
       "      <td>IEEE Multimedia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>EventShop: Recognizing situations in web data ...</td>\n",
       "      <td>WWW 2013 Companion - Proceedings of the 22nd I...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>317</th>\n",
       "      <td>Adversary aware surveillance systems</td>\n",
       "      <td>IEEE Transactions on Information Forensics and...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Toward harmonizing self-reported and logged so...</td>\n",
       "      <td>Conference on Human Factors in Computing Syste...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>Predicting privacy attitudes using phone metadata</td>\n",
       "      <td>Lecture Notes in Computer Science (including s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>Preface</td>\n",
       "      <td>Geo-Intelligence and Visualization through Big...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>Probing the interconnections between geo-explo...</td>\n",
       "      <td>UbiComp 2016 - Proceedings of the 2016 ACM Int...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 title  \\\n",
       "103  Assessing personality using demographic inform...   \n",
       "44   Examining information search behaviors in smal...   \n",
       "44   Examining information search behaviors in smal...   \n",
       "18           Social bridges in urban purchase behavior   \n",
       "62   If it looks like a spammer and behaves like a ...   \n",
       "152  EventShop: Recognizing situations in web data ...   \n",
       "5     Are you altruistic? Your mobile phone could tell   \n",
       "103  Assessing personality using demographic inform...   \n",
       "247  EventShop: From heterogeneous web streams to p...   \n",
       "61   LTA 2016 - The first workshop on lifelogging t...   \n",
       "34   Effect of gamma exposure on chalcogenide glass...   \n",
       "95   Physical-Cyber-Social Computing: Looking Back,...   \n",
       "298      Structural analysis of the emerging event-web   \n",
       "17   New Signals in Multimedia Systems and Applicat...   \n",
       "152  EventShop: Recognizing situations in web data ...   \n",
       "317               Adversary aware surveillance systems   \n",
       "30   Toward harmonizing self-reported and logged so...   \n",
       "74   Predicting privacy attitudes using phone metadata   \n",
       "89                                             Preface   \n",
       "64   Probing the interconnections between geo-explo...   \n",
       "\n",
       "                                      publication_name  \n",
       "103     ACM International Conference Proceeding Series  \n",
       "44   Proceedings of the Association for Information...  \n",
       "44   Proceedings of the Association for Information...  \n",
       "18   ACM Transactions on Intelligent Systems and Te...  \n",
       "62       International Journal of Information Security  \n",
       "152  WWW 2013 Companion - Proceedings of the 22nd I...  \n",
       "5    2017 IEEE SmartWorld Ubiquitous Intelligence a...  \n",
       "103     ACM International Conference Proceeding Series  \n",
       "247  Proceedings of the 4th Annual ACM Web Science ...  \n",
       "61   MM 2016 - Proceedings of the 2016 ACM Multimed...  \n",
       "34                IEEE Radiation Effects Data Workshop  \n",
       "95                             IEEE Internet Computing  \n",
       "298  Proceedings of the 19th International Conferen...  \n",
       "17                                     IEEE Multimedia  \n",
       "152  WWW 2013 Companion - Proceedings of the 22nd I...  \n",
       "317  IEEE Transactions on Information Forensics and...  \n",
       "30   Conference on Human Factors in Computing Syste...  \n",
       "74   Lecture Notes in Computer Science (including s...  \n",
       "89   Geo-Intelligence and Visualization through Big...  \n",
       "64   UbiComp 2016 - Proceedings of the 2016 ACM Int...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filterd_pub_df.iloc[pd.np.random.randint(0, high=59, size=20)][['title', 'publication_name']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it is much better and we can use this cleaned paper list for this focal author."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "335px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}