src/Github_Repo_Finder.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f39c31ae",
   "metadata": {},
   "source": [
    "# Github Repository Finder - with sample dataset\n",
    "\n",
    "## Introduction\n",
    "This notebook gives information about the function used to fetch Github repositories based on a keyword and a determined timeframe starting from the day of fetching.\n",
    "\n",
    "## Running instruction\n",
    "\n",
    "### Requirements:\n",
    "1. Generate your own Github Token: [Creating a personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)\n",
    "2. Save your token as an enviroment variable, remember to name the variable as 'GITHUBTOKEN': [Configuring Environment Variables](https://tilburgsciencehub.com/building-blocks/store-and-document-your-data/store-data/environment-variables/) \n",
    "3. Make sure you have installed the following packages in python: requests, math, datetime, dateutil, csv, pandas, json, os, time. Installation instruction can be found at [Python website](https://www.python.org/) \n",
    "\n",
    "## Function structure:\n",
    "GRF collects data by operating 4 separate steps accquired via 4 functions: find_repo, export_repo_list; save_column; save_dt. The working of these functions is illustrated in the following diagram:\n",
    "\n",
    "\n",
    "![Fig1 GitHub Repository Finder components](../docs/Function_documentation/fig1-grf-component-diagram.png \"GitHub Repository Finder components\")\n",
    "\n",
    "## Sample dataset:\n",
    "A sample dataset was obtained by using the following command:\n",
    "```\n",
    "grf(\"python\", 3, 8)\n",
    "```\n",
    "+ Detailed datasheet can be found here: [Datasheet for GRF sample dataset](/~https://github.com/thtbui/github-repository-finder/docs/Datasheet_for_GitHub_Repository_Finder_Sample_Dataset.pdf)\n",
    "+ Below is a preview of the final dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a86db6b8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>name</th>\n",
       "      <th>url</th>\n",
       "      <th>language</th>\n",
       "      <th>created</th>\n",
       "      <th>stars</th>\n",
       "      <th>watch</th>\n",
       "      <th>forks</th>\n",
       "      <th>readme</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>416977797</td>\n",
       "      <td>AI_Project</td>\n",
       "      <td>/~https://github.com/pbl4team/AI_Project</td>\n",
       "      <td>Python</td>\n",
       "      <td>2021-10-14T03:37:01Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>Project AI Systeam - Computer Vision with pyth...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>416995331</td>\n",
       "      <td>python</td>\n",
       "      <td>/~https://github.com/Cam0411/python</td>\n",
       "      <td>Python</td>\n",
       "      <td>2021-10-14T05:06:00Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>416908634</td>\n",
       "      <td>Python</td>\n",
       "      <td>/~https://github.com/psplendid61/Python</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-10-13T21:53:43Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>416963376</td>\n",
       "      <td>python</td>\n",
       "      <td>/~https://github.com/iAMSe/python</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-10-14T02:28:55Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>416996896</td>\n",
       "      <td>python</td>\n",
       "      <td>/~https://github.com/rakeshk67/python</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-10-14T05:13:42Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>416961346</td>\n",
       "      <td>python</td>\n",
       "      <td>/~https://github.com/colddie/python</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-10-14T02:19:55Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>416990435</td>\n",
       "      <td>Python</td>\n",
       "      <td>/~https://github.com/mahdidahmani/Python</td>\n",
       "      <td>Python</td>\n",
       "      <td>2021-10-14T04:41:11Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>416952467</td>\n",
       "      <td>python</td>\n",
       "      <td>/~https://github.com/grace-th3/python</td>\n",
       "      <td>Python</td>\n",
       "      <td>2021-10-14T01:38:42Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>416928589</td>\n",
       "      <td>Python</td>\n",
       "      <td>/~https://github.com/Cheung-man/Python</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-10-13T23:36:27Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>416935834</td>\n",
       "      <td>python</td>\n",
       "      <td>/~https://github.com/mygithuang/python</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-10-14T00:15:25Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>python mygithuang</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          id        name                                     url language  \\\n",
       "0  416977797  AI_Project  /~https://github.com/pbl4team/AI_Project   Python   \n",
       "1  416995331      python       /~https://github.com/Cam0411/python   Python   \n",
       "2  416908634      Python   /~https://github.com/psplendid61/Python      NaN   \n",
       "3  416963376      python         /~https://github.com/iAMSe/python      NaN   \n",
       "4  416996896      python     /~https://github.com/rakeshk67/python      NaN   \n",
       "5  416961346      python       /~https://github.com/colddie/python      NaN   \n",
       "6  416990435      Python  /~https://github.com/mahdidahmani/Python   Python   \n",
       "7  416952467      python     /~https://github.com/grace-th3/python   Python   \n",
       "8  416928589      Python    /~https://github.com/Cheung-man/Python      NaN   \n",
       "9  416935834      python    /~https://github.com/mygithuang/python      NaN   \n",
       "\n",
       "                created  stars  watch  forks  \\\n",
       "0  2021-10-14T03:37:01Z      0      0      4   \n",
       "1  2021-10-14T05:06:00Z      0      0      0   \n",
       "2  2021-10-13T21:53:43Z      0      0      0   \n",
       "3  2021-10-14T02:28:55Z      0      0      0   \n",
       "4  2021-10-14T05:13:42Z      0      0      0   \n",
       "5  2021-10-14T02:19:55Z      0      0      0   \n",
       "6  2021-10-14T04:41:11Z      0      0      0   \n",
       "7  2021-10-14T01:38:42Z      0      0      0   \n",
       "8  2021-10-13T23:36:27Z      0      0      0   \n",
       "9  2021-10-14T00:15:25Z      0      0      0   \n",
       "\n",
       "                                              readme  \n",
       "0  Project AI Systeam - Computer Vision with pyth...  \n",
       "1                                            python   \n",
       "2                                                NaN  \n",
       "3                                            python   \n",
       "4                                            python   \n",
       "5                                                NaN  \n",
       "6                                                NaN  \n",
       "7                                                NaN  \n",
       "8                                                NaN  \n",
       "9                                 python mygithuang   "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.read_csv(\"../data/dt.csv\", delimiter= \";\",nrows=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e920fe5e",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "# Function source code:\n",
    "\n",
    "## 1. Find_repo:\n",
    "This is the main function that crawls repositories from GitHub based on the use of time and page parameters. The API queries are generated as illustrated in the following diagram:\n",
    "\n",
    "![Fig2 Search queries generated by find_repo using time and page parameters](../docs/Function_documentation/fig2-search-query-with-time-page-parameters.png \"Search queries generated by find_repo using time and page parameters\")\n",
    "\n",
    "After each page query, the call status is recorded into a log file. This log file is saved into user's working directory as log.txt.\n",
    "\n",
    "The main result is saved into a single json file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b064d1b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Step1: Searching for repositories, crawling data, storing responses in a .json file\n",
    "\n",
    "def find_repo(term, d, h):\n",
    "    '''\n",
    "    First component of GRF function.\n",
    "    Searches for repositories on GitHub, returns name of a .json file of original data.\n",
    "\n",
    "            Parameters:\n",
    "                    term (str): A string defining the search term\n",
    "                    d (int): An integer defining the timeframe to search (in days)\n",
    "                    h (int): An integer defining the duration (in hours) of each query\n",
    "\n",
    "            Returns:\n",
    "                    filename (str): Name of a single .json file which stores the original response data\n",
    "                    .json file: A file of raw data stored in working directory\n",
    "                    log.txt: Log file recording all API calls and their status\n",
    "    \n",
    "    Check GRF documenation for more details.\n",
    "    Source code available at: /~https://github.com/thtbui/github-repository-finder \n",
    "    ''' \n",
    "    #importing required libraries\n",
    "    import requests\n",
    "    import math\n",
    "    from datetime import datetime, timedelta\n",
    "    from dateutil.relativedelta import relativedelta\n",
    "    import csv\n",
    "    import pandas as pd \n",
    "    import json\n",
    "    import os\n",
    "    import time\n",
    "    \n",
    "    #constructing URLs\n",
    "    token=os.environ['GITHUBTOKEN']\n",
    "    URL = f'https://api.github.com/search/repositories?q={term}+created:SINCE..UNTIL&per_page=100'\n",
    "    HEADERS = {'Authorization':f'{token}'}\n",
    "    \n",
    "    #setting timeframe and search segments\n",
    "    since = datetime.today() - relativedelta(days = d)  \n",
    "    until = since + timedelta(hours = h)\n",
    "    \n",
    "    #setting name for the final json file\n",
    "    fetch_time=f'{datetime.today()}'.replace(\":\",\"_\")\n",
    "    filename = f'{term}_{d}days_{fetch_time}.json' \n",
    "    \n",
    "    # creating the log file to monitor error\n",
    "    log_file = 'log.txt'\n",
    "    f = open(log_file, 'w')\n",
    "    f.close()\n",
    "    \n",
    "    # creating an empty json file to combine all json files returned from each call:\n",
    "    f= open(filename,'w',encoding='utf-8')\n",
    "    f.close()\n",
    "        \n",
    "    while until < datetime.today():\n",
    "        day_url = URL.replace('SINCE', since.strftime('%Y-%m-%dT%H:%M:%SZ')).replace('UNTIL', until.strftime('%Y-%m-%dT%H:%M:%SZ'))\n",
    "        search_response = requests.get(day_url, headers=HEADERS)\n",
    "        time.sleep(7)\n",
    "        \n",
    "        #print out No of repos in every h-hour request\n",
    "        print(f'No of repositories created between {since} and {until}:{search_response.json().get(\"total_count\")}')\n",
    "        \n",
    "        #If no Total_count found, print out the failed link and additional information\n",
    "        if search_response.json().get('total_count') == None:\n",
    "            print(f'Failed request: {day_url}')\n",
    "            \n",
    "            #saving status in log file:\n",
    "            f = open(log_file, 'a')\n",
    "            f.write(str(since)+'\\t'+page_url+'\\t'+str(search_response.status_code)+'\\n')\n",
    "            f.close()\n",
    "            \n",
    "            print(search_response.text)\n",
    "            print(search_response.headers) \n",
    "        \n",
    "        #if extraction successful, do pagination:\n",
    "        else:\n",
    "            #calculating the total No. of pages\n",
    "            no_page = math.ceil(search_response.json().get(\"total_count\")/100) \n",
    "            \n",
    "            #pagination: running a loop to retrieve each page\n",
    "            for i in range(1, no_page + 1): \n",
    "                page_url = f'{day_url}&page={i}'\n",
    "                page_response = requests.get(page_url, headers=HEADERS)\n",
    "                time.sleep(7)\n",
    "                \n",
    "                #writing all requests in a single json file (filename):\n",
    "                repo_request = page_response.json()\n",
    "                converted_to_string = json.dumps(repo_request)\n",
    "                f = open(filename,'a',encoding='utf-8')\n",
    "                f.write(converted_to_string + '\\n')\n",
    "                f.close()\n",
    "        \n",
    "                #saving status in log file:\n",
    "                f = open(log_file, 'a')\n",
    "                f.write(str(since)+'\\t'+page_url+'\\t'+str(page_response.status_code)+'\\n')\n",
    "                f.close()\n",
    "                \n",
    "        #update dates for the next search - moving {since} and {until} up /h/ hours\n",
    "        since = until \n",
    "        until += timedelta(hours = h)\n",
    "       \n",
    "    return filename"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "811ad8ff",
   "metadata": {},
   "source": [
    "## 2. extract_repo_list:\n",
    "This function reads the json file saved by find_repo and return a list of repositories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3b8c7229",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Step 2:  Extracting the list of repositories from the json file\n",
    "\n",
    "def extract_repo_list(filename):\n",
    "    '''\n",
    "    Second component of GRF function.\n",
    "    Reads .json file of repository data, returns a list of repositories.\n",
    "\n",
    "            Parameters:\n",
    "                    filename (str): Name of the .json file obtained from find_repo()\n",
    "                    \n",
    "            Returns:\n",
    "                    repo_list (list): List of all repositories found in the search queries \n",
    "                    \n",
    "    Check GRF documenation for more details.\n",
    "    Source code available at: /~https://github.com/thtbui/github-repository-finder\n",
    "    ''' \n",
    "    import json \n",
    "    f=open(filename,'r',encoding='utf-8')\n",
    "    content=f.readlines()\n",
    "    repo_list = []\n",
    "\n",
    "    for item in content:\n",
    "        jsonobj=json.loads(item)\n",
    "        repo_list.extend(jsonobj[\"items\"])\n",
    "    return repo_list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b820ddc",
   "metadata": {},
   "source": [
    "## 3. save_column:\n",
    "This function does two tasks:\n",
    "1. Extracting information that is available in the json file provided by GitHub API\n",
    "2. Looping through all the URLs of repositories to scrape the readme content\n",
    "\n",
    "It returns a list of relevant information (id, url, name, language, creation date, stars, forks, watch, readme)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a3341fc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Step 3: save_column: Extracting relevant information from the repository list\n",
    "\n",
    "def save_column(repo_list):\n",
    "    '''\n",
    "    Third component of GRF function.\n",
    "    Extracts relevant repository information, saves it in a list.\n",
    "\n",
    "            Parameters:\n",
    "                    repo_list (list): List obtained from extract_repo_list()\n",
    "                    \n",
    "            Returns:\n",
    "                    dt (list): List of repositories with only desired information \n",
    "    \n",
    "    Check GRF documenation for more details.\n",
    "    Source code available at: /~https://github.com/thtbui/github-repository-finder\n",
    "    ''' \n",
    "    dt =[]\n",
    "    \n",
    "    #extracting data directly from the list\n",
    "    for repo in repo_list:\n",
    "        id = repo.get(\"id\")\n",
    "        name =repo.get(\"name\")\n",
    "        url = repo.get(\"html_url\")\n",
    "        created = repo.get(\"created_at\")\n",
    "        stars = repo.get(\"stargazers_count\")\n",
    "        watch = repo.get(\"watchers_count\")\n",
    "        language = repo.get(\"language\")\n",
    "        forks = repo.get(\"forks_count\")\n",
    "        \n",
    "        #scraping readme by going through each repository url\n",
    "        import requests\n",
    "        from bs4 import BeautifulSoup\n",
    "        readme=requests.get(url)\n",
    "        soup =BeautifulSoup(readme.text,\"html.parser\")\n",
    "        readme=soup.find(class_=\"markdown-body entry-content container-lg\")\n",
    "        #if readme is empty, recording as blank (NaN)\n",
    "        if readme == None:\n",
    "            readme =(\"\")\n",
    "        else: \n",
    "            readme=soup.find(class_=\"markdown-body entry-content container-lg\").get_text().replace('\\n',\" \")\n",
    "\n",
    "        dt.append({\"id\": id,\n",
    "                     \"name\": name,\n",
    "                     \"url\": url,\n",
    "                     \"created\": created,\n",
    "                     \"stars\": stars,\n",
    "                     \"watch\": watch,\n",
    "                     \"language\": language,\n",
    "                     \"forks\": forks,\n",
    "                     \"readme\":readme})\n",
    "    return dt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a09e03d",
   "metadata": {},
   "source": [
    "## 4. save_dt:\n",
    "This function saves the data into csv file in user's working directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "dfc1a004",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Step 4: Saving dataset in .csv file and displaying the dataset:\n",
    "\n",
    "def save_dt(dt):\n",
    "    '''\n",
    "    Forth component of GRF function.\n",
    "    Saves desired data in a .csv file, returns a preview of the final dataset.\n",
    "\n",
    "            Parameters:\n",
    "                    dt (list): List obtained from save_column()\n",
    "                    \n",
    "            Returns:\n",
    "                    data.csv: Final dataset stored in working directory\n",
    "    \n",
    "    Check GRF documenation for more details.\n",
    "    Source code available at: /~https://github.com/thtbui/github-repository-finder\n",
    "    ''' \n",
    "    import csv\n",
    "    import pandas as pd \n",
    "\n",
    "    with open(\"dt.csv\", \"w\", encoding='utf-8', newline='') as csv_file:\n",
    "        writer = csv.writer(csv_file, delimiter = \";\")\n",
    "        writer.writerow([\"id\", \"name\", \"url\", \"language\", \"created\", \"stars\", \"watch\", \"forks\", \"readme\"])\n",
    "\n",
    "        for item in dt:\n",
    "            writer.writerow([item['id'], item['name'], item['url'], item['language'], item['created'], item['stars'], item['watch'], item['forks'], item['readme']])\n",
    "\n",
    "    data = pd.read_csv(\"dt.csv\", delimiter= \";\")\n",
    "    return data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4229368",
   "metadata": {},
   "source": [
    "# Applying the functions: \n",
    "\n",
    "We can apply 4 functions separately as below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "b29ca92f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No of repositories created between 2021-10-13 21:44:12.482296 and 2021-10-14 05:44:12.482296:452\n",
      "No of repositories created between 2021-10-14 05:44:12.482296 and 2021-10-14 13:44:12.482296:614\n",
      "No of repositories created between 2021-10-14 13:44:12.482296 and 2021-10-14 21:44:12.482296:765\n",
      "No of repositories created between 2021-10-14 21:44:12.482296 and 2021-10-15 05:44:12.482296:403\n",
      "No of repositories created between 2021-10-15 05:44:12.482296 and 2021-10-15 13:44:12.482296:546\n",
      "No of repositories created between 2021-10-15 13:44:12.482296 and 2021-10-15 21:44:12.482296:600\n",
      "No of repositories created between 2021-10-15 21:44:12.482296 and 2021-10-16 05:44:12.482296:361\n",
      "No of repositories created between 2021-10-16 05:44:12.482296 and 2021-10-16 13:44:12.482296:509\n",
      "No of repositories created between 2021-10-16 13:44:12.482296 and 2021-10-16 21:44:12.482296:536\n"
     ]
    }
   ],
   "source": [
    "# Step1: Search for repositories\n",
    "filename = find_repo(\"python\",3,8)\n",
    "\n",
    "# Step2: Extract repository lists\n",
    "repo_list = extract_repo_list(filename)\n",
    "\n",
    "# Step3: Saving relevant information\n",
    "dt = save_column(repo_list)\n",
    "\n",
    "# Step4: Saving the dataset\n",
    "data = save_dt(dt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3984ef91",
   "metadata": {},
   "source": [
    "Or we can also combine all sub-functions into a single main function, Github Repository Finder (grf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "41aa8658",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Wrapping up into one single function: Github Repository Finder (grf)\n",
    "def grf(term, d, h):\n",
    "    '''\n",
    "    Searches for repositories on GitHub, returns a .csv file of final dataset.\n",
    "\n",
    "            Parameters:\n",
    "                    term (str): A string defining the search term\n",
    "                    d (int): An integer defining the timeframe to search (in days)\n",
    "                    h (int): An integer defining the duration (in hours) of each query\n",
    "\n",
    "            Returns:\n",
    "                    data.csv: Final dataset stored in working directory\n",
    "                    .json file: A file of raw data stored in working directory\n",
    "                    log.txt: Log file recording all API calls and their status\n",
    "    \n",
    "    Check GRF documenation for more details on the function and its components.\n",
    "    Source code available at: /~https://github.com/thtbui/github-repository-finder\n",
    "    ''' \n",
    "    filename = find_repo(term, d, h)\n",
    "    repo_list = extract_repo_list(filename)\n",
    "    dt = save_column(repo_list)\n",
    "    data = save_dt(dt)\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d3b2be33",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No of repositories created between 2021-10-14 08:02:37.465760 and 2021-10-14 16:02:37.465760:673\n",
      "No of repositories created between 2021-10-14 16:02:37.465760 and 2021-10-15 00:02:37.465760:639\n",
      "No of repositories created between 2021-10-15 00:02:37.465760 and 2021-10-15 08:02:37.465760:418\n",
      "No of repositories created between 2021-10-15 08:02:37.465760 and 2021-10-15 16:02:37.465760:626\n",
      "No of repositories created between 2021-10-15 16:02:37.465760 and 2021-10-16 00:02:37.465760:503\n",
      "No of repositories created between 2021-10-16 00:02:37.465760 and 2021-10-16 08:02:37.465760:376\n",
      "No of repositories created between 2021-10-16 08:02:37.465760 and 2021-10-16 16:02:37.465760:575\n",
      "No of repositories created between 2021-10-16 16:02:37.465760 and 2021-10-17 00:02:37.465760:542\n",
      "No of repositories created between 2021-10-17 00:02:37.465760 and 2021-10-17 08:02:37.465760:261\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>name</th>\n",
       "      <th>url</th>\n",
       "      <th>language</th>\n",
       "      <th>created</th>\n",
       "      <th>stars</th>\n",
       "      <th>watch</th>\n",
       "      <th>forks</th>\n",
       "      <th>readme</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>417092394</td>\n",
       "      <td>Python</td>\n",
       "      <td>/~https://github.com/Srinath-sirpangi/Python</td>\n",
       "      <td>Python</td>\n",
       "      <td>2021-10-14T10:50:18Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>417052007</td>\n",
       "      <td>Python</td>\n",
       "      <td>/~https://github.com/sharmil01/Python</td>\n",
       "      <td>Python</td>\n",
       "      <td>2021-10-14T08:35:23Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>417083542</td>\n",
       "      <td>python</td>\n",
       "      <td>/~https://github.com/mricot/python</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-10-14T10:18:27Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>417098380</td>\n",
       "      <td>Python</td>\n",
       "      <td>/~https://github.com/rlcunha/Python</td>\n",
       "      <td>Python</td>\n",
       "      <td>2021-10-14T11:11:00Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>417077438</td>\n",
       "      <td>Python</td>\n",
       "      <td>/~https://github.com/FilipRusiecki/Python</td>\n",
       "      <td>Python</td>\n",
       "      <td>2021-10-14T09:57:45Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4608</th>\n",
       "      <td>418005510</td>\n",
       "      <td>privfiles</td>\n",
       "      <td>/~https://github.com/WardPearce/privfiles</td>\n",
       "      <td>Python</td>\n",
       "      <td>2021-10-17T03:06:34Z</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Privfiles - Encrypted file storage using Ferne...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4609</th>\n",
       "      <td>417991269</td>\n",
       "      <td>TerminalRPG</td>\n",
       "      <td>/~https://github.com/cordeirossauro/TerminalRPG</td>\n",
       "      <td>Python</td>\n",
       "      <td>2021-10-17T01:37:38Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>TerminalRPG Simple Python text-based RPG that ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4610</th>\n",
       "      <td>418005505</td>\n",
       "      <td>scrfd-opencv</td>\n",
       "      <td>/~https://github.com/hpc203/scrfd-opencv</td>\n",
       "      <td>Python</td>\n",
       "      <td>2021-10-17T03:06:33Z</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>scrfd-opencv 使用OpenCV部署SCRFD人脸检测，包含C++和Python两...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4611</th>\n",
       "      <td>418003595</td>\n",
       "      <td>misteriusxx</td>\n",
       "      <td>/~https://github.com/sllumenikmati21/misteriusxx</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-10-17T02:55:06Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>misteriusxx KUMPULAN CODE TERMUX LENGKAP   | S...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4612</th>\n",
       "      <td>418011701</td>\n",
       "      <td>Password-generator</td>\n",
       "      <td>/~https://github.com/Haneen-Heeba/Password-gener...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-10-17T03:47:49Z</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Password-generator Security is one of the most...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4613 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             id                name  \\\n",
       "0     417092394              Python   \n",
       "1     417052007              Python   \n",
       "2     417083542              python   \n",
       "3     417098380              Python   \n",
       "4     417077438              Python   \n",
       "...         ...                 ...   \n",
       "4608  418005510           privfiles   \n",
       "4609  417991269         TerminalRPG   \n",
       "4610  418005505        scrfd-opencv   \n",
       "4611  418003595         misteriusxx   \n",
       "4612  418011701  Password-generator   \n",
       "\n",
       "                                                    url language  \\\n",
       "0            /~https://github.com/Srinath-sirpangi/Python   Python   \n",
       "1                   /~https://github.com/sharmil01/Python   Python   \n",
       "2                      /~https://github.com/mricot/python      NaN   \n",
       "3                     /~https://github.com/rlcunha/Python   Python   \n",
       "4               /~https://github.com/FilipRusiecki/Python   Python   \n",
       "...                                                 ...      ...   \n",
       "4608            /~https://github.com/WardPearce/privfiles   Python   \n",
       "4609      /~https://github.com/cordeirossauro/TerminalRPG   Python   \n",
       "4610             /~https://github.com/hpc203/scrfd-opencv   Python   \n",
       "4611     /~https://github.com/sllumenikmati21/misteriusxx      NaN   \n",
       "4612  /~https://github.com/Haneen-Heeba/Password-gener...      NaN   \n",
       "\n",
       "                   created  stars  watch  forks  \\\n",
       "0     2021-10-14T10:50:18Z      0      0      0   \n",
       "1     2021-10-14T08:35:23Z      0      0      0   \n",
       "2     2021-10-14T10:18:27Z      0      0      0   \n",
       "3     2021-10-14T11:11:00Z      0      0      0   \n",
       "4     2021-10-14T09:57:45Z      0      0      0   \n",
       "...                    ...    ...    ...    ...   \n",
       "4608  2021-10-17T03:06:34Z      1      1      0   \n",
       "4609  2021-10-17T01:37:38Z      0      0      0   \n",
       "4610  2021-10-17T03:06:33Z      1      1      0   \n",
       "4611  2021-10-17T02:55:06Z      0      0      0   \n",
       "4612  2021-10-17T03:47:49Z      0      0      0   \n",
       "\n",
       "                                                 readme  \n",
       "0                                               Python   \n",
       "1                                                   NaN  \n",
       "2                                                   NaN  \n",
       "3                                                   NaN  \n",
       "4                                               Python   \n",
       "...                                                 ...  \n",
       "4608  Privfiles - Encrypted file storage using Ferne...  \n",
       "4609  TerminalRPG Simple Python text-based RPG that ...  \n",
       "4610  scrfd-opencv 使用OpenCV部署SCRFD人脸检测，包含C++和Python两...  \n",
       "4611  misteriusxx KUMPULAN CODE TERMUX LENGKAP   | S...  \n",
       "4612  Password-generator Security is one of the most...  \n",
       "\n",
       "[4613 rows x 9 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grf(\"python\", 3, 8)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}