03_data_science/07_Pandas数据类型.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6d81753b",
   "metadata": {},
   "source": [
    "| [03_data_science/07_Pandas数据类型.ipynb](/~https://github.com/shibing624/python-tutorial/blob/master/03_data_science/07_Pandas数据类型.ipynb)  | Pandas数据类型  |[Open In Colab](https://colab.research.google.com/github/shibing624/python-tutorial/blob/master/03_data_science/07_Pandas数据类型.ipynb) |\n",
    "\n",
    "# Pandas数据分析\n",
    "\n",
    "Pandas是Python的一个用于数据分析的库： http://pandas.pydata.org\n",
    "\n",
    "API速查：http://pandas.pydata.org/pandas-docs/stable/api.html\n",
    "\n",
    "基于NumPy,SciPy的功能，在其上补充了大量的数据操作（Data Manipulation）功能。\n",
    "\n",
    "统计、分组、排序、透视表自由转换，如果你已经很熟悉结构化数据库（RDBMS）与Excel的功能，就会知道Pandas有过之而无不及！"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66bef52d",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 为啥用Pandas?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0062339",
   "metadata": {},
   "source": [
    "普通的程序员看到一份数据会怎么做？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a125af99",
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_file = '../data/numpy/iris.data.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6ed065bd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.1,3.5,1.4,0.2,Iris-setosa\n",
      "\n",
      "4.9,3.0,1.4,0.2,Iris-setosa\n",
      "\n",
      "4.7,3.2,1.3,0.2,Iris-setosa\n",
      "\n",
      "4.6,3.1,1.5,0.2,Iris-setosa\n",
      "\n",
      "5.0,3.6,1.4,0.2,Iris-setosa\n",
      "\n",
      "5.4,3.9,1.7,0.4,Iris-setosa\n",
      "\n",
      "4.6,3.4,1.4,0.3,Iris-setosa\n",
      "\n",
      "5.0,3.4,1.5,0.2,Iris-setosa\n",
      "\n",
      "4.4,2.9,1.4,0.2,Iris-setosa\n",
      "\n",
      "4.9,3.1,1.5,0.1,Iris-setosa\n",
      "\n",
      "5.4,3.7,1.5,0.2,Iris-setosa\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(iris_file, 'r', encoding='utf-8') as f:\n",
    "    lines = f.readlines()\n",
    "\n",
    "for idx, line in enumerate(lines):\n",
    "    print(line)\n",
    "    if idx == 10:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21f5feb1",
   "metadata": {},
   "source": [
    "Pandas的意义就在于\n",
    "\n",
    "### 快速的识别结构化数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7bec99dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.3</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>6.3</td>\n",
       "      <td>2.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.9</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>6.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>6.2</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.4</td>\n",
       "      <td>2.3</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>5.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.1</td>\n",
       "      <td>1.8</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>150 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       0    1    2    3               4\n",
       "0    5.1  3.5  1.4  0.2     Iris-setosa\n",
       "1    4.9  3.0  1.4  0.2     Iris-setosa\n",
       "2    4.7  3.2  1.3  0.2     Iris-setosa\n",
       "3    4.6  3.1  1.5  0.2     Iris-setosa\n",
       "4    5.0  3.6  1.4  0.2     Iris-setosa\n",
       "..   ...  ...  ...  ...             ...\n",
       "145  6.7  3.0  5.2  2.3  Iris-virginica\n",
       "146  6.3  2.5  5.0  1.9  Iris-virginica\n",
       "147  6.5  3.0  5.2  2.0  Iris-virginica\n",
       "148  6.2  3.4  5.4  2.3  Iris-virginica\n",
       "149  5.9  3.0  5.1  1.8  Iris-virginica\n",
       "\n",
       "[150 rows x 5 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import scipy as sp\n",
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv(iris_file, header=None, encoding='utf-8')\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21c22815",
   "metadata": {},
   "source": [
    "### 快速的操作元数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9b9eb2a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.3</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>6.3</td>\n",
       "      <td>2.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.9</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>6.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>6.2</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.4</td>\n",
       "      <td>2.3</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>5.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.1</td>\n",
       "      <td>1.8</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>150 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     sepal_length  sepal_width  petal_length  petal_width           class\n",
       "0             5.1          3.5           1.4          0.2     Iris-setosa\n",
       "1             4.9          3.0           1.4          0.2     Iris-setosa\n",
       "2             4.7          3.2           1.3          0.2     Iris-setosa\n",
       "3             4.6          3.1           1.5          0.2     Iris-setosa\n",
       "4             5.0          3.6           1.4          0.2     Iris-setosa\n",
       "..            ...          ...           ...          ...             ...\n",
       "145           6.7          3.0           5.2          2.3  Iris-virginica\n",
       "146           6.3          2.5           5.0          1.9  Iris-virginica\n",
       "147           6.5          3.0           5.2          2.0  Iris-virginica\n",
       "148           6.2          3.4           5.4          2.3  Iris-virginica\n",
       "149           5.9          3.0           5.1          1.8  Iris-virginica\n",
       "\n",
       "[150 rows x 5 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cnames = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'class']\n",
    "data.columns = cnames\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d3ae297",
   "metadata": {},
   "source": [
    "### 快速过滤"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f592c3f9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>6.3</td>\n",
       "      <td>3.3</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>109</th>\n",
       "      <td>7.2</td>\n",
       "      <td>3.6</td>\n",
       "      <td>6.1</td>\n",
       "      <td>2.5</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.3</td>\n",
       "      <td>5.7</td>\n",
       "      <td>2.5</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     sepal_length  sepal_width  petal_length  petal_width           class\n",
       "100           6.3          3.3           6.0          2.5  Iris-virginica\n",
       "109           7.2          3.6           6.1          2.5  Iris-virginica\n",
       "144           6.7          3.3           5.7          2.5  Iris-virginica"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[data['petal_width'] == data.petal_width.max()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b12984c1",
   "metadata": {},
   "source": [
    "### 快速切片"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "aa6389b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>4.8</td>\n",
       "      <td>3.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>5.5</td>\n",
       "      <td>2.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>6.9</td>\n",
       "      <td>3.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     sepal_length  sepal_width\n",
       "0             5.1          3.5\n",
       "30            4.8          3.1\n",
       "60            5.0          2.0\n",
       "90            5.5          2.6\n",
       "120           6.9          3.2"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.iloc[::30, :2]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e3b41d3",
   "metadata": {},
   "source": [
    "### 快速统计"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5f46e3dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Iris-virginica     50\n",
       "Iris-setosa        50\n",
       "Iris-versicolor    50\n",
       "Name: class, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['class'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f481b6b3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sepal_length  Statistics:    7.9    4.3   5.84   0.83\n",
      "sepal_width   Statistics:    4.4    2.0   3.05   0.43\n",
      "petal_length  Statistics:    6.9    1.0   3.76   1.76\n",
      "petal_width   Statistics:    2.5    0.1    1.2   0.76\n"
     ]
    }
   ],
   "source": [
    "for x in range(4):\n",
    "    s = data.iloc[:,x]\n",
    "    print('{0:<12}'.format(s.name), \" Statistics: \",\n",
    "    '{0:>5}  {1:>5}  {2:>5}  {3:>5}'.format(s.max(), s.min(), round(s.mean(),2),round(s.std(),2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54c28edf",
   "metadata": {},
   "source": [
    "### 快速“MapReduce”"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9c1648d6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>class</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Iris-setosa</th>\n",
       "      <td>49.878745</td>\n",
       "      <td>49.695242</td>\n",
       "      <td>49.654909</td>\n",
       "      <td>45.810069</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iris-versicolor</th>\n",
       "      <td>49.815081</td>\n",
       "      <td>49.680665</td>\n",
       "      <td>49.694505</td>\n",
       "      <td>49.452305</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iris-virginica</th>\n",
       "      <td>49.772059</td>\n",
       "      <td>49.714500</td>\n",
       "      <td>49.761700</td>\n",
       "      <td>49.545918</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 sepal_length  sepal_width  petal_length  petal_width\n",
       "class                                                                \n",
       "Iris-setosa         49.878745    49.695242     49.654909    45.810069\n",
       "Iris-versicolor     49.815081    49.680665     49.694505    49.452305\n",
       "Iris-virginica      49.772059    49.714500     49.761700    49.545918"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slogs = lambda x:np.log(x)*x\n",
    "entpy = lambda x:np.exp((slogs(x.sum())-x.map(slogs).sum())/x.sum())\n",
    "data.groupby('class').agg(entpy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf43083c",
   "metadata": {},
   "source": [
    "# 1. Pandas(大熊猫)基础"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20ea5898",
   "metadata": {},
   "source": [
    "Pandas的重要数据类型\n",
    "\n",
    "* DataFrame(二维表)\n",
    "* Series(一维序列)\n",
    "* Index(行索引，行级元数据)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6caca792",
   "metadata": {},
   "source": [
    "### 1.1 Series：pandas的长枪(数据表中的一列或一行,观测向量,一维数组...)\n",
    "\n",
    "数据世界中对于任意一个个体的全面观测，或者对于任意一组个体某一属性的观测，全部可以抽象为Series的概念。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3b0f620",
   "metadata": {},
   "source": [
    "用值构建一个Series：\n",
    "\n",
    "由默认index和values组成。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "be916d06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0   -0.803673\n",
       "1    0.360522\n",
       "2   -2.659308\n",
       "3   -1.810764\n",
       "dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "series1 = pd.Series(np.random.randn(4))\n",
    "series1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fe4256a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.series.Series'>\n",
      "RangeIndex(start=0, stop=4, step=1)\n",
      "[-0.80367297  0.36052177 -2.65930787 -1.81076401]\n"
     ]
    }
   ],
   "source": [
    "print(type(series1))\n",
    "print(series1.index)\n",
    "print(series1.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "522fcf87",
   "metadata": {},
   "source": [
    "#### Series支持过滤的原理就如同NumPy："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "49337101",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1     True\n",
       "2    False\n",
       "3    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "series1 > 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f375294a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    0.360522\n",
       "dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "series1[series1 > 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12926ea9",
   "metadata": {},
   "source": [
    "#### 当然也支持Broadcasting："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a41f2062",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0   -1.607346\n",
       "1    0.721044\n",
       "2   -5.318616\n",
       "3   -3.621528\n",
       "dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "series1*2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "49429e28",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    4.196327\n",
       "1    5.360522\n",
       "2    2.340692\n",
       "3    3.189236\n",
       "dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "series1+5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55c17582",
   "metadata": {},
   "source": [
    "#### 以及Universal Function："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4b1e7a03",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0   -0.803673\n",
      "1    0.360522\n",
      "2   -2.659308\n",
      "3   -1.810764\n",
      "dtype: float64\n",
      "0    0.447682\n",
      "1    1.434077\n",
      "2    0.069997\n",
      "3    0.163529\n",
      "dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0    29.744792\n",
       "1    305.22327\n",
       "2     0.727155\n",
       "3     3.968833\n",
       "dtype: object"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(series1)\n",
    "print(np.exp(series1))\n",
    "\n",
    "#NumPy Universal Function\n",
    "f_np = np.frompyfunc(lambda x:np.exp(x*2 + 5), 1, 1)\n",
    "f_np(series1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ce74fdd",
   "metadata": {},
   "source": [
    "在序列上就使用行标，而不是创建一个2列的数据表，能够轻松辨别哪里是数据，哪里是元数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5c7b1123",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "norm_0   -0.803673\n",
      "norm_1    0.360522\n",
      "norm_2   -2.659308\n",
      "norm_3   -1.810764\n",
      "dtype: float64 <class 'pandas.core.series.Series'>\n",
      "Index(['norm_0', 'norm_1', 'norm_2', 'norm_3'], dtype='object')\n",
      "<class 'pandas.core.indexes.base.Index'>\n",
      "[-0.80367297  0.36052177 -2.65930787 -1.81076401]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "norm_0   -0.803673\n",
       "norm_1    0.360522\n",
       "norm_2   -2.659308\n",
       "norm_3   -1.810764\n",
       "dtype: float64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "series2 = pd.Series(series1.values, index=['norm_' + str(i) for i in range(4)])\n",
    "print(series2, type(series2))\n",
    "print(series2.index)\n",
    "print(type(series2.index))\n",
    "print(series2.values)\n",
    "series2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d003b4e",
   "metadata": {},
   "source": [
    "虽然行是有顺序的，但是仍然能够通过行级的index来访问到数据：\n",
    "\n",
    "（当然也不尽然像Ordered Dict，因为行索引甚至可以重复，不推荐重复的行索引不代表不能用）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "aa32f5b2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "norm_0   -0.803673\n",
       "norm_3   -1.810764\n",
       "dtype: float64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "series2[['norm_0','norm_3']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "0b85c46f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'norm_0' in series2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "5b02be67",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'norm_6' in series2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a989abbf",
   "metadata": {},
   "source": [
    "默认行索引就像行号一样："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "5397e826",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=4, step=1)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "series1.index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03089558",
   "metadata": {},
   "source": [
    "从Key不重复的Ordered Dict或者从Dict来定义Series就不需要担心行索引重复："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "2a1b6e08",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Japan', 'Korea', 'China'], dtype='object')\n",
      "['Tokyo' 'Seoul' 'Beijing']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Japan      Tokyo\n",
       "Korea      Seoul\n",
       "China    Beijing\n",
       "dtype: object"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s_dict = {\"Japan\":\"Tokyo\", \"Korea\":\"Seoul\", \"China\":\"Beijing\"}\n",
    "series3 = pd.Series(s_dict)\n",
    "\n",
    "print(series3.index)\n",
    "print(series3.values)\n",
    "series3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0625d8cc",
   "metadata": {},
   "source": [
    "与Dict区别一： 有序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "58c91360",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Japan          Tokyo\n",
       "China        Beijing\n",
       "Singapore        NaN\n",
       "Korea          Seoul\n",
       "dtype: object"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lst = [\"Japan\", \"China\", \"Singapore\", \"Korea\"]\n",
    "series4 = pd.Series(s_dict, index =lst)\n",
    "series4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "c3a138df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Tokyo' 'Beijing' nan 'Seoul']\n",
      "Index(['Japan', 'China', 'Singapore', 'Korea'], dtype='object')\n",
      "Japan        False\n",
      "China        False\n",
      "Singapore     True\n",
      "Korea        False\n",
      "dtype: bool\n",
      "Japan         True\n",
      "China         True\n",
      "Singapore    False\n",
      "Korea         True\n",
      "dtype: bool\n"
     ]
    }
   ],
   "source": [
    "print(series4.values)\n",
    "print(series4.index)\n",
    "print(series4.isnull())\n",
    "print(series4.notnull())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76c2ec72",
   "metadata": {},
   "source": [
    "与Dict区别二： index内值可以重复，尽管不推荐。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "44423078",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A   -0.803673\n",
       "B    0.360522\n",
       "B   -2.659308\n",
       "C   -1.810764\n",
       "dtype: float64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lst = ['A', 'B', 'B', 'C']\n",
    "series5 = pd.Series(series1.values, index=lst)\n",
    "series5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "419704e1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "B    0.360522\n",
       "B   -2.659308\n",
       "A   -0.803673\n",
       "dtype: float64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "series5[['B', 'A']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38c42a35",
   "metadata": {},
   "source": [
    "整个序列级别的元数据信息：name\n",
    "\n",
    "当数据序列以及index本身有了名字，就可以更方便的进行后续的数据关联啦！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "5db3ab3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "series4.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "e6c7b0f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "series4.index.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "98a3b25d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Nation\n",
       "Japan          Tokyo\n",
       "China        Beijing\n",
       "Singapore        NaN\n",
       "Korea          Seoul\n",
       "Name: Capital Series, dtype: object"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "series4.name = \"Capital Series\"\n",
    "series4.index.name = \"Nation\"\n",
    "series4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "be5fbee3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Capital Series</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nation</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Japan</th>\n",
       "      <td>Tokyo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>China</th>\n",
       "      <td>Beijing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Singapore</th>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Korea</th>\n",
       "      <td>Seoul</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Capital Series\n",
       "Nation                  \n",
       "Japan              Tokyo\n",
       "China            Beijing\n",
       "Singapore            NaN\n",
       "Korea              Seoul"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(series4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ae577ca",
   "metadata": {},
   "source": [
    "### 1.2 DataFrame：pandas的战锤(数据表，二维数组)\n",
    "\n",
    "Series的有序集合，就像R的DataFrame一样方便。\n",
    "\n",
    "仔细想想，绝大部分的数据形式都可以表现为DataFrame。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "822d1377",
   "metadata": {},
   "source": [
    "#### 从Numpy二维数组、从文件或者从数据库定义：数据虽好，勿忘列名"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "7e7e6ec6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nation</th>\n",
       "      <th>capital</th>\n",
       "      <th>GDP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Japan</td>\n",
       "      <td>Tokyo</td>\n",
       "      <td>4000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Korea</td>\n",
       "      <td>Seoul</td>\n",
       "      <td>1300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>Beijing</td>\n",
       "      <td>9100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  nation  capital   GDP\n",
       "0  Japan    Tokyo  4000\n",
       "1  Korea    Seoul  1300\n",
       "2  China  Beijing  9100"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_np = np.asarray([('Japan', 'Tokyo', 4000),\n",
    "                      ('Korea', 'Seoul', 1300),\n",
    "                      ('China', 'Beijing', 9100)])\n",
    "df1 = pd.DataFrame(data_np, columns=['nation','capital','GDP'])\n",
    "df1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe5e79ac",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### 等长的列数据保存在一个字典里（JSON）：很不幸，字典key是无序的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "6b7493ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nation</th>\n",
       "      <th>capital</th>\n",
       "      <th>GDP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Japan</td>\n",
       "      <td>Tokyo</td>\n",
       "      <td>4900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Korea</td>\n",
       "      <td>Seoul</td>\n",
       "      <td>1300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>Beijing</td>\n",
       "      <td>9100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  nation  capital   GDP\n",
       "0  Japan    Tokyo  4900\n",
       "1  Korea    Seoul  1300\n",
       "2  China  Beijing  9100"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_dict = {'nation': ['Japan', 'Korea', 'China'],\n",
    "             'capital': ['Tokyo', 'Seoul', 'Beijing'],\n",
    "             'GDP': [4900, 1300, 9100]}\n",
    "df2 = pd.DataFrame(data_dict)\n",
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fda0bc0",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### 从另一个DataFrame定义DataFrame：啊，强迫症犯了！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "69f87913",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nation</th>\n",
       "      <th>capital</th>\n",
       "      <th>GDP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Japan</td>\n",
       "      <td>Tokyo</td>\n",
       "      <td>4900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Korea</td>\n",
       "      <td>Seoul</td>\n",
       "      <td>1300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>Beijing</td>\n",
       "      <td>9100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  nation  capital   GDP\n",
       "0  Japan    Tokyo  4900\n",
       "1  Korea    Seoul  1300\n",
       "2  China  Beijing  9100"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df21 = pd.DataFrame(df2, columns=['nation', 'capital', 'GDP'])\n",
    "df21"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "152efa89",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nation</th>\n",
       "      <th>capital</th>\n",
       "      <th>GDP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>Beijing</td>\n",
       "      <td>9100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Japan</td>\n",
       "      <td>Tokyo</td>\n",
       "      <td>4900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Korea</td>\n",
       "      <td>Seoul</td>\n",
       "      <td>1300</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  nation  capital   GDP\n",
       "2  China  Beijing  9100\n",
       "0  Japan    Tokyo  4900\n",
       "1  Korea    Seoul  1300"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df22 = pd.DataFrame(df2, columns=['nation', 'capital', 'GDP'], index = [2, 0, 1])\n",
    "df22"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c12ec1fd",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### 从DataFrame中取出列？两种方法（与JavaScript完全一致！）\n",
    "\n",
    "* '.'的写法容易与其他预留关键字产生冲突\n",
    "* '[ ]'的写法最安全。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "152d4d24",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2    China\n",
      "0    Japan\n",
      "1    Korea\n",
      "Name: nation, dtype: object\n",
      "2    Beijing\n",
      "0      Tokyo\n",
      "1      Seoul\n",
      "Name: capital, dtype: object\n",
      "2    9100\n",
      "0    4900\n",
      "1    1300\n",
      "Name: GDP, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(df22.nation)\n",
    "print(df22.capital)\n",
    "print(df22['GDP'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "940123d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2    Beijing\n",
       "0      Tokyo\n",
       "1      Seoul\n",
       "Name: capital, dtype: object"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df22['capital']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0108c103",
   "metadata": {},
   "source": [
    "#### 从DataFrame中取出行？（至少）两种方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "0e22869c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nation</th>\n",
       "      <th>capital</th>\n",
       "      <th>GDP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>Beijing</td>\n",
       "      <td>9100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  nation  capital   GDP\n",
       "2  China  Beijing  9100"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df22[0:1] # 给出的实际是DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "be10956a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nation       China\n",
       "capital    Beijing\n",
       "GDP           9100\n",
       "Name: 2, dtype: object"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df22.iloc[0] # 通过对应Index给出行"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55781445",
   "metadata": {},
   "source": [
    "#### 像Numpy切片一样的终极招式：iloc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "5d7c93f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nation       China\n",
       "capital    Beijing\n",
       "GDP           9100\n",
       "Name: 2, dtype: object"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df22.iloc[0,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "3cc3d315",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2    China\n",
       "0    Japan\n",
       "1    Korea\n",
       "Name: nation, dtype: object"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df22.iloc[:,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6393d9ea",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### 听说你从Table地狱来，大熊猫笑了\n",
    "\n",
    "然而动态增加列无法用\".\"的方式完成，只能用\"[ ]\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "f8f8280f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nation</th>\n",
       "      <th>capital</th>\n",
       "      <th>GDP</th>\n",
       "      <th>population</th>\n",
       "      <th>region</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>Beijing</td>\n",
       "      <td>9100</td>\n",
       "      <td>1600</td>\n",
       "      <td>East_Asian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Japan</td>\n",
       "      <td>Tokyo</td>\n",
       "      <td>4900</td>\n",
       "      <td>130</td>\n",
       "      <td>East_Asian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Korea</td>\n",
       "      <td>Seoul</td>\n",
       "      <td>1300</td>\n",
       "      <td>55</td>\n",
       "      <td>East_Asian</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  nation  capital   GDP  population      region\n",
       "2  China  Beijing  9100        1600  East_Asian\n",
       "0  Japan    Tokyo  4900         130  East_Asian\n",
       "1  Korea    Seoul  1300          55  East_Asian"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df22['population'] = [1600, 130, 55]\n",
    "df22['region'] = 'East_Asian'\n",
    "df22"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13ad5f58",
   "metadata": {},
   "source": [
    "### 1.3 Index：pandas进行数据操作的鬼牌（行级索引）\n",
    "\n",
    "行级索引是\n",
    "\n",
    "* 元数据\n",
    "* 可能由真实数据产生，因此可以视作数据\n",
    "* 可以由多重索引也就是多个列组合而成\n",
    "* 可以和列名进行交换，也可以进行堆叠和展开，达到Excel透视表效果\n",
    "\n",
    "Index有四种...哦不，很多种写法，一些重要的索引类型包括\n",
    "\n",
    "* pd.Index（普通）\n",
    "* Int64Index（数值型索引）\n",
    "* MultiIndex（多重索引，在数据操纵中更详细描述）\n",
    "* DatetimeIndex（以时间格式作为索引）\n",
    "* PeriodIndex （含周期的时间格式作为索引）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2af26e87",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### 直接定义普通索引，长得就和普通的Series一样"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "9c18d887",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['a', 'b', 'c'], dtype='object')\n",
      "Index(['a', 'b', 'c'], dtype='object')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0    a\n",
       "1    b\n",
       "2    c\n",
       "dtype: object"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index_names = ['a','b','c']\n",
    "s = pd.Series(index_names)\n",
    "print(pd.Index(index_names))\n",
    "print(pd.Index(s))\n",
    "s"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "377a2f62",
   "metadata": {},
   "source": [
    "#### Immutable，牢记！\n",
    "pd.Index是不可变的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "3f88fb6b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['a' 'b' 'c']\n"
     ]
    }
   ],
   "source": [
    "index_names = ['a', 'b', 'c']\n",
    "index0 = pd.Index(index_names)\n",
    "print(index0.values)\n",
    "# index0[2] = 'd' # 改变值会出错"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adde2f04",
   "metadata": {},
   "source": [
    "#### 扔进去一个含有多元组的List，就有了MultiIndex\n",
    "MyltiIndex 也是immutable的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "2e5d00a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([['Row_1', 'Col_1'], ['Row_1', 'Col_2'], ['Row_1', 'Col_3'],\n",
       "       ['Row_1', 'Col_4'], ['Row_2', 'Col_1'], ['Row_2', 'Col_2'],\n",
       "       ['Row_2', 'Col_3'], ['Row_2', 'Col_4'], ['Row_3', 'Col_1'],\n",
       "       ['Row_3', 'Col_2'], ['Row_3', 'Col_3'], ['Row_3', 'Col_4'],\n",
       "       ['Row_4', 'Col_1'], ['Row_4', 'Col_2'], ['Row_4', 'Col_3'],\n",
       "       ['Row_4', 'Col_4']],\n",
       "      dtype='object', name=('index1', 'index2'))"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multi1 = pd.Index([['Row_' + str(x + 1), 'Col_' + str(y + 1)] for x in range(4) for y in range(4)])\n",
    "multi1.name = ('index1', 'index2')\n",
    "multi1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "bc10425f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultiIndex([('Row_1', 'Col_1'),\n",
       "            ('Row_1', 'Col_2'),\n",
       "            ('Row_1', 'Col_3'),\n",
       "            ('Row_1', 'Col_4'),\n",
       "            ('Row_2', 'Col_1'),\n",
       "            ('Row_2', 'Col_2'),\n",
       "            ('Row_2', 'Col_3'),\n",
       "            ('Row_2', 'Col_4'),\n",
       "            ('Row_3', 'Col_1'),\n",
       "            ('Row_3', 'Col_2'),\n",
       "            ('Row_3', 'Col_3'),\n",
       "            ('Row_3', 'Col_4'),\n",
       "            ('Row_4', 'Col_1'),\n",
       "            ('Row_4', 'Col_2'),\n",
       "            ('Row_4', 'Col_3'),\n",
       "            ('Row_4', 'Col_4')],\n",
       "           )"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multi2 = pd.Index([('Row_' + str(x + 1), 'Col_' + str(y + 1)) for x in range(4) for y in range(4)])\n",
    "multi2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "8093bf6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# multi2.name = ['index1', 'index2'] # 出错"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e356f679",
   "metadata": {},
   "source": [
    "#### 对于Series来说，如果拥有了多重Index，数据，变形！\n",
    "\n",
    "下列代码说明：\n",
    "\n",
    "* 二重MultiIndex的Series可以unstack()成DataFrame\n",
    "* DataFrame可以stack成拥有二重MultiIndex的Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "9691bb23",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row_1  Col_1     0\n",
       "       Col_2     1\n",
       "       Col_3     2\n",
       "       Col_4     3\n",
       "Row_2  Col_1     4\n",
       "       Col_2     5\n",
       "       Col_3     6\n",
       "       Col_4     7\n",
       "Row_3  Col_1     8\n",
       "       Col_2     9\n",
       "       Col_3    10\n",
       "       Col_4    11\n",
       "Row_4  Col_1    12\n",
       "       Col_2    13\n",
       "       Col_3    14\n",
       "       Col_4    15\n",
       "dtype: int64"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_for_multi1 = pd.Series(range(0, 16), index=multi2)\n",
    "data_for_multi1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "3ad0f03d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Col_1</th>\n",
       "      <th>Col_2</th>\n",
       "      <th>Col_3</th>\n",
       "      <th>Col_4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Row_1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Row_2</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Row_3</th>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Row_4</th>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Col_1  Col_2  Col_3  Col_4\n",
       "Row_1      0      1      2      3\n",
       "Row_2      4      5      6      7\n",
       "Row_3      8      9     10     11\n",
       "Row_4     12     13     14     15"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_for_multi1.unstack()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "b15ce9cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row_1  Col_1     0\n",
       "       Col_2     1\n",
       "       Col_3     2\n",
       "       Col_4     3\n",
       "Row_2  Col_1     4\n",
       "       Col_2     5\n",
       "       Col_3     6\n",
       "       Col_4     7\n",
       "Row_3  Col_1     8\n",
       "       Col_2     9\n",
       "       Col_3    10\n",
       "       Col_4    11\n",
       "Row_4  Col_1    12\n",
       "       Col_2    13\n",
       "       Col_3    14\n",
       "       Col_4    15\n",
       "dtype: int64"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_for_multi1.unstack().stack()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "823c8a60",
   "metadata": {},
   "source": [
    "我们来看一下非平衡数据的例子：\n",
    "\n",
    "Row_1,2,3,4和Col_1,2,3,4并不是全组合的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "c8e64834",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultiIndex([('Row_1', 'Col_1'),\n",
       "            ('Row_2', 'Col_1'),\n",
       "            ('Row_2', 'Col_2'),\n",
       "            ('Row_3', 'Col_1'),\n",
       "            ('Row_3', 'Col_2'),\n",
       "            ('Row_3', 'Col_3'),\n",
       "            ('Row_4', 'Col_1'),\n",
       "            ('Row_4', 'Col_2'),\n",
       "            ('Row_4', 'Col_3'),\n",
       "            ('Row_4', 'Col_4')],\n",
       "           )"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multi2 = pd.Index([('Row_' + str(x), 'Col_' + str(y + 1)) for x in range(5) for y in range(x)])\n",
    "multi2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "97654dd5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row_1  Col_1    0\n",
       "Row_2  Col_1    1\n",
       "       Col_2    2\n",
       "Row_3  Col_1    3\n",
       "       Col_2    4\n",
       "       Col_3    5\n",
       "Row_4  Col_1    6\n",
       "       Col_2    7\n",
       "       Col_3    8\n",
       "       Col_4    9\n",
       "dtype: int64"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_for_multi2 = pd.Series(np.arange(10), index=multi2)\n",
    "data_for_multi2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "6adb5029",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Col_1</th>\n",
       "      <th>Col_2</th>\n",
       "      <th>Col_3</th>\n",
       "      <th>Col_4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Row_1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Row_2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Row_3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Row_4</th>\n",
       "      <td>6.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Col_1  Col_2  Col_3  Col_4\n",
       "Row_1    0.0    NaN    NaN    NaN\n",
       "Row_2    1.0    2.0    NaN    NaN\n",
       "Row_3    3.0    4.0    5.0    NaN\n",
       "Row_4    6.0    7.0    8.0    9.0"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_for_multi2.unstack()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "28f4bb95",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row_1  Col_1    0.0\n",
       "Row_2  Col_1    1.0\n",
       "       Col_2    2.0\n",
       "Row_3  Col_1    3.0\n",
       "       Col_2    4.0\n",
       "       Col_3    5.0\n",
       "Row_4  Col_1    6.0\n",
       "       Col_2    7.0\n",
       "       Col_3    8.0\n",
       "       Col_4    9.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_for_multi2.unstack().stack()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d1f7181",
   "metadata": {},
   "source": [
    "#### DateTime标准库如此好用，你值得拥有"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "7c5117aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatetimeIndex(['2021-01-01', '2021-01-08', '2021-01-30'], dtype='datetime64[ns]', freq=None)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import datetime\n",
    "dates = [datetime.datetime(2021, 1, 1), datetime.datetime(2021, 1, 8), datetime.datetime(2021, 1, 30)]\n",
    "pd.DatetimeIndex(dates)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ead7175",
   "metadata": {},
   "source": [
    "#### 如果你不仅需要时间格式统一，时间频率也要统一的话"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "db958c1d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PeriodIndex(['2021-01', '2021-02', '2021-03', '2021-04'], dtype='period[M]', freq='M')"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "periodindex1 = pd.period_range('2021-01', '2021-04', freq='M')\n",
    "periodindex1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e00c4169",
   "metadata": {},
   "source": [
    "#### 月级精度和日级精度如何转换？\n",
    "\n",
    "有的公司统一以1号代表当月，有的公司统一以最后一天代表当月，转化起来很麻烦，可以asfreq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "64f2e41c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PeriodIndex(['2021-01-01', '2021-02-01', '2021-03-01', '2021-04-01'], dtype='period[D]', freq='D')"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "periodindex1.asfreq('D', how='start')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "d805ef08",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PeriodIndex(['2021-01-31', '2021-02-28', '2021-03-31', '2021-04-30'], dtype='period[D]', freq='D')"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "periodindex1.asfreq('D', how='end')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f7d561d",
   "metadata": {},
   "source": [
    "#### 最后的最后，我要真正把两种频率的时间精度匹配上？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "72ada0ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PeriodIndex(['2021-01-01', '2021-02-01', '2021-03-01'], dtype='period[D]', freq='D')"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "periodindex_mon = pd.period_range('2021-01', '2021-03', freq='M').asfreq('D', how='start')\n",
    "periodindex_day = pd.period_range('2021-01-01', '2021-03-01', freq='D')\n",
    "\n",
    "periodindex_mon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "bd2718af",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PeriodIndex(['2021-01-01', '2021-01-02', '2021-01-03', '2021-01-04',\n",
       "             '2021-01-05', '2021-01-06', '2021-01-07', '2021-01-08',\n",
       "             '2021-01-09', '2021-01-10', '2021-01-11', '2021-01-12',\n",
       "             '2021-01-13', '2021-01-14', '2021-01-15', '2021-01-16',\n",
       "             '2021-01-17', '2021-01-18', '2021-01-19', '2021-01-20',\n",
       "             '2021-01-21', '2021-01-22', '2021-01-23', '2021-01-24',\n",
       "             '2021-01-25', '2021-01-26', '2021-01-27', '2021-01-28',\n",
       "             '2021-01-29', '2021-01-30', '2021-01-31', '2021-02-01',\n",
       "             '2021-02-02', '2021-02-03', '2021-02-04', '2021-02-05',\n",
       "             '2021-02-06', '2021-02-07', '2021-02-08', '2021-02-09',\n",
       "             '2021-02-10', '2021-02-11', '2021-02-12', '2021-02-13',\n",
       "             '2021-02-14', '2021-02-15', '2021-02-16', '2021-02-17',\n",
       "             '2021-02-18', '2021-02-19', '2021-02-20', '2021-02-21',\n",
       "             '2021-02-22', '2021-02-23', '2021-02-24', '2021-02-25',\n",
       "             '2021-02-26', '2021-02-27', '2021-02-28', '2021-03-01'],\n",
       "            dtype='period[D]', freq='D')"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "periodindex_day"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba8c721c",
   "metadata": {},
   "source": [
    "#### 粗粒度数据＋reindex＋ffill/bfill"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "5692262a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2021-01-01    2021-01-01\n",
       "2021-01-02           NaT\n",
       "2021-01-03           NaT\n",
       "2021-01-04           NaT\n",
       "2021-01-05           NaT\n",
       "Freq: D, dtype: period[D]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_ts = pd.Series(periodindex_mon, index=periodindex_mon).reindex(periodindex_day)\n",
    "full_ts.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "6c275989",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2021-01-01    2021-01-01\n",
       "2021-01-02    2021-01-01\n",
       "2021-01-03    2021-01-01\n",
       "2021-01-04    2021-01-01\n",
       "2021-01-05    2021-01-01\n",
       "Freq: D, dtype: period[D]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_ts = pd.Series(periodindex_mon, index=periodindex_mon).reindex(periodindex_day, method='ffill')\n",
    "full_ts.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83bfefa1",
   "metadata": {},
   "source": [
    "#### 关于索引，方便的操作有？\n",
    "\n",
    "前面描述过了，索引有序，重复，但一定程度上又能通过key来访问，也就是说，某些集合操作都是可以支持的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "a940ccd8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['A', 'B', 'B', 'C', 'C', 'C', 'D', 'E', 'E', 'F'], dtype='object')\n",
      "Index(['A', 'B'], dtype='object')\n",
      "Index(['C'], dtype='object')\n",
      "Index(['A', 'B', 'B', 'C', 'C', 'D', 'E', 'E', 'F'], dtype='object')\n",
      "[False False False  True  True]\n",
      "Index(['A', 'B', 'C', 'C'], dtype='object')\n",
      "Index(['K', 'A', 'B', 'B', 'C', 'C'], dtype='object')\n",
      "Index(['B', 'C'], dtype='object')\n",
      "True True False\n",
      "False False True\n"
     ]
    }
   ],
   "source": [
    "index1 = pd.Index(['A', 'B', 'B', 'C', 'C'])\n",
    "index2 = pd.Index(['C', 'D', 'E', 'E', 'F'])\n",
    "index3 = pd.Index(['B', 'C', 'A'])\n",
    "print(index1.append(index2))\n",
    "print(index1.difference(index2))\n",
    "print(index1.intersection(index2))\n",
    "print(index1.union(index2)) # Support unique-value Index well\n",
    "print(index1.isin(index2))\n",
    "print(index1.delete(2))\n",
    "print(index1.insert(0, 'K')) # Not suggested\n",
    "print(index3.drop('A')) # Support unique-value Index well\n",
    "print(index1.is_monotonic, index2.is_monotonic, index3.is_monotonic)\n",
    "print(index1.is_unique, index2.is_unique, index3.is_unique)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4483bf4b",
   "metadata": {},
   "source": [
    "# 2. Pandas的I/O"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b6e24a9",
   "metadata": {},
   "source": [
    "老生常谈，从基础来看，我们仍然关心pandas对于与外部数据是如何交互的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33f3fe49",
   "metadata": {},
   "source": [
    "### 2.1 结构化数据输入输出"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57f8539f",
   "metadata": {},
   "source": [
    "* read_csv与to_csv 是一对输入输出的工具，read_csv直接返回pandas.DataFrame，而to_csv只要执行命令即可写文件\n",
    "  * read_table：功能类似\n",
    "  * read_fwf：操作fixed width file\n",
    "* read_excel与to_excel方便的与excel交互"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41f95c5c",
   "metadata": {},
   "source": [
    "还记得刚开始的例子吗？\n",
    "\n",
    "* header 表示数据中是否存在列名，如果在第0行就写就写0，并且开始读数据时跳过相应的行数，不存在可以写none\n",
    "* names 表示要用给定的列名来作为最终的列名\n",
    "* encoding 表示数据集的字符编码，通常而言一份数据为了方便的进行文件传输都以utf-8作为标准\n",
    "\n",
    "提问：下列例子中，header=4，names=cnames时，究竟会读到怎样的数据？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "b9502356",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cnames: ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'class']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>4.8</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.6</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Iris-versicolor</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>5.5</td>\n",
       "      <td>2.6</td>\n",
       "      <td>4.4</td>\n",
       "      <td>1.2</td>\n",
       "      <td>Iris-versicolor</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>6.9</td>\n",
       "      <td>3.2</td>\n",
       "      <td>5.7</td>\n",
       "      <td>2.3</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     sepal_length  sepal_width  petal_length  petal_width            class\n",
       "0             5.1          3.5           1.4          0.2      Iris-setosa\n",
       "30            4.8          3.1           1.6          0.2      Iris-setosa\n",
       "60            5.0          2.0           3.5          1.0  Iris-versicolor\n",
       "90            5.5          2.6           4.4          1.2  Iris-versicolor\n",
       "120           6.9          3.2           5.7          2.3   Iris-virginica"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('cnames:', cnames)\n",
    "irisdata = pd.read_csv(iris_file, header=None, names=cnames, encoding='utf-8')\n",
    "irisdata[::30]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c168348",
   "metadata": {},
   "source": [
    "希望了解全部参数的请移步API：\n",
    "\n",
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html#pandas.read_csv\n",
    "\n",
    "这里介绍一些常用的参数：\n",
    "\n",
    "读取处理：\n",
    "\n",
    "* skiprows：跳过一定的行数\n",
    "* nrows：仅读取一定的行数\n",
    "* skipfooter：尾部有固定的行数永不读取\n",
    "* skip_blank_lines：空行跳过\n",
    "\n",
    "内容处理：\n",
    "\n",
    "* sep/delimiter：分隔符很重要，常见的有逗号，空格和Tab('\\t')\n",
    "* na_values：指定应该被当作na_values的数值\n",
    "* thousands：处理数值类型时，每千位分隔符并不统一 (1.234.567,89或者1,234,567.89都可能)，此时要把字符串转化为数字需要指明千位分隔符\n",
    "\n",
    "收尾处理：\n",
    "\n",
    "* index_col：将真实的某列（列的数目，甚至列名）当作index\n",
    "* squeeze：仅读到一列时，不再保存为pandas.DataFrame而是pandas.Series\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c99cdf9d",
   "metadata": {},
   "source": [
    "### 2.1.x Excel ... ?\n",
    "\n",
    "对于存储着极为规整数据的Excel而言，其实是没必要一定用Excel来存，尽管Pandas也十分友好的提供了I/O接口。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "18f4b2f7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-66-edc26f1e2690>:2: FutureWarning: As the xlwt package is no longer maintained, the xlwt engine will be removed in a future version of pandas. This is the only engine in pandas that supports writing in the xls format. Install openpyxl and write to an xlsx file instead. You can set the option io.excel.xls.writer to 'xlwt' to silence this warning. While this option is deprecated and will also raise a warning, it can be globally set and the warning suppressed.\n",
      "  irisdata.to_excel(iris_excel_file, index=None)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>4.8</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.6</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Iris-versicolor</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>5.5</td>\n",
       "      <td>2.6</td>\n",
       "      <td>4.4</td>\n",
       "      <td>1.2</td>\n",
       "      <td>Iris-versicolor</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>6.9</td>\n",
       "      <td>3.2</td>\n",
       "      <td>5.7</td>\n",
       "      <td>2.3</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     sepal_length  sepal_width  petal_length  petal_width            class\n",
       "0             5.1          3.5           1.4          0.2      Iris-setosa\n",
       "30            4.8          3.1           1.6          0.2      Iris-setosa\n",
       "60            5.0          2.0           3.5          1.0  Iris-versicolor\n",
       "90            5.5          2.6           4.4          1.2  Iris-versicolor\n",
       "120           6.9          3.2           5.7          2.3   Iris-virginica"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iris_excel_file = 'irisdata.xls'\n",
    "irisdata.to_excel(iris_excel_file, index=None)\n",
    "irisdata_from_excel = pd.read_excel(iris_excel_file, header=0)\n",
    "irisdata_from_excel[::30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "c6c5f916",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.remove(iris_excel_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb4674f2",
   "metadata": {},
   "source": [
    "唯一重要的参数：sheetname=k，标志着一个excel的第k个sheet页将会被取出。（从0开始）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4e38685",
   "metadata": {},
   "source": [
    "### 2.2 半结构化数据"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08b772f8",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "JSON：网络传输中常用的一种数据格式。\n",
    "\n",
    "仔细看一下，实际上这就是我们平时收集到异源数据的风格是一致的：\n",
    "\n",
    "* 列名不能完全匹配\n",
    "* 关联键可能并不唯一\n",
    "* 元数据被保存在数据里"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "9e8e0b53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>job</th>\n",
       "      <th>sal</th>\n",
       "      <th>report</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Wang</td>\n",
       "      <td>VP</td>\n",
       "      <td>50000.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Zhang</td>\n",
       "      <td>Manager</td>\n",
       "      <td>NaN</td>\n",
       "      <td>VP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Li</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5000.0</td>\n",
       "      <td>Manager</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    name      job      sal   report\n",
       "0   Wang       VP  50000.0      NaN\n",
       "1  Zhang  Manager      NaN       VP\n",
       "2     Li      NaN   5000.0  Manager"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "json_data = [{'name': 'Wang', 'sal': 50000, 'job': 'VP'},\n",
    "             {'name': 'Zhang', 'job': 'Manager', 'report': 'VP'},\n",
    "             {'name': 'Li', 'sal': 5000, 'report': 'Manager'}]\n",
    "data_employee = pd.read_json(json.dumps(json_data))\n",
    "data_employee_ri = data_employee.reindex(columns=['name', 'job', 'sal', 'report'])\n",
    "data_employee_ri"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f1e424b",
   "metadata": {},
   "source": [
    "本节完。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6025865a",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}