handwritten-digit-classification-with-random-forests.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handwritten Digits Classification with Random Forests\n",
    "\n",
    "_Author: Michael Niemeyer_\n",
    "\n",
    "This work is an easy example of how the random forest classifier can be employed to learn a model to classify handwritten digits.\n",
    "\n",
    "The data set is taken from a kaggle.com challenge, and you can find the respective kernel here: https://www.kaggle.com/michaelnie/simple-random-forest-classification/\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Training Phase\n",
    "***\n",
    "Let's start with training the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from IPython.display import display\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Data Set\n",
    "\n",
    "First, we import the training set and have a look at the data structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>pixel0</th>\n",
       "      <th>pixel1</th>\n",
       "      <th>pixel2</th>\n",
       "      <th>pixel3</th>\n",
       "      <th>pixel4</th>\n",
       "      <th>pixel5</th>\n",
       "      <th>pixel6</th>\n",
       "      <th>pixel7</th>\n",
       "      <th>pixel8</th>\n",
       "      <th>...</th>\n",
       "      <th>pixel774</th>\n",
       "      <th>pixel775</th>\n",
       "      <th>pixel776</th>\n",
       "      <th>pixel777</th>\n",
       "      <th>pixel778</th>\n",
       "      <th>pixel779</th>\n",
       "      <th>pixel780</th>\n",
       "      <th>pixel781</th>\n",
       "      <th>pixel782</th>\n",
       "      <th>pixel783</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 785 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   label  pixel0  pixel1  pixel2  pixel3  pixel4  pixel5  pixel6  pixel7  \\\n",
       "0      1       0       0       0       0       0       0       0       0   \n",
       "1      0       0       0       0       0       0       0       0       0   \n",
       "2      1       0       0       0       0       0       0       0       0   \n",
       "\n",
       "   pixel8    ...     pixel774  pixel775  pixel776  pixel777  pixel778  \\\n",
       "0       0    ...            0         0         0         0         0   \n",
       "1       0    ...            0         0         0         0         0   \n",
       "2       0    ...            0         0         0         0         0   \n",
       "\n",
       "   pixel779  pixel780  pixel781  pixel782  pixel783  \n",
       "0         0         0         0         0         0  \n",
       "1         0         0         0         0         0  \n",
       "2         0         0         0         0         0  \n",
       "\n",
       "[3 rows x 785 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = pd.read_csv('train.csv')\n",
    "display(data.head(n=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the first column indicates the respective number, and the following 784 columns contain a intensity value ranging from 0 to 255. They describe a 28 x 28 image that contains the handwritten number.\n",
    "\n",
    "Let's have a look at an example image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAB2dJREFUeJzt3U+Ijnsfx3EjEWkoizFkw0pkYSOlyIZSEnayIGWhpKSs\nRNRQShZ2SFlIpET5syElfxZqSrKxmmRISZQRZp6NZ3F6nvt74Z65Z5zP67X9nGvmas55n2vxu/90\njYyMTALyTB7vGwDGh/ghlPghlPghlPghlPghlPghlPghlPgh1JQO/z4vJ4Sx1/Ur/5AnP4QSP4QS\nP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QS\nP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QS\nP4QSP4QSP4QSP4QSP4QSP4SaMt43wNj6/v17uQ8NDbX18x8/flzu69ata+vnt2N4eLjltn379vLa\nvr6+cp8/f/4f3dNE4skPocQPocQPocQPocQPocQPobpGRkY6+fs6+stSfPz4seW2e/fu8tqrV6+2\n9bub/vvp6upq6+e3o7q3pvtasGBBuff395d7d3d3uY+xX/qje/JDKPFDKPFDKPFDKPFDKPFDKPFD\nKG/p/Qu8f/++3E+ePNlya/ccfzw1nbVfuXKl3Ldt29Zye/XqVXntwMBAuV+8eLHc9+zZU+4TgSc/\nhBI/hBI/hBI/hBI/hBI/hBI/hHLOPwE8evSo3Lds2VLu7969G83bmTAWLlxY7suWLSv3FStWtNya\nzvmbzJw5s63rJwJPfgglfgglfgglfgglfgglfgglfgjlnL8DBgcHy33Dhg3lXn0u/6RJ4/vZ+GPp\n6dOn5X7+/Plyf/PmzWjezj+0+zqBicCTH0KJH0KJH0KJH0KJH0KJH0KJH0J1NX2/+ijr6C/rlO/f\nv5f7ypUry/3Zs2flPjw8XO6TJ4/d/8Pnzp1b7k3fQ3///v2WW09PT3ntjRs3yn3Tpk3lXv3dmv5m\nq1atKvdbt26V+4wZM8p9jP3SCz88+SGU+CGU+CGU+CGU+CGU+CGUt/SOgqGhoXJv9y25TcdS7byl\nt+njrx8+fFju06dP/+Pf/eHDh3I/dOhQubfzd1u0aFF57YULF8p9nI/yRoUnP4QSP4QSP4QSP4QS\nP4QSP4QSP4Tylt4OuHPnTrlv3ry53JteR9DOOX/Tx2MvX7683Jvezvzy5cuW2969e8trHzx4UO5N\nduzY0XI7depUee1f/hXc3tILtCZ+CCV+CCV+CCV+CCV+CCV+COWcfwJ4+/Ztuc+bN6/c2znnnzNn\nTrmfO3eu3O/du1fup0+f/u17+q/FixeX+5EjR8q96fUT/2LO+YHWxA+hxA+hxA+hxA+hxA+hxA+h\nnPP/BQ4fPlzux44d68yN/B9N//309va23M6cOVNeu379+nKfNm1auQdzzg+0Jn4IJX4IJX4IJX4I\nJX4IJX4I5Zz/L/D58+dynzVrVofu5H8NDw+X+/79+1tux48fL6+dMmXKH90TzvmBgvghlPghlPgh\nlPghlPghlKO+CWBwcLDc7969W+47d+5suXV3d5fXfvv2rdy/fPlS7u28pbe/v7+8tuljxWnJUR/Q\nmvghlPghlPghlPghlPghlPghlPdMjoKms/CDBw+W+9mzZ8t9aGio3Ldu3dpyO3nyZHntixcvyr3p\na66b7q16DUPT6xuc848tT34IJX4IJX4IJX4IJX4IJX4IJX4I5Zx/FDx//rzcr169Wu5fv34t99Wr\nV5d7X19fy23BggXltU370aNHy/3AgQPlXnny5Em5L1my5I9/Ns08+SGU+CGU+CGU+CGU+CGU+CGU\n+CGUc/5fVL33fM2aNeW1Tef4a9euLfebN2+W+9SpU8u9HT09PWP2s1esWDFmP5tmnvwQSvwQSvwQ\nSvwQSvwQSvwQylHfLzpx4kTLrenjqzdu3Fjuly9fLvexPMpr0vT14MPDw+U+ebLny0Tl3wyEEj+E\nEj+EEj+EEj+EEj+EEj+Ecs7/048fP8r9w4cPLbeurq7y2k2bNpV70zl+0729ffu23CsXL14s92vX\nrpV70zl+09+G8ePJD6HED6HED6HED6HED6HED6HED6Gc8/80MjJS7k3v2a8cPny43O/cuVPuTR/9\nff369d+9pY7p7u7+o42x58kPocQPocQPocQPocQPocQPocQPoZzz/9T0+fNLly5tuT158qS8dmBg\noK296TUI4/me+du3b5f78uXLW25z5swZ7dvhN3jyQyjxQyjxQyjxQyjxQyjxQyjxQ6iupjPkUdbR\nX9Ypr1+/LvdLly6V+7Fjx8r906dP5d7b29ty27dvX3ltk127dpX77Nmz2/r5jIlfeuGHJz+EEj+E\nEj+EEj+EEj+EEj+EctQH/z6O+oDWxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+h\nxA+hxA+hOv0V3eP3XdLAP3jyQyjxQyjxQyjxQyjxQyjxQyjxQyjxQyjxQyjxQyjxQyjxQyjxQyjx\nQyjxQyjxQyjxQyjxQyjxQyjxQyjxQyjxQyjxQ6j/AGpFWUIHG2TnAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0fa2f3a750>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ex_row = data.iloc[[10]].values.ravel()\n",
    "ex_img = np.array(ex_row[1:].reshape(28, 28), dtype=\"float32\")\n",
    "plt.imshow(ex_img, cmap = 'Greys')\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like an 8! Let's have a look what was indicated in the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(ex_row[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perfect!\n",
    "\n",
    "## 1.2 Restructuring the Data Set and Learning the Classifier\n",
    "\n",
    "We can now move to the interesting part as we have already understood the data we are analysing. First, we have to set up our data in such a way that the following random forest classifier can be applied. \n",
    "\n",
    "As so often, we have to construct an array X of the form [n_samples, k_features], and in our case, we have 784 features - each pixel is one feature.\n",
    "For the y values, we have to construct a [n_samples, n_outputs] array, and in our case we only have one output for each image (the number) resulting in a n-dim vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_matrix = data.as_matrix()\n",
    "X = data_matrix[:, 1:]\n",
    "y = data_matrix[:, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we train our classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rfc = RandomForestClassifier(n_estimators=10).fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing\n",
    "***\n",
    "\n",
    "We import the testing data and apply our classfier to the new data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(28000, 784)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_data = pd.read_csv('test.csv').as_matrix()\n",
    "display(test_data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2, 0, 9, ..., 3, 9, 2])"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predicted_numbers = rfc.predict(test_data)\n",
    "display(predicted_numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can save the results in a csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_data = pd.DataFrame({'ImageId': range(1, len(predicted_numbers)+1), 'Label': predicted_numbers})\n",
    "results_data.to_csv('results.csv', sep=',', index=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "***\n",
    "\n",
    "Done! The produced predictions achieved an accuracy score of 94% which is quite remarkable for such a simple classifier in my opinion.\n",
    "\n",
    "I hope you enjoyed the introduction to handwritten digit classification with random forests. If you have any questions, please just let me know!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}