GoogLeNet_Incepton-v1-PyTorch.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fQYTKcCS9LYB"
      },
      "source": [
        "# Implementing GoogLeNet and train it on CIFAR-10.\n",
        "\n",
        "## [Link to my Youtube Video Explaining this whole Notebook](https://www.youtube.com/watch?v=CZNYrkdDrmQ&list=PLxqBkZuBynVRyOJs4RWmB_fKlOVe5S8CR&index=12)\n",
        "\n",
        "[![Imgur](https://imgur.com/a5lSG5y.png)](https://youtu.be/AIK6Gi3NUhI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "### Original Problem\n",
        "\n",
        "Salient parts in the image can have extremely large variation in size. For instance, an image with a cat can be either of the following, as shown below. The area occupied by the cat is different in each image.\n",
        "\n",
        "![Imgur](https://imgur.com/uAWiQaC.png)\n",
        "\n",
        "- Because of this huge variation in the location of the information, choosing the right kernel size for the convolution operation becomes tough. A larger kernel is preferred for information that is distributed more globally, and a smaller kernel is preferred for information that is distributed more locally.\n",
        "\n",
        "- Very deep networks are prone to overfitting. It also hard to pass gradient updates through the entire network.\n",
        "\n",
        "- Naively stacking large convolution operations is computationally expensive.\n",
        "\n",
        "The Solution:\n",
        "\n",
        "Why not have filters with multiple sizes operate on the same level? The network essentially would get a bit “wider” rather than “deeper”. The authors designed the inception module to reflect the same.\n",
        "\n",
        "The below image is the “naive” inception module. It performs convolution on an input, with 3 different sizes of filters (1x1, 3x3, 5x5). Additionally, max pooling is also performed. The outputs are concatenated and sent to the next inception module.\n",
        "\n",
        "![Imgur](https://imgur.com/x5pl2QB.png)\n",
        "\n",
        "However,  deep neural networks are computationally expensive. To make it cheaper, the authors limit the number of input channels by adding an extra 1x1 convolution before the 3x3 and 5x5 convolutions. Though adding an extra operation may seem counterintuitive, 1x1 convolutions are far more cheaper than 5x5 convolutions, and the reduced number of input channels also help. Do note that however, the 1x1 convolution is introduced after the max pooling layer, rather than before.\n",
        "\n",
        "\n",
        "![Imgur](https://imgur.com/zqrJ2jo.png)\n",
        "\n",
        "Using the dimension reduced inception module, a neural network architecture was built. This was popularly known as GoogLeNet (Inception v1). The architecture is shown below:\n",
        "\n",
        "\n",
        "Proposed Architectural Details\n",
        "The paper proposes a new type of architecture – GoogLeNet or Inception v1. It is basically a convolutional neural network (CNN) which is 27 layers deep. Below is the model summary:\n",
        "\n",
        "\n",
        "![Imgur](https://imgur.com/8Br0HLk.png)\n",
        "\n",
        "Notice in the above image that there is a layer called inception layer. This is actually the main idea behind the paper’s approach.\n",
        "\n",
        "![Imgur](https://imgur.com/B281dyr.png)\n",
        "\n",
        "\n",
        "### (Inception Layer) is a combination of all those layers (namely, 1×1 Convolutional layer, 3×3 Convolutional layer, 5×5 Convolutional layer) with their output filter banks concatenated into a single output vector forming the input of the next stage.\n",
        "\n",
        "\n",
        "===========================================================================\n",
        "\n",
        "### Inception Module:\n",
        "\n",
        "The inception module is different from previous architectures such as AlexNet, ZF-Net. In this architecture, there is a fixed convolution size for each layer.\n",
        "\n",
        "In theory you can have as many filter sizes as possible, but the Inception Architecture is restricted to filter sizes 1 × 1, 3 × 3 and 5 × 5. The small filters help capture the local details and features whereas spread out features of higher abstraction are captured by the larger filters. A 3 × 3 max pooling is also added to the Inception architecture, because, why not? Historically, it has been found that pooling layers make the network work better, so might as well add them!\n",
        "\n",
        "#### In the Inception module 1×1, 3×3, 5×5 convolution and 3×3 max pooling performed in a parallel way at the input and the output of these are stacked together to generate final output. The idea behind is - that the convolution filters of different sizes will handle objects at multiple scale better.\n",
        "\n",
        "\n",
        "==========================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Imgur](https://imgur.com/U7HLYco.png)\n",
        "\n",
        "### The orange box is the stem, which has some preliminary convolutions. The purple boxes are auxiliary classifiers. The wide parts are the inception modules.\n",
        "\n",
        "\n",
        "GoogLeNet has 9 such inception modules stacked linearly. It is 22 layers deep (27, including the pooling layers). It uses global average pooling at the end of the last inception module.\n",
        "Needless to say, it is a pretty deep classifier. As with any very deep network, it is subject to the vanishing gradient problem.\n",
        "\n",
        "\n",
        "The network is 22 layers deep. The initial layers are simple convolution layers.\n",
        "\n",
        "After that there are multiple blocks of inception modules with layers of max pooling following some of the blocks. **The spatial dimensions get affected by these max pooling layers only.**\n",
        "\n",
        "Another interesting addition to the architecture is to change the second last fully-connected layer with an average pooling layer. This layer spatially averages the feature map, converting 7 × 7 × 1024 input to 1 × 1 × 1024. Doing this not only reduces the computation and the number of parameters, by a factor of 49, of the network but also improves the accuracy of the model, improving top-1 accuracy by 0.6%.\n",
        "\n",
        "This average pooling layer is finally followed by a normal fully-connected layer with 1000 neurons (and 1024 × 1000 parameters), for the 1000 ImageNet classes.\n",
        "\n",
        "### Two auxiliary classifier\n",
        "\n",
        "From the Original Paper\n",
        "\n",
        "![Imgur](https://imgur.com/DlXDcFO.png)\n",
        "\n",
        "Auxiliary Classifiers are type of architectural component that seek to improve the convergence of very deep networks. They are classifier heads we attach to layers before the end of the network. The motivation is to push useful gradients to the lower layers to make them immediately useful and improve the convergence during training by combatting the vanishing gradient problem.\n",
        "\n",
        "To prevent the middle part of the network from “dying out”, the authors introduced two auxiliary classifiers (The purple boxes in the image). They essentially applied softmax to the outputs of two of the inception modules, and computed an auxiliary loss over the same labels.\n",
        "\n",
        "\n",
        "So, the two auxiliary classifier layers are connected to the output of Inception (4a) and Inception (4d) layers.\n",
        "\n",
        "The architectural details of auxiliary classifiers as follows:\n",
        "\n",
        "\n",
        "• An average pooling layer with 5x5 filter size and stride 3, resulting in an\n",
        "4x4x512 output or the (4a), and 4x4x528 for the (4d) stage.\n",
        "\n",
        "• A 1x1 convolution with 128 filters for dimension reduction and rectified linear (ReLU) activation.\n",
        "\n",
        "• A fully connected layer with 1024 units and rectified linear activation.\n",
        "\n",
        "• A dropout layer with 70% ratio of dropped outputs.\n",
        "\n",
        "• A linear layer with softmax loss as the classifier (predicting the same\n",
        "1000 classes as the main classifier, but removed at inference time)\n",
        "\n",
        "\n",
        "### Loss Function\n",
        "\n",
        "The total loss function is a weighted sum of the auxiliary loss and the real loss. Weight value used in the paper was 0.3 for each auxiliary loss.\n",
        "\n",
        "\n",
        "### The total loss used by the inception net during training.\n",
        "\n",
        "### total_loss = real_loss + 0.3 * aux_loss_1 + 0.3 * aux_loss_2\n",
        "\n",
        "\n",
        "What’s novel in Inception v1?\n",
        "\n",
        "Instead of stacking convolutional layers, we stack modules or blocks, within which are convolutional layers.\n",
        "\n",
        "===========================================================================\n",
        "\n",
        "### Google Inception model:why there are 3 softmax ?\n",
        "\n",
        "When creating deeper networks, there arises a problem coined as the \"vanishing gradients\" problem.\n",
        "\n",
        "Intuitively, you can think about this problem lik this that - gradients carrying less and less information the deeper we go inside the network, which is of course a major concern, since we tune the network's parameters (weights) based solely on the gradients, using the \"back-prop\" algorithm.\n",
        "\n",
        "How did the developers of GoogLeNet handle this problem ? They recognized the fact that it's not only the features of the final layers that carry all the discriminatory information: intermediate features are also capable of discriminating different labels; and, most importantly, their values are more \"reliable\" since they are extracted from earlier layers in which the gradient carry more information. Building on this intuition, they added \"auxiliary classifiers\" in two intermediate layers. This is the reason for the \"early escape\" loss layers in the middle of the network\n",
        "\n",
        "\n",
        "The total loss is then a combination of these three loss layers.\n",
        "\n",
        "\n",
        "### total_loss = real_loss + 0.3 * aux_loss_1 + 0.3 * aux_loss_2\n",
        "\n",
        "\n",
        "I quote from the original article:\n",
        "\n",
        "\n",
        "These classifiers take the form of smaller convolutional networks put on top of the output of the Inception (4a) and (4d) modules. During training, their loss gets added to the total loss of the network with a discount weight (the losses of the auxiliary classi- fiers were weighted by 0.3). At inference time, these auxiliary networks are discarded.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Lh8f1QBFOP5O",
        "outputId": "434e8ca1-0a92-42f0-bc32-11b16347181e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "thwqjGZx6CbQ"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "from matplotlib import pyplot as plt\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "from torch.utils.data import Dataset, DataLoader, random_split\n",
        "from torchvision import transforms, datasets\n",
        "from torchsummary import summary\n",
        "\n",
        "device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ByGkcE0j6Cc2"
      },
      "source": [
        "## Building the initial Convolutional Block"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "kk3QXQjL6Cc9"
      },
      "outputs": [],
      "source": [
        "class ConvBlock(nn.Module):\n",
        "    \n",
        "    def __init__(self, in_channels, out_channels, kernel_size, stride, padding):\n",
        "        super(ConvBlock, self).__init__()\n",
        "        \n",
        "        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding)\n",
        "        self.bn = nn.BatchNorm2d(out_channels)\n",
        "        self .activation = nn.ReLU()\n",
        "        \n",
        "    def forward(self, x):\n",
        "        x = self.conv(x)\n",
        "        x = self.bn(x)\n",
        "        x = self .activation(x)\n",
        "        return x"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V6w3mimipHvC"
      },
      "source": [
        "## Building the Inception Block\n",
        "\n",
        "### “#3×3 reduce” and “#5×5 reduce”\n",
        "\n",
        "From Paper - In the above “#3 × 3 reduce” and “#5 × 5 reduce” stands for the number of 1 × 1 filters in the reduction layer used before the 3 × 3 and 5 × 5 convolutions. One can see the number of 1 × 1 filters in the projection layer after the built-in max-pooling in the “pool proj” column. All these reduction/ projection layers use rectified linear (ReLU) activation. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "uBqQN88P6CdH"
      },
      "outputs": [],
      "source": [
        "class Inception(nn.Module):\n",
        "    \n",
        "    def __init__(self, in_channels, num1x1, num3x3_reduce, num3x3, num5x5_reduce, num5x5, pool_proj):\n",
        "        super(Inception, self).__init__()\n",
        "        \n",
        "        # Four output channel for each parallel block of network\n",
        "        # Note, within Inception the individual blocks are running parallely\n",
        "        # NOT sequentially. \n",
        "        self.block1 = nn.Sequential(\n",
        "            ConvBlock(in_channels, num1x1, kernel_size=1, stride=1, padding=0)\n",
        "        )\n",
        "        \n",
        "        self.block2 = nn.Sequential(\n",
        "            ConvBlock(in_channels, num3x3_reduce, kernel_size=1, stride=1, padding=0),\n",
        "            ConvBlock(num3x3_reduce, num3x3, kernel_size=3, stride=1, padding=1)\n",
        "        )\n",
        "        \n",
        "        self.block3 = nn.Sequential(\n",
        "            ConvBlock(in_channels, num5x5_reduce, kernel_size=1, stride=1, padding=0),\n",
        "            ConvBlock(num5x5_reduce, num5x5, kernel_size=5, stride=1, padding=2)\n",
        "        )\n",
        "        \n",
        "        self.block4 = nn.Sequential(\n",
        "            nn.MaxPool2d(3, stride=1, padding=1, ceil_mode=True),\n",
        "            ConvBlock(in_channels, pool_proj, kernel_size=1, stride=1, padding=0)\n",
        "        )\n",
        "        \n",
        "    def forward(self, x):\n",
        "        # Note the different way this forward function \n",
        "        # calculates the output.\n",
        "        block1 = self.block1(x)\n",
        "        block2 = self.block2(x)\n",
        "        block3 = self.block3(x)\n",
        "        block4 = self.block4(x)\n",
        "        \n",
        "        return torch.cat([block1, block2, block3, block4], 1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "vIlaEO6b6CdT"
      },
      "outputs": [],
      "source": [
        "class Auxiliary(nn.Module):\n",
        "    \n",
        "    def __init__(self, in_channels, num_classes):\n",
        "        super(Auxiliary, self).__init__()\n",
        "        \n",
        "        self.pool = nn.AdaptiveAvgPool2d((4,4))\n",
        "        self.conv = nn.Conv2d(in_channels, 128, kernel_size=1, stride=1, padding=0)\n",
        "        self .activation = nn.ReLU()\n",
        "        self.fc1 = nn.Linear(2048, 1024)\n",
        "        self.dropout = nn.Dropout(0.7)\n",
        "        self.fc2 = nn.Linear(1024, num_classes)\n",
        "    \n",
        "    def forward(self, x):\n",
        "        out = self.pool(x)\n",
        "        \n",
        "        out = self.conv(out)\n",
        "        out = self .activation(out)\n",
        "    \n",
        "        out = torch.flatten(out, 1)\n",
        "        \n",
        "        out = self.fc1(out)\n",
        "        out = self .activation(out)\n",
        "        out = self.dropout(out)\n",
        "        \n",
        "        out = self.fc2(out)\n",
        "        \n",
        "        return out"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "az-c6q6i6Cde"
      },
      "outputs": [],
      "source": [
        "class GoogLeNet(nn.Module):\n",
        "    \n",
        "    def __init__(self, num_classes = 10):\n",
        "        super(GoogLeNet, self).__init__()\n",
        "      \n",
        "        self.conv1 = ConvBlock(3, 64, kernel_size=7, stride=2, padding=3)\n",
        "        self.pool1 = nn.MaxPool2d(3, stride=2, padding=0, ceil_mode=True)\n",
        "        self.conv2 = ConvBlock(64, 64, kernel_size=1, stride=1, padding=0)\n",
        "        self.conv3 = ConvBlock(64, 192, kernel_size=3, stride=1, padding=1)\n",
        "        self.pool3 = nn.MaxPool2d(3, stride=2, padding=0, ceil_mode=True)\n",
        "        \n",
        "        self.inception3A = Inception(in_channels=192, num1x1=64, num3x3_reduce=96, num3x3=128, num5x5_reduce=16, num5x5=32, pool_proj=32)\n",
        "        self.inception3B = Inception(in_channels=256, num1x1=128, num3x3_reduce=128, num3x3=192, num5x5_reduce=32, num5x5=96, pool_proj=64)\n",
        "        self.pool4 = nn.MaxPool2d(3, stride=2, padding=0, ceil_mode=True)\n",
        "        \n",
        "        self.inception4A = Inception(in_channels=480, num1x1=192, num3x3_reduce=96, num3x3=208, num5x5_reduce=16, num5x5=48, pool_proj=64)\n",
        "        self.inception4B = Inception(in_channels=512, num1x1=160, num3x3_reduce=112, num3x3=224, num5x5_reduce=24, num5x5=64, pool_proj=64)\n",
        "        self.inception4C = Inception(in_channels=512, num1x1=128, num3x3_reduce=128, num3x3=256, num5x5_reduce=24, num5x5=64, pool_proj=64)\n",
        "        self.inception4D = Inception(in_channels=512, num1x1=112, num3x3_reduce=144, num3x3=288, num5x5_reduce=32, num5x5=64, pool_proj=64)\n",
        "        self.inception4E = Inception(in_channels=528, num1x1=256, num3x3_reduce=160, num3x3=320, num5x5_reduce=32, num5x5=128, pool_proj=128)\n",
        "        self.pool5 = nn.MaxPool2d(3, stride=2, padding=0, ceil_mode=True)\n",
        "        \n",
        "        self.inception5A = Inception(in_channels=832, num1x1=256, num3x3_reduce=160, num3x3=320, num5x5_reduce=32, num5x5=128, pool_proj=128)\n",
        "        self.inception5B = Inception(in_channels=832, num1x1=384, num3x3_reduce=192, num3x3=384, num5x5_reduce=48, num5x5=128, pool_proj=128)\n",
        "        self.pool6 = nn.AdaptiveAvgPool2d((1,1))\n",
        "        \n",
        "        self.dropout = nn.Dropout(0.4)\n",
        "        self.fc = nn.Linear(1024, num_classes)\n",
        "        \n",
        "        self.aux4A = Auxiliary(512, num_classes) \n",
        "        self.aux4D = Auxiliary(528, num_classes)\n",
        "\n",
        "    def forward(self, x):\n",
        "        out = self.conv1(x)\n",
        "        out = self.pool1(out)\n",
        "        out = self.conv2(out)\n",
        "        out = self.conv3(out)\n",
        "        out = self.pool3(out)\n",
        "        out = self.inception3A(out)\n",
        "        out = self.inception3B(out)\n",
        "        out = self.pool4(out)\n",
        "        out = self.inception4A(out)\n",
        "  \n",
        "        aux1 = self.aux4A(out)\n",
        "        \n",
        "        out = self.inception4B(out)\n",
        "        out = self.inception4C(out)\n",
        "        out = self.inception4D(out)\n",
        "  \n",
        "        aux2 = self.aux4D(out)\n",
        "        \n",
        "        out = self.inception4E(out)\n",
        "        out = self.pool5(out)\n",
        "        out = self.inception5A(out)\n",
        "        out = self.inception5B(out)\n",
        "        out = self.pool6(out)\n",
        "        out = torch.flatten(out,1)\n",
        "        out = self.dropout(out)\n",
        "        out = self.fc(out)\n",
        "        \n",
        "        return out, aux1, aux2\n",
        "        "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "3eXvTYZNIIrU"
      },
      "outputs": [],
      "source": [
        "model = GoogLeNet()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w_Iwek5h6Cdx",
        "outputId": "278db859-3db6-414a-ba88-1744c51a83ac",
        "scrolled": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "----------------------------------------------------------------\n",
            "        Layer (type)               Output Shape         Param #\n",
            "================================================================\n",
            "            Conv2d-1           [-1, 64, 48, 48]           9,472\n",
            "       BatchNorm2d-2           [-1, 64, 48, 48]             128\n",
            "              ReLU-3           [-1, 64, 48, 48]               0\n",
            "         ConvBlock-4           [-1, 64, 48, 48]               0\n",
            "         MaxPool2d-5           [-1, 64, 24, 24]               0\n",
            "            Conv2d-6           [-1, 64, 24, 24]           4,160\n",
            "       BatchNorm2d-7           [-1, 64, 24, 24]             128\n",
            "              ReLU-8           [-1, 64, 24, 24]               0\n",
            "         ConvBlock-9           [-1, 64, 24, 24]               0\n",
            "           Conv2d-10          [-1, 192, 24, 24]         110,784\n",
            "      BatchNorm2d-11          [-1, 192, 24, 24]             384\n",
            "             ReLU-12          [-1, 192, 24, 24]               0\n",
            "        ConvBlock-13          [-1, 192, 24, 24]               0\n",
            "        MaxPool2d-14          [-1, 192, 12, 12]               0\n",
            "           Conv2d-15           [-1, 64, 12, 12]          12,352\n",
            "      BatchNorm2d-16           [-1, 64, 12, 12]             128\n",
            "             ReLU-17           [-1, 64, 12, 12]               0\n",
            "        ConvBlock-18           [-1, 64, 12, 12]               0\n",
            "           Conv2d-19           [-1, 96, 12, 12]          18,528\n",
            "      BatchNorm2d-20           [-1, 96, 12, 12]             192\n",
            "             ReLU-21           [-1, 96, 12, 12]               0\n",
            "        ConvBlock-22           [-1, 96, 12, 12]               0\n",
            "           Conv2d-23          [-1, 128, 12, 12]         110,720\n",
            "      BatchNorm2d-24          [-1, 128, 12, 12]             256\n",
            "             ReLU-25          [-1, 128, 12, 12]               0\n",
            "        ConvBlock-26          [-1, 128, 12, 12]               0\n",
            "           Conv2d-27           [-1, 16, 12, 12]           3,088\n",
            "      BatchNorm2d-28           [-1, 16, 12, 12]              32\n",
            "             ReLU-29           [-1, 16, 12, 12]               0\n",
            "        ConvBlock-30           [-1, 16, 12, 12]               0\n",
            "           Conv2d-31           [-1, 32, 12, 12]          12,832\n",
            "      BatchNorm2d-32           [-1, 32, 12, 12]              64\n",
            "             ReLU-33           [-1, 32, 12, 12]               0\n",
            "        ConvBlock-34           [-1, 32, 12, 12]               0\n",
            "        MaxPool2d-35          [-1, 192, 12, 12]               0\n",
            "           Conv2d-36           [-1, 32, 12, 12]           6,176\n",
            "      BatchNorm2d-37           [-1, 32, 12, 12]              64\n",
            "             ReLU-38           [-1, 32, 12, 12]               0\n",
            "        ConvBlock-39           [-1, 32, 12, 12]               0\n",
            "        Inception-40          [-1, 256, 12, 12]               0\n",
            "           Conv2d-41          [-1, 128, 12, 12]          32,896\n",
            "      BatchNorm2d-42          [-1, 128, 12, 12]             256\n",
            "             ReLU-43          [-1, 128, 12, 12]               0\n",
            "        ConvBlock-44          [-1, 128, 12, 12]               0\n",
            "           Conv2d-45          [-1, 128, 12, 12]          32,896\n",
            "      BatchNorm2d-46          [-1, 128, 12, 12]             256\n",
            "             ReLU-47          [-1, 128, 12, 12]               0\n",
            "        ConvBlock-48          [-1, 128, 12, 12]               0\n",
            "           Conv2d-49          [-1, 192, 12, 12]         221,376\n",
            "      BatchNorm2d-50          [-1, 192, 12, 12]             384\n",
            "             ReLU-51          [-1, 192, 12, 12]               0\n",
            "        ConvBlock-52          [-1, 192, 12, 12]               0\n",
            "           Conv2d-53           [-1, 32, 12, 12]           8,224\n",
            "      BatchNorm2d-54           [-1, 32, 12, 12]              64\n",
            "             ReLU-55           [-1, 32, 12, 12]               0\n",
            "        ConvBlock-56           [-1, 32, 12, 12]               0\n",
            "           Conv2d-57           [-1, 96, 12, 12]          76,896\n",
            "      BatchNorm2d-58           [-1, 96, 12, 12]             192\n",
            "             ReLU-59           [-1, 96, 12, 12]               0\n",
            "        ConvBlock-60           [-1, 96, 12, 12]               0\n",
            "        MaxPool2d-61          [-1, 256, 12, 12]               0\n",
            "           Conv2d-62           [-1, 64, 12, 12]          16,448\n",
            "      BatchNorm2d-63           [-1, 64, 12, 12]             128\n",
            "             ReLU-64           [-1, 64, 12, 12]               0\n",
            "        ConvBlock-65           [-1, 64, 12, 12]               0\n",
            "        Inception-66          [-1, 480, 12, 12]               0\n",
            "        MaxPool2d-67            [-1, 480, 6, 6]               0\n",
            "           Conv2d-68            [-1, 192, 6, 6]          92,352\n",
            "      BatchNorm2d-69            [-1, 192, 6, 6]             384\n",
            "             ReLU-70            [-1, 192, 6, 6]               0\n",
            "        ConvBlock-71            [-1, 192, 6, 6]               0\n",
            "           Conv2d-72             [-1, 96, 6, 6]          46,176\n",
            "      BatchNorm2d-73             [-1, 96, 6, 6]             192\n",
            "             ReLU-74             [-1, 96, 6, 6]               0\n",
            "        ConvBlock-75             [-1, 96, 6, 6]               0\n",
            "           Conv2d-76            [-1, 208, 6, 6]         179,920\n",
            "      BatchNorm2d-77            [-1, 208, 6, 6]             416\n",
            "             ReLU-78            [-1, 208, 6, 6]               0\n",
            "        ConvBlock-79            [-1, 208, 6, 6]               0\n",
            "           Conv2d-80             [-1, 16, 6, 6]           7,696\n",
            "      BatchNorm2d-81             [-1, 16, 6, 6]              32\n",
            "             ReLU-82             [-1, 16, 6, 6]               0\n",
            "        ConvBlock-83             [-1, 16, 6, 6]               0\n",
            "           Conv2d-84             [-1, 48, 6, 6]          19,248\n",
            "      BatchNorm2d-85             [-1, 48, 6, 6]              96\n",
            "             ReLU-86             [-1, 48, 6, 6]               0\n",
            "        ConvBlock-87             [-1, 48, 6, 6]               0\n",
            "        MaxPool2d-88            [-1, 480, 6, 6]               0\n",
            "           Conv2d-89             [-1, 64, 6, 6]          30,784\n",
            "      BatchNorm2d-90             [-1, 64, 6, 6]             128\n",
            "             ReLU-91             [-1, 64, 6, 6]               0\n",
            "        ConvBlock-92             [-1, 64, 6, 6]               0\n",
            "        Inception-93            [-1, 512, 6, 6]               0\n",
            "AdaptiveAvgPool2d-94            [-1, 512, 4, 4]               0\n",
            "           Conv2d-95            [-1, 128, 4, 4]          65,664\n",
            "             ReLU-96            [-1, 128, 4, 4]               0\n",
            "           Linear-97                 [-1, 1024]       2,098,176\n",
            "             ReLU-98                 [-1, 1024]               0\n",
            "          Dropout-99                 [-1, 1024]               0\n",
            "          Linear-100                   [-1, 10]          10,250\n",
            "       Auxiliary-101                   [-1, 10]               0\n",
            "          Conv2d-102            [-1, 160, 6, 6]          82,080\n",
            "     BatchNorm2d-103            [-1, 160, 6, 6]             320\n",
            "            ReLU-104            [-1, 160, 6, 6]               0\n",
            "       ConvBlock-105            [-1, 160, 6, 6]               0\n",
            "          Conv2d-106            [-1, 112, 6, 6]          57,456\n",
            "     BatchNorm2d-107            [-1, 112, 6, 6]             224\n",
            "            ReLU-108            [-1, 112, 6, 6]               0\n",
            "       ConvBlock-109            [-1, 112, 6, 6]               0\n",
            "          Conv2d-110            [-1, 224, 6, 6]         226,016\n",
            "     BatchNorm2d-111            [-1, 224, 6, 6]             448\n",
            "            ReLU-112            [-1, 224, 6, 6]               0\n",
            "       ConvBlock-113            [-1, 224, 6, 6]               0\n",
            "          Conv2d-114             [-1, 24, 6, 6]          12,312\n",
            "     BatchNorm2d-115             [-1, 24, 6, 6]              48\n",
            "            ReLU-116             [-1, 24, 6, 6]               0\n",
            "       ConvBlock-117             [-1, 24, 6, 6]               0\n",
            "          Conv2d-118             [-1, 64, 6, 6]          38,464\n",
            "     BatchNorm2d-119             [-1, 64, 6, 6]             128\n",
            "            ReLU-120             [-1, 64, 6, 6]               0\n",
            "       ConvBlock-121             [-1, 64, 6, 6]               0\n",
            "       MaxPool2d-122            [-1, 512, 6, 6]               0\n",
            "          Conv2d-123             [-1, 64, 6, 6]          32,832\n",
            "     BatchNorm2d-124             [-1, 64, 6, 6]             128\n",
            "            ReLU-125             [-1, 64, 6, 6]               0\n",
            "       ConvBlock-126             [-1, 64, 6, 6]               0\n",
            "       Inception-127            [-1, 512, 6, 6]               0\n",
            "          Conv2d-128            [-1, 128, 6, 6]          65,664\n",
            "     BatchNorm2d-129            [-1, 128, 6, 6]             256\n",
            "            ReLU-130            [-1, 128, 6, 6]               0\n",
            "       ConvBlock-131            [-1, 128, 6, 6]               0\n",
            "          Conv2d-132            [-1, 128, 6, 6]          65,664\n",
            "     BatchNorm2d-133            [-1, 128, 6, 6]             256\n",
            "            ReLU-134            [-1, 128, 6, 6]               0\n",
            "       ConvBlock-135            [-1, 128, 6, 6]               0\n",
            "          Conv2d-136            [-1, 256, 6, 6]         295,168\n",
            "     BatchNorm2d-137            [-1, 256, 6, 6]             512\n",
            "            ReLU-138            [-1, 256, 6, 6]               0\n",
            "       ConvBlock-139            [-1, 256, 6, 6]               0\n",
            "          Conv2d-140             [-1, 24, 6, 6]          12,312\n",
            "     BatchNorm2d-141             [-1, 24, 6, 6]              48\n",
            "            ReLU-142             [-1, 24, 6, 6]               0\n",
            "       ConvBlock-143             [-1, 24, 6, 6]               0\n",
            "          Conv2d-144             [-1, 64, 6, 6]          38,464\n",
            "     BatchNorm2d-145             [-1, 64, 6, 6]             128\n",
            "            ReLU-146             [-1, 64, 6, 6]               0\n",
            "       ConvBlock-147             [-1, 64, 6, 6]               0\n",
            "       MaxPool2d-148            [-1, 512, 6, 6]               0\n",
            "          Conv2d-149             [-1, 64, 6, 6]          32,832\n",
            "     BatchNorm2d-150             [-1, 64, 6, 6]             128\n",
            "            ReLU-151             [-1, 64, 6, 6]               0\n",
            "       ConvBlock-152             [-1, 64, 6, 6]               0\n",
            "       Inception-153            [-1, 512, 6, 6]               0\n",
            "          Conv2d-154            [-1, 112, 6, 6]          57,456\n",
            "     BatchNorm2d-155            [-1, 112, 6, 6]             224\n",
            "            ReLU-156            [-1, 112, 6, 6]               0\n",
            "       ConvBlock-157            [-1, 112, 6, 6]               0\n",
            "          Conv2d-158            [-1, 144, 6, 6]          73,872\n",
            "     BatchNorm2d-159            [-1, 144, 6, 6]             288\n",
            "            ReLU-160            [-1, 144, 6, 6]               0\n",
            "       ConvBlock-161            [-1, 144, 6, 6]               0\n",
            "          Conv2d-162            [-1, 288, 6, 6]         373,536\n",
            "     BatchNorm2d-163            [-1, 288, 6, 6]             576\n",
            "            ReLU-164            [-1, 288, 6, 6]               0\n",
            "       ConvBlock-165            [-1, 288, 6, 6]               0\n",
            "          Conv2d-166             [-1, 32, 6, 6]          16,416\n",
            "     BatchNorm2d-167             [-1, 32, 6, 6]              64\n",
            "            ReLU-168             [-1, 32, 6, 6]               0\n",
            "       ConvBlock-169             [-1, 32, 6, 6]               0\n",
            "          Conv2d-170             [-1, 64, 6, 6]          51,264\n",
            "     BatchNorm2d-171             [-1, 64, 6, 6]             128\n",
            "            ReLU-172             [-1, 64, 6, 6]               0\n",
            "       ConvBlock-173             [-1, 64, 6, 6]               0\n",
            "       MaxPool2d-174            [-1, 512, 6, 6]               0\n",
            "          Conv2d-175             [-1, 64, 6, 6]          32,832\n",
            "     BatchNorm2d-176             [-1, 64, 6, 6]             128\n",
            "            ReLU-177             [-1, 64, 6, 6]               0\n",
            "       ConvBlock-178             [-1, 64, 6, 6]               0\n",
            "       Inception-179            [-1, 528, 6, 6]               0\n",
            "AdaptiveAvgPool2d-180            [-1, 528, 4, 4]               0\n",
            "          Conv2d-181            [-1, 128, 4, 4]          67,712\n",
            "            ReLU-182            [-1, 128, 4, 4]               0\n",
            "          Linear-183                 [-1, 1024]       2,098,176\n",
            "            ReLU-184                 [-1, 1024]               0\n",
            "         Dropout-185                 [-1, 1024]               0\n",
            "          Linear-186                   [-1, 10]          10,250\n",
            "       Auxiliary-187                   [-1, 10]               0\n",
            "          Conv2d-188            [-1, 256, 6, 6]         135,424\n",
            "     BatchNorm2d-189            [-1, 256, 6, 6]             512\n",
            "            ReLU-190            [-1, 256, 6, 6]               0\n",
            "       ConvBlock-191            [-1, 256, 6, 6]               0\n",
            "          Conv2d-192            [-1, 160, 6, 6]          84,640\n",
            "     BatchNorm2d-193            [-1, 160, 6, 6]             320\n",
            "            ReLU-194            [-1, 160, 6, 6]               0\n",
            "       ConvBlock-195            [-1, 160, 6, 6]               0\n",
            "          Conv2d-196            [-1, 320, 6, 6]         461,120\n",
            "     BatchNorm2d-197            [-1, 320, 6, 6]             640\n",
            "            ReLU-198            [-1, 320, 6, 6]               0\n",
            "       ConvBlock-199            [-1, 320, 6, 6]               0\n",
            "          Conv2d-200             [-1, 32, 6, 6]          16,928\n",
            "     BatchNorm2d-201             [-1, 32, 6, 6]              64\n",
            "            ReLU-202             [-1, 32, 6, 6]               0\n",
            "       ConvBlock-203             [-1, 32, 6, 6]               0\n",
            "          Conv2d-204            [-1, 128, 6, 6]         102,528\n",
            "     BatchNorm2d-205            [-1, 128, 6, 6]             256\n",
            "            ReLU-206            [-1, 128, 6, 6]               0\n",
            "       ConvBlock-207            [-1, 128, 6, 6]               0\n",
            "       MaxPool2d-208            [-1, 528, 6, 6]               0\n",
            "          Conv2d-209            [-1, 128, 6, 6]          67,712\n",
            "     BatchNorm2d-210            [-1, 128, 6, 6]             256\n",
            "            ReLU-211            [-1, 128, 6, 6]               0\n",
            "       ConvBlock-212            [-1, 128, 6, 6]               0\n",
            "       Inception-213            [-1, 832, 6, 6]               0\n",
            "       MaxPool2d-214            [-1, 832, 3, 3]               0\n",
            "          Conv2d-215            [-1, 256, 3, 3]         213,248\n",
            "     BatchNorm2d-216            [-1, 256, 3, 3]             512\n",
            "            ReLU-217            [-1, 256, 3, 3]               0\n",
            "       ConvBlock-218            [-1, 256, 3, 3]               0\n",
            "          Conv2d-219            [-1, 160, 3, 3]         133,280\n",
            "     BatchNorm2d-220            [-1, 160, 3, 3]             320\n",
            "            ReLU-221            [-1, 160, 3, 3]               0\n",
            "       ConvBlock-222            [-1, 160, 3, 3]               0\n",
            "          Conv2d-223            [-1, 320, 3, 3]         461,120\n",
            "     BatchNorm2d-224            [-1, 320, 3, 3]             640\n",
            "            ReLU-225            [-1, 320, 3, 3]               0\n",
            "       ConvBlock-226            [-1, 320, 3, 3]               0\n",
            "          Conv2d-227             [-1, 32, 3, 3]          26,656\n",
            "     BatchNorm2d-228             [-1, 32, 3, 3]              64\n",
            "            ReLU-229             [-1, 32, 3, 3]               0\n",
            "       ConvBlock-230             [-1, 32, 3, 3]               0\n",
            "          Conv2d-231            [-1, 128, 3, 3]         102,528\n",
            "     BatchNorm2d-232            [-1, 128, 3, 3]             256\n",
            "            ReLU-233            [-1, 128, 3, 3]               0\n",
            "       ConvBlock-234            [-1, 128, 3, 3]               0\n",
            "       MaxPool2d-235            [-1, 832, 3, 3]               0\n",
            "          Conv2d-236            [-1, 128, 3, 3]         106,624\n",
            "     BatchNorm2d-237            [-1, 128, 3, 3]             256\n",
            "            ReLU-238            [-1, 128, 3, 3]               0\n",
            "       ConvBlock-239            [-1, 128, 3, 3]               0\n",
            "       Inception-240            [-1, 832, 3, 3]               0\n",
            "          Conv2d-241            [-1, 384, 3, 3]         319,872\n",
            "     BatchNorm2d-242            [-1, 384, 3, 3]             768\n",
            "            ReLU-243            [-1, 384, 3, 3]               0\n",
            "       ConvBlock-244            [-1, 384, 3, 3]               0\n",
            "          Conv2d-245            [-1, 192, 3, 3]         159,936\n",
            "     BatchNorm2d-246            [-1, 192, 3, 3]             384\n",
            "            ReLU-247            [-1, 192, 3, 3]               0\n",
            "       ConvBlock-248            [-1, 192, 3, 3]               0\n",
            "          Conv2d-249            [-1, 384, 3, 3]         663,936\n",
            "     BatchNorm2d-250            [-1, 384, 3, 3]             768\n",
            "            ReLU-251            [-1, 384, 3, 3]               0\n",
            "       ConvBlock-252            [-1, 384, 3, 3]               0\n",
            "          Conv2d-253             [-1, 48, 3, 3]          39,984\n",
            "     BatchNorm2d-254             [-1, 48, 3, 3]              96\n",
            "            ReLU-255             [-1, 48, 3, 3]               0\n",
            "       ConvBlock-256             [-1, 48, 3, 3]               0\n",
            "          Conv2d-257            [-1, 128, 3, 3]         153,728\n",
            "     BatchNorm2d-258            [-1, 128, 3, 3]             256\n",
            "            ReLU-259            [-1, 128, 3, 3]               0\n",
            "       ConvBlock-260            [-1, 128, 3, 3]               0\n",
            "       MaxPool2d-261            [-1, 832, 3, 3]               0\n",
            "          Conv2d-262            [-1, 128, 3, 3]         106,624\n",
            "     BatchNorm2d-263            [-1, 128, 3, 3]             256\n",
            "            ReLU-264            [-1, 128, 3, 3]               0\n",
            "       ConvBlock-265            [-1, 128, 3, 3]               0\n",
            "       Inception-266           [-1, 1024, 3, 3]               0\n",
            "AdaptiveAvgPool2d-267           [-1, 1024, 1, 1]               0\n",
            "         Dropout-268                 [-1, 1024]               0\n",
            "          Linear-269                   [-1, 10]          10,250\n",
            "================================================================\n",
            "Total params: 10,348,590\n",
            "Trainable params: 10,348,590\n",
            "Non-trainable params: 0\n",
            "----------------------------------------------------------------\n",
            "Input size (MB): 0.11\n",
            "Forward/backward pass size (MB): 22.05\n",
            "Params size (MB): 39.48\n",
            "Estimated Total Size (MB): 61.64\n",
            "----------------------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "model.to(device)\n",
        "summary(model, (3, 96, 96))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FvmCqZtRpHvG"
      },
      "source": [
        "## Loading CIFAR-10"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "R8lAVfrwpHvG"
      },
      "outputs": [],
      "source": [
        "def cifar_dataloader():\n",
        "    \n",
        "    transform = transforms.Compose([transforms.ToTensor(),\n",
        "                                    transforms.Normalize(mean=[0.5], std=[0.5])])\n",
        "            \n",
        "    # Input Data in Local Machine\n",
        "    # train_dataset = datasets.CIFAR10('../input_data', train=True, download=True, transform=transform)\n",
        "    # test_dataset = datasets.CIFAR10('../input_data', train=False, download=True, transform=transform)\n",
        "    \n",
        "    # Input Data in Google Drive\n",
        "    train_dataset = datasets.CIFAR10('/content/drive/MyDrive/All_Datasets/CIFAR10', train=True, download=True, transform=transform)\n",
        "    test_dataset = datasets.CIFAR10('/content/drive/MyDrive/All_Datasets/CIFAR10', train=False, download=True, transform=transform)\n",
        "\n",
        "    # Split dataset into training set and validation set.\n",
        "    train_dataset, val_dataset = random_split(train_dataset, (45000, 5000))\n",
        "    \n",
        "    print(\"Image shape of a random sample image : {}\".format(train_dataset[0][0].numpy().shape), end = '\\n\\n')\n",
        "    \n",
        "    print(\"Training Set:   {} images\".format(len(train_dataset)))\n",
        "    print(\"Validation Set:   {} images\".format(len(val_dataset)))\n",
        "    print(\"Test Set:       {} images\".format(len(test_dataset)))\n",
        "    \n",
        "    BATCH_SIZE = 128\n",
        "\n",
        "    # Generate dataloader\n",
        "    train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True)\n",
        "    val_loader = DataLoader(val_dataset, batch_size=BATCH_SIZE, shuffle=True)\n",
        "    test_loader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=True)\n",
        "    \n",
        "    return train_loader, val_loader, test_loader"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rsO079VsrW8h",
        "outputId": "918cc332-f06b-4298-a22c-ef27d8bd09ef"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Image Shape: (3, 96, 96)\n",
            "\n",
            "Training Set:   45000 samples\n",
            "Validation Set:   5000 samples\n",
            "Test Set:       10000 samples\n"
          ]
        }
      ],
      "source": [
        "train_loader, val_loader, test_loader = cifar_dataloader()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "8Iy7e81D6Cd9"
      },
      "outputs": [],
      "source": [
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = optim.Adam(model.parameters(), lr=0.001)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5zYLT95o6CeN"
      },
      "source": [
        "## c) Training the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "SBkLVkjl6CeQ"
      },
      "outputs": [],
      "source": [
        "def train_model():\n",
        "    EPOCHS = 15\n",
        "    train_samples_num = 45000\n",
        "    val_samples_num = 5000\n",
        "    train_epoch_loss_history, val_epoch_loss_history = [], []\n",
        "    \n",
        "    for epoch in range(EPOCHS):\n",
        "\n",
        "        train_running_loss = 0\n",
        "        correct_train = 0\n",
        "        \n",
        "        model.train().cuda()\n",
        "        \n",
        "        for inputs, labels in train_loader:\n",
        "            inputs, labels = inputs.to(device), labels.to(device)\n",
        "            \n",
        "            \"\"\" for every mini-batch during the training phase, we typically want to explicitly set the gradients \n",
        "            to zero before starting to do backpropragation \"\"\"\n",
        "            optimizer.zero_grad()\n",
        "            \n",
        "            # Start the forward pass\n",
        "            prediction0, aux_pred_1, aux_pred_2 = model(inputs)\n",
        "            \n",
        "            # Compute the loss.\n",
        "            real_loss = criterion(prediction0, labels)\n",
        "            aux_loss_1 = criterion(aux_pred_1, labels)\n",
        "            aux_loss_2 = criterion(aux_pred_2, labels)\n",
        "            \n",
        "            loss = real_loss + 0.3 * aux_loss_1 + 0.3 * aux_loss_2\n",
        "            \n",
        "            # do backpropagation and update weights with step()# Backward pass.\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "            \n",
        "            # Update the running corrects \n",
        "            _, predicted = torch.max(prediction0.data, 1)\n",
        "            \n",
        "            correct_train += (predicted == labels).float().sum().item()\n",
        "            \n",
        "            ''' Compute batch loss\n",
        "            multiply each average batch loss with batch-length. \n",
        "            The batch-length is inputs.size(0) which gives the number total images in each batch. \n",
        "            Essentially I am un-averaging the previously calculated Loss '''\n",
        "            train_running_loss += (loss.data.item() * inputs.shape[0])\n",
        "\n",
        "\n",
        "        train_epoch_loss = train_running_loss / train_samples_num\n",
        "        \n",
        "        train_epoch_loss_history.append(train_epoch_loss)\n",
        "        \n",
        "        train_acc =  correct_train / train_samples_num\n",
        "\n",
        "        val_loss = 0\n",
        "        correct_val = 0\n",
        "          \n",
        "        model.eval().cuda()\n",
        "        \n",
        "        with torch.no_grad():\n",
        "          for inputs, labels in val_loader:\n",
        "              inputs, labels = inputs.to(device), labels.to(device)\n",
        "\n",
        "              # Forward pass.\n",
        "              prediction0, aux_pred_1, aux_pred_2 = model(inputs)\n",
        "              \n",
        "              # Compute the loss.\n",
        "              real_loss = criterion(prediction0, labels)\n",
        "              aux_loss_1 = criterion(aux_pred_1, labels)\n",
        "              aux_loss_2 = criterion(aux_pred_2, labels)\n",
        "              \n",
        "              loss = real_loss + 0.3 * aux_loss_1 + 0.3 * aux_loss_2\n",
        "              \n",
        "              # Compute training accuracy.\n",
        "              _, predicted = torch.max(prediction0.data, 1)\n",
        "              correct_val += (predicted == labels).float().sum().item()\n",
        "              \n",
        "              # Compute batch loss.\n",
        "              val_loss += (loss.data.item() * inputs.shape[0])\n",
        "\n",
        "          val_loss /= val_samples_num\n",
        "          val_epoch_loss_history.append(val_loss)\n",
        "          val_acc =  correct_val / val_samples_num\n",
        "        \n",
        "        info = \"[For Epoch {}/{}]: train-loss = {:0.5f} | train-acc = {:0.3f} | val-loss = {:0.5f} | val-acc = {:0.3f}\"\n",
        "        \n",
        "        print(info.format(epoch+1, EPOCHS, train_epoch_loss, train_acc, val_loss, val_acc))\n",
        "        \n",
        "        torch.save(model.state_dict(), '/content/sample_data/checkpoint{}'.format(epoch + 1)) \n",
        "                      \n",
        "    torch.save(model.state_dict(), '/content/sample_data/googlenet_model')  \n",
        "        \n",
        "    return train_epoch_loss_history, val_epoch_loss_history"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 327
        },
        "id": "LxT7jmjb6Cea",
        "outputId": "382b0cdd-c982-4483-93e6-69303b8a9e93"
      },
      "outputs": [],
      "source": [
        "# train_epoch_loss_history, val_epoch_loss_history = train_model()\n",
        "train_epoch_loss_history, val_epoch_loss_history = train_model(model, train_loader, val_loader, criterion, optimizer)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "STo-5WyWQk9_"
      },
      "source": [
        "## Evaluating model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CUQcUcsccRMg"
      },
      "outputs": [],
      "source": [
        "model = GoogLeNet()\n",
        "model.load_state_dict(torch.load('/content/sample_data/googlenet_weights_gpu'))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wD4kZBywAScv"
      },
      "outputs": [],
      "source": [
        "num_test_samples = 10000\n",
        "correct = 0 \n",
        "\n",
        "model.eval().cuda()\n",
        "\n",
        "with  torch.no_grad():\n",
        "    for inputs, labels in test_loader:\n",
        "        inputs, labels = inputs.to(device), labels.to(device)\n",
        "        # Make predictions.\n",
        "        prediction, _, _ = model(inputs)\n",
        "\n",
        "        # Retrieve predictions indexes.\n",
        "        _, predicted_class = torch.max(prediction.data, 1)\n",
        "\n",
        "        # Compute number of correct predictions.\n",
        "        correct += (predicted_class == labels).float().sum().item()\n",
        "\n",
        "test_accuracy = correct / num_test_samples\n",
        "\n",
        "print('Test accuracy: {}'.format(test_accuracy))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Bonus Points - Understanding 1 x 1 Convolutions\n",
        "\n",
        "1x1 convolution sometimes referred to as “Network In Network”, was introduced in 2013 in this paper by Lin et al.\n",
        "\n",
        "A 1x1 convolution takes the element-wise product of all pixel values of an image. A convolution operation occurs between the image(input data) and the conv 1x1 filter, to create an output with the dimensions 1 x 1 x n (where ‘n’ is the number of filters).\n",
        "\n",
        "Although a 1x1 filter does not learn any spatial patterns that occur within the image, it does learn patterns across the depth(cross channel) of the input image. Therefore not only do 1x1 convolution filters provide a method for dimension reduction, but they also provide the additional benefit of enabling the network to learn more.\n",
        "\n",
        "The input channels are reduced by the 1x1 convolution, creating output with a reduced number of channels. This part of the Inception network is the bottleneck layer\n",
        "\n",
        "\n",
        "### Example of how 1x1 Conv reduces dimension / number of parameters For Inception Module in GoogleNet\n",
        "\n",
        "First WITHOUT using 1x1 Conv\n",
        "\n",
        "![Imgur](https://imgur.com/JQIKIk2.png)\n",
        "\n",
        "\n",
        "In the above diagram, the input is of size 28x28x192 convoluted with 5x5 filters of channel size 192 with 32 filters of 5x5.\n",
        "\n",
        "We got the output of size 28x28x32.\n",
        "\n",
        "How we got the output as 28x28x32? We can use 2 formulas for calculating the output size after applying convolution using a filter on the input image, they are:\n",
        "\n",
        "result image (Height) = ((original image height + 2 * padding value — filter size (height))/stride value) +1\n",
        "result image (width) = ((original image width + 2 * padding value — filter size (width))/stride value) +1\n",
        "\n",
        "Here,\n",
        "\n",
        "Original image height = 28\n",
        "Original image width = 28\n",
        "Padding value = 0\n",
        "Filter size = 1\n",
        "Stride value = 1\n",
        "\n",
        "result image (Height) = 28 + 2*0 -1 + 1 = 28\n",
        "result image (width) = 28 + 2*0 -1 + 1 = 28\n",
        "\n",
        "As we used 32 filters of 5x5, the output has 32 channels\n",
        "\n",
        "So finally, we get the output size of 28x28x32\n",
        "\n",
        "So, When we apply a 5x5 filter on 28x28x192, the number of operations to be performed is (28x28x32)+(5x5x192)= 120 million operations.\n",
        "\n",
        "\n",
        "Now using 1x1 Conv\n",
        "\n",
        "\n",
        "![Imgur](https://imgur.com/qAFxZcZ.png)\n",
        "\n",
        "\n",
        "In the above image where we first applied 16 filters of 1x1 and then 32 filters of 5x5.\n",
        "\n",
        "Here we just need 12.4 million operations\n",
        "\n",
        "\n",
        "((28x28x16x1x1x192)+(28x28x32x5x5x1)) to complete the same task.\n",
        "\n",
        "### To summarize the reasons to use 1x1 Convolution in Inception Module\n",
        "\n",
        "A filter of 3×3 cannot capture a feature in a 5×5 window. And a 5×5 filter has a hard time modeling a 3×3 filter. So we try to combine patterns on different scales. Because we are doing so much computation, and then joining all this information, we dont want to be very large hence we do a 1×1 before so that we reduce the dimension.\n",
        "\n",
        "\n",
        "The whole point of using 1x1 convolutions is to reduce the dimension along the direction of the number of channels while - keeping other dimensions same, not losing lots of useful information and not having to learn lots of new parameters to do this.\n",
        "\n",
        "\n",
        "This, \"Pointwise Convolution\", aka 1x1 Convolution (kernel size=1, stride=1), is currently used in the vast majority of well-known CNN architectures."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "machine_shape": "hm",
      "name": "MyWork_Final_GoogLeNet.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.10"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}