PyTorch-Techniques/53-Visualizing-Activation-Functions-PyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization of Activation FUnction\n",
    "\n",
    "### [Link to my Youtube Video Explaining this whole Notebook](https://www.youtube.com/watch?v=SWqDVkYnSYQ&list=PLxqBkZuBynVQqJTE9nRM2p7Tb12TDPlnq&index=9)\n",
    "\n",
    "[![Imgur](https://imgur.com/VxrKTU5.png)](https://www.youtube.com/watch?v=SWqDVkYnSYQ&list=PLxqBkZuBynVQqJTE9nRM2p7Tb12TDPlnq&index=9)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RELU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "from torch.autograd import Variable\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TORCH.LINSPACE\n",
    "\n",
    "![](assets/2022-02-22-22-16-19.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2000,)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.linspace(-10, 10, 2000) \n",
    "x.shape\n",
    "\n",
    "x = Variable(x)\n",
    "x_numpy = x.data.numpy() \n",
    "x_numpy.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define all the Activation Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch_relu = F.relu(x).data.numpy()\n",
    "\n",
    "torch_tanh = torch.tanh(x).data.numpy()\n",
    "\n",
    "torch_sigmoid = torch.sigmoid(x).data.numpy()\n",
    "\n",
    "torch_softplus = F.softplus(x).data.numpy()\n",
    "\n",
    "torch_leaky_relu = F.leaky_relu(x).data.numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RELU Plot\n",
    "\n",
    "The range in which\n",
    "this activation function operates is from 0 to infinity. It is mostly used\n",
    "between different hidden layers in a neural network model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fe538b55f70>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAD8CAYAAAAi2jCVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAd1UlEQVR4nO3de5yV4/7/8dfHdE4671AodhI20XwR294ROsuhkr1RROSYr0Pajpt8HSPtHe2IRCqKpIN0dBZFpVS7w24zQweRJJ2v3x/Xym+MmZpmHa57rfV+Ph7zaM2671nrPffczWfu+77uz2XOOURERKJmn9ABREREiqICJSIikaQCJSIikaQCJSIikaQCJSIikaQCJSIikbTHAmVmz5rZGjNbUOC5GmY2xcyWxv6tntyYIiKSbUpyBDUUaFXouduAac65hsC02OciIiIJYyW5UdfM6gPjnXNHxz5fAjR3zn1jZgcAM51zjZKaVEREskqZUn5dHefcN7HHq4A6xa1oZj2AHgCVK1duesQRR5TyLUVEJBJWrYL8fKhbF/bfP66XmjNnzrfOudpFLSttgfqFc86ZWbGHYc65wcBggNzcXDd79ux431JEREJ5/nno1g26dIHhw2Gf+Mbamdl/i1tW2ldeHTu1R+zfNaV8HRERSReTJkH37tCiBQwdGndx2pPSvvo4oGvscVfg9cTEERGRSPr4Y+jYEf7wB3j1VShfPulvWZJh5iOAD4FGZpZnZt2BB4EzzWwpcEbscxERyURLl0LbtlCnjj+K2m+/lLztHq9BOecuLGZRi0QE2LZtG3l5eWzevDkRLxd5FSpUoF69epQtWzZ0FBGRPVu1Clq29I/ffDPuQRF7I+5BEvHKy8ujSpUq1K9fHzMLHSepnHOsW7eOvLw8GjRoEDqOiMjubdgAbdrA6tUwYwYcfnhK3z54q6PNmzdTs2bNjC9OAGZGzZo1s+ZoUUTS2NatcN55MH8+jB4NJ5yQ8gjBj6CArChOu2TT9yoiaWrnTj+UfNo0P1qvdesgMYIfQYmISMTccguMGAEPPABdu+55/SRRgQJycnJo0qQJRx99NO3bt2f9+vW7Xf+ee+7h0Ucf/dVz3bp1Y/To0b96bt999010VBGR5OrXDx57DK67Dnr3DhpFBQqoWLEic+fOZcGCBdSoUYOBAweGjiQiknrDh8PNN0OnTvD44xD4koQKVCHNmjUjPz8fgOXLl9OqVSuaNm3KqaeeyuLFiwOnExFJkrfe8tedmjeHYcMgJyd0omgMkvhFr14wd25iX7NJE+jfv0Sr7tixg2nTptG9e3cAevTowaBBg2jYsCGzZs3i6quvZvr06YnNJyIS2pw5cP75cOSRMHYsVKgQOhEQtQIVyM8//0yTJk3Iz8+ncePGnHnmmWzcuJEPPviATp06/bLeli1bin2NokbnacSeiETe8uX+XqcaNXyXiKpVQyf6RbQKVAmPdBJt1zWoTZs20bJlSwYOHEi3bt2oVq0ac0t4RFezZk2+//77Xz7/7rvvqFWrVpISi4gkwJo10KoVbN/uu0QceGDoRL+ia1AFVKpUiQEDBtCvXz8qVapEgwYNeOWVVwDfBWLevHnFfm3z5s0ZNWoUW7duBWDo0KGcdtppKcktIrLXNm70/fXy82H8eGjcOHSi34jWEVQEHHfccRxzzDGMGDGC4cOH07NnT/r27cu2bdvo0qULxx57LAB9+/alf4Ejvry8PObMmUPTpk3JycnhsMMOY9CgQYG+CxGR3di2zXcm//RTf82pWbPQiYpUoinfE6WoCQsXLVpE4whW7mTKxu9ZRCLCOX/z7QsvwNNPw+WXB41jZnOcc7lFLdMpPhGRbNKnjy9O994bvDjtiQqUiEi2eOIJeOghuOoquOOO0Gn2KBIFKpWnGUPLpu9VRCJk1Ci48UY45xz45z+Dd4koieAFqkKFCqxbty4rfnHvmg+qQkRughORLDF9OlxyCZxyCrz0UiS6RJRE8FF89erVIy8vj7Vr14aOkhK7ZtQVEUmJuXP9UVPDhjBuHFSsGDpRiQUvUGXLltXssiIiybBypZ/LqWpV3yWievXQifZK8AIlIiJJ8O230LIlbN4M770HBx0UOtFeU4ESEck0P/0E7drBl1/ClClw1FGhE5WKCpSISCbZtg06d4ZPPoExY+CPfwydqNRUoEREMoVzcOWVMHEiPPWUHxyRxoIPMxcRkQS580547jm46y5/M26aU4ESEckETz4J99/v2xfdc0/oNAmhAiUiku5efRWuvdYPjHjqqbToElESKlAiIunsnXfgL3+BE0/07YzKZM7QAhUoEZF09fnncPbZUL++n3SwUqXQiRJKBUpEJB19+aXvElGpEkyeDDVrhk6UcJlzLCgiki2++w5atYIff4R334VDDgmdKClUoERE0snPP0P79rB8uT9yOuaY0ImSJq5TfGZ2o5ktNLMFZjbCzDSPhIhIsmzfDl26wIcfwosvQvPmoRMlVakLlJnVBa4Hcp1zRwM5QJdEBRMRkQKcg2uu8VNmPPEEdOoUOlHSxTtIogxQ0czKAJWAr+OPJCIiv3HvvTB4MPTpA9ddFzpNSpS6QDnn8oFHgS+Bb4AfnHNvFV7PzHqY2Wwzm50tkxKKiCTU4MG+O0TXrr5bRJaI5xRfdaAD0AA4EKhsZhcVXs85N9g5l+ucy61du3bpk4qIZKPXX4eePf2Q8qefzpguESURzym+M4D/OOfWOue2Aa8CJycmloiI8P77flBE06bwyitQtmzoRCkVT4H6EjjJzCqZmQEtgEWJiSUikuW++MIPJz/oIJgwASpXDp0o5eK5BjULGA18Cnwee63BCcolIpK98vL8jbjly/t7nbL08khcN+o65+4G7k5QFhERWb/eX29avx7efhsaNAidKBh1khARiYrNm6FDB1iyBCZNguOOC50oKBUoEZEo2LEDLrrIT5/x0kvQokXoRMGpm7mISGjOwQ03wJgx8NhjcOGFoRNFggqUiEhoDzwAAwfCzTfDjTeGThMZKlAiIiE99xzcfjv89a/w0EOh00SKCpSISCgTJsAVV8CZZ8Kzz8I++pVckLaGiEgIs2b5juTHHuuvPZUrFzpR5KhAiYik2pIl0LYtHHAATJwIVaqEThRJKlAiIqn09dfQsqU/nTd5MtSpEzpRZOk+KBGRVPnhB2jTBr79FmbOhN//PnSiSFOBEhFJhS1b4NxzYeFCGD8ecnNDJ4o8FSgRkWTbuRMuuQRmzIBhw/wpPtkjXYMSEUkm5/zNty+/DA8/DBdfHDpR2lCBEhFJpkcegQEDoFcv3ylCSkwFSkQkWV54AXr3hgsugH79smq69kRQgRIRSYbJk+Gyy+C00+D559UlohS0xUREEm32bDj/fDjqKHjtNT8zruw1FSgRkURatszf61Srlp90sGrV0InSlgqUiEiirF7th5Dv3OlP8R1wQOhEaU33QYmIJMKPP/ojp1WrYPp0aNQodKK0pwIlIhKvrVv9Nad58+D11+HEE0MnyggqUCIi8di504/WmzLFz+nUtm3oRBlD16BEROLRuzcMHw733w+XXho6TUZRgRIRKa3HHoNHH4VrroE+fUKnyTgqUCIipTFiBNx0k7/29MQT6hKRBCpQIiJ7a+pU6NoV/vQnePFFyMkJnSgjqUCJiOyNzz7z8zo1auRH7FWoEDpRxlKBEhEpqRUroHVrqF4d3nwTqlULnSijaZi5iEhJrF3ru0Rs3eonHqxbN3SijKcCJSKyJxs3+vub8vJg2jRo3Dh0oqwQ1yk+M6tmZqPNbLGZLTKzZokKJiISCdu2QefOMGcOjBoFJ58cOlHWiPcI6gngTedcRzMrB1RKQCYRkWhwDq64wnclHzwYzj47dKKsUuoCZWZVgT8B3QCcc1uBrYmJJSISAbff7icbvOceX6gkpeI5xdcAWAs8Z2afmdkzZla58Epm1sPMZpvZ7LVr18bxdiIiKfSPf8ADD0CPHnDXXaHTZKV4ClQZ4HjgKefcccBPwG2FV3LODXbO5TrncmvXrh3H24mIpMgrr8ANN0CHDjBwoLpEBBJPgcoD8pxzs2Kfj8YXLBGR9DVzJlx0ETRr5tsZldFg51BKXaCcc6uAr8xs16xcLYAvEpJKRCSE+fP9UdNhh8Ebb0DFiqETZbV4/zS4DhgeG8G3AlCveRFJT//9L7RqBVWq+C4RNWqETpT14ipQzrm5QG5iooiIBLJune8SsWkTvPceHHxw6ESCOkmISLbbtAnatYOVK+Gtt+Doo0MnkhgVKBHJXtu3wwUXwKxZMHq0nz5DIkMFSkSyk3Nw1VUwfjw8+SScd17oRFKIptsQkex0990wZAjccQf07Bk6jRRBBUpEss+gQXDffXDZZXDvvaHTSDFUoEQku7z2GlxzjR8Y8a9/qUtEhKlAiUj2eO89uPBCOOEEP3WGukREmgqUiGSHhQuhfXuoX993iaik2YGiTgVKRDLfV1/5LhEVKvguEbVqhU4kJaDjWxHJbN9/74vThg3wzjv+CErSggqUiGSun3/2s+AuW+aPnI49NnQi2QsqUCKSmXbsgL/8Bd5/H0aOhNNOC51I9pIKlIhkHuf8UPKxY+GJJ6Bz59CJpBQ0SEJEMk/fvv4ep9694frrQ6eRUlKBEpHM8swzcNddcMkl8MADodNIHFSgRCRzjBsHV17p53Z65hl1iUhzKlAikhk++MBPnXH88X7qjLJlQyeSOKlAiUj6W7TId4moVw8mTIB99w2dSBJABUpE0lt+vr8Rt0wZmDwZfve70IkkQTTMXETS1/r10Lo1fPcdvP02HHpo6ESSQCpQIpKeNm+Gc86BxYv9ab3jjw+dSBJMBUpE0s+OHXDxxf6oafhwOPPM0IkkCXQNSkTSi3PQq5cfqdevn29nJBlJBUpE0stDD8E//wn/+7/+QzKWCpSIpI+hQ6FPHz8r7iOPhE4jSaYCJSLpYdIkuPxyOOMMX6j20a+vTKefsIhE38cfQ8eOcMwxMGYMlCsXOpGkgAqUiETbv/8NbdtCnTowcSLst1/oRJIiKlAiEl2rVvnGr+C7ROy/f9g8klK6D0pEomnDBt8lYs0amDkTGjYMnUhSTAVKRKJnyxY47zxYsADeeAP+539CJ5IA4j7FZ2Y5ZvaZmY1PRCARyXI7d0K3bjBtGgwZ4hvBSlZKxDWoG4BFCXgdEcl2zsFNN8HIkfDgg35WXMlacRUoM6sHtAWeSUwcEclq/fpB//5w/fVw662h00hg8R5B9QduBXYWt4KZ9TCz2WY2e+3atXG+nYhkrBdfhFtugc6d4fHHNV27lL5AmVk7YI1zbs7u1nPODXbO5TrncmvXrl3atxORTPbWW3DppdC8OQwbpi4RAsR3BHUKcLaZrQRGAqeb2YsJSSUi2WPOHDj/fDjySBg7FsqXD51IIqLUBco518c5V885Vx/oAkx3zl2UsGQikvmWL4c2baBmTd9rr2rV0IkkQnQcLSJhrFnju0Ts2OG7RBx4YOhEEjEJuVHXOTcTmJmI1xKRLLBxoz9y+vprmD4dGjUKnUgiSJ0kRCS1tm7115zmzvXXnE46KXQiiSgVKBFJnZ07oXt3P2rvmWegXbvQiSTCdA1KRFKnTx9/v9N99/lCJbIbKlAikhr9+8PDD0PPnnD77aHTSBpQgRKR5Bs5Em680Xco/8c/1CVCSkQFSkSSa/p03/T11FNh+HDIyQmdSNKECpSIJM/cuXDOOXD44fD661ChQuhEkkZUoEQkOf7zHz8jbtWq8OabUL166ESSZjTMXEQSb+1a3yViyxY/8WC9eqETSRpSgRKRxPrpJ39/01dfwdSpvgmsSCmoQIlI4mzb5udzmj0bxoyBU04JnUjSmAqUiCSGc3DllTBxIgwa5AdHiMRBgyREJDHuvBOeew7uvtsXKpE4qUCJSPwGDoT774crrvAFSiQBVKBEJD6jR8N118HZZ8OTT6pLhCSMCpSIlN7bb8Nf/+qnzBgxAsrosrYkjgqUiJTO559Dhw5w6KHwxhtQqVLoRJJhVKBEZO99+SW0agWVK/suETVrhk4kGUjH4yKyd9at810ifvoJ3n0XDjkkdCLJUCpQIlJymzZB+/awYoWfFfcPfwidSDKYCpSIlMz27XDhhfDRR/Dyy/DnP4dOJBlOBUpE9sw5uPpqGDfOTzjYsWPoRJIFNEhCRPbs73+Hp5+Gv/0Nrr02dBrJEipQIrJ7//qXL1DdukHfvqHTSBZRgRKR4o0d60/ttWkDgwerS4SklAqUiBTt/ff9oIjcXD8oomzZ0Ikky6hAichvffGFH05+8MEwYYK/IVckxVSgROTX8vJ8l4jy5X2XiFq1QieSLKVh5iLy/33/PbRuDevXwzvvQIMGoRNJFlOBEhFv82bf/HXJEn/k1KRJ6ESS5Up9is/MDjKzGWb2hZktNLMbEhlMRFJoxw4/bca778KwYXD66aETicR1BLUduMk596mZVQHmmNkU59wXCcomIqngHFx/Pbz6Kjz+OHTpEjqRCBDHEZRz7hvn3Kexxz8Ci4C6iQomIinyf//nZ8K95Rbo1St0GpFfJGQUn5nVB44DZhWxrIeZzTaz2WvXrk3E24lIojz7LNxxB1x0ETz4YOg0Ir8Sd4Eys32BMUAv59yGwsudc4Odc7nOudzatWvH+3Yikijjx0OPHnDWWTBkCOyju04kWuLaI82sLL44DXfOvZqYSCKSdB99BJ07+5F6o0dDuXKhE4n8Rjyj+AwYAixyzj2WuEgiklSLF0PbtnDggb5LRJUqoROJFCmeI6hTgIuB081sbuyjTYJyiUgyfP217xJRpgxMngx16oROJFKsUg8zd869B6i1sUi6+OEH3yXi22/h7bfhsMNCJxLZLXWSEMkGW7bAuef6JrATJkDTpqETieyRCpRIptu5Ey65BGbMgBde8KP2RNKAxpWKZDLn4MYb/XxOjzzi73cSSRMqUCKZ7JFHYMAAX6Ruuil0GpG9ogIlkqmGDYPevX1vvUcf1XTtknZUoEQy0ZtvQvfu0KIFDB2qLhGSlrTXimSaTz6Bjh3h6KN9h/Ly5UMnEikVFSiRTLJ0qe8SUbs2TJoE++0XOpFIqalAiWSKVaugZUs/cm/yZNh//9CJROKi+6BEMsGPP0KbNrB6NUyfDocfHjqRSNxUoETS3datcN55MH8+jBsHJ54YOpFIQqhAiaSznTvh0kth6lR47jl/FCWSIXQNSiSd3XorvPSSn7a9W7fQaUQSSgVKJF316+c/rr0WbrstdBqRhFOBEklHL70EN9/s73fq319dIiQjqUCJpJupU/3pvD//2Xcnz8kJnUgkKVSgRNLJZ5/5eZ2OOALGjoUKFUInEkkaFSiRdLFihZ8Rt0YN3yWiWrXQiUSSSsPMRdLBmjW+S8S2bX7iwbp1QycSSToVKJGo27gR2rWD/HyYNg0aNw6dSCQlVKBEomzbNujUCebM8decmjULnUgkZVSgRKLKObj8cj+309NPQ/v2oROJpJQGSYhE1d/+5mfFvfdeX6hEsowKlEgUDRgADz4IV14Jd9wROo1IECpQIlHz8svQqxeccw4MHKguEZK1VKBEomTGDLj4Yjj5ZN/OSF0iJIupQIlExbx5/qjp97/38zpVrBg6kUhQKlAiUbBype8SUaWKH7VXo0boRCLBaZi5SGjffgutWsHPP8N778FBB4VOJBIJKlAiIW3a5O9vWrkSpkyBo44KnUgkMlSgRELZvh0uuAA+/hhGj4ZTTw2dSCRS4roGZWatzGyJmS0zM03pKVJSzsFVV8H48X4o+bnnhk4kEjmlLlBmlgMMBFoDRwIXmtmRiQomktHuvhuGDIE77/SFSkR+I55TfCcAy5xzKwDMbCTQAfgiEcGKtHKlv7O+KM7t/mt3tzwTlkUtj77H4pft2AELF0L37vD3vxe/nkiWi6dA1QW+KvB5HnBi4ZXMrAfQA+Dggw+O4+3w/7E3bCh++Z7uuN/d8qgsM8uM7yOeZVHLk4zvo1076NtXXSJEdiPpgyScc4OBwQC5ubl7+PN4Dw47DD78MBGxREQk4uIZJJEPFLxho17sORERkbjFU6A+ARqaWQMzKwd0AcYlJpaIiGS7Up/ic85tN7NrgclADvCsc25hwpKJiEhWi+salHNuIjAxQVlERER+oWaxIiISSSpQIiISSSpQIiISSSpQIiISSSpQIiISSSpQIiISSSpQIiISSSpQIiISSSpQIiISSSpQIiISSSpQIiISSSpQIiISSSpQIiISSSpQIiISSeZcfLOw79Wbma0F/puAl6oFfJuA10kV5U2+dMucbnkh/TIrb/IlIvMhzrnaRS1IaYFKFDOb7ZzLDZ2jpJQ3+dItc7rlhfTLrLzJl+zMOsUnIiKRpAIlIiKRlK4FanDoAHtJeZMv3TKnW15Iv8zKm3xJzZyW16BERCTzpesRlIiIZDgVKBERiaRIFigz62RmC81sp5nlFlrWx8yWmdkSM2tZzNc3MLNZsfVGmVm51CT/5f1Hmdnc2MdKM5tbzHorzezz2HqzU5mxUI57zCy/QOY2xazXKrbdl5nZbanOWSjLI2a22Mzmm9lrZlatmPWCbuM9bTMzKx/bX5bF9tn6qc5YIMtBZjbDzL6I/f+7oYh1mpvZDwX2lbtCZC2Uabc/Y/MGxLbxfDM7PkTOWJZGBbbdXDPbYGa9Cq0TfBub2bNmtsbMFhR4roaZTTGzpbF/qxfztV1j6yw1s65xBXHORe4DaAw0AmYCuQWePxKYB5QHGgDLgZwivv5loEvs8SCgZ8DvpR9wVzHLVgK1IrC97wFu3sM6ObHtfShQLvZzODJg5rOAMrHHDwEPRW0bl2SbAVcDg2KPuwCjAm7TA4DjY4+rAP8uIm9zYHyojKX5GQNtgEmAAScBs0JnLrB/rMLfqBqpbQz8CTgeWFDguYeB22KPbyvq/xxQA1gR+7d67HH10uaI5BGUc26Rc25JEYs6ACOdc1ucc/8BlgEnFFzBzAw4HRgde+p54Jwkxi1WLEtnYESI90+wE4BlzrkVzrmtwEj8zyMI59xbzrntsU8/AuqFyrIbJdlmHfD7KPh9tkVsv0k559w3zrlPY49/BBYBdUNkSbAOwDDnfQRUM7MDQocCWgDLnXOJ6K6TUM65d4DvCj1dcF8t7vdqS2CKc+4759z3wBSgVWlzRLJA7UZd4KsCn+fx2/9ANYH1BX55FbVOqpwKrHbOLS1muQPeMrM5ZtYjhbmKcm3s9MezxRy6l2Tbh3IZ/i/kooTcxiXZZr+sE9tnf8Dvw0HFTjUeB8wqYnEzM5tnZpPM7KjUJivSnn7GUd13u1D8H69R28YAdZxz38QerwLqFLFOQrd1mdJ+YbzMbCqwfxGLbnfOvZ7qPHurhPkvZPdHT390zuWb2e+AKWa2OPaXS8LtLi/wFHAf/j/6ffjTkpclI8feKMk2NrPbge3A8GJeJmXbOFOY2b7AGKCXc25DocWf4k9JbYxdqxwLNExxxMLS7mccuy5+NtCniMVR3Ma/4pxzZpb0e5SCFSjn3Bml+LJ84KACn9eLPVfQOvwhfJnYX6RFrRO3PeU3szLAeUDT3bxGfuzfNWb2Gv6UUFL+Y5V0e5vZ08D4IhaVZNsnVAm2cTegHdDCxU6AF/EaKdvGRSjJNtu1Tl5sn6mK34eDMLOy+OI03Dn3auHlBQuWc26imT1pZrWcc8GanJbgZ5zyfbcEWgOfOudWF14QxW0cs9rMDnDOfRM7RbqmiHXy8dfQdqmHH0tQKul2im8c0CU28qkB/q+KjwuuEPtFNQPoGHuqKxDiiOwMYLFzLq+ohWZW2cyq7HqMv+i/oKh1k63Q+fhzi8nxCdDQ/AjJcvjTE+NSka8oZtYKuBU42zm3qZh1Qm/jkmyzcfh9FPw+O724YptssWtfQ4BFzrnHilln/13XyMzsBPzvkJAFtSQ/43HAJbHRfCcBPxQ4VRVKsWdXoraNCyi4rxb3e3UycJaZVY9dKjgr9lzphBwpUtwH/pdkHrAFWA1MLrDsdvzIqCVA6wLPTwQOjD0+FF+4lgGvAOUDfA9DgasKPXcgMLFAxnmxj4X401ahtvcLwOfA/NhOeEDhvLHP2+BHdi0PmTeWZRn+XPfc2MeukXCR2sZFbTPgXnxhBagQ20eXxfbZQwNu0z/iT/POL7Bd2wBX7dqXgWtj23IefnDKyYH3gyJ/xoUyGzAw9jP4nAIjgwNlrowvOFULPBepbYwvnt8A22K/i7vjr41OA5YCU4EasXVzgWcKfO1lsf15GXBpPDnU6khERCIp3U7xiYhIllCBEhGRSFKBEhGRSFKBEhGRSFKBEhGRSFKBEhGRSFKBEhGRSPp/4KAeMuzSf6MAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 504x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(7, 4))\n",
    "plt.plot(x_numpy, torch_relu, c='r', label='ReLU')\n",
    "plt.ylim((-1, 10))\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sigmoid Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fe5369302e0>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbQAAAD8CAYAAAAfSFHzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAg70lEQVR4nO3deZwU1bn/8c8DDCCKKAJGNoErRBDU4EhM5CegqCgRJIpLNC5EUCM3GjeIGvXqdV+SaFSChGh8KSAY/Y0sAZeoREVZRBGICoo6gIKICgrIwHP/OD3SjD0LTE1Xd833/XrVq6urTvd5pqZnnj6nTp0yd0dERCTf1Yk7ABERkSgooYmISCIooYmISCIooYmISCIooYmISCIooYmISCJEktDMbKyZrTKzt8vZf4aZvWVmC8zsFTM7KIp6RURESkXVQnsI6FfB/g+AXu7eDbgRGB1RvSIiIgDUi+JN3P0lM2tXwf5X0p7OAlpHUa+IiEipSBLaDvoVMC3TDjMbBgwD2HXXXQ/Zf//9sxmXiIjkuLlz537m7s0z7ctqQjOzPoSE1jPTfncfTao7srCw0OfMmZPF6EREJNeZ2Yfl7ctaQjOzA4ExwHHuviZb9YqISO2QlWH7ZtYW+AfwS3d/Nxt1iohI7RJJC83MxgG9gWZmVgxcBxQAuPso4FpgL+B+MwMocffCKOoWERGB6EY5nl7J/vOA86KoS0Qkn23evJni4mI2btwYdyg5rWHDhrRu3ZqCgoIqvyaOUY4iIrVWcXExjRs3pl27dqR6rKQMd2fNmjUUFxfTvn37Kr9OU1+JiGTRxo0b2WuvvZTMKmBm7LXXXjvcilVCExHJMiWzyu3MMVJCExGRRFBCExGp5c477zwWLVpUo3Ucf/zxfPHFF9/bfv3113PnnXdGUocGhYiI1HJjxoyp8TqmTp1a43WohSYiUot8/fXX9O/fn4MOOoiuXbsyYcIEevfuTelUg3/961/p1KkTPXr0YOjQoQwfPhyAc845hwsvvJDDDjuMDh068MILLzBkyBA6d+7MOeec8937jxs3jm7dutG1a1dGjBjx3fZ27drx2WefAXDTTTfRqVMnevbsyTvvvBPZz6YWmohIXC65BObPj/Y9Dz4Y/vjHcnf/85//pGXLlkyZMgWAL7/8kgceeACAFStWcOONNzJv3jwaN27MkUceyUEHbbt95dq1a3n11VcpKipiwIABvPzyy4wZM4ZDDz2U+fPn06JFC0aMGMHcuXPZc889OeaYY3jqqac48cQTv3uPuXPnMn78eObPn09JSQndu3fnkEMOieRHVwtNRKQW6datG8888wwjRoxg5syZNGnS5Lt9r7/+Or169aJp06YUFBQwePDg7V57wgknYGZ069aNvffem27dulGnTh0OOOAAli1bxuzZs+nduzfNmzenXr16nHHGGbz00kvbvcfMmTMZNGgQjRo1Yvfdd2fAgAGR/WxqoYmIxKWCllRN6dSpE/PmzWPq1Klcc801HHXUUVV+bYMGDQCoU6fOd+ulz0tKSnZoVo+aoBaaiEgtsmLFCho1asSZZ57JFVdcwbx5877bd+ihh/Liiy+ydu1aSkpKeOKJJ3bovXv06MGLL77IZ599xpYtWxg3bhy9evXarswRRxzBU089xYYNG1i3bh1PP/10JD8XqIUmIlKrLFiwgCuuuII6depQUFDAAw88wOWXXw5Aq1atuOqqq+jRowdNmzZl//33365LsjL77LMPt956K3369MHd6d+/PwMHDtyuTPfu3Tn11FM56KCDaNGiBYceemhkP5u5e2RvFiXd4FNEkmjx4sV07tw57jDKtX79enbbbTdKSkoYNGgQQ4YMYdCgQbHEkulYmdnc8u7Woi5HERH5zvXXX8/BBx9M165dad++/XYjFHOduhxFROQ7Uc3aEQe10EREsixXT/Xkkp05RkpoIiJZ1LBhQ9asWaOkVoHS+6E1bNhwh16nLkcRkSxq3bo1xcXFrF69Ou5QclrpHat3hBKaiEgWFRQU7NBdmKXq1OUoIiKJoIQmIiKJoIQmIiKJEElCM7OxZrbKzN4uZ7+Z2T1mtsTM3jKz7lHUKyIiUiqqFtpDQL8K9h8HdEwtw4AHIqpXREQEiGiUo7u/ZGbtKigyEPi7hwsvZpnZHma2j7uvjKJ+EZGc4w4lJfDtt7BlS1i2bt3+MdO2iva5b1tK6yh9rMp6lGUr+rnLM2AA1Ku5wfXZGrbfCvg47Xlxatt2Cc3MhhFacLRt2zZLoYlIrVNSAl98AWvXhuXzz8Pzdevg66/hm2/CY/pSum3DhpCkMi2bN2//XLa3bh3stluNvX1OXYfm7qOB0RBm2485HBHJN+vXw/vvw0cfwcqV318+/TQksK++qvy96taFXXeFRo3CY+myyy7QuDHUr1+1paAgtErq1AnvWfYx07by9tVJnSUy2/a4o+tRli1Peft22aXy414N2Upoy4E2ac9bp7aJiOyYLVtg6VJYsCAsS5aEJLZ0Kaxa9f3ye+0F++wTlk6doGnTsOy55/eXxo23Ja769Sv+py05J1sJrQgYbmbjgR8DX+r8mYhUyh3efRdeeSUsb7wBixaFbj8ICadtW/iv/wrnZzp0COtt20LLlrD33tCgQbw/g2RNJAnNzMYBvYFmZlYMXAcUALj7KGAqcDywBPgGODeKekUkYdxDi2v6dJgxA15+OZzfgtCC6t4dLrgAunWDAw+ELl1qvBtL8kdUoxxPr2S/AxdFUZeIJMzWrTBzJkycCNOmhe5DgP32g0GD4Cc/gZ/+FH74w23nkEQyyKlBISJSi8ybB3//e0hkK1aEllbfvnDZZXDssaHrUGQHKKGJSPZ8/TVMmACjRsHs2eH81nHHwamnws9+VqNDuiX5lNBEpOZ9/jncey/86U9h2HyXLnDPPfDLX8Iee8QdnSSEEpqI1JzPP4fbboP77w/XiA0cGLoUe/bUkHiJnBKaiERv0ya47z648cZwEfOpp8LvfhdGJ4rUECU0EYnWs8+GofVLl8Ixx8CddyqRSVZoDKyIRGPtWhgyBI4+OgyvnzYtXE+mZCZZohaaiFTfjBlw9tmwenXoWrz2WmjYMO6opJZRC01Edt7mzSGBHXtsmDNx9my4+WYlM4mFWmgisnM++QROOinMsTh0KPzxj2FmepGYKKGJyI5780044QRYswYeewxOr3D2O5GsUJejiOyYyZPh8MO3zcGoZCY5QglNRKrukUfgxBNh//3h9dfD7PciOUIJTUSqZtQoOOss6NULXngh3G9MJIcooYlI5f7wB7jwwjCB8JQpmkRYcpISmohU7MEH4dJLw4jGf/xDQ/IlZymhiUj5Hn8czj8/3OLlscegoCDuiETKpYQmIpnNmAFnnhlGNE6aBPXrxx2RSIWU0ETk+xYuhJNPDvctmzxZF0xLXlBCE5HtrV4dLpredVd4+mlo0iTuiESqRDOFiMg2mzbBz38OK1fCiy9CmzZxRyRSZUpoIrLNpZfCv/8N48ZBjx5xRyOyQyLpcjSzfmb2jpktMbORGfa3NbN/mdkbZvaWmR0fRb0iEqEJE+D+++Gyy+C00+KORmSHVTuhmVld4D7gOKALcLqZdSlT7BrgcXf/EXAacH916xWRCL33Xpgx/yc/gVtuiTsakZ0SRQutB7DE3d9392+B8cDAMmUc2D213gRYEUG9IhKFjRth8OBwjdmECbrWTPJWFOfQWgEfpz0vBn5cpsz1wAwz+29gV6Bvpjcys2HAMIC2bdtGEJqIVOrqq8PtYCZP1iAQyWvZGrZ/OvCQu7cGjgceMbPv1e3uo9290N0LmzdvnqXQRGqxF1/cNk9j//5xRyNSLVEktOVA+te61qlt6X4FPA7g7q8CDYFmEdQtIjtr3To45xzo0AHuuCPuaESqLYqENhvoaGbtzaw+YdBHUZkyHwFHAZhZZ0JCWx1B3SKysy67DD78EB5+OFxELZLnqp3Q3L0EGA5MBxYTRjMuNLMbzGxAqthlwFAzexMYB5zj7l7dukVkJz3zTJhF/4orwlyNIglguZpXCgsLfc6cOXGHIZI8GzZAt25Qt24YDKLbwUgeMbO57l6YaZ9mChGpbW66CZYuheefVzKTRNHkxCK1yeLFcPvtcNZZ0KdP3NGIREoJTaS22Lo13KyzcWO48864oxGJnLocRWqLRx+FmTNhzBjQdZ6SQGqhidQG69fDyJFhBv1zz407GpEaoRaaSG1w++2wYgVMmgR19D1WkkmfbJGk+/DDMBPI6aeH2fRFEkoJTSTpRowAM7jttrgjEalRSmgiSfbyy+GWMFdeqZn0JfGU0ESSyj20zlq2DFNciSScBoWIJNWUKaGF9pe/aPJhqRXUQhNJoq1b4aqrYL/9NExfag210ESSaNw4WLAAxo+HgoK4oxHJCrXQRJLm22/h97+HH/0IBg+OOxqRrFELTSRpxoyBDz6AadN0EbXUKvq0iyTJN9/AjTdCr15w7LFxRyOSVWqhiSTJ6NHwyScwcWK4mFqkFlELTSQpNmwIs4EceST07Bl3NCJZpxaaSFKMGRNaZ+PHxx2JSCzUQhNJgo0b4dZbw7mzXr3ijkYkFmqhiSTB2LHh9jCPPBJ3JCKxUQtNJN9t2gS33AKHHw59+sQdjUhsIkloZtbPzN4xsyVmNrKcMqeY2SIzW2hmj0VRr4gADz0ExcVw3XUa2Si1WrW7HM2sLnAfcDRQDMw2syJ3X5RWpiPwO+Bwd19rZi2qW6+IEGYFuflmOOww6Ns37mhEYhXFObQewBJ3fx/AzMYDA4FFaWWGAve5+1oAd18VQb0i8uij8NFHMGqUWmdS60XR5dgK+DjteXFqW7pOQCcze9nMZplZvwjqFandtm4N150dfDD005+USLZGOdYDOgK9gdbAS2bWzd2/SC9kZsOAYQBt27bNUmgieaqoCN55Bx57TK0zEaJpoS0H0u/t3jq1LV0xUOTum939A+BdQoLbjruPdvdCdy9s3rx5BKGJJJR7aJ21b68Z9UVSokhos4GOZtbezOoDpwFFZco8RWidYWbNCF2Q70dQt0jtNHMmzJoFl10G9XQ5qQhEkNDcvQQYDkwHFgOPu/tCM7vBzAakik0H1pjZIuBfwBXuvqa6dYvUWrfdBs2a6W7UImki+Wrn7lOBqWW2XZu27sClqUVEqmPBApg6FW64ARo1ijsakZyhmUJE8s3tt8Ouu8JFF8UdiUhOUUITyScffgjjxsHQodC0adzRiOQUJTSRfHL33WGI/qXqvRcpSwlNJF+sWRPuefaLX0CbNpWXF6lllNBE8sWf/wzffANXXhl3JCI5SQlNJB98/TXcey/87GdwwAFxRyOSk5TQRPLB2LGhy3HEiLgjEclZSmgiuW7zZrjrLvjpT6Fnz7ijEclZmjNHJNc9/ngYrn/PPXFHIpLT1EITyWXu4ULqzp3D+TMRKZdaaCK5bMYMeOutcA6tjr5/ilREfyEiuez226Fly3DtmYhUSAlNJFfNmQPPPw+//S00aBB3NCI5TwlNJFfdcQfsvjsMGxZ3JCJ5QQlNJBctXQqTJsGFF4akJiKVUkITyUV33x3uRP2b38QdiUjeUEITyTWrV4dRjb/8ZRgQIiJVooQmkmv+/GfYuBEuvzzuSETyihKaSC75+uuQ0AYOhP33jzsakbyihCaSS8aOhc8/1y1iRHaCEppIrigpCZMQ9+wZJiIWkR2iqa9EcsXEiWES4nvvjTsSkbwUSQvNzPqZ2TtmtsTMRlZQ7iQzczMrjKJekcRIn4S4f/+4oxHJS9VuoZlZXeA+4GigGJhtZkXuvqhMucbAxcBr1a1TJHGmTYP58+Fvf9MkxCI7KYq/nB7AEnd/392/BcYDAzOUuxG4DdgYQZ0iyeEO//u/sO++cMYZcUcjkreiSGitgI/Tnhentn3HzLoDbdx9SgT1iSTLCy/Aq6+GkY0FBXFHI5K3arxvw8zqAHcDl1Wh7DAzm2Nmc1avXl3ToYnkhptugh/8AIYMiTsSkbwWRUJbDrRJe946ta1UY6Ar8IKZLQMOA4oyDQxx99HuXujuhc2bN48gNJEcN2sWPPdcmBWkYcO4oxHJa1EktNlARzNrb2b1gdOAotKd7v6luzdz93bu3g6YBQxw9zkR1C2S3266CZo2hfPPjzsSkbxX7YTm7iXAcGA6sBh43N0XmtkNZjaguu8vkljz58PkyeEGnrvtFnc0InnP3D3uGDIqLCz0OXPUiJMEO+UUmD49XEy9xx5xRyOSF8xsrrtnvJZZF7yIxGHx4nADz4suUjITiYgSmkgc/ud/oFGj0N0oIpFQQhPJtgULYMIEuPhi0GhekcgooYlk23XXwe67w2WVXpopIjtACU0km+bNgyefhEsvDcP1RSQySmgi2XTttbDnnnDJJXFHIpI4Smgi2TJrFkyZAldcAU2axB2NSOIooYlkgztcc00YBPLf/x13NCKJpDtWi2TDjBlhzsY//EGzgojUELXQRGrali2hm7FDB/j1r+OORiSx1EITqWmPPLLt2rP69eOORiSx1EITqUnffBPOnfXoAYMHxx2NSKKphSZSk/70J1i+HB57DMzijkYk0dRCE6kpn3wCt9wCJ5wARxwRdzQiiaeEJlJTRo6ETZvgrrvijkSkVlBCE6kJr7wCDz8c5mvs2DHuaERqBSU0kaht2QLDh0OrVnDVVXFHI1JraFCISNTGjIE33oDx43URtUgWqYUmEqVVq0KrrHdvOOWUuKMRqVWU0ESidPHFsH493H+/humLZJkSmkhUJk8O3YzXXAOdO8cdjUito4QmEoWvvoILL4SuXWHEiLijEamVNChEJAojRoQZQZ54QvM1isQkkhaamfUzs3fMbImZjcyw/1IzW2Rmb5nZc2a2bxT1iuSEadNg1Cj47W/DnI0iEotqJzQzqwvcBxwHdAFON7MuZYq9ARS6+4HAJOD26tYrkhNWr4Zzz4Vu3eCmm+KORqRWi6KF1gNY4u7vu/u3wHhgYHoBd/+Xu3+TejoLaB1BvSLxcofzzoMvvoBHH4WGDeOOSKRWiyKhtQI+TntenNpWnl8B0zLtMLNhZjbHzOasXr06gtBEatCDD0JREdx6a2ihiUissjrK0czOBAqBOzLtd/fR7l7o7oXNmzfPZmgiO2bePPjNb+CYY8KjiMQuilGOy4E2ac9bp7Ztx8z6AlcDvdx9UwT1isRjzRo46SRo0SJ0NdbR1S8iuSCKhDYb6Ghm7QmJ7DTgF+kFzOxHwF+Afu6+KoI6ReKxZQuceSasWAEzZ0KzZnFHJCIp1U5o7l5iZsOB6UBdYKy7LzSzG4A57l5E6GLcDZhoYTqgj9x9QHXrFsm63/8e/vnPMExfQ/RFckokF1a7+1Rgaplt16at942iHpFYjR0b7kA9dCgMGxZ3NCJShjr/RarimWfg/PPDIJD77tPEwyI5SAlNpDJvvAEnnxwmHJ44EQoK4o5IRDJQQhOpyMKFcPTRsMceMGUK7L573BGJSDmU0ETK8957cNRRYbLh556DNm0qf42IxEYJTSSTRYugT58wTP/ZZ2G//eKOSEQqoYQmUtacOXDEEVBSElpmXcrOtS0iuUgJTSTdc8/BkUdC48bw73/DgQfGHZGIVJESmkip+++HY4+FffcNyUzdjCJ5RQlNZPNm+PWv4aKL4Pjj4eWXoVVFN4wQkVykhCa129Kl0LMnPPAAjBwJTz6pofkieSqSqa9E8tJjj8EFF0DdujBpUphBX0TyllpoUvssXw6DBsEZZ4RBH/PnK5mJJIASmtQeW7aEgR+dO8P06eFO0y+8EAaBiEjeU5ejJJ87TJsGV165bSqrUaOgQ4e4IxORCKmFJsnlDi+9BH37Qv/+sGlTOFc2fbqSmUgCKaFJ8mzdCkVFcPjh0KsXvP023HtvmM7qpJN06xeRhFKXoyTHJ5/A3/4GDz4IH3wA7dqFe5edey7sskvc0YlIDVNCk/z21VehNTZxIkydGuZf7NMHbr453MOsnj7iIrWF/tol/3z4YbiD9NNPh/NhmzaFmT0uuQSGDoVOneKOUERioIQmuc0dPv4YXnstDPCYMQPefTfsa90aLrwQBg+Gww6DOjolLFKbKaFJ7ti6NZz7WrgwDOSYPTskspUrw/5GjaB375DEjj463NZFAzxEJEUJTbLr229Di2vZstB1uGwZvP9+GIH4n//Ahg3bynbsGO4Y/eMfh+Wgg8Ldo0VEMogkoZlZP+BPQF1gjLvfWmZ/A+DvwCHAGuBUd18WRd0Soy1bYP36MDDjyy+3Pa5atf3y6afbHleuDN2IperUCV2HnTuH1tcBB4SlSxdNEiwiO6TaCc3M6gL3AUcDxcBsMyty90VpxX4FrHX3/czsNOA24NTq1l2pkpLQjeW+7Z9oReuV7d+RslG+V9myW7eGn23Llqo9VrTv229h48bQMtq4seJlwwZYty4krq++CusVadgQ9t4bWrSAli3h4IOhbdswnH7ffcNj69ZQUFDx+4iIVEEULbQewBJ3fx/AzMYDA4H0hDYQuD61Pgn4s5mZe/pX9RowciTcdVeNVpEo9euHJFR22WWX8LjHHtCmDTRpElpPpY9l11u0CMtuu+kcl4hkTRQJrRXwcdrzYuDH5ZVx9xIz+xLYC/gsvZCZDQOGAbRt27b6kR13HDRtWvrm2/65VrQeZdmaqtcsXF9Vr1649cmOPJbd1qBBSFYNGmiUoIjktZwaFOLuo4HRAIWFhdVvvR11VFhERCTxovhKvhxok/a8dWpbxjJmVg9oQhgcIiIiEokoEtpsoKOZtTez+sBpQFGZMkXA2an1k4Hna/z8mYiI1CrV7nJMnRMbDkwnDNsf6+4LzewGYI67FwF/BR4xsyXA54SkJyIiEplIzqG5+1Rgaplt16atbwQGR1GXiIhIJhrWJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiaCEJiIiiVCthGZmTc3sGTN7L/W4Z4YyB5vZq2a20MzeMrNTq1OniIhIJtVtoY0EnnP3jsBzqedlfQOc5e4HAP2AP5rZHtWsV0REZDvVTWgDgYdT6w8DJ5Yt4O7vuvt7qfUVwCqgeTXrFRER2U69ar5+b3dfmVr/BNi7osJm1gOoDywtZ/8wYFjq6Xoze6ea8QE0Az6L4H2yKd9iVrw1L99izrd4If9irq3x7lveDnP3Cl9pZs8CP8iw62rgYXffI63sWnf/3nm01L59gBeAs919VuUxR8PM5rh7Ybbqi0K+xax4a16+xZxv8UL+xax4v6/SFpq79y1vn5l9amb7uPvKVMJaVU653YEpwNXZTGYiIlJ7VPccWhFwdmr9bOD/ly1gZvWBJ4G/u/ukatYnIiKSUXUT2q3A0Wb2HtA39RwzKzSzMakypwBHAOeY2fzUcnA1690Ro7NYV1TyLWbFW/PyLeZ8ixfyL2bFW0al59BERETygWYKERGRRFBCExGRREhEQjOzwamptbaaWWGZfb8zsyVm9o6ZHVvO69ub2WupchNSA1myJlVn6fnFZWY2v5xyy8xsQarcnGzGWCaO681seVrMx5dTrl/quC8xs0yzyGSFmd1hZv9JTb32ZHkz1cR9fCs7XmbWIPVZWZL6vLbLdoxl4mljZv8ys0Wpv7+LM5TpbWZfpn1Wro0j1rR4KvwdW3BP6hi/ZWbd44gzLZ4fph27+Wb2lZldUqZMrMfYzMaa2SozezttW6XTIqbKnZ0q856ZnZ2pzA5x97xfgM7ADwnXuRWmbe8CvAk0ANoTLuium+H1jwOnpdZHARfG+LPcBVxbzr5lQLMcON7XA5dXUqZu6nh3IFxM/ybQJaZ4jwHqpdZvA27LteNbleMF/BoYlVo/DZgQ8+dgH6B7ar0x8G6GmHsDk+OMc0d+x8DxwDTAgMOA1+KOucxn5BNg31w6xoRBf92Bt9O23Q6MTK2PzPQ3BzQF3k897pla37M6sSSihebui90906wiA4Hx7r7J3T8AlgA90guYmQFHAqWXFGScwisbUrGcAoyLo/6I9QCWuPv77v4tMJ7w+8g6d5/h7iWpp7OA1nHEUYmqHK/0qeYmAUelPjOxcPeV7j4vtb4OWAy0iiueiAwkXGLkHq6Z3SN1jW0uOApY6u4fxh1IOnd/Cfi8zOZKp0UEjgWecffP3X0t8Axhvt+dloiEVoFWwMdpz4v5/h/cXsAXaf/wMpXJlv8HfOqpuS8zcGCGmc21ME1YnIanumTGltOdUJVjH4chhG/gmcR5fKtyvL4rk/q8fkn4/MYu1f35I+C1DLt/YmZvmtk0Mzsgu5F9T2W/41z93EJolZf3ZTeXjjFUbVrEyI91dedyzBqrYAoud//eBd25porxn07FrbOe7r7czFoAz5jZf1LfjiJXUbzAA8CNhH8ONxK6SYfURBxVVZXja2ZXAyXAo+W8TdaOb5KY2W7AE8Al7v5Vmd3zCF1k61PnWp8COmY5xHR5+TtOndcfAPwuw+5cO8bbcXc3s6xcH5Y3Cc0rmIKrAsuBNmnPW6e2pVtD6Faol/rWm6lMtVUWv5nVA34OHFLBeyxPPa4ysycJ3VQ18sdY1eNtZg8CkzPsqsqxj0wVju85wM+AozzVgZ/hPbJ2fDOoyvEqLVOc+rw0IXx+Y2NmBYRk9qi7/6Ps/vQE5+5Tzex+M2vm7rFMqluF33FWP7c74Dhgnrt/WnZHrh3jlKpMi7iccP6vVGvCOIidlvQuxyLgtNTosPaEby2vpxdI/XP7F3ByalPGKbyyoC/wH3cvzrTTzHY1s8al64SBDm9nKlvTypxTGFROHLOBjhZGkNYndJcUZSO+ssysH3AlMMDdvymnTNzHtyrHK32quZOB58tLztmQOn/3V2Cxu99dTpkflJ7ns3C3jTrElISr+DsuAs5KjXY8DPgyressTuX23uTSMU5T6bSIwHTgGDPbM3Xa4pjUtp0X18iYKBfCP9ViYBPwKTA9bd/VhNFj7wDHpW2fCrRMrXcgJLolwESgQQw/w0PABWW2tQSmpsX4ZmpZSOhKi+t4PwIsAN5KfXD3KRtv6vnxhJFvS2OOdwmhr35+aikdKZhTxzfT8QJuICRigIapz+eS1Oe1Q1zHNBVPT0K381tpx/Z44ILSzzIwPHU83yQMyPlpjPFm/B2XideA+1K/gwWkjZqOMe5dCQmqSdq2nDnGhES7Etic+j/8K8K53eeA94BngaapsoXAmLTXDkl9npcA51Y3Fk19JSIiiZD0LkcREakllNBERCQRlNBERCQRlNBERCQRlNBERCQRlNBERCQRlNBERCQR/g8FTF2d7ODpNwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 504x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(7, 4))\n",
    "plt.plot(x_numpy, torch_sigmoid, c='r', label='sigmoid')\n",
    "plt.ylim((-0.2, 1.2))\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tanh Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fe536923af0>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbQAAAD4CAYAAACE2RPlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbSUlEQVR4nO3de3RV9Z338feXEEgrKHJREYRgH1BELGpErdRaUUTqINixg1OXWDtlemE6Ou16RmvHVmfWGu2Mdvp0qhUvM9qxaq1aKIUFqCiOgpCwkIuiXMQSjBBjRVSQS77PH3uHHuIJSTiX39n7fF5rnbVvv5z9zc5OPtl7/87e5u6IiIgkXZfQBYiIiOSDAk1ERFJBgSYiIqmgQBMRkVRQoImISCp0DV1AW/r27evV1dWhyxARkRJSV1f3jrv3y7asZAOturqa2tra0GWIiEgJMbM321qmU44iIpIKCjQREUkFBZqIiKRCyV5Dy2bPnj3U19eza9eu0KUUVFVVFQMHDqSysjJ0KSIiiZGoQKuvr6dnz55UV1djZqHLKQh3p6mpifr6eoYMGRK6HBGRxEjUKcddu3bRp0+f1IYZgJnRp0+f1B+FiojkW6ICDUh1mLUoh+9RRCTfEhdoIiIi2SjQOum9997jzjvvPOSvP++88/SBcRGRAlCgdVKugSYiIoWhQOuk66+/ng0bNjBq1Ciuu+46xo4dy2mnncbIkSOZOXMmAJs2bWL48OF84xvfYMSIEYwbN46dO3fuf4/HHnuM0aNHM2zYMJ5//vlQ34qISKokqtv+Aa69FlasyO97jhoF//EfB21y6623snr1alasWMHevXv56KOPOPzww3nnnXc466yzmDhxIgDr1q3j4Ycf5p577uErX/kKjz/+OFdeeSUAe/fuZenSpcyZM4ebb76Zp556Kr/fh4hIGUpuoJUAd+cHP/gBixYtokuXLmzZsoWtW7cCMGTIEEaNGgXA6aefzqZNm/Z/3WWXXZZ1voiIHLq8BJqZ3Q9cAmxz95OzLDfgZ8AE4CPgandfntNK2zmSKoaHHnqIxsZG6urqqKyspLq6ev/nx7p3776/XUVFxQGnHFuWVVRUsHfv3uIWLSKSUvm6hvbfwPiDLL8YGBq/pgF35Wm9RdezZ0927NgBwPbt2znqqKOorKxk4cKFvPlmm081EBGRAsvLEZq7LzKz6oM0uRR40N0dWGJmvcysv7s35GP9xdSnTx/OOeccTj75ZM444wzWrl3LyJEjqamp4cQTTwxdnkiyNTfDxx/D7t3Rq2V8375oWXMzuP95vPUr27K2uHdufiGWHexr0uiLX4SuhbvSVaxraAOAzRnT9fG8AwLNzKYRHcExaNCgIpXWeb/+9a/bbbN69er949///vf3jz/77LP7x/v27atraJI+7tDQAJs2RcO3346GW7fC9u3w/vsHvnbsODC4JL127IAePQr29iXVKcTdZwAzAGpqasrsXxeRBNq5E5Yvh5degtpaWLsWXn8dPvzwwHZdusBRR0GvXnD44dGrf/9o2KMHVFVBt27QvXs0bD3etWv0HmbRMNsr2zKz6NWWtpYdytcc6rJyutXdpz5V0LcvVqBtAY7LmB4YzxORpNmwAWbOhDlzYNEi2LMnmn/ccTBiBJx7LgwbBkOGRKHVvz/07QsVFWHrltQrVqDNAqab2SPAmcD2Q71+5u6pv3mvl9t5dSl9u3fDI4/AffdFIQZReF17LYwZA6NHwzHHBC1RJF/d9h8GzgP6mlk98COgEsDdfwnMIeqyv56o2/7XDmU9VVVVNDU1pfoRMi3PQ6uqqgpdikgUZDNmwG23QX09DB0K//qvMGUKVFeHrk7kAPnq5XhFO8sd+E6u6xk4cCD19fU0Njbm+lYlreWJ1SJBPfUUTJ8Or70WHYXdey+MG1de13wkUUqqU0h7Kisr9RRnkULbtQu+9z248074zGeia2UXXxy6KpF2JSrQRKTA/vhHmDgRXn45CrV/+ZeoB6JIAijQRCSyahWMHx91uZ89G770pdAViXSKAk1EYOVK+MIX4LDD4PnnYeTI0BWJdJoCTaTcrV8fdfbo0SMKM/VelIRSoImUs+3bYcKE6JZTzz6rMJNEU6CJlKvmZpg6Fd54AxYuBN1cWxJOgSZSru64I7qF1c9+Fn3OTCTh8vU8NBFJkrVr4Yc/hEmT4O/+LnQ1InmhQBMpN/v2wTXXRD0a77pLd/6Q1NApR5Fyc999sHgxPPigbigsqaIjNJFy8v778E//BJ//PFx5ZehqRPJKR2gi5eS222DbtuhOIDrVKCmjIzSRctHQEPVs/Ou/hjPOCF2NSN4p0ETKxR13RM83u+WW0JWIFIQCTaQcvPtu1KNxypTokTAiKaRAEykHP/95dBf9G24IXYlIwSjQRNJu164o0CZOhJNPDl2NSMEo0ETS7rHHoKkJvvvd0JWIFJQCTSTt7roLhg2D888PXYlIQSnQRNJsxYroriDf+pY+dyapp0ATSbO774ZPfSp6TIxIyinQRNLq44/h0Udh8mQ48sjQ1YgUnAJNJK3mzoU//Un3bJSyoUATSauHHoJ+/eDCC0NXIlIUCjSRNNq+HX7/e7jiCuiqe5BLeVCgiaTRE09E19C++tXQlYgUjQJNJI2efBIGD9Zd9aWsKNBE0uaDD2D+fJg0SZ89k7KiQBNJm/nzo9ONkyaFrkSkqBRoImnzu99B794wZkzoSkSKSoEmkiZ79sDs2XDJJerdKGVHgSaSJs8/H32YWqcbpQwp0ETSZO5c6NZNH6aWsqRAE0mTefOia2c9eoSuRKToFGgiadHQAKtWwbhxoSsRCUKBJpIW8+dHw4suCluHSCB5CTQzG29mr5nZejO7Psvyq82s0cxWxK+/ycd6RSTD/Plw9NFwyimhKxEJIud+vWZWAfwCuBCoB5aZ2Sx3f6VV00fdfXqu6xORLJqbYcGC6Oisi068SHnKx54/Gljv7hvdfTfwCHBpHt5XRDpqxQpobNTpRilr+Qi0AcDmjOn6eF5rXzazlWb2WzM7Ltsbmdk0M6s1s9rGxsY8lCZSJp55JhqOHRu2DpGAinVu4vdAtbufAiwAHsjWyN1nuHuNu9f069evSKWJpMBzz8GwYdC/f+hKRILJR6BtATKPuAbG8/Zz9yZ3/zievBc4PQ/rFRGAffuiO4R84QuhKxEJKh+BtgwYamZDzKwbMAWYldnAzDL/bZwIvJqH9YoIwMqV0ROqFWhS5nLu5ejue81sOjAPqADud/c1ZnYLUOvus4DvmtlEYC/wLnB1rusVkdhzz0VDBZqUOXP30DVkVVNT47W1taHLECl9kydHR2kbNoSuRKTgzKzO3WuyLdMHVkSSrLkZFi3S0ZkICjSRZFuzBt59V4EmggJNJNkWLYqG554btg6REqBAE0myF1+EY4+F6urQlYgEp0ATSbLFi+Hss8EsdCUiwSnQRJJq61Z44w0466zQlYiUBAWaSFItXhwNzz47bB0iJUKBJpJUS5ZAZSWcrjvJiYACTSS5Fi+GU0+FqqrQlYiUBAWaSBLt2QPLlul0o0gGBZpIEq1cCTt3KtBEMijQRJKopUOIejiK7KdAE0mixYujh3kOGhS6EpGSoUATSaIlS/SBapFWFGgiSdPUBBs3wujRoSsRKSkKNJGkqauLhmecEbYOkRKjQBNJmpYH3552Wtg6REqMAk0kaWprYehQ6NUrdCUiJUWBJpI0tbVQk/UJ9CJlTYEmkiRbt8LmzQo0kSwUaCJJ0tIhRIEm8gkKNJEkqa2NPnt26qmhKxEpOQo0kSSprYUTT4SePUNXIlJyFGgiSaIOISJtUqCJJMVbb0FDgwJNpA0KNJGkaPlAtQJNJCsFmkhS1NZCly4walToSkRKkgJNJClqa2HECPj0p0NXIlKSFGgiSeCuDiEi7VCgiSTB5s3Q2KhAEzkIBZpIErR0CNEjY0TapEATSYLaWqishFNOCV2JSMlSoIkkQW0tnHwydO8euhKRkqVAEyl17rB8OZx+euhKREqaAk2k1P3xj9DUpCdUi7RDgSZS6pYvj4Y6QhM5qLwEmpmNN7PXzGy9mV2fZXl3M3s0Xv6SmVXnY70iZaGuDioq1CFEpB05B5qZVQC/AC4GTgKuMLOTWjX7OvAnd/8/wE+B23Jdr0jZqKuL7hBSVRW6EpGSlo8jtNHAenff6O67gUeAS1u1uRR4IB7/LTDWzCwP6xZJN/co0HS6UaRd+Qi0AcDmjOn6eF7WNu6+F9gO9Gn9RmY2zcxqzay2sbExD6WJJNyWLdEdQtQhRKRdJdUpxN1nuHuNu9f069cvdDki4alDiEiH5SPQtgDHZUwPjOdlbWNmXYEjgKY8rFsk3erqokfGfPazoSsRKXn5CLRlwFAzG2Jm3YApwKxWbWYBU+PxvwSecXfPw7pF0q2uDoYP1yNjRDog50CLr4lNB+YBrwK/cfc1ZnaLmU2Mm90H9DGz9cA/AJ/o2i8iWegOISId1jUfb+Luc4A5rebdlDG+C7g8H+sSKRsNDdFLHUJEOqSkOoWISAZ1CBHpFAWaSKmqqwMzGDUqdCUiiaBAEylVy5fDCSdAjx6hKxFJBAWaSKnSHUJEOkWBJlKKtm2D+np1CBHpBAWaSClShxCRTlOgiZSiurpoqA4hIh2mQBMpRcuXw9ChcMQRoSsRSQwFmkgpqqvT9TORTlKgiZSapiZ4801dPxPpJAWaSKlpuX6mIzSRTlGgiZSaZcuiYU1N2DpEEkaBJlJqli6N7hCiDiEinaJAEykl7lGgjR4duhKRxFGgiZSSLVvg7bcVaCKHQIEmUkqWLo2GZ5wRtg6RBFKgiZSSpUuhshI++9nQlYgkjgJNpJQsWxaFWVVV6EpEEkeBJlIqmpujQNP1M5FDokATKRWvvQY7duj6mcghUqCJlIqWDiE6QhM5JAo0kVKxbBn07Bl9qFpEOk2BJlIqli6NbndVURG6EpFEUqCJlIKdO2HFCl0/E8mBAk2kFNTVwZ49cM45oSsRSSwFmkgpeOGFaHj22WHrEEkwBZpIKXjhBRg2DPr1C12JSGIp0ERCc4cXX9TpRpEcKdBEQnv9dWhqgs99LnQlIommQBMJreX6mY7QRHKiQBMJ7cUXoXdvfaBaJEcKNJHQXnghOt3YRb+OIrnQb5BISE1NsHatrp+J5IECTSQkXT8TyRsFmkhICxdGD/M888zQlYgkngJNJKSFC6PTjd27h65EJPFyCjQz621mC8xsXTw8so12+8xsRfyalcs6RVKjqQlefhm++MXQlYikQq5HaNcDT7v7UODpeDqbne4+Kn5NzHGdIunw3HPRUIEmkhe5BtqlwAPx+APApBzfT6R8LFwIn/60Hhkjkie5BtrR7t4Qj78NHN1GuyozqzWzJWY2qa03M7NpcbvaxsbGHEsTKXELF8KYMdCtW+hKRFKha3sNzOwp4Jgsi27MnHB3NzNv420Gu/sWMzseeMbMVrn7htaN3H0GMAOgpqamrfcSSb5t22DNGrjyytCViKRGu4Hm7he0tczMtppZf3dvMLP+wLY23mNLPNxoZs8CpwKfCDSRsrFgQTQ8//ywdYikSK6nHGcBU+PxqcDM1g3M7Egz6x6P9wXOAV7Jcb0iyTZ3LvTtCzU1oSsRSY1cA+1W4EIzWwdcEE9jZjVmdm/cZjhQa2YvAwuBW91dgSblq7kZ5s2Diy7S/RtF8qjdU44H4+5NwNgs82uBv4nHXwRG5rIekVSprYV33oGLLw5diUiq6N9DkWKbOxfMoiM0EckbBZpIsc2dG332rG/f0JWIpIoCTaSY3nkHli7V6UaRAlCgiRTT7NngDl/6UuhKRFJHgSZSTI8/DoMGqbu+SAEo0ESKZccOmD8fLrss6hQiInmlQBMpljlzYPfuKNBEJO8UaCLF8vjjcPTR0QM9RSTvFGgixfDBB9ER2uTJUFERuhqRVFKgiRTDk0/Chx/CV78auhKR1FKgiRTDgw/CkCFwzjmhKxFJLQWaSKHV18PTT8NVV6l3o0gBKdBECu2hh6IPU191VehKRFJNgSZSSM3NcN99MGYMHH986GpEUk2BJlJICxbAunXw7W+HrkQk9RRoIoX0n/8Zffbsy18OXYlI6inQRApl40b4wx9g2jTo1i10NSKpp0ATKZR//3fo2hX+9m9DVyJSFhRoIoXQ0AD33w9XXw0DBoSuRqQsKNBECuH222HPHvjHfwxdiUjZUKCJ5Ntbb8Fdd8EVV8BnPhO6GpGyoUATybebboqOzm6+OXQlImVFgSaST6tWRdfOpk/X0ZlIkSnQRPKluTn6AHWvXvDDH4auRqTsdA1dgEhq3H03/O//wn/9F/TuHboakbKjIzSRfHjjjahH4wUXwNSpoasRKUsKNJFc7d4Nf/VX0KUL3HOPHhEjEohOOYrkwh2uuw6WLYMnnoDq6tAViZQtHaGJ5OKnP4U774TvfQ8mTw5djUhZU6CJHKpf/SoKsssvh5/8JHQ1ImVPgSZyKO6+O+r8cf758OCD0fUzEQlKv4UinbFvH9xwA3zzmzBhAsyeDVVVoasSEdQpRKTjNm+Gr30Nnn46esbZz3+u55yJlBAdoYm0Z+/eqOPHiBGweDHce290ylFhJlJSFGgibdm7Fx5+GE46Cb7zHTjzTFi9Gr7+9dCViUgWOuUo0tqGDfA//wMzZkSPghk5EmbOhL/4C31oWqSEKdBE3n8fliyBRYui4Fq9Opo/fnz0XLNLLlEvRpEEyCnQzOxy4MfAcGC0u9e20W488DOgArjX3W/NZb0indbcDI2N0RHXxo3wyivw6qtReK1ZEy3v0gXGjIk+LD15MgweHLpqEemEXI/QVgOXAXe31cDMKoBfABcC9cAyM5vl7q/kuO72ffgh7Nr152n3A5druvSn9+2L7pWY7bVnz5/Hd+6MjrS2b4+GLePvvReFWENDdE0s0+DBMHx4FF5jxkTXyA4/HBFJppwCzd1fBbCDX1cYDax3941x20eAS4HCB9qPfgS3317w1UgJOeywKJSOOCIa9uoFJ54Ixx4LAwZEw8GD4YQTorYikhrFuIY2ANicMV0PnJmtoZlNA6YBDBo0KPc1T5r0ydNGrcNX06U93bVr1D2+WzeorPzzeOtX9+5RgHXVZWGRctXub7+ZPQUck2XRje4+M5/FuPsMYAZATU2Nt9O8fWPGRC8REUm9dgPN3S/IcR1bgOMypgfG80RERPKmGH2RlwFDzWyImXUDpgCzirBeEREpIzkFmplNNrN64GzgD2Y2L55/rJnNAXD3vcB0YB7wKvAbd1+TW9kiIiIHyrWX45PAk1nmvwVMyJieA8zJZV0iIiIHo9sfiIhIKijQREQkFRRoIiKSCgo0ERFJBQWaiIikggJNRERSQYEmIiKpoEATEZFUUKCJiEgqKNBERCQVFGgiIpIKCjQREUkFc8/9OZqFYGaNwJt5eKu+wDt5eJ9iSlrNqrfwklZz0uqF5NVcrvUOdvd+2RaUbKDli5nVuntN6Do6I2k1q97CS1rNSasXklez6v0knXIUEZFUUKCJiEgqlEOgzQhdwCFIWs2qt/CSVnPS6oXk1ax6W0n9NTQRESkP5XCEJiIiZUCBJiIiqZCKQDOzy81sjZk1m1lNq2U3mNl6M3vNzC5q4+uHmNlLcbtHzaxbcSrfv/5HzWxF/NpkZivaaLfJzFbF7WqLWWOrOn5sZlsyap7QRrvx8XZfb2bXF7vOjDr+zczWmtlKM3vSzHq10S7o9m1ve5lZ93hfWR/vr9XFrrFVPceZ2UIzeyX+/fv7LG3OM7PtGfvKTSFqzajnoD9ji/y/eBuvNLPTQtSZUc8JGdtuhZm9b2bXtmoTdBub2f1mts3MVmfM621mC8xsXTw8so2vnRq3WWdmU3Muxt0T/wKGAycAzwI1GfNPAl4GugNDgA1ARZav/w0wJR7/JfCtgN/L7cBNbSzbBPQtge39Y+D77bSpiLf38UC3+OdwUqB6xwFd4/HbgNtKbft2ZHsB3wZ+GY9PAR4NvB/0B06Lx3sCr2ep+Txgdsg6O/MzBiYAcwEDzgJeCl1zq33kbaIPFpfMNgbOBU4DVmfM+wlwfTx+fbbfOaA3sDEeHhmPH5lLLak4QnP3V939tSyLLgUecfeP3f0NYD0wOrOBmRlwPvDbeNYDwKQCltumuJavAA+HWH+ejQbWu/tGd98NPEL08yg6d5/v7nvjySXAwBB1tKMj2+tSov0Tov11bLzPBOHuDe6+PB7fAbwKDAhVT55cCjzokSVALzPrH7qo2Fhgg7vn4w5KeePui4B3W83O3Ffb+pt6EbDA3d919z8BC4DxudSSikA7iAHA5ozpej75C9cHeC/jD162NsXyeWCru69rY7kD882szsymFbGubKbHp2Tub+N0Qke2fQjXEP0Hnk3I7duR7bW/Tby/bifaf4OLT3+eCryUZfHZZvaymc01sxHFrewT2vsZl+p+C9FReVv/7JbSNgY42t0b4vG3gaOztMn7tu6ayxcXk5k9BRyTZdGN7j6z2PV0Vgfrv4KDH52NcfctZnYUsMDM1sb/HeXdweoF7gL+meiPwz8TnSa9phB1dFRHtq+Z3QjsBR5q422Ktn3TxMx6AI8D17r7+60WLyc6RfZBfK31d8DQIpeYKZE/4/i6/kTghiyLS20bH8Dd3cyK8vmwxASau19wCF+2BTguY3pgPC9TE9Fpha7xf73Z2uSsvfrNrCtwGXD6Qd5jSzzcZmZPEp2mKsgvY0e3t5ndA8zOsqgj2z5vOrB9rwYuAcZ6fAI/y3sUbftm0ZHt1dKmPt5fjiDaf4Mxs0qiMHvI3Z9ovTwz4Nx9jpndaWZ93T3ITXU78DMu6n7bCRcDy919a+sFpbaNY1vNrL+7N8SnbLdlabOF6Ppfi4FE/SAOWdpPOc4CpsS9w4YQ/deyNLNB/MdtIfCX8aypQIgjvguAte5en22hmR1mZj1bxok6OqzO1rbQWl1TmNxGHcuAoRb1IO1GdLpkVjHqa83MxgP/F5jo7h+10Sb09u3I9ppFtH9CtL8+01Y4F0N8/e4+4FV3v6ONNse0XOczs9FEf3OChHAHf8azgKvi3o5nAdszTp2F1ObZm1Laxhky99W2/qbOA8aZ2ZHxZYtx8bxDF6pnTD5fRH9U64GPga3AvIxlNxL1HnsNuDhj/hzg2Hj8eKKgWw88BnQP8D38N/DNVvOOBeZk1Phy/FpDdCot1Pb+FbAKWBnvuP1b1xtPTyDq+bYhcL3ric7Vr4hfLT0FS2r7ZttewC1EQQxQFe+f6+P99fhQ2zSuZwzRaeeVGdt2AvDNln0ZmB5vz5eJOuR8LmC9WX/Greo14Bfxz2AVGb2mA9Z9GFFAHZExr2S2MVHQNgB74r/DXye6tvs0sA54Cugdt60B7s342mvi/Xk98LVca9Gtr0REJBXSfspRRETKhAJNRERSQYEmIiKpoEATEZFUUKCJiEgqKNBERCQVFGgiIpIK/x/jJxEGvh7D8gAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 504x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(7, 4))\n",
    "plt.plot(x_numpy, torch_tanh, c='r', label='tanh')\n",
    "plt.ylim((-1.2, 1.2))\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Softplus Plot\n",
    "\n",
    "![](assets/2022-02-22-22-15-06.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fe536894970>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAD4CAYAAAC5S3KDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgCElEQVR4nO3de5zOZf7H8dcn1GCJ0EFl0aYISztqO+yW2i2nKKkoEbV+tauyRai0diudbKWTY+JXKopSyLEl9dABSQfapImR1dCPHNZxrt8f163GmGHMfbjuw/v5eMxj7rnv78z9nu/cM+/5fu/rvi5zziEiIpJsDgsdQEREpCgqKBERSUoqKBERSUoqKBERSUoqKBERSUplE3ln1atXd7Vr107kXYqISBJbtGjReudcjaJuS2hB1a5dm4ULFybyLkVEJImZ2bfF3aZTfCIikpRUUCIikpRUUCIikpQS+hxUUXbt2kVubi7bt28PHSWpZWVlccIJJ1CuXLnQUUREEiJ4QeXm5lKpUiVq166NmYWOk5Scc2zYsIHc3Fzq1KkTOo6ISEIEP8W3fft2qlWrpnI6ADOjWrVqOsoUkYwSvKAAlVMJaB+JSKZJioISEREpTAV1CObPn89pp51GkyZNWLBgAdOmTTvo5+Tk5NCwYcMEpBMRSS8qqEMwbtw4+vfvz5IlS/jyyy9LVFAiIlI6GV9QW7dupXXr1vz617+mYcOGjB8/njlz5tC0aVMaNWpE9+7d2bFjB6NGjWLChAkMGDCATp06cc899zB+/HiaNGnC+PHjGThwINdeey1nnXUWJ598MiNHjtzvvsaMGUPPnj1/+rhNmzbMnTuXPXv2cN1119GwYUMaNWrEY489lshdICKSlIIPM99Hr16wZElsv2aTJvD448XePH36dGrWrMnUqVMB2LRpEw0bNmTOnDnUq1ePLl26MHToUHr16sW7775LmzZt6NChA2PGjGHhwoU89dRTAAwcOJClS5fy/vvvs3XrVpo2bUrr1q1LFHHJkiWsWbOGzz77DICNGzdG8x2LiKSFjD+CatSoEbNmzaJv377Mnz+fnJwc6tSpQ7169QDo2rUr77zzTom+Vrt27ShfvjzVq1enefPmfPjhhyX6vLp167Jy5Upuvvlmpk+fTuXKlUv9/YiIpIvkOoI6wJFOvNSrV4/Fixczbdo07r77bi644IJSf63CQ8ELf1y2bFny8/N/+njv65qqVq3KJ598wowZMxg2bBgTJkxg9OjRpc4hIpIOMv4I6rvvvqNChQp07tyZPn36sGDBAnJyclixYgUAzz//POedd95+n1epUiU2b968z3WTJ09m+/btbNiwgblz59KsWbN9bq9duzZLliwhPz+f1atX/3SEtX79evLz87n88su57777WLx4cZy+WxGR1HHQIygzGw20Ab53zjWMXHcUMB6oDeQAVzrn/i9+MePn008/pU+fPhx22GGUK1eOoUOHsmnTJq644gp2795Ns2bNuPHGG/f7vObNm/Pggw/SpEkT+vfvD0Djxo1p3rw569evZ8CAAdSsWZOcnJyfPuecc86hTp06NGjQgPr163P66acDsGbNGrp16/bT0dUDDzwQ/29cRCTJmXPuwBuY/R7YAvxvgYJ6GPjBOfegmfUDqjrn+h7szrKzs13hBQuXLVtG/fr1S5s/aQwcOJBf/OIX9O7dO273kS77SkTSQF4e1ChyIdxDYmaLnHPZRd120FN8zrl3gB8KXd0OGBu5PBa4NJqAIiKSQj7/HE45BYYOjevdlHaQxDHOubWRy/8BjiluQzPrAfQAqFWrVinvLvkNHDgwdAQRkfj79lu46CLIyoIWLeJ6V1EPknD+HGGx5wmdcyOcc9nOuewaxRwOHuw0o2gfiUgSyMvz5bRtG8yYAXFe/qe0BbXOzI4DiLz/vrQBsrKy2LBhg/4AH8De9aCysrJCRxGRTLV5M7RqBatWwZtvQqNGcb/L0p7iewPoCjwYeT+5tAFOOOEEcnNzycvLK+2XyAh7V9QVEUm4HTvgssvg44/h9dfh3HMTcrclGWb+EnA+UN3McoG/4YtpgpldD3wLXFnaAOXKldMqsSIiyWrPHujcGebMgbFjoU2bhN31QQvKOdepmJsujHEWERFJJs5Bz57w6qsweDB06ZLQu8/4mSRERKQYAwfCsGFwxx1w++0Jv3sVlIiI7O+pp+Af/4Du3eHBB4NEUEGJiMi+Xn4ZbrkF2raF4cOh0MTXiaKCEhGRn82c6Z9rOvdcX1Rlwy16oYISERHvgw+gfXto0ADeeAPKlw8aRwUlIiKwbBm0bg3HHAPTp0OVKqETqaBERDLe6tVw8cX+dN7MmXDssaETAcm2oq6IiCTWhg2+nDZtgnnz4KSTQif6iQpKRCRTbdniT+utXOknf23SJHSifaigREQy0c6d0KEDfPQRTJwI550XOtF+VFAiIpkmPx+uu84fNY0aBZdeGjpRkTRIQkQkkzgHvXrBSy/BAw/A9deHTlQsFZSISCa5/3548km47Tbo2zd0mgNSQYmIZIrhw2HAALj2WnjkkWBTGJWUCkpEJBO8+ircdJMftffss3BY8v/5T/6EIiISnTlz4Jpr4KyzYMIEKFcudKISUUGJiKSzRYv8KL169WDKFKhQIXSiElNBiYikq3//G1q2hGrV/Px6VauGTnRIVFAiIulozRq46CJ/eeZMOP74sHlKQS/UFRFJNz/84OfX27AB5s71p/dSkApKRCSdbNsGl1wCX30Fb70Fv/lN6ESlpoISEUkXu3bBFVfAggV+tN4FF4ROFBUVlIhIOsjPh+7dYdo0GDbMTwSb4jRIQkQk1TkHvXvDCy/AvffC//xP6EQxoYISEUl1Dz8Mjz0GN98Md90VOk3MqKBERFLZs89Cv37QqRM8/njSz693KFRQIiKp6vXXoUcPP6R8zJiUmF/vUKTXdyMikinmzYOOHaFZM78i7uGHh04UcyooEZFU8/HH0LYt1K0LU6dCxYqhE8VFVAVlZn81s8/N7DMze8nMsmIVTEREirBiBbRoAUce6Zdsr1YtdKK4KXVBmdnxwC1AtnOuIVAG6BirYCIiUsjatf75pj17/Px6J54YOlFcRftC3bJAeTPbBVQAvos+koiI7GfjRj8z+bp18PbbcOqpoRPFXamPoJxza4DBwCpgLbDJOTczVsFERCTiv//1zzl98QVMmgRnnBE6UUJEc4qvKtAOqAPUBCqaWecituthZgvNbGFeXl7pk4qIZKLdu/1ovXffheef/3kJjQwQzSCJPwDfOOfynHO7gEnA2YU3cs6NcM5lO+eya9SoEcXdiYhkGOf865zeeAOeeAKuuip0ooSKpqBWAb81swpmZsCFwLLYxBIREfr1g+eeg3vugZ49Q6dJuGieg/oAeBVYDHwa+VojYpRLRCSzDR7s59i76SYYODB0miCiGsXnnPsb8LcYZREREYCxY6FPH7+205NPptX8eodCM0mIiCSTKVPg+uvhwgv9oIgyZUInCkYFJSKSLN591x81NW0Kr70GRxwROlFQKigRkWSwdCm0aQO1avlVcStVCp0oOBWUiEho33zj59erWNFPYaSX5ADRT3UkIiLR+P57/+Lb7dth/nz45S9DJ0oaKigRkVB+/NEfOa1ZA7Nnw2mnhU6UVFRQIiIhbN8Ol14Kn37qZ4o4e7+JeDKeCkpEJNH27IFrroF//csPJW/ZMnSipKRBEiIiieScnx1i0iR47DHovN8c2xKhghIRSaQBA2DkSLjzTujVK3SapKaCEhFJlCFD4P774YYb4L77QqdJeiooEZFEGDfOHzFddhkMHZqx8+sdChWUiEi8vfUWXHcdnH8+vPgilNX4tJJQQYmIxNOCBXD55dCoEUyeDFlZoROlDBWUiEi8fP45tG4NNWv6o6jKlUMnSikqKBGRePj2W7j4Yj8j+cyZcMwxoROlHJ0IFRGJtbw8P7/eli3wzjtQt27oRClJBSUiEkubN0OrVrBqlT9yatw4dKKUpYISEYmVHTugfXv4+GO/4ODvfhc6UUpTQYmIxMKePdCli5+V/Lnn4JJLQidKeRokISISLefglltgwgR45BH/mieJmgpKRCRa//gHPPMM9OkDvXuHTpM2VFAiItF45hkYONAfNT30UOg0aUUFJSJSWuPHQ8+e0Latn6Fc8+vFlApKRKQ0Zs2Ca6+Fc86Bl1/W/HpxoIISETlUH33kZyU/9VR4800oXz50orSkghIRORTLl/sl2o8+GmbMgCpVQidKWyooEZGSys31UxiVKeNniTjuuNCJ0ppOmoqIlMSGDb6cNm6EefPgV78KnSjtqaBERA5m61Zo0wZWroTp06Fp09CJMkJUp/jMrIqZvWpmy81smZmdFatgIiJJYedOv+Dghx/CSy/5VXElIaI9ghoCTHfOdTCzw4EKMcgkIpIc8vOhWzc/GGLkSD9yTxKm1AVlZkcCvweuA3DO7QR2xiaWiEhgzsFf/wovvgiDBsENN4ROlHGiOcVXB8gDnjOzj81slJlVLLyRmfUws4VmtjAvLy+KuxMRSaBBg+CJJ6BXL+jXL3SajBRNQZUFTgeGOueaAluB/X6KzrkRzrls51x2jRo1org7EZEEGTEC7r4bOneGf/5TUxgFEk1B5QK5zrkPIh+/ii8sEZHUNXEi3HSTfzHu6NFwmF4uGkqp97xz7j/AajM7JXLVhcAXMUklIhLCnDlw9dVw5pnwyitQrlzoRBkt2lF8NwPjIiP4VgLdoo8kIhLAwoVw6aVQrx5MmQIV93tKXRIsqoJyzi0BsmMTRUQkkL3z61Wv7oeUH3VU6ESC5uITkUy3erWfwuiww/wSGjVrhk4kEZrqSEQy1/r1vpw2bYK5czW/XpJRQYlIZtq8GVq1gm++8af1NL9e0lFBiUjm2bED2reHxYth0iQ477zQiaQIKigRySx79vil2mfPhjFjoG3b0ImkGBokISKZwzn4y1/8a5wGD4auXUMnkgNQQYlI5hgwAIYP93Pr3X576DRyECooEckMjz8O99/vZyUfNCh0GikBFZSIpL/nn/dLZ7RvD8OGafLXFKGCEpH0NnWqX3Twggtg3DgoUyZ0IikhFZSIpK/586FDB2jSBF5/HbKyQieSQ6CCEpH09MkncMklUKsWvPUWVKoUOpEcIhWUiKSfr7+GFi18Kc2aBVosNSXphboikl7WrvXz6+3cCW+/7Y+gJCWpoEQkfWzc6I+c1q3ziw/Wrx86kURBBSUi6WHbNv+c07JlfuTemWeGTiRRUkGJSOrbtQuuvBLeew/Gj4c//jF0IokBFZSIpLb8fOje3R81DR0KV1wROpHEiEbxiUjqcg5uuw1eeAHuvRduvDF0IokhFZSIpK5Bg2DIELj1VrjrrtBpJMZUUCKSmoYPh7vvhs6d4dFHNb9eGlJBiUjqeeUVuOkmaN0aRo+Gw/SnLB3ppyoiqWXWLLjmGjj7bJgwAcqVC51I4kQFJSKp48MP4bLL/Atwp0yBChVCJ5I4UkGJSGpYtgxatoRjjoHp06FKldCJJM5UUCKS/HJy/Itvy5WDmTPhuONCJ5IE0At1RSS5rVvny2nrVpg3D046KXQiSRAVlIgkr40b4eKL4bvvYPZsaNw4dCJJIBWUiCSnrVv9MPIvvvDTGJ11VuhEkmAqKBFJPjt3wuWXw/vva/LXDBb1IAkzK2NmH5vZlFgEEpEMt2ePnx1ixgwYMQI6dAidSAKJxSi+W4FlMfg6IpLpnPMTvr7yCgweDNdfHzqRBBRVQZnZCUBrYFRs4ohIxnIO+vaFUaP8xK+33x46kQQW7RHU48AdQH5xG5hZDzNbaGYL8/Lyorw7EUlbDz0EjzwCf/6zXzpDMl6pC8rM2gDfO+cWHWg759wI51y2cy67Ro0apb07EUlnw4ZB//5w9dXw5JOamVyA6I6gzgHamlkO8DJwgZm9EJNUIpI5XnrJHzW1bg1jxmhmcvlJqR8Jzrn+zrkTnHO1gY7A2865zjFLJiLpb9o06NIFfvc7PzBCM5NLAfpXRUTCmD/fv9apcWN4800oXz50IkkyMXmhrnNuLjA3Fl9LRDLA4sXQpg3Uru1nJq9cOXQiSUI6ghKRxPryS2jRwi+XMXMmaPCUFEMFJSKJs2qVn7bIzK+Me+KJoRNJEtNcfCKSGN9/78vpxx9h7lyoVy90IklyKigRib8ffvDltHq1P63XpEnoRJICVFAiEl+bN/ul2pcvhylT4NxzQyeSFKGCEpH42bbNj9ZbtAgmTtSyGXJIVFAiEh87dvjXOc2fD+PGQbt2oRNJilFBiUjs7d7t59WbPt3PTt6pU+hEkoI0zFxEYis/H7p1g0mT4PHHtaaTlJoKSkRixzk/8esLL8D998Ott4ZOJClMBSUiseEc9O4Nw4dDv35w552hE0mKU0GJSGz8/e/w6KPQsycMGhQ6jaQBFZSIRG/wYF9Q3brBkCFacFBiQgUlItEZOhT69IGrroKRI7XgoMSMHkkiUnpjx/pBEZdcAs8/D2XKhE4kaUQFJSKlM26cP6X3hz/AhAlaDVdiTgUlIodu/Hi/VPv558PkyZCVFTqRpCEVlIgcmokT4Zpr4Jxz/FLtFSqETiRpSgUlIiU3eTJ07AhnnglTp0LFiqETSRpTQYlIyUydCldcAaefDm+9BZUqhU4kaU4FJSIHN2MGtG8PjRv7y5Urh04kGUAFJSIHNmcOXHopNGjgV8OtUiV0IskQKigRKd68ef41TiefDLNmwVFHhU4kGUQFJSJFmzcPWreGOnVg9myoXj10IskwKigR2d/bb0PLllCrlj/Fd/TRoRNJBlJBici+Zs3yR04nnQRz58Kxx4ZOJBlKBSUiP5s+3T/nVK+eP4rSkZMEpIISEW/qVGjXzo/We/ttqFEjdCLJcCooEfEzRFx2mX+d05w5UK1a6EQiKiiRjDdpEnToAE2b+uefqlYNnUgEiKKgzOxEM/uXmX1hZp+b2a2xDCYiCfDyy3DlldCsmV6EK0knmiOo3cDtzrkGwG+Bv5hZg9jEEpG4Gz4crr7az0o+YwYceWToRCL7KHVBOefWOucWRy5vBpYBx8cqmIjE0cMPw403+tc6TZ+uiV8lKcXkOSgzqw00BT4o4rYeZrbQzBbm5eXF4u5EpLScgzvvhL594aqr4LXXoHz50KlEihR1QZnZL4CJQC/n3I+Fb3fOjXDOZTvnsmto2KpIOPn50LMnPPAA/OlPfsn2ww8PnUqkWFEVlJmVw5fTOOfcpNhEEpGY27ULunaFZ56B3r39809lyoROJXJAZUv7iWZmwLPAMufco7GLJCIxtXWrXwV3yhS47z5/is8sdCqRgyp1QQHnANcCn5rZksh1dzrnpkWdSkRiIy/PT1304Yfw9NPw5z+HTiRSYqUuKOfcu4D+DRNJVitXQosWsHo1TJzoZ4oQSSHRHEGJSLJatAhatfLPPc2e7V/rJJJiNNWRSLqZMQPOOw+ysuC991ROkrJUUCLpZPhwaNMGfvUrWLAA6tcPnUik1FRQIulg927o1cvPDvHHP8I770DNmqFTiURFBSWS6jZt8iP1hgzxJfXmm1C5cuhUIlHTIAmRVPb1176cvvrKn97r0SN0IpGYUUGJpKpZs6BTJz+F0cyZ0Lx56EQiMaVTfCKpJj8fBg2Ciy+GY4+FDz5QOUla0hGUSCrZuBG6dPHPM3XqBCNHQsWKoVOJxIUKSiRVLF0Kl18OOTl+QMTNN2tOPUlrOsUnkuycg6eegjPO8BO/zp0Lt9yicpK0p4ISSWbr10O7dv5o6cILYckSzQwhGUMFJZKsZs+Gxo391EVDhvjlMo4+OnQqkYRRQYkkmy1b4C9/8TNCVKnil8rQKT3JQCookWQyezY0bAhDh/pZIRYuhF//OnQqkSBUUCLJYONG+NOf/FFTVha8+y489hhUqBA6mUgwKiiRkPLz4bnnoF49GD0a+vb1AyHOPjt0MpHg9DookVAWLoSePf1MEGef7QdDNG0aOpVI0tARlEiirVoF3br51zXl5MDYsf6UnspJZB86ghJJlPXr/Rx6Tz/tR+T99a9wzz1w5JGhk4kkJRWUSLzl5cETT/jXMm3dCl27wsCBUKtW6GQiSU0FJRIvq1bB4MEwahT897/Qvj3cey80aBA6mUhKUEGJxJJz8N57/nVMEyb46zp3hjvugPr1w2YTSTEqKJFY+PFHeOEFX0yffeafV+rZE267DU48MXQ6kZSkghIprR07YNo0ePFFP0/e9u1w+un+lF7HjlqnSSRKKiiRQ7Fli19q/c03YdIk2LQJatSAG26Aa6+FZs00Z55IjKigRA7EOVi+3M+RN2WKX4tp506oXNkvg3HNNX4ZjLL6VRKJNf1WiRS0axd8/jnMnw/z5sE77/hh4gCnnOLXZWrTxq/JVK5c2KwiaU4FJZlr82Z/dLRkCSxaBIsX+2XVd+zwt//yl9CyJfz+93D++XDSSSHTimQcFZSkL+fghx9g9Wr/9vXX8OWX/m35cli79udtq1TxAxxuvtm/P/tsX1AiEkxUBWVmLYAhQBlglHPuwZikEilOfr4/8snL81MHrV//8+W8PP+Wm/tzKW3btu/nV6kCp54KF13k359yil9vqU4dDW4QSTKlLigzKwM8DfwRyAU+MrM3nHNfxCqcBOYc7Nmz79vu3ftfV9zb7t3+dNn27f79gS7v/XjrVl9Ae99+/HHfj7ds8bmKcsQRfkTd8cdDo0bQqpV/DdLetzp1/O0qIpGUEM0R1BnACufcSgAzexloB8SvoL76yo+cgn3/SMXzcqLuJxmy5OfvWzDFFUE8HHaYL5iKFaFSpZ/fatSAunX3ve7II/31NWpA9eo/v69YUeUjkkaiKajjgdUFPs4Fziy8kZn1AHoA1Ip2csysLL8c9s9fPDGXE3U/obOYQZkyB34rW/bg2xTcNivLF88RRxR9ee97DdMWkULi/lfBOTcCGAGQnZ0d3b/kJ5748/xmIiKS1qJZsHANUHCSsRMi14mIiEQtmoL6CDjZzOqY2eFAR+CN2MQSEZFMZy6KJ8LNrBXwOH6Y+Wjn3P0H2T4P+LbUd/iz6sD6GHydRFHe+Eu1zKmWF1Ivs/LGXywy/9I5V6OoG6IqqFDMbKFzLjt0jpJS3vhLtcyplhdSL7Pyxl+8M0dzik9ERCRuVFAiIpKUUrWgRoQOcIiUN/5SLXOq5YXUy6y88RfXzCn5HJSIiKS/VD2CEhGRNKeCEhGRpJSUBWVmV5jZ52aWb2bZhW7rb2YrzOxLM7u4mM+vY2YfRLYbH3khccJE7nNJ5C3HzJYUs12OmX0a2W5hIjMWyjHQzNYUyNyqmO1aRPb7CjPrl+ichbI8YmbLzWypmb1mZlWK2S7oPj7YPjOzIyKPlxWRx2ztRGcskOVEM/uXmX0R+f27tYhtzjezTQUeK/eEyFoo0wF/xuY9EdnHS83s9BA5I1lOKbDvlpjZj2bWq9A2wfexmY02s+/N7LMC1x1lZrPM7KvI+6rFfG7XyDZfmVnXqII455LuDagPnALMBbILXN8A+AQ4AqgDfA2UKeLzJwAdI5eHATcF/F7+CdxTzG05QPUk2N8Dgd4H2aZMZH/XBQ6P/BwaBMx8EVA2cvkh4KFk28cl2WfAn4FhkcsdgfEB9+lxwOmRy5WAfxeR93xgSqiMpfkZA62AtwADfgt8EDpzgcfHf/AvVE2qfQz8Hjgd+KzAdQ8D/SKX+xX1OwccBayMvK8auVy1tDmS8gjKObfMOfdlETe1A152zu1wzn0DrMAv+/ETMzPgAuDVyFVjgUvjGLdYkSxXAi+FuP8Y+2l5FefcTmDv8ipBOOdmOud2Rz58Hz8XZLIpyT5rh3+Mgn/MXhh53CScc26tc25x5PJmYBl+1YJU1w74X+e9D1Qxs+NChwIuBL52zsVidp2Ycs69A/xQ6OqCj9Xi/q5eDMxyzv3gnPs/YBbQorQ5krKgDqCoJT4K/wJVAzYW+ONV1DaJ8jtgnXPuq2Jud8BMM1sUWZYkpJ6R0x+jizl0L8m+D6U7/j/kooTcxyXZZz9tE3nMbsI/hoOKnGpsCnxQxM1nmdknZvaWmZ2W2GRFOtjPOFkfux0p/p/XZNvHAMc459ZGLv8HOKaIbWK6r4MtwmNms4Fji7jpLufc5ETnOVQlzN+JAx89neucW2NmRwOzzGx55D+XmDtQXmAocC/+F/1e/GnJ7vHIcShKso/N7C5gNzCumC+TsH2cLszsF8BEoJdz7sdCNy/Gn5LaEnmu8nXg5ARHLCzlfsaR58XbAv2LuDkZ9/E+nHPOzOL+GqVgBeWc+0MpPq0kS3xswB/Cl438RxqXZUAOlt/MygLtgd8c4Gusibz/3sxew58SissvVkn3t5mNBKYUcVPCl1cpwT6+DmgDXOgiJ8CL+BoJ28dFKMk+27tNbuQxcyT+MRyEmZXDl9M459ykwrcXLCzn3DQze8bMqjvngk1yWoKfcTIuDdQSWOycW1f4hmTcxxHrzOw459zayCnS74vYZg3+ObS9TsCPJSiVVDvF9wbQMTLyqQ7+v4oPC24Q+UP1L6BD5KquQIgjsj8Ay51zuUXdaGYVzazS3sv4J/0/K2rbeCt0Pv6yYnIk1fIqZtYCuANo65zbVsw2ofdxSfbZG/jHKPjH7NvFlW28RZ77ehZY5px7tJhtjt37HJmZnYH/GxKyUEvyM34D6BIZzfdbYFOBU1WhFHt2Jdn2cQEFH6vF/V2dAVxkZlUjTxVcFLmudEKOFCnuDf9HMhfYAawDZhS47S78yKgvgZYFrp8G1IxcrosvrhXAK8ARAb6HMcCNha6rCUwrkPGTyNvn+NNWofb388CnwNLIg/C4wnkjH7fCj+z6OmTeSJYV+HPdSyJve0fCJdU+LmqfAf/AFytAVuQxuiLymK0bcJ+eiz/Nu7TAfm0F3Lj3sQz0jOzLT/CDU84O/Dgo8mdcKLMBT0d+Bp9SYGRwoMwV8YVzZIHrkmof48tzLbAr8rf4evxzo3OAr4DZwFGRbbOBUQU+t3vk8bwC6BZNDk11JCIiSSnVTvGJiEiGUEGJiEhSUkGJiEhSUkGJiEhSUkGJiEhSUkGJiEhSUkGJiEhS+n8baWKaCbjmuAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 504x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(7, 4))\n",
    "plt.plot(x_numpy, torch_softplus, c='r', label='softplus')\n",
    "plt.ylim((-0.2, 11))\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Leaky Relu\n",
    "\n",
    "![](assets/2022-02-22-22-19-14.png)\n",
    "\n",
    "In a standard neural network model, a dying gradient problem is common.\n",
    "To avoid this issue, leaky ReLU is applied. Leaky ReLU allows a small and\n",
    "non-zero gradient when the unit is not active.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fe536801d30>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAD4CAYAAAC5S3KDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfP0lEQVR4nO3deZgU1bnH8e/rACKCskYJoKDhGtzCMldjxESD7MiighsqimLciTG4XkOIcYlCEBUVEYGIG6CyKaAobokEEFAEDIjbIOIIiAqyzrl/nIYMYw/MTC+nuvv3eZ556OmunvpR3dPvVNWp95hzDhERkajZJ3QAERGReFSgREQkklSgREQkklSgREQkklSgREQkkiqlc2V169Z1jRs3TucqRUQkwubPn/+1c65evMfSWqAaN27MvHnz0rlKERGJMDP7tLTHdIhPREQiSQVKREQiSQVKREQiKa3noOLZtm0bBQUFbN68OXSUnFC1alUaNmxI5cqVQ0cREdmj4AWqoKCAGjVq0LhxY8wsdJys5pxj7dq1FBQU0KRJk9BxRET2KPghvs2bN1OnTh0VpzQwM+rUqaO9VRHJCMELFKDilEba1iKSKSJRoEREREpSgQKqV6+e8M8YOHAg9957b4Wem5eXR/PmzTn66KM57bTT+Oabb8q9rj59+jBhwoTd7kvG/0tEJBQVqAjYb7/9WLhwIYsXL6Z27do8+OCDoSOJiASnAlWKjz76iA4dOtCqVStOOukkli1bBsCUKVM4/vjjadGiBaeeeipr1qz50XMfffRROnbsyB//+EeGDh266/5bbrmF++67b4/rPeGEE1i1atUeM4iI5ILgw8x3078/LFyY3J/ZvDkUKxJl1a9fPx5++GGaNm3KnDlzuOKKK3j11Vdp3bo177zzDmbGyJEj+dvf/sbgwYN3Pe+BBx7g5Zdf5oUXXmD16tWcfvrp9O/fn6KiIp5++mn+/e9/l7rOHTt2MGvWLPr27bvHDCIiuSBaBSoivv/+e/75z3/Ss2fPXfdt2bIF8NdtnXXWWaxevZqtW7fudj3R2LFjadSoES+88AKVK1emcePG1KlThwULFrBmzRpatGhBnTp1frS+H374gebNm7Nq1SqaNWtG27Zt95ghnnij8zRiT0QyWbQKVAX2dFKhqKiImjVrsjDO3tzVV1/NddddR9euXZk9ezYDBw7c9dgxxxzDwoULd7sQ9pJLLmH06NF8+eWXXHzxxXHXt/Mc1KZNm2jfvj0PPvggffr0KTVDPHXq1GH9+vW7vl+3bh1169Yt8/9ZRCRqdA4qjgMOOIAmTZowfvx4wHdgWLRoEQAbNmygQYMGAIwZM2a357Vo0YJHHnmErl278sUXXwDQo0cPpk+fzty5c2nfvv0e11utWjWGDRvG4MGDqVatWqkZ4jn55JN55pln2Lp1KwCjR4/mlFNOqcD/XkQkGvZaoMxslJl9ZWaLi91X28xeNrPlsX9rpTZmam3atImGDRvu+hoyZAjjxo3jscce4xe/+AVHHXUUkyZNAvwQ7549e9KqVau4eyitW7fm3nvvpXPnznz99ddUqVKFU045hV69epGXl7fXLC1atODYY4/lqaeeKjUDwO23375b5i5dunDSSSfRqlUrmjdvzttvv83dd9+dvI0kIpJm5pzb8wJmvwa+B8Y6546O3fc3YJ1z7i4zuxGo5Zy7YW8ry8/PdyUnLFy6dCnNmjWraP7IKyoqomXLlowfP56mTZuGjgNk/zYXkTQoLIR6cSfCLRczm++cy4/32F73oJxzbwDrStzdDdh5fGsM0D2RgNlqyZIl/OxnP6NNmzaRKU4iIgn74AM44gh46KGUrqaigyQOcs6tjt3+EjiotAXNrB/QD+CQQw6p4Ooy05FHHsnKlStDxxARSZ5PP4X27aFqVejYMaWrSniQhPPHCEs9TuicG+Gcy3fO5dcrZXdwb4cZJXm0rUWkwgoLoV072LgRZsyAxo1TurqKFqg1ZlYfIPbvVxUNULVqVdauXasPzjTYOR9U1apVQ0cRkUzz3XfQqRN89hlMnQrHHJPyVVb0EN9k4ELgrti/k/a8eOkaNmxIQUEBhYWFFf0RUg47Z9QVESmzLVugRw9YsABeeAFOPDEtq91rgTKzp4CTgbpmVgD8CV+YnjWzvsCnQK+KBqhcubJmdxURiaodO+D882HWLBgzBrp0Sduq91qgnHPnlPJQmyRnERGRKHEOrrkGxo+He++FCy5I6+rVSUJEROIbNAiGD4cBA+APf0j76lWgRETkx4YPh4ED4aKL4K67gkRQgRIRkd09+yxcdRV07QojRkCgmRFUoERE5L9efhl694bWreHpp6FSuEkvVKBERMSbO9cPJ2/WDCZPhv32CxpHBUpERGDZMt+66Cc/genToWbN0IlUoEREcl5Bge+vl5cHM2dC/fqhEwFRm1FXRETSa906X5zWr4fXX4ef/Sx0ol1UoEREctXGjb4zxIoVvvlrixahE+1GBUpEJBdt2wY9e8KcOb5TxMknh070IypQIiK5pqgILr4YXnrJX+d0+umhE8WlQRIiIrnEOd+26Ikn4K9/hUsvDZ2oVCpQIiK55K67YOhQ6N8fbropdJo9UoESEckVI0fCzTfDeefB4MHBWhiVlQqUiEgueP55uOwyfzHu44/DPtH/+I9+QhERSczs2XDOOXDccX7EXuXKoROViQqUiEg2W7DAdyU//HCYNg323z90ojJTgRIRyVYrVkCHDr6v3owZULt26ETlogIlIpKNVq/2LYx27PD99Ro2DJ2o3HShrohItvnmGz8YYs0aePVV+PnPQyeqEBUoEZFs8sMP/pzTkiX+nNNxx4VOVGEqUCIi2WL7djj7bHjrLT8bbtu2oRMlRAVKRCQbOAf9+vmZcB98EHr1Cp0oYRokISKSDW66yV+A+6c/wRVXhE6TFCpQIiKZbvBguPtuuPxyX6CyhAqUiEgmGzsWrr/ez+10//2R769XHipQIiKZato0P69Tmzbwj39AXl7oREmlAiUikoneftvvNTVv7hvB7rtv6ERJpwIlIpJp3n8funSBRo38rLg1aoROlBIJFSgz+72ZfWBmi83sKTOrmqxgIiISxyef+BZG1ar5Fkb16oVOlDIVLlBm1gC4Bsh3zh0N5AFnJyuYiIiU8NVX0K4dbN7si9Ohh4ZOlFKJXqhbCdjPzLYB1YAvEo8kIiI/8u23vr9eQQG88gocdVToRClX4T0o59wq4F7gM2A1sME5NzNZwUREJGbLFujRAxYtggkT4Fe/Cp0oLRI5xFcL6AY0AX4K7G9mveMs18/M5pnZvMLCwoonFRHJRTt2QO/eviv5449Dp06hE6VNIoMkTgU+ds4VOue2Ac8BPyrrzrkRzrl851x+vSw+mSciknTOwZVX+r2mIUPg/PNDJ0qrRArUZ8AvzayamRnQBlianFgiIsLAgfDII3DjjfD734dOk3aJnIOaA0wA3gXej/2sEUnKJSKS2x54AAYN8p0i7rgjdJogEhrF55z7E5A9nQlFRKLgqafgmmuge3e/B5VF/fXKQ50kRESiZMYMuOAC+PWvfaGqlLvT9qlAiYhExZw5cMYZ/hqnSZOgam4351GBEhGJgqVLoXNnOOggmD4dDjwwdKLgVKBEREL7/HPfX69SJd/C6OCDQyeKhNw9uCkiEgVr1/ritGEDvP46HH546ESRoQIlIhLKxo3+sN7KlX5wRPPmoRNFigqUiEgIW7f6ARFz58LEifCb34ROFDkqUCIi6VZUBH36+L2mkSP99U7yIxokISKSTs5B//7+Gqc774S+fUMniiwVKBGRdLrjDrj/frjuOrjhhtBpIk0FSkQkXUaMgFtv9V3J77knZ1sYlZUKlIhIOkycCJdf7udzeuwx2Ecfv3ujLSQikmqvvQbnngu//CWMHw+VK4dOlBFUoEREUundd6FbN2jaFKZMgWrVQifKGCpQIiKpsnw5dOgAtWr5IeW1a4dOlFFUoEREUuGLL6BdOz+sfOZMaNAgdKKMowt1RUSSbf16v+f09df+/NMRR4ROlJFUoEREkmnTJjjtNPjwQ5g2DfLzQyfKWCpQIiLJsm0bnHUW/POf8MwzcOqpoRNlNBUoEZFkcA4uvRSmToXhw6Fnz9CJMp4GSYiIJMMNN8CYMfDnP/sLciVhKlAiIom65x7/deWV8H//FzpN1lCBEhFJxOjRMGCAP/c0bJj66yWRCpSISEVNmQKXXAJt28LYseqvl2TamiIiFfHmm9CrF7Rs6RvBVqkSOlHWUYESESmv997z1zodeii8+CLUqBE6UVZSgRIRKY+PP4b27aF6dd/CqG7d0Imylq6DEhEpqzVrfH+9LVvgrbfgkENCJ8pqKlAiImXx7bfQsaNvAvvKK3DkkaETZT0VKBGRvdm8Gbp3h/ffh8mT4YQTQifKCQmdgzKzmmY2wcyWmdlSM9OrJiLZZccOOO8835V89Gi/FyVpkege1H3AdOfcmWZWBdBUkSKSPZyDK66A556DoUN9oZK0qXCBMrMDgV8DfQCcc1uBrcmJJSISAbfdBiNGwM03w7XXhk6TcxI5xNcEKAQeN7MFZjbSzPYvuZCZ9TOzeWY2r7CwMIHViYik0bBhcPvtvlPE7beHTpOTEilQlYCWwEPOuRbARuDGkgs550Y45/Kdc/n16tVLYHUiImkybpzfY+rRAx56SP31AkmkQBUABc65ObHvJ+ALlohI5po+Hfr0gZNPhiefhEoa7BxKhQuUc+5L4HMzOyJ2VxtgSVJSiYiE8M47cMYZcMwxMGkSVK0aOlFOS/RPg6uBcbERfCuBixKPJCISwJIl0Lkz1K8PL70EBxwQOlHOS6hAOecWAvnJiSIiEshnn/n+elWq+P56Bx0UOpGgThIikuu+/toXp+++gzfegMMOC51IYlSgRCR3ff+9P6z3ySd+z+nYY0MnkmJUoEQkN23d6gdEzJ/vO0WcdFLoRFKCCpSI5J6iIrjgAr/XNGoUdO0aOpHEoQkLRSS3OOcvwn3mGbj7brhIg4+jSgVKRHLL7bfDAw/A9dfDgAGh08geqECJSO54+GHfAPbCC/3ek0SaCpSI5IYJE/zUGV26wKOPwj76+Is6vUIikv1mzfJzOf3qV/7cU+XKoRNJGahAiUh2mz/fT9f+P/8DU6ZANc2rmilUoEQke/3nP36K9jp1YMYMqFUrdCIpBxUoEclOq1ZBu3b+9syZ8NOfhs0j5aYLdUUk+6xb5/vrrV0Ls2f7w3uScVSgRCS7bNoEp50Gy5f7aTNatQqdSCpIBUpEsse2bdCrF/zrXzB+PPz2t6ETSQJUoEQkOxQVQd++MG2avyD3jDNCJ5IEaZCEiGQ+5+CPf4R//AP+8he47LLQiSQJVKBEJPPdcw8MGQJXXw233BI6jSSJCpSIZLZRo+CGG+Ccc2DoUDALnUiSRAVKRDLXpElw6aX+eqfRo9VfL8vo1RSRzPTGG3DWWZCfDxMnQpUqoRNJkqlAiUjmWbTIX+vUpIkftVe9euhEkgIqUCKSWVau9F0iDjjAtzCqWzd0IkkRXQclIpnjyy/9+aZt2+C116BRo9CJJIVUoEQkM2zY4DuTr14Nr74KzZqFTiQppgIlItG3eTN06waLF8PUqXD88aETSRqoQIlItG3f7q9xev11ePJJf/5JcoIGSYhIdDkHl18OL7wA993nC5XkDBUoEYmuW26BkSPh1lvhmmtCp5E0U4ESkWj6+9/hzjuhXz8YNCh0Ggkg4QJlZnlmtsDMpiYjkIgITzwB113np8wYPlz99XJUMvagrgWWJuHniIjAiy/CRRf5yQbHjYO8vNCJJJCECpSZNQQ6AyOTE0dEctq//gVnngnHHgvPPw/77hs6kQSU6B7UUGAAUFTaAmbWz8zmmdm8wsLCBFcnIlnrgw+gc2do0ABeesm3MpKcVuECZWZdgK+cc/P3tJxzboRzLt85l1+vXr2Krk5Estmnn/rrm6pW9f31fvKT0IkkAhK5UPdEoKuZdQKqAgeY2RPOud7JiSYiOaGw0PfX27jRT6HRpEnoRBIRFd6Dcs7d5Jxr6JxrDJwNvKriJCLl8t130KkTfPYZTJkCxxwTOpFEiFodiUgYW7ZAjx6wYIEfENG6dehEEjFJKVDOudnA7GT8LBHJATt2wPnnw6xZfqr2004LnUgiSJ0kRCS9nPNti8aPh3vugQsvDJ1IIkoFSkTSa9Ag3x1iwAC4/vrQaSTCVKBEJH2GD4eBA32niLvuCp1GIk4FSkTS49ln4aqroGtXGDFC/fVkr1SgRCT1XnkFeveGE0+Ep5+GShpALHunAiUiqTV3LnTvDj//ub/Wab/9QieSDKECJSKps2yZvxC3Xj2YMQNq1gydSDKICpSIpEZBge+vt88+vr9e/fqhE0mG0YFgEUm+det8cVq/HmbPhqZNQyeSDKQCJSLJtXEjdOkCK1bA9OnQsmXoRJKhVKBEJHm2bYOePWHOHN8p4pRTQieSDKYCJSLJUVQEF1/sJxscMQJOPz10IslwGiQhIolzDv7wB3jiCfjrX+HSS0MnkiygAiUiibv7bhg6FK69Fm66KXQayRIqUCKSmJEjfVE691wYMkQtjCRpVKBEpOKefx4uuww6dIDHH/fXPIkkid5NIlIxs2fDOefA//4vTJgAVaqETiRZRgVKRMpvwQLflfyww2DaNNh//9CJJAupQIlI+axY4Q/p1azpWxjVqRM6kWQpFSgRKbvVq30Lox07fHFq2DB0IsliulBXRMrmm2+gY0dYswZefdVPnyGSQipQIrJ3P/wA3brBkiX+nNNxx4VOJDlABUpE9mz7dj9a78034amnoG3b0IkkR6hAiUjpnPPXOU2aBPffD2edFTqR5BANkhCR0t10E4waBbfdBlddFTqN5BgVKBGJb/Bg32Pvd7+DgQNDp5EcpAIlIj82dixcfz2ceSY88ID660kQKlAisrtp0/y8Tm3a+Okz8vJCJ5IcpQIlIv/19tt+RtzmzX0j2H33DZ1IcpgKlIh4ixdDly7QqJGfFbdGjdCJJMdVuECZWSMze83MlpjZB2Z2bTKDiUgaffKJb2FUrRrMmAH16oVOJJLQdVDbgT845941sxrAfDN72Tm3JEnZRCQdvvoK2rWDTZv8xbiNG4dOJAIksAflnFvtnHs3dvs7YCnQIFnBRCQNvv3W99crKICpU+Hoo0MnEtklKeegzKwx0AKYE+exfmY2z8zmFRYWJmN1IpIMW7ZAjx6waBGMHw8nnhg6kchuEi5QZlYdmAj0d859W/Jx59wI51y+cy6/no5ri0TDjh3Qu7fvSj5qFHTuHDqRyI8kVKDMrDK+OI1zzj2XnEgiklLOwZVX+mnaBw+GCy4InUgkrkRG8RnwGLDUOTckeZFEJKUGDoRHHoEbb4TrrgudRqRUiexBnQicD/zWzBbGvjolKZeIpMIDD8CgQb5TxB13hE4jskcVHmbunHsLUIMukUzx9NNwzTV+4sFHHlF/PYk8dZIQyQUzZ/pzTSed5CcdrKSp4CT6VKBEst2cOXD66XDkkTB5Muy3X+hEImWiAiWSzZYu9UPIDzoIpk+HAw8MnUikzFSgRLLV55/7/nqVKvlDfAcfHDqRSLnoQLRINlq71henDRtg9mw4/PDQiUTKTQVKJNts3OgP661c6TuTt2gROpFIhahAiWSTrVvhjDNg7lyYOBF+85vQiUQqTAVKJFsUFcFFF/m9ppEjoXv30IlEEqJBEiLZwDn4/e/hySfhzjuhb9/QiUQSpgIlkg3uuAOGDfNF6oYbQqcRSQoVKJFMN2IE3Hqrnz7j3nvVwkiyhgqUSCabOBEuv9zPijtqFOyjX2nJHno3i2Sq116Dc8+F44/3M+JWrhw6kUhSqUCJZKJ33/VdyZs2halTYf/9QycSSToVKJFMs3w5dOgAtWr5IeW1a4dOJJISKlAimeSLL6BdOz+sfOZMaNAgdCKRlNGFuiKZ4ptv/J7T11/7809HHBE6kUhKqUCJZIIffoDTToNly+DFFyE/P3QikZRTgRKJuu3b4ayz4O23/bTtp54aOpFIWqhAiUSZc3DJJTBlCgwfDr16hU4kkjYaJCESZTfcAGPGwMCB/oJckRyiAiUSVffc47+uuAJuuy10GpG0U4ESiaLRo2HAAH/uadgw9deTnKQCJRI1U6b4805t28LYsZCXFzqRSBAqUCJR8tZbfiBEy5a+EWyVKqETiQSjAiUSFe+9B126wKGHwrRpUKNG6EQiQalAiUTBxx/7LhHVq/v+evXqhU4kEpyugxIJbc0a319v82Z4802/ByUiKlAiQX37rZ9scNUqmDULjjoqdCKRyFCBEgll82bo3h3efx8mT4YTTgidSCRSEjoHZWYdzOxDM1thZjcmK5RI1tuxA847z3clf/xxvxclIrupcIEyszzgQaAjcCRwjpkdmaxgIlnLOd8d4rnn4O9/h969QycSiaREDvEdB6xwzq0EMLOngW7AkmQEkwgrKvJ7APG+9vRYWb4SeX7IdZfn+du3w/r1cPPN0L9/6FdTJLISKVANgM+LfV8AHF9yITPrB/QDOOSQQxJYXTlV9MMmUz7kQj4/05j5bgxl+dpnn7Ivu+++FX9+s2Z+L0pESpXyQRLOuRHACID8/HyX0A9btgzatCnbB2ym2fnBVp4PyLJ8wMb7EE3WB3SmPF997EQyUiIFahXQqNj3DWP3pU6NGv5kcqZ9QJblOfoQFRHZTSIFai7Q1Mya4AvT2cC5SUlVmgYNYOTIlK5CRESiwZyr+FE3M+sEDAXygFHOub/uZflC4NMKr/C/6gJfJ+HnpIvypl6mZc60vJB5mZU39ZKR+VDnXNzeXgkVqFDMbJ5zLj90jrJS3tTLtMyZlhcyL7Pypl6qM6tZrIiIRJIKlIiIRFKmFqgRoQOUk/KmXqZlzrS8kHmZlTf1Upo5I89BiYhI9svUPSgREclyKlAiIhJJkSxQZtbTzD4wsyIzyy/x2E2x6T0+NLP2pTy/iZnNiS33jJlVSU/yXet/xswWxr4+MbOFpSz3iZm9H1tuXjozlsgx0MxWFcvcqZTlIjO9ipndY2bLzOw9M3vezGqWslzQbby3bWZm+8beLyti79nG6c5YLEsjM3vNzJbEfv+ujbPMyWa2odh75bYQWUtk2uNrbN6w2DZ+z8xahsgZy3JEsW230My+NbP+JZYJvo3NbJSZfWVmi4vdV9vMXjaz5bF/a5Xy3Atjyyw3swsTCuKci9wX0Aw4ApgN5Be7/0hgEbAv0AT4CMiL8/xngbNjtx8GLg/4fxkM3FbKY58AdSOwvQcC1+9lmbzY9j4MqBJ7HY4MmLkdUCl2+27g7qht47JsM+AK4OHY7bOBZwJu0/pAy9jtGsB/4uQ9GZgaKmNFXmOgE/ASYMAvgTmhMxd7f3yJv1A1UtsY+DXQElhc7L6/ATfGbt8Y73cOqA2sjP1bK3a7VkVzRHIPyjm31Dn3YZyHugFPO+e2OOc+Blbgp/3YxcwM+C0wIXbXGKB7CuOWKpalF/BUiPUn2a7pVZxzW4Gd06sE4Zyb6ZzbHvv2HXwvyKgpyzbrhn+Pgn/Ptom9b9LOObfaOfdu7PZ3wFL8rAWZrhsw1nnvADXNrH7oUEAb4CPnXDK66ySVc+4NYF2Ju4u/V0v7XG0PvOycW+ecWw+8DHSoaI5IFqg9iDfFR8lfoDrAN8U+vOItky4nAWucc8tLedwBM81sfmxakpCuih3+GFXKrntZtn0oF+P/Qo4n5DYuyzbbtUzsPbsB/x4OKnaosQUwJ87DJ5jZIjN7ycyOSm+yuPb2Gkf1vXs2pf/xGrVtDHCQc2517PaXwEFxlknqtk75dBulMbNXgIPjPHSLc25SuvOUVxnzn8Oe955aO+dWmdlPgJfNbFnsL5ek21Ne4CHgL/hf9L/gD0tenIoc5VGWbWxmtwDbgXGl/Ji0beNsYWbVgYlAf+fctyUefhd/SOr72LnKF4CmaY5YUsa9xrHz4l2Bm+I8HMVtvBvnnDOzlF+jFKxAOedOrcDTyjLFx1r8Lnyl2F+kKZkGZG/5zawScDrQag8/Y1Xs36/M7Hn8IaGU/GKVdXub2aPA1DgPpX16lTJs4z5AF6CNix0Aj/Mz0raN4yjLNtu5TEHsPXMg/j0chJlVxhencc6550o+XrxgOedeNLPhZlbXOResyWkZXuP0Tw20dx2Bd51za0o+EMVtHLPGzOo751bHDpF+FWeZVfhzaDs1xI8lqJBMO8Q3GTg7NvKpCf6vin8XXyD2QfUacGbsrguBEHtkpwLLnHMF8R40s/3NrMbO2/iT/ovjLZtqJY7H9yglx67pVWJ//Z2Nfz2CMLMOwACgq3NuUynLhN7GZdlmk/HvUfDv2VdLK7apFjv39Riw1Dk3pJRlDt55jszMjsN/hoQsqGV5jScDF8RG8/0S2FDsUFUopR5dido2Lqb4e7W0z9UZQDszqxU7VdAudl/FhBwpUtoX/kOyANgCrAFmFHvsFvzIqA+BjsXufxH4aez2YfjCtQIYD+wb4P8wGvhdift+CrxYLOOi2NcH+MNWobb3P4D3gfdib8L6JfPGvu+EH9n1Uci8sSwr8Me6F8a+do6Ei9Q2jrfNgEH4wgpQNfYeXRF7zx4WcJu2xh/mfa/Ydu0E/G7nexm4KrYtF+EHp/wq8Psg7mtcIrMBD8Zeg/cpNjI4UOb98QXnwGL3RWob44vnamBb7LO4L/7c6CxgOfAKUDu2bD4wsthzL469n1cAFyWSQ62OREQkkjLtEJ+IiOQIFSgREYkkFSgREYkkFSgREYkkFSgREYkkFSgREYkkFSgREYmk/wdIOPfFZ9mW+wAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 504x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(7, 4))\n",
    "plt.plot(x_numpy, torch_leaky_relu, c='r', label='Leaky ReLU')\n",
    "plt.ylim((-0.2, 11))\n",
    "plt.legend(loc='best')"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "36cf16204b8548560b1c020c4e8fb5b57f0e4c58016f52f2d4be01e192833930"
  },
  "kernelspec": {
   "display_name": "Python 3.9.10 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}