session-3/optimization-lecture.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0ffb3ecf",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Julia for Economists\n",
    "## Automatic Differentiation and Optimization\n",
    "### Cameron Pfiffer (cpfiffer@stanford.edu)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67862896",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Who am I?\n",
    "\n",
    "- I'm Cameron Pfiffer, a PhD student in finance at the University of Oregon.\n",
    "- I'm visiting at the Stanford GSB\n",
    "- I work on Bayesian econometrics, asset pricing, industrial organization, and general computer go-fast stuff\n",
    "- I'm doing a 5-part series on Julia for economists. \n",
    "- Howdy!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "860b42bf",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What are we doing today?\n",
    "\n",
    "- **Automatic differentiation**: how to get exact derivatives for arbitrary code\n",
    "- **Optimization**: how to optimize most functions with Julia tools\n",
    "- Lecturing with intermittent periods for you to practice concepts\n",
    "- We go from 9am-1pm, with breaks at the top of each hour\n",
    "- Ask questions when needed! Feel free to interrupt me.\n",
    "- If you need a refresher about basics, please ask!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb21f41d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Resources\n",
    "\n",
    "Everything done here will be available on the GitHub repo for this course:\n",
    "    \n",
    "/~https://github.com/cpfiffer/julia-bootcamp-2022"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d846b693",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Automatic differentiation\n",
    "\n",
    "We often have functions we want to take the derivative/gradient/Jacobian/Hessian of for various reasons:\n",
    "\n",
    "- Derivatives help us optimize functions (the big one for our purposes today)\n",
    "- Derivatives often have economic intuition on their own (marginal utility/marginal cost/etc.)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22dcd12c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A common way one might work with derivatives in their research is simply to take an analytic derivative.\n",
    "\n",
    "For example, let's look at the following function $f(x)$:\n",
    "\n",
    "$$\n",
    "f(x) = \\sin(x^2)\n",
    "$$\n",
    "\n",
    "If we want $d f(x) / dx$, we can calculate this with the chain rule:\n",
    "\n",
    "$$\n",
    "\\frac{df(x)}{dx} = \\frac{d \\sin(x^2)}{dx} = \\cos(x^2) \\frac{d x^2}{dx} = 2x \\cos(x^2) \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "680b95a8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In Julia, this would be written up as"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "478d0adf",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "df (generic function with 1 method)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f(x) = sin(x^2)\n",
    "df(x) = 2 * x * cos(x^2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "869a29e4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip600\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip600)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip601\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip600)\" d=\"\n",
       "M149.191 1486.45 L2352.76 1486.45 L2352.76 47.2441 L149.191 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip602\">\n",
       "    <rect x=\"149\" y=\"47\" width=\"2205\" height=\"1440\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip602)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  211.556,1486.45 211.556,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip602)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  904.501,1486.45 904.501,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip602)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1597.45,1486.45 1597.45,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip602)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2290.39,1486.45 2290.39,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip600)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  149.191,1486.45 2352.76,1486.45 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip600)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  211.556,1486.45 211.556,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip600)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  904.501,1486.45 904.501,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip600)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1597.45,1486.45 1597.45,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip600)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2290.39,1486.45 2290.39,1467.55 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip600)\" d=\"M211.556 1517.37 Q207.945 1517.37 206.116 1520.93 Q204.311 1524.47 204.311 1531.6 Q204.311 1538.71 206.116 1542.27 Q207.945 1545.82 211.556 1545.82 Q215.19 1545.82 216.996 1542.27 Q218.824 1538.71 218.824 1531.6 Q218.824 1524.47 216.996 1520.93 Q215.19 1517.37 211.556 1517.37 M211.556 1513.66 Q217.366 1513.66 220.422 1518.27 Q223.5 1522.85 223.5 1531.6 Q223.5 1540.33 220.422 1544.94 Q217.366 1549.52 211.556 1549.52 Q205.746 1549.52 202.667 1544.94 Q199.612 1540.33 199.612 1531.6 Q199.612 1522.85 202.667 1518.27 Q205.746 1513.66 211.556 1513.66 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M894.883 1544.91 L902.522 1544.91 L902.522 1518.55 L894.212 1520.21 L894.212 1515.95 L902.475 1514.29 L907.151 1514.29 L907.151 1544.91 L914.79 1544.91 L914.79 1548.85 L894.883 1548.85 L894.883 1544.91 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M1592.1 1544.91 L1608.42 1544.91 L1608.42 1548.85 L1586.47 1548.85 L1586.47 1544.91 Q1589.14 1542.16 1593.72 1537.53 Q1598.33 1532.88 1599.51 1531.53 Q1601.75 1529.01 1602.63 1527.27 Q1603.53 1525.51 1603.53 1523.82 Q1603.53 1521.07 1601.59 1519.33 Q1599.67 1517.6 1596.57 1517.6 Q1594.37 1517.6 1591.91 1518.36 Q1589.48 1519.13 1586.71 1520.68 L1586.71 1515.95 Q1589.53 1514.82 1591.98 1514.24 Q1594.44 1513.66 1596.47 1513.66 Q1601.84 1513.66 1605.04 1516.35 Q1608.23 1519.03 1608.23 1523.52 Q1608.23 1525.65 1607.42 1527.57 Q1606.64 1529.47 1604.53 1532.07 Q1603.95 1532.74 1600.85 1535.95 Q1597.75 1539.15 1592.1 1544.91 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M2294.64 1530.21 Q2297.99 1530.93 2299.87 1533.2 Q2301.77 1535.47 2301.77 1538.8 Q2301.77 1543.92 2298.25 1546.72 Q2294.73 1549.52 2288.25 1549.52 Q2286.07 1549.52 2283.76 1549.08 Q2281.47 1548.66 2279.01 1547.81 L2279.01 1543.29 Q2280.96 1544.43 2283.27 1545.01 Q2285.59 1545.58 2288.11 1545.58 Q2292.51 1545.58 2294.8 1543.85 Q2297.12 1542.11 2297.12 1538.8 Q2297.12 1535.75 2294.96 1534.03 Q2292.83 1532.3 2289.01 1532.3 L2284.99 1532.3 L2284.99 1528.45 L2289.2 1528.45 Q2292.65 1528.45 2294.48 1527.09 Q2296.31 1525.7 2296.31 1523.11 Q2296.31 1520.45 2294.41 1519.03 Q2292.53 1517.6 2289.01 1517.6 Q2287.09 1517.6 2284.89 1518.01 Q2282.69 1518.43 2280.06 1519.31 L2280.06 1515.14 Q2282.72 1514.4 2285.03 1514.03 Q2287.37 1513.66 2289.43 1513.66 Q2294.75 1513.66 2297.86 1516.09 Q2300.96 1518.5 2300.96 1522.62 Q2300.96 1525.49 2299.31 1527.48 Q2297.67 1529.45 2294.64 1530.21 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip602)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  149.191,1255.91 2352.76,1255.91 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip602)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  149.191,997.11 2352.76,997.11 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip602)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  149.191,738.307 2352.76,738.307 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip602)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  149.191,479.505 2352.76,479.505 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip602)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  149.191,220.703 2352.76,220.703 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip600)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  149.191,1486.45 149.191,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip600)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  149.191,1255.91 168.089,1255.91 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip600)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  149.191,997.11 168.089,997.11 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip600)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  149.191,738.307 168.089,738.307 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip600)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  149.191,479.505 168.089,479.505 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip600)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  149.191,220.703 168.089,220.703 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip600)\" d=\"M50.9921 1256.36 L80.6679 1256.36 L80.6679 1260.3 L50.9921 1260.3 L50.9921 1256.36 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M103.608 1242.71 L91.8021 1261.16 L103.608 1261.16 L103.608 1242.71 M102.381 1238.63 L108.26 1238.63 L108.26 1261.16 L113.191 1261.16 L113.191 1265.04 L108.26 1265.04 L108.26 1273.19 L103.608 1273.19 L103.608 1265.04 L88.0058 1265.04 L88.0058 1260.53 L102.381 1238.63 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M53.0754 997.561 L82.7512 997.561 L82.7512 1001.5 L53.0754 1001.5 L53.0754 997.561 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M96.8715 1010.45 L113.191 1010.45 L113.191 1014.39 L91.2465 1014.39 L91.2465 1010.45 Q93.9086 1007.7 98.4919 1003.07 Q103.098 998.418 104.279 997.075 Q106.524 994.552 107.404 992.816 Q108.307 991.057 108.307 989.367 Q108.307 986.612 106.362 984.876 Q104.441 983.14 101.339 983.14 Q99.14 983.14 96.6863 983.904 Q94.2558 984.668 91.478 986.219 L91.478 981.497 Q94.3021 980.362 96.7558 979.784 Q99.2095 979.205 101.246 979.205 Q106.617 979.205 109.811 981.89 Q113.006 984.575 113.006 989.066 Q113.006 991.196 112.196 993.117 Q111.408 995.015 109.302 997.608 Q108.723 998.279 105.621 1001.5 Q102.52 1004.69 96.8715 1010.45 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M101.246 724.106 Q97.6354 724.106 95.8067 727.671 Q94.0012 731.213 94.0012 738.342 Q94.0012 745.449 95.8067 749.013 Q97.6354 752.555 101.246 752.555 Q104.881 752.555 106.686 749.013 Q108.515 745.449 108.515 738.342 Q108.515 731.213 106.686 727.671 Q104.881 724.106 101.246 724.106 M101.246 720.402 Q107.057 720.402 110.112 725.009 Q113.191 729.592 113.191 738.342 Q113.191 747.069 110.112 751.675 Q107.057 756.259 101.246 756.259 Q95.4363 756.259 92.3576 751.675 Q89.3021 747.069 89.3021 738.342 Q89.3021 729.592 92.3576 725.009 Q95.4363 720.402 101.246 720.402 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M96.8715 492.85 L113.191 492.85 L113.191 496.785 L91.2465 496.785 L91.2465 492.85 Q93.9086 490.095 98.4919 485.466 Q103.098 480.813 104.279 479.47 Q106.524 476.947 107.404 475.211 Q108.307 473.452 108.307 471.762 Q108.307 469.007 106.362 467.271 Q104.441 465.535 101.339 465.535 Q99.14 465.535 96.6863 466.299 Q94.2558 467.063 91.478 468.614 L91.478 463.892 Q94.3021 462.757 96.7558 462.179 Q99.2095 461.6 101.246 461.6 Q106.617 461.6 109.811 464.285 Q113.006 466.97 113.006 471.461 Q113.006 473.591 112.196 475.512 Q111.408 477.41 109.302 480.003 Q108.723 480.674 105.621 483.892 Q102.52 487.086 96.8715 492.85 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M103.608 207.497 L91.8021 225.946 L103.608 225.946 L103.608 207.497 M102.381 203.423 L108.26 203.423 L108.26 225.946 L113.191 225.946 L113.191 229.834 L108.26 229.834 L108.26 237.983 L103.608 237.983 L103.608 229.834 L88.0058 229.834 L88.0058 225.321 L102.381 203.423 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip602)\" style=\"stroke:#009af9; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  211.556,738.307 232.554,738.189 253.553,737.832 274.551,737.238 295.549,736.406 316.548,735.337 337.546,734.031 358.544,732.487 379.543,730.707 400.541,728.691 \n",
       "  421.539,726.442 442.538,723.959 463.536,721.246 484.534,718.306 505.533,715.143 526.531,711.761 547.529,708.167 568.528,704.369 589.526,700.373 610.524,696.193 \n",
       "  631.523,691.839 652.521,687.326 673.519,682.671 694.518,677.892 715.516,673.011 736.514,668.052 757.513,663.042 778.511,658.01 799.509,652.989 820.508,648.016 \n",
       "  841.506,643.129 862.504,638.37 883.503,633.785 904.501,629.42 925.499,625.327 946.498,621.559 967.496,618.171 988.494,615.22 1009.49,612.765 1030.49,610.862 \n",
       "  1051.49,609.573 1072.49,608.954 1093.49,609.062 1114.48,609.95 1135.48,611.668 1156.48,614.262 1177.48,617.77 1198.48,622.224 1219.48,627.647 1240.47,634.052 \n",
       "  1261.47,641.441 1282.47,649.804 1303.47,659.115 1324.47,669.334 1345.47,680.408 1366.46,692.261 1387.46,704.805 1408.46,717.932 1429.46,731.514 1450.46,745.41 \n",
       "  1471.46,759.459 1492.45,773.484 1513.45,787.295 1534.45,800.69 1555.45,813.458 1576.45,825.381 1597.45,836.239 1618.44,845.814 1639.44,853.894 1660.44,860.28 \n",
       "  1681.44,864.789 1702.44,867.259 1723.44,867.56 1744.43,865.593 1765.43,861.298 1786.43,854.663 1807.43,845.72 1828.43,834.556 1849.43,821.314 1870.42,806.191 \n",
       "  1891.42,789.441 1912.42,771.373 1933.42,752.347 1954.42,732.771 1975.42,713.087 1996.41,693.773 2017.41,675.321 2038.41,658.233 2059.41,643 2080.41,630.092 \n",
       "  2101.41,619.941 2122.4,612.921 2143.4,609.338 2164.4,609.409 2185.4,613.252 2206.4,620.872 2227.4,632.157 2248.39,646.867 2269.39,664.637 2290.39,684.979 \n",
       "  \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip602)\" style=\"stroke:#e26f46; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  211.556,738.307 232.554,730.465 253.553,722.623 274.551,714.781 295.549,706.941 316.548,699.105 337.546,691.278 358.544,683.466 379.543,675.676 400.541,667.92 \n",
       "  421.539,660.213 442.538,652.572 463.536,645.019 484.534,637.58 505.533,630.286 526.531,623.172 547.529,616.279 568.528,609.652 589.526,603.344 610.524,597.413 \n",
       "  631.523,591.92 652.521,586.936 673.519,582.534 694.518,578.797 715.516,575.808 736.514,573.658 757.513,572.443 778.511,572.259 799.509,573.208 820.508,575.391 \n",
       "  841.506,578.91 862.504,583.867 883.503,590.358 904.501,598.476 925.499,608.307 946.498,619.927 967.496,633.398 988.494,648.772 1009.49,666.077 1030.49,685.326 \n",
       "  1051.49,706.503 1072.49,729.57 1093.49,754.454 1114.48,781.051 1135.48,809.222 1156.48,838.786 1177.48,869.525 1198.48,901.176 1219.48,933.431 1240.47,965.939 \n",
       "  1261.47,998.306 1282.47,1030.1 1303.47,1060.83 1324.47,1089.99 1345.47,1117.04 1366.46,1141.41 1387.46,1162.51 1408.46,1179.75 1429.46,1192.55 1450.46,1200.32 \n",
       "  1471.46,1202.53 1492.45,1198.68 1513.45,1188.35 1534.45,1171.18 1555.45,1146.91 1576.45,1115.4 1597.45,1076.64 1618.44,1030.76 1639.44,978.062 1660.44,919.017 \n",
       "  1681.44,854.275 1702.44,784.676 1723.44,711.247 1744.43,635.198 1765.43,557.913 1786.43,480.931 1807.43,405.928 1828.43,334.679 1849.43,269.03 1870.42,210.846 \n",
       "  1891.42,161.968 1912.42,124.154 1933.42,99.0191 1954.42,87.9763 1975.42,92.1705 1996.41,112.417 2017.41,149.144 2038.41,202.336 2059.41,271.493 2080.41,355.6 \n",
       "  2101.41,453.103 2122.4,561.911 2143.4,679.416 2164.4,802.529 2185.4,927.744 2206.4,1051.22 2227.4,1168.88 2248.39,1276.57 2269.39,1370.16 2290.39,1445.72 \n",
       "  \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip600)\" d=\"\n",
       "M1929.32 250.738 L2279.3 250.738 L2279.3 95.2176 L1929.32 95.2176  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip600)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1929.32,250.738 2279.3,250.738 2279.3,95.2176 1929.32,95.2176 1929.32,250.738 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip600)\" style=\"stroke:#009af9; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1953.81,147.058 2100.71,147.058 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip600)\" d=\"M2141.7 128.319 L2141.7 131.861 L2137.63 131.861 Q2135.34 131.861 2134.43 132.787 Q2133.55 133.713 2133.55 136.12 L2133.55 138.412 L2140.57 138.412 L2140.57 141.722 L2133.55 141.722 L2133.55 164.338 L2129.27 164.338 L2129.27 141.722 L2125.2 141.722 L2125.2 138.412 L2129.27 138.412 L2129.27 136.606 Q2129.27 132.278 2131.29 130.31 Q2133.3 128.319 2137.67 128.319 L2141.7 128.319 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M2155.5 128.366 Q2152.4 133.69 2150.89 138.898 Q2149.39 144.106 2149.39 149.453 Q2149.39 154.801 2150.89 160.055 Q2152.42 165.287 2155.5 170.588 L2151.79 170.588 Q2148.32 165.148 2146.59 159.893 Q2144.87 154.639 2144.87 149.453 Q2144.87 144.291 2146.59 139.06 Q2148.3 133.828 2151.79 128.366 L2155.5 128.366 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M2185.31 138.412 L2175.94 151.027 L2185.8 164.338 L2180.78 164.338 L2173.23 154.152 L2165.68 164.338 L2160.66 164.338 L2170.73 150.773 L2161.52 138.412 L2166.54 138.412 L2173.41 147.648 L2180.29 138.412 L2185.31 138.412 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M2191.15 128.366 L2194.85 128.366 Q2198.32 133.828 2200.03 139.06 Q2201.77 144.291 2201.77 149.453 Q2201.77 154.639 2200.03 159.893 Q2198.32 165.148 2194.85 170.588 L2191.15 170.588 Q2194.22 165.287 2195.73 160.055 Q2197.26 154.801 2197.26 149.453 Q2197.26 144.106 2195.73 138.898 Q2194.22 133.69 2191.15 128.366 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip600)\" style=\"stroke:#e26f46; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1953.81,198.898 2100.71,198.898 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip600)\" d=\"M2144.11 194.187 L2144.11 180.159 L2148.37 180.159 L2148.37 216.178 L2144.11 216.178 L2144.11 212.289 Q2142.77 214.603 2140.71 215.738 Q2138.67 216.849 2135.8 216.849 Q2131.1 216.849 2128.14 213.099 Q2125.2 209.349 2125.2 203.238 Q2125.2 197.127 2128.14 193.377 Q2131.1 189.627 2135.8 189.627 Q2138.67 189.627 2140.71 190.761 Q2142.77 191.872 2144.11 194.187 M2129.6 203.238 Q2129.6 207.937 2131.52 210.622 Q2133.46 213.284 2136.84 213.284 Q2140.22 213.284 2142.16 210.622 Q2144.11 207.937 2144.11 203.238 Q2144.11 198.539 2142.16 195.877 Q2140.22 193.192 2136.84 193.192 Q2133.46 193.192 2131.52 195.877 Q2129.6 198.539 2129.6 203.238 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M2170.27 180.159 L2170.27 183.701 L2166.19 183.701 Q2163.9 183.701 2163 184.627 Q2162.12 185.553 2162.12 187.96 L2162.12 190.252 L2169.13 190.252 L2169.13 193.562 L2162.12 193.562 L2162.12 216.178 L2157.84 216.178 L2157.84 193.562 L2153.76 193.562 L2153.76 190.252 L2157.84 190.252 L2157.84 188.446 Q2157.84 184.118 2159.85 182.15 Q2161.86 180.159 2166.24 180.159 L2170.27 180.159 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M2184.06 180.206 Q2180.96 185.53 2179.46 190.738 Q2177.95 195.946 2177.95 201.293 Q2177.95 206.641 2179.46 211.895 Q2180.98 217.127 2184.06 222.428 L2180.36 222.428 Q2176.89 216.988 2175.15 211.733 Q2173.44 206.479 2173.44 201.293 Q2173.44 196.131 2175.15 190.9 Q2176.86 185.668 2180.36 180.206 L2184.06 180.206 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M2213.88 190.252 L2204.5 202.867 L2214.36 216.178 L2209.34 216.178 L2201.79 205.992 L2194.25 216.178 L2189.22 216.178 L2199.29 202.613 L2190.08 190.252 L2195.1 190.252 L2201.98 199.488 L2208.85 190.252 L2213.88 190.252 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip600)\" d=\"M2219.71 180.206 L2223.41 180.206 Q2226.89 185.668 2228.6 190.9 Q2230.34 196.131 2230.34 201.293 Q2230.34 206.479 2228.6 211.733 Q2226.89 216.988 2223.41 222.428 L2219.71 222.428 Q2222.79 217.127 2224.29 211.895 Q2225.82 206.641 2225.82 201.293 Q2225.82 195.946 2224.29 190.738 Q2222.79 185.53 2219.71 180.206 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /></svg>\n"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Plots\n",
    "\n",
    "# 100 values of x from 0,3\n",
    "xs = range(0, 3, length=100)\n",
    "plot(xs, f, label=\"f(x)\")\n",
    "plot!(xs, df, label=\"df(x)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9a5bbc4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Sometimes, however, it's a massive pain to calculate analytic derivatives. Maybe you have a lot of parameters to keep track of. \n",
    "\n",
    "Enter _automatic differentiation_. This is how we can get `df(x)` without doing boring old _normal_ math!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54a38558",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Automatic differentiation (autodiff)\n",
    "\n",
    "Automatic differentiation is simply a programmatic way of handling the chain rule. There are two \"modes\" of autodiff, called \"reverse mode\" and \"forward mode\".\n",
    "\n",
    "Both modes yield the same derivative, but get there different ways.\n",
    "\n",
    "- Forward mode calculates derivatives from the __inside out__\n",
    "- Reverse mode calculates derivatives from the __outside in__\n",
    "\n",
    "I will talk later about which is \"better\", but for right now the gist is that forward mode is good for small numbers of parameters, and reverse mode is good for large numbers of parameters (for scalar functions)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46f500f5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "I'm not going to dig into how these work! \n",
    "\n",
    "I just want you to limp along for now, and going through the details of AD is not particularly important until you start writing your own AD rules.\n",
    "\n",
    "There's oodles of resources on AD and how they work, but for our purposes I just want you to understand how to hook them up.\n",
    "\n",
    "A good resource is the `ChainRules.jl` [documentation](https://juliadiff.org/ChainRulesCore.jl/dev/maths/propagators.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be55ae80",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's do some tinkering around. Please install ReverseDiff.jl and ForwardDiff.jl using\n",
    "\n",
    "```julia\n",
    "] add ReverseDiff ForwardDiff\n",
    "```\n",
    "\n",
    "or \n",
    "\n",
    "```julia\n",
    "import Pkg\n",
    "Pkg.add([\"ReverseDiff\", \"ForwardDiff\"])\n",
    "```\n",
    "\n",
    "There are other packages that work differently (`Zyogte.jl`, `Tracker.jl`, and a few others) but I'm going to leave these aside for now."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2499dea",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Once those are installed, let's tinker around with how they work. \n",
    "\n",
    "\n",
    "Let's only `import` them -- don't call `using` for now since the packages export similarly named functions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f221936b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For `ForwardDiff`, we can use `ForwardDiff.derivative(f, x)`.\n",
    "\n",
    "`ReverseDiff` only works with array inputs, so I'm going to use `ReverseDiff.gradient(f, x)`, with some modifications.\n",
    "\n",
    "For completeness I'll show how to use `gradient` for ForwardDiff as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "44084ab8",
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Info: AD Examples\n",
      "│   exact_df(1.0) = 1.0806046117362795\n",
      "│   reverse_df(1.0) = 1.0806046117362795\n",
      "│   forward_df(1.0) = 1.0806046117362795\n",
      "│   forward_grad_df(1.0) = 1.0806046117362795\n",
      "└ @ Main In[36]:10\n"
     ]
    }
   ],
   "source": [
    "import ReverseDiff, ForwardDiff\n",
    "\n",
    "f(x) = sin(x^2)\n",
    "\n",
    "exact_df(x) = 2*x*cos(x^2)\n",
    "forward_df(x) = ForwardDiff.derivative(f, x)\n",
    "forward_grad_df(x) = ForwardDiff.gradient(z -> f(z[1]), [x])[1]\n",
    "reverse_df(x) = ReverseDiff.gradient(z -> f(z[1]), [x])[1]\n",
    "\n",
    "@info \"AD Examples\" exact_df(1.0) reverse_df(1.0) forward_df(1.0) forward_grad_df(1.0)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "94ff0c21",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip640\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip640)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip641\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip640)\" d=\"\n",
       "M149.191 1486.45 L2352.76 1486.45 L2352.76 47.2441 L149.191 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip642\">\n",
       "    <rect x=\"149\" y=\"47\" width=\"2205\" height=\"1440\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip642)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  211.556,1486.45 211.556,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip642)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  904.501,1486.45 904.501,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip642)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1597.45,1486.45 1597.45,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip642)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2290.39,1486.45 2290.39,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip640)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  149.191,1486.45 2352.76,1486.45 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip640)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  211.556,1486.45 211.556,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip640)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  904.501,1486.45 904.501,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip640)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1597.45,1486.45 1597.45,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip640)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2290.39,1486.45 2290.39,1467.55 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip640)\" d=\"M211.556 1517.37 Q207.945 1517.37 206.116 1520.93 Q204.311 1524.47 204.311 1531.6 Q204.311 1538.71 206.116 1542.27 Q207.945 1545.82 211.556 1545.82 Q215.19 1545.82 216.996 1542.27 Q218.824 1538.71 218.824 1531.6 Q218.824 1524.47 216.996 1520.93 Q215.19 1517.37 211.556 1517.37 M211.556 1513.66 Q217.366 1513.66 220.422 1518.27 Q223.5 1522.85 223.5 1531.6 Q223.5 1540.33 220.422 1544.94 Q217.366 1549.52 211.556 1549.52 Q205.746 1549.52 202.667 1544.94 Q199.612 1540.33 199.612 1531.6 Q199.612 1522.85 202.667 1518.27 Q205.746 1513.66 211.556 1513.66 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M894.883 1544.91 L902.522 1544.91 L902.522 1518.55 L894.212 1520.21 L894.212 1515.95 L902.475 1514.29 L907.151 1514.29 L907.151 1544.91 L914.79 1544.91 L914.79 1548.85 L894.883 1548.85 L894.883 1544.91 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M1592.1 1544.91 L1608.42 1544.91 L1608.42 1548.85 L1586.47 1548.85 L1586.47 1544.91 Q1589.14 1542.16 1593.72 1537.53 Q1598.33 1532.88 1599.51 1531.53 Q1601.75 1529.01 1602.63 1527.27 Q1603.53 1525.51 1603.53 1523.82 Q1603.53 1521.07 1601.59 1519.33 Q1599.67 1517.6 1596.57 1517.6 Q1594.37 1517.6 1591.91 1518.36 Q1589.48 1519.13 1586.71 1520.68 L1586.71 1515.95 Q1589.53 1514.82 1591.98 1514.24 Q1594.44 1513.66 1596.47 1513.66 Q1601.84 1513.66 1605.04 1516.35 Q1608.23 1519.03 1608.23 1523.52 Q1608.23 1525.65 1607.42 1527.57 Q1606.64 1529.47 1604.53 1532.07 Q1603.95 1532.74 1600.85 1535.95 Q1597.75 1539.15 1592.1 1544.91 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M2294.64 1530.21 Q2297.99 1530.93 2299.87 1533.2 Q2301.77 1535.47 2301.77 1538.8 Q2301.77 1543.92 2298.25 1546.72 Q2294.73 1549.52 2288.25 1549.52 Q2286.07 1549.52 2283.76 1549.08 Q2281.47 1548.66 2279.01 1547.81 L2279.01 1543.29 Q2280.96 1544.43 2283.27 1545.01 Q2285.59 1545.58 2288.11 1545.58 Q2292.51 1545.58 2294.8 1543.85 Q2297.12 1542.11 2297.12 1538.8 Q2297.12 1535.75 2294.96 1534.03 Q2292.83 1532.3 2289.01 1532.3 L2284.99 1532.3 L2284.99 1528.45 L2289.2 1528.45 Q2292.65 1528.45 2294.48 1527.09 Q2296.31 1525.7 2296.31 1523.11 Q2296.31 1520.45 2294.41 1519.03 Q2292.53 1517.6 2289.01 1517.6 Q2287.09 1517.6 2284.89 1518.01 Q2282.69 1518.43 2280.06 1519.31 L2280.06 1515.14 Q2282.72 1514.4 2285.03 1514.03 Q2287.37 1513.66 2289.43 1513.66 Q2294.75 1513.66 2297.86 1516.09 Q2300.96 1518.5 2300.96 1522.62 Q2300.96 1525.49 2299.31 1527.48 Q2297.67 1529.45 2294.64 1530.21 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip642)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  149.191,1255.91 2352.76,1255.91 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip642)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  149.191,997.11 2352.76,997.11 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip642)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  149.191,738.307 2352.76,738.307 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip642)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  149.191,479.505 2352.76,479.505 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip642)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  149.191,220.703 2352.76,220.703 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip640)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  149.191,1486.45 149.191,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip640)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  149.191,1255.91 168.089,1255.91 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip640)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  149.191,997.11 168.089,997.11 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip640)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  149.191,738.307 168.089,738.307 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip640)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  149.191,479.505 168.089,479.505 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip640)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  149.191,220.703 168.089,220.703 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip640)\" d=\"M50.9921 1256.36 L80.6679 1256.36 L80.6679 1260.3 L50.9921 1260.3 L50.9921 1256.36 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M103.608 1242.71 L91.8021 1261.16 L103.608 1261.16 L103.608 1242.71 M102.381 1238.63 L108.26 1238.63 L108.26 1261.16 L113.191 1261.16 L113.191 1265.04 L108.26 1265.04 L108.26 1273.19 L103.608 1273.19 L103.608 1265.04 L88.0058 1265.04 L88.0058 1260.53 L102.381 1238.63 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M53.0754 997.561 L82.7512 997.561 L82.7512 1001.5 L53.0754 1001.5 L53.0754 997.561 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M96.8715 1010.45 L113.191 1010.45 L113.191 1014.39 L91.2465 1014.39 L91.2465 1010.45 Q93.9086 1007.7 98.4919 1003.07 Q103.098 998.418 104.279 997.075 Q106.524 994.552 107.404 992.816 Q108.307 991.057 108.307 989.367 Q108.307 986.612 106.362 984.876 Q104.441 983.14 101.339 983.14 Q99.14 983.14 96.6863 983.904 Q94.2558 984.668 91.478 986.219 L91.478 981.497 Q94.3021 980.362 96.7558 979.784 Q99.2095 979.205 101.246 979.205 Q106.617 979.205 109.811 981.89 Q113.006 984.575 113.006 989.066 Q113.006 991.196 112.196 993.117 Q111.408 995.015 109.302 997.608 Q108.723 998.279 105.621 1001.5 Q102.52 1004.69 96.8715 1010.45 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M101.246 724.106 Q97.6354 724.106 95.8067 727.671 Q94.0012 731.213 94.0012 738.342 Q94.0012 745.449 95.8067 749.013 Q97.6354 752.555 101.246 752.555 Q104.881 752.555 106.686 749.013 Q108.515 745.449 108.515 738.342 Q108.515 731.213 106.686 727.671 Q104.881 724.106 101.246 724.106 M101.246 720.402 Q107.057 720.402 110.112 725.009 Q113.191 729.592 113.191 738.342 Q113.191 747.069 110.112 751.675 Q107.057 756.259 101.246 756.259 Q95.4363 756.259 92.3576 751.675 Q89.3021 747.069 89.3021 738.342 Q89.3021 729.592 92.3576 725.009 Q95.4363 720.402 101.246 720.402 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M96.8715 492.85 L113.191 492.85 L113.191 496.785 L91.2465 496.785 L91.2465 492.85 Q93.9086 490.095 98.4919 485.466 Q103.098 480.813 104.279 479.47 Q106.524 476.947 107.404 475.211 Q108.307 473.452 108.307 471.762 Q108.307 469.007 106.362 467.271 Q104.441 465.535 101.339 465.535 Q99.14 465.535 96.6863 466.299 Q94.2558 467.063 91.478 468.614 L91.478 463.892 Q94.3021 462.757 96.7558 462.179 Q99.2095 461.6 101.246 461.6 Q106.617 461.6 109.811 464.285 Q113.006 466.97 113.006 471.461 Q113.006 473.591 112.196 475.512 Q111.408 477.41 109.302 480.003 Q108.723 480.674 105.621 483.892 Q102.52 487.086 96.8715 492.85 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M103.608 207.497 L91.8021 225.946 L103.608 225.946 L103.608 207.497 M102.381 203.423 L108.26 203.423 L108.26 225.946 L113.191 225.946 L113.191 229.834 L108.26 229.834 L108.26 237.983 L103.608 237.983 L103.608 229.834 L88.0058 229.834 L88.0058 225.321 L102.381 203.423 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip642)\" style=\"stroke:#009af9; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  211.556,738.307 232.554,738.189 253.553,737.832 274.551,737.238 295.549,736.406 316.548,735.337 337.546,734.031 358.544,732.487 379.543,730.707 400.541,728.691 \n",
       "  421.539,726.442 442.538,723.959 463.536,721.246 484.534,718.306 505.533,715.143 526.531,711.761 547.529,708.167 568.528,704.369 589.526,700.373 610.524,696.193 \n",
       "  631.523,691.839 652.521,687.326 673.519,682.671 694.518,677.892 715.516,673.011 736.514,668.052 757.513,663.042 778.511,658.01 799.509,652.989 820.508,648.016 \n",
       "  841.506,643.129 862.504,638.37 883.503,633.785 904.501,629.42 925.499,625.327 946.498,621.559 967.496,618.171 988.494,615.22 1009.49,612.765 1030.49,610.862 \n",
       "  1051.49,609.573 1072.49,608.954 1093.49,609.062 1114.48,609.95 1135.48,611.668 1156.48,614.262 1177.48,617.77 1198.48,622.224 1219.48,627.647 1240.47,634.052 \n",
       "  1261.47,641.441 1282.47,649.804 1303.47,659.115 1324.47,669.334 1345.47,680.408 1366.46,692.261 1387.46,704.805 1408.46,717.932 1429.46,731.514 1450.46,745.41 \n",
       "  1471.46,759.459 1492.45,773.484 1513.45,787.295 1534.45,800.69 1555.45,813.458 1576.45,825.381 1597.45,836.239 1618.44,845.814 1639.44,853.894 1660.44,860.28 \n",
       "  1681.44,864.789 1702.44,867.259 1723.44,867.56 1744.43,865.593 1765.43,861.298 1786.43,854.663 1807.43,845.72 1828.43,834.556 1849.43,821.314 1870.42,806.191 \n",
       "  1891.42,789.441 1912.42,771.373 1933.42,752.347 1954.42,732.771 1975.42,713.087 1996.41,693.773 2017.41,675.321 2038.41,658.233 2059.41,643 2080.41,630.092 \n",
       "  2101.41,619.941 2122.4,612.921 2143.4,609.338 2164.4,609.409 2185.4,613.252 2206.4,620.872 2227.4,632.157 2248.39,646.867 2269.39,664.637 2290.39,684.979 \n",
       "  \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip642)\" style=\"stroke:#e26f46; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  211.556,738.307 232.554,730.465 253.553,722.623 274.551,714.781 295.549,706.941 316.548,699.105 337.546,691.278 358.544,683.466 379.543,675.676 400.541,667.92 \n",
       "  421.539,660.213 442.538,652.572 463.536,645.019 484.534,637.58 505.533,630.286 526.531,623.172 547.529,616.279 568.528,609.652 589.526,603.344 610.524,597.413 \n",
       "  631.523,591.92 652.521,586.936 673.519,582.534 694.518,578.797 715.516,575.808 736.514,573.658 757.513,572.443 778.511,572.259 799.509,573.208 820.508,575.391 \n",
       "  841.506,578.91 862.504,583.867 883.503,590.358 904.501,598.476 925.499,608.307 946.498,619.927 967.496,633.398 988.494,648.772 1009.49,666.077 1030.49,685.326 \n",
       "  1051.49,706.503 1072.49,729.57 1093.49,754.454 1114.48,781.051 1135.48,809.222 1156.48,838.786 1177.48,869.525 1198.48,901.176 1219.48,933.431 1240.47,965.939 \n",
       "  1261.47,998.306 1282.47,1030.1 1303.47,1060.83 1324.47,1089.99 1345.47,1117.04 1366.46,1141.41 1387.46,1162.51 1408.46,1179.75 1429.46,1192.55 1450.46,1200.32 \n",
       "  1471.46,1202.53 1492.45,1198.68 1513.45,1188.35 1534.45,1171.18 1555.45,1146.91 1576.45,1115.4 1597.45,1076.64 1618.44,1030.76 1639.44,978.062 1660.44,919.017 \n",
       "  1681.44,854.275 1702.44,784.676 1723.44,711.247 1744.43,635.198 1765.43,557.913 1786.43,480.931 1807.43,405.928 1828.43,334.679 1849.43,269.03 1870.42,210.846 \n",
       "  1891.42,161.968 1912.42,124.154 1933.42,99.0191 1954.42,87.9763 1975.42,92.1705 1996.41,112.417 2017.41,149.144 2038.41,202.336 2059.41,271.493 2080.41,355.6 \n",
       "  2101.41,453.103 2122.4,561.911 2143.4,679.416 2164.4,802.529 2185.4,927.744 2206.4,1051.22 2227.4,1168.88 2248.39,1276.57 2269.39,1370.16 2290.39,1445.72 \n",
       "  \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip642)\" style=\"stroke:#3da44d; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  211.556,738.307 232.554,730.465 253.553,722.623 274.551,714.781 295.549,706.941 316.548,699.105 337.546,691.278 358.544,683.466 379.543,675.676 400.541,667.92 \n",
       "  421.539,660.213 442.538,652.572 463.536,645.019 484.534,637.58 505.533,630.286 526.531,623.172 547.529,616.279 568.528,609.652 589.526,603.344 610.524,597.413 \n",
       "  631.523,591.92 652.521,586.936 673.519,582.534 694.518,578.797 715.516,575.808 736.514,573.658 757.513,572.443 778.511,572.259 799.509,573.208 820.508,575.391 \n",
       "  841.506,578.91 862.504,583.867 883.503,590.358 904.501,598.476 925.499,608.307 946.498,619.927 967.496,633.398 988.494,648.772 1009.49,666.077 1030.49,685.326 \n",
       "  1051.49,706.503 1072.49,729.57 1093.49,754.454 1114.48,781.051 1135.48,809.222 1156.48,838.786 1177.48,869.525 1198.48,901.176 1219.48,933.431 1240.47,965.939 \n",
       "  1261.47,998.306 1282.47,1030.1 1303.47,1060.83 1324.47,1089.99 1345.47,1117.04 1366.46,1141.41 1387.46,1162.51 1408.46,1179.75 1429.46,1192.55 1450.46,1200.32 \n",
       "  1471.46,1202.53 1492.45,1198.68 1513.45,1188.35 1534.45,1171.18 1555.45,1146.91 1576.45,1115.4 1597.45,1076.64 1618.44,1030.76 1639.44,978.062 1660.44,919.017 \n",
       "  1681.44,854.275 1702.44,784.676 1723.44,711.247 1744.43,635.198 1765.43,557.913 1786.43,480.931 1807.43,405.928 1828.43,334.679 1849.43,269.03 1870.42,210.846 \n",
       "  1891.42,161.968 1912.42,124.154 1933.42,99.0191 1954.42,87.9763 1975.42,92.1705 1996.41,112.417 2017.41,149.144 2038.41,202.336 2059.41,271.493 2080.41,355.6 \n",
       "  2101.41,453.103 2122.4,561.911 2143.4,679.416 2164.4,802.529 2185.4,927.744 2206.4,1051.22 2227.4,1168.88 2248.39,1276.57 2269.39,1370.16 2290.39,1445.72 \n",
       "  \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip642)\" style=\"stroke:#c271d2; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  211.556,738.307 232.554,730.465 253.553,722.623 274.551,714.781 295.549,706.941 316.548,699.105 337.546,691.278 358.544,683.466 379.543,675.676 400.541,667.92 \n",
       "  421.539,660.213 442.538,652.572 463.536,645.019 484.534,637.58 505.533,630.286 526.531,623.172 547.529,616.279 568.528,609.652 589.526,603.344 610.524,597.413 \n",
       "  631.523,591.92 652.521,586.936 673.519,582.534 694.518,578.797 715.516,575.808 736.514,573.658 757.513,572.443 778.511,572.259 799.509,573.208 820.508,575.391 \n",
       "  841.506,578.91 862.504,583.867 883.503,590.358 904.501,598.476 925.499,608.307 946.498,619.927 967.496,633.398 988.494,648.772 1009.49,666.077 1030.49,685.326 \n",
       "  1051.49,706.503 1072.49,729.57 1093.49,754.454 1114.48,781.051 1135.48,809.222 1156.48,838.786 1177.48,869.525 1198.48,901.176 1219.48,933.431 1240.47,965.939 \n",
       "  1261.47,998.306 1282.47,1030.1 1303.47,1060.83 1324.47,1089.99 1345.47,1117.04 1366.46,1141.41 1387.46,1162.51 1408.46,1179.75 1429.46,1192.55 1450.46,1200.32 \n",
       "  1471.46,1202.53 1492.45,1198.68 1513.45,1188.35 1534.45,1171.18 1555.45,1146.91 1576.45,1115.4 1597.45,1076.64 1618.44,1030.76 1639.44,978.062 1660.44,919.017 \n",
       "  1681.44,854.275 1702.44,784.676 1723.44,711.247 1744.43,635.198 1765.43,557.913 1786.43,480.931 1807.43,405.928 1828.43,334.679 1849.43,269.03 1870.42,210.846 \n",
       "  1891.42,161.968 1912.42,124.154 1933.42,99.0191 1954.42,87.9763 1975.42,92.1705 1996.41,112.417 2017.41,149.144 2038.41,202.336 2059.41,271.493 2080.41,355.6 \n",
       "  2101.41,453.103 2122.4,561.911 2143.4,679.416 2164.4,802.529 2185.4,927.744 2206.4,1051.22 2227.4,1168.88 2248.39,1276.57 2269.39,1370.16 2290.39,1445.72 \n",
       "  \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip640)\" d=\"\n",
       "M222.643 354.418 L779.612 354.418 L779.612 95.2176 L222.643 95.2176  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip640)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  222.643,354.418 779.612,354.418 779.612,95.2176 222.643,95.2176 222.643,354.418 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip640)\" style=\"stroke:#009af9; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  247.127,147.058 394.031,147.058 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip640)\" d=\"M435.02 128.319 L435.02 131.861 L430.946 131.861 Q428.654 131.861 427.752 132.787 Q426.872 133.713 426.872 136.12 L426.872 138.412 L433.886 138.412 L433.886 141.722 L426.872 141.722 L426.872 164.338 L422.59 164.338 L422.59 141.722 L418.515 141.722 L418.515 138.412 L422.59 138.412 L422.59 136.606 Q422.59 132.278 424.603 130.31 Q426.617 128.319 430.992 128.319 L435.02 128.319 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M448.816 128.366 Q445.714 133.69 444.21 138.898 Q442.705 144.106 442.705 149.453 Q442.705 154.801 444.21 160.055 Q445.738 165.287 448.816 170.588 L445.113 170.588 Q441.64 165.148 439.904 159.893 Q438.191 154.639 438.191 149.453 Q438.191 144.291 439.904 139.06 Q441.617 133.828 445.113 128.366 L448.816 128.366 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M478.631 138.412 L469.256 151.027 L479.117 164.338 L474.094 164.338 L466.548 154.152 L459.001 164.338 L453.978 164.338 L464.048 150.773 L454.835 138.412 L459.858 138.412 L466.733 147.648 L473.608 138.412 L478.631 138.412 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M484.464 128.366 L488.168 128.366 Q491.64 133.828 493.353 139.06 Q495.089 144.291 495.089 149.453 Q495.089 154.639 493.353 159.893 Q491.64 165.148 488.168 170.588 L484.464 170.588 Q487.543 165.287 489.048 160.055 Q490.575 154.801 490.575 149.453 Q490.575 144.106 489.048 138.898 Q487.543 133.69 484.464 128.366 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip640)\" style=\"stroke:#e26f46; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  247.127,198.898 394.031,198.898 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip640)\" d=\"M442.543 202.15 L442.543 204.233 L422.96 204.233 Q423.238 208.631 425.599 210.946 Q427.983 213.238 432.219 213.238 Q434.673 213.238 436.964 212.636 Q439.279 212.034 441.548 210.83 L441.548 214.858 Q439.256 215.83 436.849 216.34 Q434.441 216.849 431.964 216.849 Q425.761 216.849 422.127 213.238 Q418.515 209.627 418.515 203.469 Q418.515 197.104 421.941 193.377 Q425.39 189.627 431.224 189.627 Q436.455 189.627 439.488 193.006 Q442.543 196.363 442.543 202.15 M438.284 200.9 Q438.238 197.405 436.316 195.321 Q434.418 193.238 431.27 193.238 Q427.705 193.238 425.552 195.252 Q423.423 197.266 423.099 200.923 L438.284 200.9 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M470.251 190.252 L460.876 202.867 L470.737 216.178 L465.714 216.178 L458.168 205.992 L450.622 216.178 L445.599 216.178 L455.668 202.613 L446.455 190.252 L451.478 190.252 L458.353 199.488 L465.228 190.252 L470.251 190.252 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M488.538 203.145 Q483.376 203.145 481.386 204.326 Q479.395 205.506 479.395 208.354 Q479.395 210.622 480.876 211.965 Q482.381 213.284 484.95 213.284 Q488.492 213.284 490.622 210.784 Q492.774 208.261 492.774 204.094 L492.774 203.145 L488.538 203.145 M497.034 201.386 L497.034 216.178 L492.774 216.178 L492.774 212.242 Q491.316 214.603 489.14 215.738 Q486.964 216.849 483.816 216.849 Q479.835 216.849 477.474 214.627 Q475.136 212.381 475.136 208.631 Q475.136 204.256 478.052 202.034 Q480.992 199.812 486.802 199.812 L492.774 199.812 L492.774 199.395 Q492.774 196.455 490.83 194.858 Q488.909 193.238 485.413 193.238 Q483.191 193.238 481.085 193.77 Q478.978 194.303 477.034 195.367 L477.034 191.432 Q479.372 190.53 481.571 190.09 Q483.77 189.627 485.853 189.627 Q491.478 189.627 494.256 192.543 Q497.034 195.46 497.034 201.386 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M524.464 191.247 L524.464 195.229 Q522.658 194.233 520.83 193.747 Q519.024 193.238 517.172 193.238 Q513.029 193.238 510.737 195.877 Q508.446 198.492 508.446 203.238 Q508.446 207.983 510.737 210.622 Q513.029 213.238 517.172 213.238 Q519.024 213.238 520.83 212.752 Q522.658 212.242 524.464 211.247 L524.464 215.182 Q522.682 216.016 520.76 216.432 Q518.862 216.849 516.709 216.849 Q510.853 216.849 507.404 213.168 Q503.955 209.488 503.955 203.238 Q503.955 196.895 507.427 193.261 Q510.922 189.627 516.987 189.627 Q518.955 189.627 520.83 190.043 Q522.705 190.437 524.464 191.247 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M536.084 182.891 L536.084 190.252 L544.857 190.252 L544.857 193.562 L536.084 193.562 L536.084 207.636 Q536.084 210.807 536.941 211.71 Q537.82 212.613 540.482 212.613 L544.857 212.613 L544.857 216.178 L540.482 216.178 Q535.552 216.178 533.677 214.349 Q531.802 212.497 531.802 207.636 L531.802 193.562 L528.677 193.562 L528.677 190.252 L531.802 190.252 L531.802 182.891 L536.084 182.891 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M570.158 224.048 L570.158 227.358 L545.529 227.358 L545.529 224.048 L570.158 224.048 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M591.223 194.187 L591.223 180.159 L595.482 180.159 L595.482 216.178 L591.223 216.178 L591.223 212.289 Q589.88 214.603 587.82 215.738 Q585.783 216.849 582.913 216.849 Q578.214 216.849 575.251 213.099 Q572.311 209.349 572.311 203.238 Q572.311 197.127 575.251 193.377 Q578.214 189.627 582.913 189.627 Q585.783 189.627 587.82 190.761 Q589.88 191.872 591.223 194.187 M576.709 203.238 Q576.709 207.937 578.63 210.622 Q580.575 213.284 583.954 213.284 Q587.334 213.284 589.278 210.622 Q591.223 207.937 591.223 203.238 Q591.223 198.539 589.278 195.877 Q587.334 193.192 583.954 193.192 Q580.575 193.192 578.63 195.877 Q576.709 198.539 576.709 203.238 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M617.38 180.159 L617.38 183.701 L613.306 183.701 Q611.014 183.701 610.112 184.627 Q609.232 185.553 609.232 187.96 L609.232 190.252 L616.246 190.252 L616.246 193.562 L609.232 193.562 L609.232 216.178 L604.95 216.178 L604.95 193.562 L600.876 193.562 L600.876 190.252 L604.95 190.252 L604.95 188.446 Q604.95 184.118 606.964 182.15 Q608.977 180.159 613.352 180.159 L617.38 180.159 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M631.176 180.206 Q628.075 185.53 626.57 190.738 Q625.065 195.946 625.065 201.293 Q625.065 206.641 626.57 211.895 Q628.098 217.127 631.176 222.428 L627.473 222.428 Q624 216.988 622.264 211.733 Q620.551 206.479 620.551 201.293 Q620.551 196.131 622.264 190.9 Q623.977 185.668 627.473 180.206 L631.176 180.206 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M660.991 190.252 L651.616 202.867 L661.477 216.178 L656.454 216.178 L648.908 205.992 L641.362 216.178 L636.338 216.178 L646.408 202.613 L637.195 190.252 L642.218 190.252 L649.093 199.488 L655.968 190.252 L660.991 190.252 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M666.824 180.206 L670.528 180.206 Q674 185.668 675.713 190.9 Q677.449 196.131 677.449 201.293 Q677.449 206.479 675.713 211.733 Q674 216.988 670.528 222.428 L666.824 222.428 Q669.903 217.127 671.408 211.895 Q672.935 206.641 672.935 201.293 Q672.935 195.946 671.408 190.738 Q669.903 185.53 666.824 180.206 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip640)\" style=\"stroke:#3da44d; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  247.127,250.738 394.031,250.738 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip640)\" d=\"M435.02 231.999 L435.02 235.541 L430.946 235.541 Q428.654 235.541 427.752 236.467 Q426.872 237.393 426.872 239.8 L426.872 242.092 L433.886 242.092 L433.886 245.402 L426.872 245.402 L426.872 268.018 L422.59 268.018 L422.59 245.402 L418.515 245.402 L418.515 242.092 L422.59 242.092 L422.59 240.286 Q422.59 235.958 424.603 233.99 Q426.617 231.999 430.992 231.999 L435.02 231.999 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M448.631 245.078 Q445.205 245.078 443.214 247.763 Q441.224 250.425 441.224 255.078 Q441.224 259.731 443.191 262.416 Q445.182 265.078 448.631 265.078 Q452.034 265.078 454.025 262.393 Q456.015 259.707 456.015 255.078 Q456.015 250.471 454.025 247.786 Q452.034 245.078 448.631 245.078 M448.631 241.467 Q454.187 241.467 457.358 245.078 Q460.529 248.689 460.529 255.078 Q460.529 261.444 457.358 265.078 Q454.187 268.689 448.631 268.689 Q443.052 268.689 439.881 265.078 Q436.733 261.444 436.733 255.078 Q436.733 248.689 439.881 245.078 Q443.052 241.467 448.631 241.467 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M482.612 246.073 Q481.895 245.657 481.038 245.471 Q480.205 245.263 479.186 245.263 Q475.575 245.263 473.631 247.624 Q471.71 249.962 471.71 254.36 L471.71 268.018 L467.427 268.018 L467.427 242.092 L471.71 242.092 L471.71 246.12 Q473.052 243.758 475.205 242.624 Q477.358 241.467 480.436 241.467 Q480.876 241.467 481.409 241.536 Q481.941 241.583 482.589 241.698 L482.612 246.073 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M484.603 242.092 L488.862 242.092 L494.186 262.323 L499.487 242.092 L504.51 242.092 L509.834 262.323 L515.135 242.092 L519.395 242.092 L512.612 268.018 L507.589 268.018 L502.01 246.768 L496.409 268.018 L491.385 268.018 L484.603 242.092 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M537.635 254.985 Q532.473 254.985 530.482 256.166 Q528.492 257.346 528.492 260.194 Q528.492 262.462 529.973 263.805 Q531.478 265.124 534.047 265.124 Q537.589 265.124 539.719 262.624 Q541.871 260.101 541.871 255.934 L541.871 254.985 L537.635 254.985 M546.131 253.226 L546.131 268.018 L541.871 268.018 L541.871 264.082 Q540.413 266.443 538.237 267.578 Q536.061 268.689 532.913 268.689 Q528.932 268.689 526.57 266.467 Q524.233 264.221 524.233 260.471 Q524.233 256.096 527.149 253.874 Q530.089 251.652 535.899 251.652 L541.871 251.652 L541.871 251.235 Q541.871 248.295 539.927 246.698 Q538.006 245.078 534.51 245.078 Q532.288 245.078 530.182 245.61 Q528.075 246.143 526.131 247.207 L526.131 243.272 Q528.469 242.37 530.668 241.93 Q532.867 241.467 534.95 241.467 Q540.575 241.467 543.353 244.383 Q546.131 247.3 546.131 253.226 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M569.927 246.073 Q569.209 245.657 568.353 245.471 Q567.519 245.263 566.501 245.263 Q562.89 245.263 560.945 247.624 Q559.024 249.962 559.024 254.36 L559.024 268.018 L554.742 268.018 L554.742 242.092 L559.024 242.092 L559.024 246.12 Q560.367 243.758 562.519 242.624 Q564.672 241.467 567.751 241.467 Q568.191 241.467 568.723 241.536 Q569.255 241.583 569.904 241.698 L569.927 246.073 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M590.621 246.027 L590.621 231.999 L594.88 231.999 L594.88 268.018 L590.621 268.018 L590.621 264.129 Q589.278 266.443 587.218 267.578 Q585.181 268.689 582.311 268.689 Q577.612 268.689 574.649 264.939 Q571.709 261.189 571.709 255.078 Q571.709 248.967 574.649 245.217 Q577.612 241.467 582.311 241.467 Q585.181 241.467 587.218 242.601 Q589.278 243.712 590.621 246.027 M576.107 255.078 Q576.107 259.777 578.029 262.462 Q579.973 265.124 583.353 265.124 Q586.732 265.124 588.677 262.462 Q590.621 259.777 590.621 255.078 Q590.621 250.379 588.677 247.717 Q586.732 245.032 583.353 245.032 Q579.973 245.032 578.029 247.717 Q576.107 250.379 576.107 255.078 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M623.352 275.888 L623.352 279.198 L598.723 279.198 L598.723 275.888 L623.352 275.888 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M644.417 246.027 L644.417 231.999 L648.676 231.999 L648.676 268.018 L644.417 268.018 L644.417 264.129 Q643.074 266.443 641.014 267.578 Q638.977 268.689 636.107 268.689 Q631.408 268.689 628.445 264.939 Q625.505 261.189 625.505 255.078 Q625.505 248.967 628.445 245.217 Q631.408 241.467 636.107 241.467 Q638.977 241.467 641.014 242.601 Q643.074 243.712 644.417 246.027 M629.903 255.078 Q629.903 259.777 631.825 262.462 Q633.769 265.124 637.149 265.124 Q640.528 265.124 642.473 262.462 Q644.417 259.777 644.417 255.078 Q644.417 250.379 642.473 247.717 Q640.528 245.032 637.149 245.032 Q633.769 245.032 631.825 247.717 Q629.903 250.379 629.903 255.078 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M670.574 231.999 L670.574 235.541 L666.5 235.541 Q664.209 235.541 663.306 236.467 Q662.426 237.393 662.426 239.8 L662.426 242.092 L669.44 242.092 L669.44 245.402 L662.426 245.402 L662.426 268.018 L658.144 268.018 L658.144 245.402 L654.07 245.402 L654.07 242.092 L658.144 242.092 L658.144 240.286 Q658.144 235.958 660.158 233.99 Q662.172 231.999 666.547 231.999 L670.574 231.999 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M684.371 232.046 Q681.269 237.37 679.764 242.578 Q678.259 247.786 678.259 253.133 Q678.259 258.481 679.764 263.735 Q681.292 268.967 684.371 274.268 L680.667 274.268 Q677.195 268.828 675.459 263.573 Q673.746 258.319 673.746 253.133 Q673.746 247.971 675.459 242.74 Q677.172 237.508 680.667 232.046 L684.371 232.046 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M714.185 242.092 L704.81 254.707 L714.671 268.018 L709.648 268.018 L702.102 257.832 L694.556 268.018 L689.533 268.018 L699.602 254.453 L690.389 242.092 L695.412 242.092 L702.287 251.328 L709.162 242.092 L714.185 242.092 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M720.018 232.046 L723.722 232.046 Q727.194 237.508 728.907 242.74 Q730.643 247.971 730.643 253.133 Q730.643 258.319 728.907 263.573 Q727.194 268.828 723.722 274.268 L720.018 274.268 Q723.097 268.967 724.602 263.735 Q726.13 258.481 726.13 253.133 Q726.13 247.786 724.602 242.578 Q723.097 237.37 720.018 232.046 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip640)\" style=\"stroke:#c271d2; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  247.127,302.578 394.031,302.578 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip640)\" d=\"M433.701 297.913 Q432.983 297.497 432.127 297.311 Q431.293 297.103 430.275 297.103 Q426.664 297.103 424.719 299.464 Q422.798 301.802 422.798 306.2 L422.798 319.858 L418.515 319.858 L418.515 293.932 L422.798 293.932 L422.798 297.96 Q424.14 295.598 426.293 294.464 Q428.446 293.307 431.525 293.307 Q431.964 293.307 432.497 293.376 Q433.029 293.423 433.677 293.538 L433.701 297.913 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M459.302 305.83 L459.302 307.913 L439.719 307.913 Q439.997 312.311 442.358 314.626 Q444.742 316.918 448.978 316.918 Q451.432 316.918 453.724 316.316 Q456.038 315.714 458.307 314.51 L458.307 318.538 Q456.015 319.51 453.608 320.02 Q451.2 320.529 448.724 320.529 Q442.52 320.529 438.886 316.918 Q435.275 313.307 435.275 307.149 Q435.275 300.784 438.701 297.057 Q442.15 293.307 447.983 293.307 Q453.214 293.307 456.247 296.686 Q459.302 300.043 459.302 305.83 M455.043 304.58 Q454.997 301.085 453.075 299.001 Q451.177 296.918 448.029 296.918 Q444.464 296.918 442.312 298.932 Q440.182 300.946 439.858 304.603 L455.043 304.58 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M463.237 293.932 L467.751 293.932 L475.853 315.691 L483.955 293.932 L488.469 293.932 L478.747 319.858 L472.96 319.858 L463.237 293.932 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M516.524 305.83 L516.524 307.913 L496.941 307.913 Q497.219 312.311 499.58 314.626 Q501.964 316.918 506.2 316.918 Q508.654 316.918 510.946 316.316 Q513.26 315.714 515.529 314.51 L515.529 318.538 Q513.237 319.51 510.83 320.02 Q508.422 320.529 505.946 320.529 Q499.742 320.529 496.108 316.918 Q492.497 313.307 492.497 307.149 Q492.497 300.784 495.922 297.057 Q499.372 293.307 505.205 293.307 Q510.436 293.307 513.469 296.686 Q516.524 300.043 516.524 305.83 M512.265 304.58 Q512.219 301.085 510.297 299.001 Q508.399 296.918 505.251 296.918 Q501.686 296.918 499.534 298.932 Q497.404 300.946 497.08 304.603 L512.265 304.58 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M538.538 297.913 Q537.82 297.497 536.964 297.311 Q536.131 297.103 535.112 297.103 Q531.501 297.103 529.557 299.464 Q527.635 301.802 527.635 306.2 L527.635 319.858 L523.353 319.858 L523.353 293.932 L527.635 293.932 L527.635 297.96 Q528.978 295.598 531.131 294.464 Q533.283 293.307 536.362 293.307 Q536.802 293.307 537.334 293.376 Q537.867 293.423 538.515 293.538 L538.538 297.913 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M559.533 294.696 L559.533 298.723 Q557.728 297.797 555.783 297.335 Q553.839 296.872 551.756 296.872 Q548.584 296.872 546.987 297.844 Q545.413 298.816 545.413 300.76 Q545.413 302.242 546.547 303.098 Q547.681 303.932 551.107 304.696 L552.566 305.02 Q557.103 305.992 559.001 307.774 Q560.922 309.534 560.922 312.705 Q560.922 316.316 558.052 318.422 Q555.205 320.529 550.205 320.529 Q548.121 320.529 545.853 320.112 Q543.607 319.719 541.107 318.908 L541.107 314.51 Q543.469 315.737 545.76 316.362 Q548.052 316.964 550.297 316.964 Q553.306 316.964 554.927 315.946 Q556.547 314.904 556.547 313.029 Q556.547 311.293 555.367 310.367 Q554.209 309.441 550.251 308.584 L548.769 308.237 Q544.811 307.404 543.052 305.691 Q541.293 303.955 541.293 300.946 Q541.293 297.288 543.885 295.298 Q546.478 293.307 551.246 293.307 Q553.607 293.307 555.691 293.654 Q557.774 294.001 559.533 294.696 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M589.88 305.83 L589.88 307.913 L570.297 307.913 Q570.575 312.311 572.936 314.626 Q575.32 316.918 579.556 316.918 Q582.01 316.918 584.302 316.316 Q586.616 315.714 588.885 314.51 L588.885 318.538 Q586.593 319.51 584.186 320.02 Q581.778 320.529 579.302 320.529 Q573.098 320.529 569.464 316.918 Q565.853 313.307 565.853 307.149 Q565.853 300.784 569.279 297.057 Q572.728 293.307 578.561 293.307 Q583.792 293.307 586.825 296.686 Q589.88 300.043 589.88 305.83 M585.621 304.58 Q585.575 301.085 583.653 299.001 Q581.755 296.918 578.607 296.918 Q575.042 296.918 572.89 298.932 Q570.76 300.946 570.436 304.603 L585.621 304.58 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M616.57 327.728 L616.57 331.038 L591.94 331.038 L591.94 327.728 L616.57 327.728 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M637.635 297.867 L637.635 283.839 L641.894 283.839 L641.894 319.858 L637.635 319.858 L637.635 315.969 Q636.292 318.283 634.232 319.418 Q632.195 320.529 629.325 320.529 Q624.625 320.529 621.663 316.779 Q618.723 313.029 618.723 306.918 Q618.723 300.807 621.663 297.057 Q624.625 293.307 629.325 293.307 Q632.195 293.307 634.232 294.441 Q636.292 295.552 637.635 297.867 M623.121 306.918 Q623.121 311.617 625.042 314.302 Q626.987 316.964 630.366 316.964 Q633.746 316.964 635.69 314.302 Q637.635 311.617 637.635 306.918 Q637.635 302.219 635.69 299.557 Q633.746 296.872 630.366 296.872 Q626.987 296.872 625.042 299.557 Q623.121 302.219 623.121 306.918 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M663.792 283.839 L663.792 287.381 L659.718 287.381 Q657.426 287.381 656.523 288.307 Q655.644 289.233 655.644 291.64 L655.644 293.932 L662.658 293.932 L662.658 297.242 L655.644 297.242 L655.644 319.858 L651.361 319.858 L651.361 297.242 L647.287 297.242 L647.287 293.932 L651.361 293.932 L651.361 292.126 Q651.361 287.798 653.375 285.83 Q655.389 283.839 659.764 283.839 L663.792 283.839 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M677.588 283.886 Q674.486 289.21 672.982 294.418 Q671.477 299.626 671.477 304.973 Q671.477 310.321 672.982 315.575 Q674.509 320.807 677.588 326.108 L673.884 326.108 Q670.412 320.668 668.676 315.413 Q666.963 310.159 666.963 304.973 Q666.963 299.811 668.676 294.58 Q670.389 289.348 673.884 283.886 L677.588 283.886 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M707.403 293.932 L698.028 306.547 L707.889 319.858 L702.866 319.858 L695.32 309.672 L687.773 319.858 L682.75 319.858 L692.82 306.293 L683.607 293.932 L688.63 293.932 L695.505 303.168 L702.38 293.932 L707.403 293.932 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip640)\" d=\"M713.236 283.886 L716.94 283.886 Q720.412 289.348 722.125 294.58 Q723.861 299.811 723.861 304.973 Q723.861 310.159 722.125 315.413 Q720.412 320.668 716.94 326.108 L713.236 326.108 Q716.315 320.807 717.819 315.575 Q719.347 310.321 719.347 304.973 Q719.347 299.626 717.819 294.418 Q716.315 289.21 713.236 283.886 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /></svg>\n"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xs = range(0, 3, length=100)\n",
    "plot(xs, f, label=\"f(x)\")\n",
    "plot!(xs, exact_df, label=\"exact_df(x)\")\n",
    "plot!(xs, forward_df, label=\"forward_df(x)\")\n",
    "plot!(xs, reverse_df, label=\"reverse_df(x)\", legend=:topleft)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e237e2e6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Concept check!\n",
    "\n",
    "Let's expand what we've worked with into general multivariate functions.\n",
    "\n",
    "Take the loss function for OLS, as an example.\n",
    "\n",
    "$$\n",
    "f(\\beta, X, Y) = (Y- X\\beta)'(Y- X\\beta)\n",
    "$$\n",
    "\n",
    "Use `ForwardDiff` and/or `ReverseDiff` to calculate the gradient w.r.t. $\\beta$, $\\nabla f(\\cdot)$, at $\\hat \\beta = (X'X)^{-1}X'Y$. This gradient should be approximately zero for both, since $\\hat \\beta$ is the exact analytic value that minimizes the gradient.\n",
    "\n",
    "__Bonus points__: Find $\\hat \\beta$ by grid search -- generate a grid of betas, evaluate their gradients, and return the beta with a gradient closest to zero.\n",
    "\n",
    "Use the following code to generate data for $X$ and $Y$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "12eabf59",
   "metadata": {},
   "outputs": [],
   "source": [
    "using LinearAlgebra # You'll need the inv(...) function to calculate an inverse\n",
    "\n",
    "n = 100\n",
    "beta = [1.0, 2.2]\n",
    "X = [ones(n) rand(n)]\n",
    "Y = X*beta + randn(n)\n",
    "beta_hat = inv(X'X)*X'Y;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "06ad42fe",
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element Vector{Float64}:\n",
       " -1.270095140171179e-13\n",
       " -1.6697754290362354e-13"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f(b) = (Y - X*b)' * (Y - X*b)\n",
    "df(b) = ReverseDiff.gradient(f, b)\n",
    "\n",
    "df(beta_hat)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1248d68",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## AD (usually) works on any function\n",
    "\n",
    "A really cool thing is you can write really gnarly functions and get their derivatives easily and quickly. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "89791901",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.7777777777777778, -0.2222222222222222, -0.18518518518518517]\n",
      "[-5.4011857784131045, 0.5106491996000274, -0.15905184440444453]\n",
      "[0.3555555555555555, -0.3555555555555555, 0.04592592592592598]\n"
     ]
    }
   ],
   "source": [
    "using ForwardDiff, Distributions\n",
    "choosy(arr) = choosy(arr...)\n",
    "function choosy(x, μ, σ)\n",
    "    if x >= 1\n",
    "        return logpdf(LogNormal(μ, σ), x)\n",
    "    elseif x <= -1\n",
    "        return logpdf(Normal(μ, σ), x) \n",
    "    else\n",
    "        return logpdf(Beta(15, 12), abs(x)) * sin(μ) * cos(σ)\n",
    "    end\n",
    "end\n",
    "\n",
    "println(ForwardDiff.gradient(choosy, [1.0, 2.0, 3.0]))\n",
    "println(ForwardDiff.gradient(choosy, [0.5, 2.0, 3.0]))\n",
    "println(ForwardDiff.gradient(choosy, [-1.2, 2.0, 3.0]))\n",
    "# rng = "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce1b6418",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# AD woes\n",
    "\n",
    "The way that our two AD packages work is that they change the underlying type of your input.\n",
    "\n",
    "What does this mean? \n",
    "\n",
    "__Do not make your functions too type constricted if you want to get their derivative!__\n",
    "\n",
    "Let me show you what I mean.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "cb6dfafe",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip840\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip840)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip841\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip840)\" d=\"\n",
       "M156.274 1486.45 L2352.76 1486.45 L2352.76 47.2441 L156.274 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip842\">\n",
       "    <rect x=\"156\" y=\"47\" width=\"2197\" height=\"1440\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip842)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  218.439,1486.45 218.439,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip842)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  909.156,1486.45 909.156,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip842)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1599.87,1486.45 1599.87,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip842)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2290.59,1486.45 2290.59,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip840)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  156.274,1486.45 2352.76,1486.45 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip840)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  218.439,1486.45 218.439,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip840)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  909.156,1486.45 909.156,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip840)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1599.87,1486.45 1599.87,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip840)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2290.59,1486.45 2290.59,1467.55 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip840)\" d=\"M218.439 1517.37 Q214.828 1517.37 212.999 1520.93 Q211.193 1524.47 211.193 1531.6 Q211.193 1538.71 212.999 1542.27 Q214.828 1545.82 218.439 1545.82 Q222.073 1545.82 223.879 1542.27 Q225.707 1538.71 225.707 1531.6 Q225.707 1524.47 223.879 1520.93 Q222.073 1517.37 218.439 1517.37 M218.439 1513.66 Q224.249 1513.66 227.304 1518.27 Q230.383 1522.85 230.383 1531.6 Q230.383 1540.33 227.304 1544.94 Q224.249 1549.52 218.439 1549.52 Q212.629 1549.52 209.55 1544.94 Q206.494 1540.33 206.494 1531.6 Q206.494 1522.85 209.55 1518.27 Q212.629 1513.66 218.439 1513.66 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M899.538 1544.91 L907.177 1544.91 L907.177 1518.55 L898.867 1520.21 L898.867 1515.95 L907.131 1514.29 L911.807 1514.29 L911.807 1544.91 L919.446 1544.91 L919.446 1548.85 L899.538 1548.85 L899.538 1544.91 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M1594.53 1544.91 L1610.85 1544.91 L1610.85 1548.85 L1588.9 1548.85 L1588.9 1544.91 Q1591.56 1542.16 1596.15 1537.53 Q1600.75 1532.88 1601.93 1531.53 Q1604.18 1529.01 1605.06 1527.27 Q1605.96 1525.51 1605.96 1523.82 Q1605.96 1521.07 1604.02 1519.33 Q1602.1 1517.6 1598.99 1517.6 Q1596.8 1517.6 1594.34 1518.36 Q1591.91 1519.13 1589.13 1520.68 L1589.13 1515.95 Q1591.96 1514.82 1594.41 1514.24 Q1596.86 1513.66 1598.9 1513.66 Q1604.27 1513.66 1607.47 1516.35 Q1610.66 1519.03 1610.66 1523.52 Q1610.66 1525.65 1609.85 1527.57 Q1609.06 1529.47 1606.96 1532.07 Q1606.38 1532.74 1603.28 1535.95 Q1600.17 1539.15 1594.53 1544.91 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M2294.84 1530.21 Q2298.2 1530.93 2300.07 1533.2 Q2301.97 1535.47 2301.97 1538.8 Q2301.97 1543.92 2298.45 1546.72 Q2294.93 1549.52 2288.45 1549.52 Q2286.27 1549.52 2283.96 1549.08 Q2281.67 1548.66 2279.21 1547.81 L2279.21 1543.29 Q2281.16 1544.43 2283.47 1545.01 Q2285.79 1545.58 2288.31 1545.58 Q2292.71 1545.58 2295 1543.85 Q2297.32 1542.11 2297.32 1538.8 Q2297.32 1535.75 2295.16 1534.03 Q2293.03 1532.3 2289.21 1532.3 L2285.19 1532.3 L2285.19 1528.45 L2289.4 1528.45 Q2292.85 1528.45 2294.68 1527.09 Q2296.51 1525.7 2296.51 1523.11 Q2296.51 1520.45 2294.61 1519.03 Q2292.73 1517.6 2289.21 1517.6 Q2287.29 1517.6 2285.09 1518.01 Q2282.89 1518.43 2280.26 1519.31 L2280.26 1515.14 Q2282.92 1514.4 2285.23 1514.03 Q2287.57 1513.66 2289.63 1513.66 Q2294.95 1513.66 2298.06 1516.09 Q2301.16 1518.5 2301.16 1522.62 Q2301.16 1525.49 2299.51 1527.48 Q2297.87 1529.45 2294.84 1530.21 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip842)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  156.274,1269.8 2352.76,1269.8 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip842)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  156.274,974.34 2352.76,974.34 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip842)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  156.274,678.886 2352.76,678.886 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip842)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  156.274,383.431 2352.76,383.431 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip842)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  156.274,87.9763 2352.76,87.9763 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip840)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  156.274,1486.45 156.274,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip840)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  156.274,1269.8 175.172,1269.8 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip840)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  156.274,974.34 175.172,974.34 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip840)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  156.274,678.886 175.172,678.886 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip840)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  156.274,383.431 175.172,383.431 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip840)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  156.274,87.9763 175.172,87.9763 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip840)\" d=\"M62.9365 1255.59 Q59.3254 1255.59 57.4967 1259.16 Q55.6912 1262.7 55.6912 1269.83 Q55.6912 1276.94 57.4967 1280.5 Q59.3254 1284.04 62.9365 1284.04 Q66.5707 1284.04 68.3763 1280.5 Q70.205 1276.94 70.205 1269.83 Q70.205 1262.7 68.3763 1259.16 Q66.5707 1255.59 62.9365 1255.59 M62.9365 1251.89 Q68.7467 1251.89 71.8022 1256.5 Q74.8809 1261.08 74.8809 1269.83 Q74.8809 1278.56 71.8022 1283.16 Q68.7467 1287.75 62.9365 1287.75 Q57.1264 1287.75 54.0477 1283.16 Q50.9921 1278.56 50.9921 1269.83 Q50.9921 1261.08 54.0477 1256.5 Q57.1264 1251.89 62.9365 1251.89 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M83.0984 1281.2 L87.9827 1281.2 L87.9827 1287.08 L83.0984 1287.08 L83.0984 1281.2 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M108.746 1267.93 Q105.598 1267.93 103.746 1270.08 Q101.918 1272.24 101.918 1275.99 Q101.918 1279.71 103.746 1281.89 Q105.598 1284.04 108.746 1284.04 Q111.895 1284.04 113.723 1281.89 Q115.575 1279.71 115.575 1275.99 Q115.575 1272.24 113.723 1270.08 Q111.895 1267.93 108.746 1267.93 M118.029 1253.28 L118.029 1257.54 Q116.27 1256.7 114.464 1256.27 Q112.682 1255.83 110.922 1255.83 Q106.293 1255.83 103.839 1258.95 Q101.409 1262.08 101.061 1268.39 Q102.427 1266.38 104.487 1265.32 Q106.547 1264.23 109.024 1264.23 Q114.233 1264.23 117.242 1267.4 Q120.274 1270.55 120.274 1275.99 Q120.274 1281.31 117.126 1284.53 Q113.978 1287.75 108.746 1287.75 Q102.751 1287.75 99.5798 1283.16 Q96.4085 1278.56 96.4085 1269.83 Q96.4085 1261.64 100.297 1256.77 Q104.186 1251.89 110.737 1251.89 Q112.496 1251.89 114.279 1252.24 Q116.084 1252.58 118.029 1253.28 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M64.0013 960.139 Q60.3902 960.139 58.5615 963.704 Q56.756 967.246 56.756 974.375 Q56.756 981.482 58.5615 985.046 Q60.3902 988.588 64.0013 988.588 Q67.6356 988.588 69.4411 985.046 Q71.2698 981.482 71.2698 974.375 Q71.2698 967.246 69.4411 963.704 Q67.6356 960.139 64.0013 960.139 M64.0013 956.435 Q69.8115 956.435 72.867 961.042 Q75.9457 965.625 75.9457 974.375 Q75.9457 983.102 72.867 987.708 Q69.8115 992.292 64.0013 992.292 Q58.1912 992.292 55.1125 987.708 Q52.0569 983.102 52.0569 974.375 Q52.0569 965.625 55.1125 961.042 Q58.1912 956.435 64.0013 956.435 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M84.1632 985.741 L89.0475 985.741 L89.0475 991.62 L84.1632 991.62 L84.1632 985.741 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M98.0521 957.06 L120.274 957.06 L120.274 959.051 L107.728 991.62 L102.844 991.62 L114.649 960.996 L98.0521 960.996 L98.0521 957.06 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M63.1911 664.684 Q59.58 664.684 57.7513 668.249 Q55.9458 671.791 55.9458 678.92 Q55.9458 686.027 57.7513 689.592 Q59.58 693.133 63.1911 693.133 Q66.8254 693.133 68.6309 689.592 Q70.4596 686.027 70.4596 678.92 Q70.4596 671.791 68.6309 668.249 Q66.8254 664.684 63.1911 664.684 M63.1911 660.981 Q69.0013 660.981 72.0568 665.587 Q75.1355 670.17 75.1355 678.92 Q75.1355 687.647 72.0568 692.254 Q69.0013 696.837 63.1911 696.837 Q57.381 696.837 54.3023 692.254 Q51.2468 687.647 51.2468 678.92 Q51.2468 670.17 54.3023 665.587 Q57.381 660.981 63.1911 660.981 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M83.3531 690.286 L88.2373 690.286 L88.2373 696.166 L83.3531 696.166 L83.3531 690.286 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M108.422 679.754 Q105.089 679.754 103.168 681.536 Q101.27 683.319 101.27 686.444 Q101.27 689.569 103.168 691.351 Q105.089 693.133 108.422 693.133 Q111.756 693.133 113.677 691.351 Q115.598 689.545 115.598 686.444 Q115.598 683.319 113.677 681.536 Q111.779 679.754 108.422 679.754 M103.746 677.763 Q100.737 677.022 99.0474 674.962 Q97.3808 672.902 97.3808 669.939 Q97.3808 665.795 100.321 663.388 Q103.284 660.981 108.422 660.981 Q113.584 660.981 116.524 663.388 Q119.464 665.795 119.464 669.939 Q119.464 672.902 117.774 674.962 Q116.108 677.022 113.121 677.763 Q116.501 678.55 118.376 680.842 Q120.274 683.133 120.274 686.444 Q120.274 691.467 117.195 694.152 Q114.14 696.837 108.422 696.837 Q102.705 696.837 99.6261 694.152 Q96.5706 691.467 96.5706 686.444 Q96.5706 683.133 98.4687 680.842 Q100.367 678.55 103.746 677.763 M102.034 670.379 Q102.034 673.064 103.7 674.569 Q105.39 676.073 108.422 676.073 Q111.432 676.073 113.121 674.569 Q114.834 673.064 114.834 670.379 Q114.834 667.694 113.121 666.189 Q111.432 664.684 108.422 664.684 Q105.39 664.684 103.7 666.189 Q102.034 667.694 102.034 670.379 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M63.2837 369.23 Q59.6726 369.23 57.8439 372.794 Q56.0384 376.336 56.0384 383.466 Q56.0384 390.572 57.8439 394.137 Q59.6726 397.679 63.2837 397.679 Q66.918 397.679 68.7235 394.137 Q70.5522 390.572 70.5522 383.466 Q70.5522 376.336 68.7235 372.794 Q66.918 369.23 63.2837 369.23 M63.2837 365.526 Q69.0939 365.526 72.1494 370.132 Q75.2281 374.716 75.2281 383.466 Q75.2281 392.193 72.1494 396.799 Q69.0939 401.382 63.2837 401.382 Q57.4736 401.382 54.3949 396.799 Q51.3393 392.193 51.3393 383.466 Q51.3393 374.716 54.3949 370.132 Q57.4736 365.526 63.2837 365.526 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M83.4457 394.831 L88.3299 394.831 L88.3299 400.711 L83.4457 400.711 L83.4457 394.831 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M98.6539 399.993 L98.6539 395.734 Q100.413 396.567 102.219 397.007 Q104.024 397.447 105.76 397.447 Q110.39 397.447 112.82 394.345 Q115.274 391.22 115.621 384.878 Q114.279 386.868 112.219 387.933 Q110.158 388.998 107.658 388.998 Q102.473 388.998 99.4409 385.873 Q96.4317 382.725 96.4317 377.285 Q96.4317 371.961 99.5798 368.744 Q102.728 365.526 107.959 365.526 Q113.955 365.526 117.103 370.132 Q120.274 374.716 120.274 383.466 Q120.274 391.637 116.385 396.521 Q112.52 401.382 105.969 401.382 Q104.209 401.382 102.404 401.035 Q100.598 400.688 98.6539 399.993 M107.959 385.341 Q111.108 385.341 112.936 383.188 Q114.788 381.035 114.788 377.285 Q114.788 373.558 112.936 371.406 Q111.108 369.23 107.959 369.23 Q104.811 369.23 102.959 371.406 Q101.131 373.558 101.131 377.285 Q101.131 381.035 102.959 383.188 Q104.811 385.341 107.959 385.341 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M53.9088 101.321 L61.5476 101.321 L61.5476 74.9555 L53.2375 76.6222 L53.2375 72.3629 L61.5013 70.6963 L66.1772 70.6963 L66.1772 101.321 L73.8161 101.321 L73.8161 105.256 L53.9088 105.256 L53.9088 101.321 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M83.2605 99.3767 L88.1447 99.3767 L88.1447 105.256 L83.2605 105.256 L83.2605 99.3767 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M108.33 73.775 Q104.719 73.775 102.89 77.3398 Q101.084 80.8814 101.084 88.011 Q101.084 95.1174 102.89 98.6822 Q104.719 102.224 108.33 102.224 Q111.964 102.224 113.77 98.6822 Q115.598 95.1174 115.598 88.011 Q115.598 80.8814 113.77 77.3398 Q111.964 73.775 108.33 73.775 M108.33 70.0713 Q114.14 70.0713 117.195 74.6777 Q120.274 79.261 120.274 88.011 Q120.274 96.7378 117.195 101.344 Q114.14 105.928 108.33 105.928 Q102.52 105.928 99.4409 101.344 Q96.3854 96.7378 96.3854 88.011 Q96.3854 79.261 99.4409 74.6777 Q102.52 70.0713 108.33 70.0713 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip842)\" style=\"stroke:#009af9; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  218.439,87.9763 239.37,89.3327 260.3,93.4008 281.231,100.177 302.162,109.654 323.093,121.822 344.024,136.669 364.955,154.176 385.885,174.321 406.816,197.077 \n",
       "  427.747,222.408 448.678,250.272 469.609,280.619 490.54,313.387 511.47,348.504 532.401,385.886 553.332,425.434 574.263,467.037 595.194,510.563 616.125,555.869 \n",
       "  637.055,602.787 657.986,651.134 678.917,700.705 699.848,751.275 720.779,802.594 741.71,854.393 762.64,906.38 783.571,958.239 804.502,1009.63 825.433,1060.2 \n",
       "  846.364,1109.56 867.295,1157.32 888.225,1203.04 909.156,1246.3 930.087,1286.63 951.018,1323.56 971.949,1356.6 992.88,1385.25 1013.81,1409.01 1034.74,1427.36 \n",
       "  1055.67,1439.77 1076.6,1445.72 1097.53,1444.68 1118.46,1436.14 1139.4,1419.59 1160.33,1394.54 1181.26,1360.51 1202.19,1317.06 1223.12,1263.8 1244.05,1200.39 \n",
       "  1264.98,1126.54 1285.91,1042.07 1306.84,946.883 1327.77,841.028 1348.7,724.693 1369.63,598.248 1390.57,462.266 1411.5,317.557 1432.43,165.189 1453.36,168.694 \n",
       "  1474.29,326.174 1495.22,480.542 1516.15,629.806 1537.08,771.996 1558.01,905.185 1578.94,1027.52 1599.87,1137.23 1620.8,1232.64 1641.74,1312.19 1662.67,1374.44 \n",
       "  1683.6,1418.07 1704.53,1441.86 1725.46,1444.75 1746.39,1425.82 1767.32,1384.32 1788.25,1319.71 1809.18,1231.71 1830.11,1120.33 1851.04,986.013 1871.97,829.655 \n",
       "  1892.91,652.754 1913.84,457.488 1934.77,246.815 1955.7,150.962 1976.63,371.248 1997.56,581.977 2018.49,778.343 2039.42,955.95 2060.35,1110.87 2081.28,1239.66 \n",
       "  2102.21,1339.36 2123.14,1407.5 2144.08,1442.03 2165.01,1441.34 2185.94,1404.31 2206.87,1330.27 2227.8,1219.2 2248.73,1071.84 2269.66,889.865 2290.59,676.165 \n",
       "  \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip840)\" d=\"\n",
       "M1983.06 198.898 L2279.54 198.898 L2279.54 95.2176 L1983.06 95.2176  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip840)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1983.06,198.898 2279.54,198.898 2279.54,95.2176 1983.06,95.2176 1983.06,198.898 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip840)\" style=\"stroke:#009af9; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2007.46,147.058 2153.89,147.058 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip840)\" d=\"M2192.14 166.745 Q2190.34 171.375 2188.62 172.787 Q2186.91 174.199 2184.04 174.199 L2180.64 174.199 L2180.64 170.634 L2183.14 170.634 Q2184.9 170.634 2185.87 169.8 Q2186.84 168.967 2188.02 165.865 L2188.78 163.921 L2178.3 138.412 L2182.81 138.412 L2190.91 158.689 L2199.02 138.412 L2203.53 138.412 L2192.14 166.745 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip840)\" d=\"M2210.82 160.402 L2218.46 160.402 L2218.46 134.037 L2210.15 135.703 L2210.15 131.444 L2218.41 129.778 L2223.09 129.778 L2223.09 160.402 L2230.73 160.402 L2230.73 164.338 L2210.82 164.338 L2210.82 160.402 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /></svg>\n"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function g_basic(x::Float64)\n",
    "    return cos(sqrt(abs(sin(x^2))))\n",
    "end\n",
    "\n",
    "xs = range(0, 3, length=100)\n",
    "plot(xs, g_basic)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6687ec9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now let's try and take a derivative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "bb823c37",
   "metadata": {},
   "outputs": [
    {
     "ename": "LoadError",
     "evalue": "MethodError: no method matching g_basic(::ForwardDiff.Dual{ForwardDiff.Tag{typeof(g_basic), Float64}, Float64, 1})\n\u001b[0mClosest candidates are:\n\u001b[0m  g_basic(\u001b[91m::Float64\u001b[39m) at In[97]:1",
     "output_type": "error",
     "traceback": [
      "MethodError: no method matching g_basic(::ForwardDiff.Dual{ForwardDiff.Tag{typeof(g_basic), Float64}, Float64, 1})\n\u001b[0mClosest candidates are:\n\u001b[0m  g_basic(\u001b[91m::Float64\u001b[39m) at In[97]:1",
      "",
      "Stacktrace:",
      " [1] derivative(f::typeof(g_basic), x::Float64)",
      "   @ ForwardDiff ~/.julia/packages/ForwardDiff/PBzup/src/derivative.jl:14",
      " [2] top-level scope",
      "   @ In[98]:1",
      " [3] eval",
      "   @ ./boot.jl:360 [inlined]",
      " [4] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)",
      "   @ Base ./loading.jl:1116"
     ]
    }
   ],
   "source": [
    "ForwardDiff.derivative(g_basic, 1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c69e2f7f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What happened here?\n",
    "\n",
    "Well, when I wrote `g`, I constricted the input to be `Float64`. This is fine when I call `g(1.0)`, but when I run \n",
    "\n",
    "```julia\n",
    "ForwardDiff.derivative(g, 1.0)\n",
    "```\n",
    "\n",
    "ForwardDiff wraps the input value `1.0` in what is called a _dual_. A dual value accumulates the derivatives from the inside of a function and carries them all the way out.\n",
    "\n",
    "Let's see how to fix it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "6cb48065",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.46765250135005926"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function g(x) # Note that I have removed the type constraint! Let multiple dispatch work for you.\n",
    "    return cos(sqrt(abs(sin(x^2))))\n",
    "end\n",
    "\n",
    "ForwardDiff.derivative(g, 1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e818207",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Pro tip: when working with automatic differentiation, __only annotate types when it is strictly necessary!__\n",
    "\n",
    "If you absolutely must add type annotations because the compiler cannot infer which function to dispatch to, it is best to only constraint your types as far as `Real`, i.e.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "66f03947",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.46765250135005926"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function g(x::Real) # Using a ::Real type constraint\n",
    "    return cos(sqrt(abs(sin(x^2))))\n",
    "end\n",
    "\n",
    "ForwardDiff.derivative(g, 1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8226b014",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "ReverseDiff does the exact same thing, though with a different type:\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "93e16837",
   "metadata": {},
   "outputs": [
    {
     "ename": "LoadError",
     "evalue": "MethodError: no method matching g_rev(::ReverseDiff.TrackedReal{Float64, Float64, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}})\n\u001b[0mClosest candidates are:\n\u001b[0m  g_rev(\u001b[91m::Float64\u001b[39m) at In[45]:1",
     "output_type": "error",
     "traceback": [
      "MethodError: no method matching g_rev(::ReverseDiff.TrackedReal{Float64, Float64, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}})\n\u001b[0mClosest candidates are:\n\u001b[0m  g_rev(\u001b[91m::Float64\u001b[39m) at In[45]:1",
      "",
      "Stacktrace:",
      " [1] (::var\"#28#29\")(m::ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}})",
      "   @ Main ./In[45]:6",
      " [2] ReverseDiff.GradientTape(f::var\"#28#29\", input::Vector{Float64}, cfg::ReverseDiff.GradientConfig{ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}})",
      "   @ ReverseDiff ~/.julia/packages/ReverseDiff/E4Tzn/src/api/tape.jl:199",
      " [3] gradient(f::Function, input::Vector{Float64}, cfg::ReverseDiff.GradientConfig{ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}}) (repeats 2 times)",
      "   @ ReverseDiff ~/.julia/packages/ReverseDiff/E4Tzn/src/api/gradients.jl:22",
      " [4] top-level scope",
      "   @ In[45]:6",
      " [5] eval",
      "   @ ./boot.jl:360 [inlined]",
      " [6] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)",
      "   @ Base ./loading.jl:1116"
     ]
    }
   ],
   "source": [
    "function g_rev(x::Float64) # Using a ::Float64 type constraint\n",
    "    println(typeof(x))\n",
    "    return cos(sqrt(abs(sin(x^2))))\n",
    "end\n",
    "\n",
    "ReverseDiff.gradient(m -> g_rev(m[1]), [1.0])[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "388ca335",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## In-place derivatives\n",
    "\n",
    "A big computational cost can be allocating memory to store gradients in arrays.\n",
    "\n",
    "In Julia, it's common to provide a _buffer_ to store gradients in. Buffers are particularly useful if you end up computing a _lot_ of gradients.\n",
    "\n",
    "From here on out I'm going to use `gradient`, because it's much more common and accessible than `derivative`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "db766a04",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function fun_function(p)\n",
    "    return sin(p[1]^2) * cos(p[2] - p[1]) + p[3]^3\n",
    "end\n",
    "\n",
    "# Version that allocates a return value each time\n",
    "result = ForwardDiff.gradient(fun_function, [1.0, 2.1, 2.1])\n",
    "\n",
    "# Version that places the result in an existing array\n",
    "buf = zeros(3)\n",
    "ForwardDiff.gradient!(buf, fun_function, [1.0, 2.1, 2.1])\n",
    "\n",
    "# Verify that the results are the same.\n",
    "buf == result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93042011",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This is __really__ valuable for performance if you run a lot of derivatives. \n",
    "\n",
    "Let's say for example that we wanted the gradient of this function across a grid of `p`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "b5bb7e98",
   "metadata": {},
   "outputs": [],
   "source": [
    "rng = range(-5, 5, length=100)\n",
    "p_grid = vec(collect([a,b,c] for a in rng, b in rng, c in rng));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "6635f041",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Vector{Vector{Float64}}:\n",
       " [-5.0, -5.0, -5.0]\n",
       " [-4.898989898989899, -5.0, -5.0]\n",
       " [-4.797979797979798, -5.0, -5.0]"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_grid[1:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "12dbd77e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "simple_grad (generic function with 1 method)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using ForwardDiff.gradient\n",
    "function simple_grad(p_grid)\n",
    "    grads = zeros(length(p_grid), 3)\n",
    "    for i in 1:length(p_grid)\n",
    "        grads[i,:] = ForwardDiff.gradient(fun_function, p_grid[i])\n",
    "    end\n",
    "    return grads\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "ab2fc957",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "buf_grad (generic function with 1 method)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using ForwardDiff.gradient! with a buffer\n",
    "function buf_grad(p_grid)\n",
    "    grads = zeros(length(p_grid), 3)\n",
    "    for i in 1:length(p_grid)\n",
    "        # Note that I use view() here -- this gives the LOCATION of an \n",
    "        # array. Using grads[i,:] copies the array.\n",
    "        ForwardDiff.gradient!(view(grads, i,:), fun_function, p_grid[i])\n",
    "    end\n",
    "    return grads\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ddf470f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "`buf_grad` is way faster that `simple_grad` -- it's got about half the amount of allocations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "6b6e7a36",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: 4 samples with 1 evaluation.\n",
       " Range \u001b[90m(\u001b[39m\u001b[36m\u001b[1mmin\u001b[22m\u001b[39m … \u001b[35mmax\u001b[39m\u001b[90m):  \u001b[39m\u001b[36m\u001b[1m1.220 s\u001b[22m\u001b[39m … \u001b[35m  1.283 s\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmin … max\u001b[90m): \u001b[39m8.59% … 8.74%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[34m\u001b[1mmedian\u001b[22m\u001b[39m\u001b[90m):     \u001b[39m\u001b[34m\u001b[1m1.275 s              \u001b[22m\u001b[39m\u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmedian\u001b[90m):    \u001b[39m8.46%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[32m\u001b[1mmean\u001b[22m\u001b[39m ± \u001b[32mσ\u001b[39m\u001b[90m):   \u001b[39m\u001b[32m\u001b[1m1.263 s\u001b[22m\u001b[39m ± \u001b[32m29.145 ms\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmean ± σ\u001b[90m):  \u001b[39m8.44% ± 0.44%\n",
       "\n",
       "  \u001b[39m█\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[32m \u001b[39m\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m█\u001b[34m \u001b[39m\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m█\u001b[39m \u001b[39m█\u001b[39m \u001b[39m \n",
       "  \u001b[39m█\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[32m▁\u001b[39m\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m█\u001b[34m▁\u001b[39m\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m█\u001b[39m▁\u001b[39m█\u001b[39m \u001b[39m▁\n",
       "  1.22 s\u001b[90m         Histogram: frequency by time\u001b[39m        1.28 s \u001b[0m\u001b[1m<\u001b[22m\n",
       "\n",
       " Memory estimate\u001b[90m: \u001b[39m\u001b[33m511.16 MiB\u001b[39m, allocs estimate\u001b[90m: \u001b[39m\u001b[33m5999491\u001b[39m."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: 9 samples with 1 evaluation.\n",
       " Range \u001b[90m(\u001b[39m\u001b[36m\u001b[1mmin\u001b[22m\u001b[39m … \u001b[35mmax\u001b[39m\u001b[90m):  \u001b[39m\u001b[36m\u001b[1m547.415 ms\u001b[22m\u001b[39m … \u001b[35m705.015 ms\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmin … max\u001b[90m): \u001b[39m 9.59% … 17.63%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[34m\u001b[1mmedian\u001b[22m\u001b[39m\u001b[90m):     \u001b[39m\u001b[34m\u001b[1m627.364 ms               \u001b[22m\u001b[39m\u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmedian\u001b[90m):    \u001b[39m 8.81%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[32m\u001b[1mmean\u001b[22m\u001b[39m ± \u001b[32mσ\u001b[39m\u001b[90m):   \u001b[39m\u001b[32m\u001b[1m628.917 ms\u001b[22m\u001b[39m ± \u001b[32m 54.273 ms\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmean ± σ\u001b[90m):  \u001b[39m11.19% ±  4.15%\n",
       "\n",
       "  \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m█\u001b[34m \u001b[39m\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[32m \u001b[39m\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \n",
       "  \u001b[39m▇\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m█\u001b[34m▁\u001b[39m\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▇\u001b[39m▁\u001b[32m▁\u001b[39m\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▇\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▇\u001b[39m▁\u001b[39m▁\u001b[39m▇\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▇\u001b[39m \u001b[39m▁\n",
       "  547 ms\u001b[90m           Histogram: frequency by time\u001b[39m          705 ms \u001b[0m\u001b[1m<\u001b[22m\n",
       "\n",
       " Memory estimate\u001b[90m: \u001b[39m\u001b[33m328.06 MiB\u001b[39m, allocs estimate\u001b[90m: \u001b[39m\u001b[33m3000002\u001b[39m."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "using BenchmarkTools\n",
    "\n",
    "bm1 = @benchmark simple_grad(p_grid)\n",
    "bm2 = @benchmark buf_grad(p_grid)\n",
    "\n",
    "display(bm1)\n",
    "display(bm2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "648709d2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Reverse mode or forward mode?\n",
    "\n",
    "People generally recommend `ForwardDiff` for gradients with less than 100 parameters, and `ReverseDiff` for more than 100 parameters.\n",
    "\n",
    "Of course, they also differ a little in implementation. `ReverseDiff` allows you to precompile a lot of your gradient computation so you can reuse parts of the gradient.\n",
    "\n",
    "Lets play around a little so you can see _when_ `ReverseDiff` is preferred to `ForwardDiff`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "69b3294e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# This function has a variable parameter size p\n",
    "# How slow/fast is it to use different AD methods?\n",
    "function big_function(p)\n",
    "    n = div(length(p), 2)\n",
    "    alphas = p[1:n]\n",
    "    betas = p[n+1:end]\n",
    "    \n",
    "    return sum(alphas .* sin.(cos.(betas)))\n",
    "end\n",
    "\n",
    "xs_2 = randn(2)\n",
    "xs_10 = randn(10)\n",
    "xs_1k = randn(1000)\n",
    "xs_10k = randn(10_000);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "ca161cef",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: 10000 samples with 43 evaluations.\n",
       " Range \u001b[90m(\u001b[39m\u001b[36m\u001b[1mmin\u001b[22m\u001b[39m … \u001b[35mmax\u001b[39m\u001b[90m):  \u001b[39m\u001b[36m\u001b[1m871.558 ns\u001b[22m\u001b[39m … \u001b[35m573.079 μs\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmin … max\u001b[90m): \u001b[39m0.00% … 99.73%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[34m\u001b[1mmedian\u001b[22m\u001b[39m\u001b[90m):     \u001b[39m\u001b[34m\u001b[1m  1.175 μs               \u001b[22m\u001b[39m\u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmedian\u001b[90m):    \u001b[39m0.00%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[32m\u001b[1mmean\u001b[22m\u001b[39m ± \u001b[32mσ\u001b[39m\u001b[90m):   \u001b[39m\u001b[32m\u001b[1m  1.485 μs\u001b[22m\u001b[39m ± \u001b[32m  5.749 μs\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmean ± σ\u001b[90m):  \u001b[39m3.85% ±  1.00%\n",
       "\n",
       "  \u001b[39m \u001b[39m \u001b[39m▂\u001b[39m \u001b[39m▂\u001b[39m█\u001b[34m▄\u001b[39m\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[32m \u001b[39m\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \n",
       "  \u001b[39m▂\u001b[39m█\u001b[39m█\u001b[39m▅\u001b[39m█\u001b[39m█\u001b[34m█\u001b[39m\u001b[39m▇\u001b[39m▅\u001b[39m▃\u001b[39m▃\u001b[39m▂\u001b[39m▂\u001b[32m▂\u001b[39m\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m \u001b[39m▂\n",
       "  872 ns\u001b[90m           Histogram: frequency by time\u001b[39m         3.81 μs \u001b[0m\u001b[1m<\u001b[22m\n",
       "\n",
       " Memory estimate\u001b[90m: \u001b[39m\u001b[33m656 bytes\u001b[39m, allocs estimate\u001b[90m: \u001b[39m\u001b[33m7\u001b[39m."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: 10000 samples with 1 evaluation.\n",
       " Range \u001b[90m(\u001b[39m\u001b[36m\u001b[1mmin\u001b[22m\u001b[39m … \u001b[35mmax\u001b[39m\u001b[90m):  \u001b[39m\u001b[36m\u001b[1m126.284 μs\u001b[22m\u001b[39m … \u001b[35m  1.433 ms\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmin … max\u001b[90m): \u001b[39m0.00% … 0.00%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[34m\u001b[1mmedian\u001b[22m\u001b[39m\u001b[90m):     \u001b[39m\u001b[34m\u001b[1m151.274 μs               \u001b[22m\u001b[39m\u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmedian\u001b[90m):    \u001b[39m0.00%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[32m\u001b[1mmean\u001b[22m\u001b[39m ± \u001b[32mσ\u001b[39m\u001b[90m):   \u001b[39m\u001b[32m\u001b[1m230.908 μs\u001b[22m\u001b[39m ± \u001b[32m184.777 μs\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmean ± σ\u001b[90m):  \u001b[39m0.00% ± 0.00%\n",
       "\n",
       "  \u001b[39m█\u001b[39m▇\u001b[34m▅\u001b[39m\u001b[39m▄\u001b[39m▅\u001b[39m▅\u001b[39m▄\u001b[39m▃\u001b[32m▂\u001b[39m\u001b[39m▂\u001b[39m▁\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m▁\n",
       "  \u001b[39m█\u001b[39m█\u001b[34m█\u001b[39m\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[32m█\u001b[39m\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m▇\u001b[39m▇\u001b[39m█\u001b[39m█\u001b[39m▇\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m▇\u001b[39m▇\u001b[39m█\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m█\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▆\u001b[39m▆\u001b[39m▆\u001b[39m▇\u001b[39m▆\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m█\u001b[39m▇\u001b[39m▇\u001b[39m▆\u001b[39m▆\u001b[39m▆\u001b[39m▆\u001b[39m▅\u001b[39m \u001b[39m█\n",
       "  126 μs\u001b[90m        \u001b[39m\u001b[90mHistogram: \u001b[39m\u001b[90m\u001b[1mlog(\u001b[22m\u001b[39m\u001b[90mfrequency\u001b[39m\u001b[90m\u001b[1m)\u001b[22m\u001b[39m\u001b[90m by time\u001b[39m        962 μs \u001b[0m\u001b[1m<\u001b[22m\n",
       "\n",
       " Memory estimate\u001b[90m: \u001b[39m\u001b[33m6.55 KiB\u001b[39m, allocs estimate\u001b[90m: \u001b[39m\u001b[33m121\u001b[39m."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 2 parameters\n",
    "display(@benchmark ForwardDiff.gradient(big_function, xs_2))\n",
    "display(@benchmark ReverseDiff.gradient(big_function, xs_2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "37ebb8cd",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: 10000 samples with 10 evaluations.\n",
       " Range \u001b[90m(\u001b[39m\u001b[36m\u001b[1mmin\u001b[22m\u001b[39m … \u001b[35mmax\u001b[39m\u001b[90m):  \u001b[39m\u001b[36m\u001b[1m1.366 μs\u001b[22m\u001b[39m … \u001b[35m 2.955 ms\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmin … max\u001b[90m): \u001b[39m 0.00% … 99.73%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[34m\u001b[1mmedian\u001b[22m\u001b[39m\u001b[90m):     \u001b[39m\u001b[34m\u001b[1m3.260 μs              \u001b[22m\u001b[39m\u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmedian\u001b[90m):    \u001b[39m 0.00%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[32m\u001b[1mmean\u001b[22m\u001b[39m ± \u001b[32mσ\u001b[39m\u001b[90m):   \u001b[39m\u001b[32m\u001b[1m4.190 μs\u001b[22m\u001b[39m ± \u001b[32m40.357 μs\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmean ± σ\u001b[90m):  \u001b[39m13.58% ±  1.41%\n",
       "\n",
       "  \u001b[39m▆\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m▅\u001b[39m█\u001b[34m▆\u001b[39m\u001b[39m▂\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[32m \u001b[39m\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \n",
       "  \u001b[39m█\u001b[39m▅\u001b[39m▃\u001b[39m▂\u001b[39m▂\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▄\u001b[39m▂\u001b[39m▂\u001b[39m▇\u001b[39m█\u001b[39m█\u001b[34m█\u001b[39m\u001b[39m█\u001b[39m▇\u001b[39m▆\u001b[39m▄\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[32m▃\u001b[39m\u001b[39m▂\u001b[39m▂\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m \u001b[39m▂\n",
       "  1.37 μs\u001b[90m        Histogram: frequency by time\u001b[39m        8.97 μs \u001b[0m\u001b[1m<\u001b[22m\n",
       "\n",
       " Memory estimate\u001b[90m: \u001b[39m\u001b[33m4.33 KiB\u001b[39m, allocs estimate\u001b[90m: \u001b[39m\u001b[33m7\u001b[39m."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: 10000 samples with 1 evaluation.\n",
       " Range \u001b[90m(\u001b[39m\u001b[36m\u001b[1mmin\u001b[22m\u001b[39m … \u001b[35mmax\u001b[39m\u001b[90m):  \u001b[39m\u001b[36m\u001b[1m129.768 μs\u001b[22m\u001b[39m … \u001b[35m  1.368 ms\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmin … max\u001b[90m): \u001b[39m0.00% … 0.00%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[34m\u001b[1mmedian\u001b[22m\u001b[39m\u001b[90m):     \u001b[39m\u001b[34m\u001b[1m151.827 μs               \u001b[22m\u001b[39m\u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmedian\u001b[90m):    \u001b[39m0.00%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[32m\u001b[1mmean\u001b[22m\u001b[39m ± \u001b[32mσ\u001b[39m\u001b[90m):   \u001b[39m\u001b[32m\u001b[1m234.990 μs\u001b[22m\u001b[39m ± \u001b[32m191.523 μs\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmean ± σ\u001b[90m):  \u001b[39m0.00% ± 0.00%\n",
       "\n",
       "  \u001b[39m█\u001b[39m▆\u001b[34m▄\u001b[39m\u001b[39m▄\u001b[39m▄\u001b[39m▄\u001b[39m▃\u001b[32m▂\u001b[39m\u001b[39m▂\u001b[39m▁\u001b[39m▁\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m▁\n",
       "  \u001b[39m█\u001b[39m█\u001b[34m█\u001b[39m\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[32m█\u001b[39m\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m▇\u001b[39m█\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▆\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▆\u001b[39m▇\u001b[39m▆\u001b[39m▇\u001b[39m▇\u001b[39m▆\u001b[39m█\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▇\u001b[39m▆\u001b[39m▆\u001b[39m▆\u001b[39m▆\u001b[39m▆\u001b[39m▅\u001b[39m▆\u001b[39m \u001b[39m█\n",
       "  130 μs\u001b[90m        \u001b[39m\u001b[90mHistogram: \u001b[39m\u001b[90m\u001b[1mlog(\u001b[22m\u001b[39m\u001b[90mfrequency\u001b[39m\u001b[90m\u001b[1m)\u001b[22m\u001b[39m\u001b[90m by time\u001b[39m        986 μs \u001b[0m\u001b[1m<\u001b[22m\n",
       "\n",
       " Memory estimate\u001b[90m: \u001b[39m\u001b[33m7.02 KiB\u001b[39m, allocs estimate\u001b[90m: \u001b[39m\u001b[33m121\u001b[39m."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 10 parameters\n",
    "display(@benchmark ForwardDiff.gradient(big_function, xs_10))\n",
    "display(@benchmark ReverseDiff.gradient(big_function, xs_10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "0c2bdb17",
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: 1276 samples with 1 evaluation.\n",
       " Range \u001b[90m(\u001b[39m\u001b[36m\u001b[1mmin\u001b[22m\u001b[39m … \u001b[35mmax\u001b[39m\u001b[90m):  \u001b[39m\u001b[36m\u001b[1m2.523 ms\u001b[22m\u001b[39m … \u001b[35m12.254 ms\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmin … max\u001b[90m): \u001b[39m 0.00% … 57.69%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[34m\u001b[1mmedian\u001b[22m\u001b[39m\u001b[90m):     \u001b[39m\u001b[34m\u001b[1m3.371 ms              \u001b[22m\u001b[39m\u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmedian\u001b[90m):    \u001b[39m 0.00%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[32m\u001b[1mmean\u001b[22m\u001b[39m ± \u001b[32mσ\u001b[39m\u001b[90m):   \u001b[39m\u001b[32m\u001b[1m3.880 ms\u001b[22m\u001b[39m ± \u001b[32m 1.672 ms\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmean ± σ\u001b[90m):  \u001b[39m10.55% ± 15.79%\n",
       "\n",
       "  \u001b[39m█\u001b[39m█\u001b[39m▆\u001b[39m▆\u001b[39m▅\u001b[39m▅\u001b[34m▅\u001b[39m\u001b[39m▅\u001b[39m▅\u001b[39m▄\u001b[32m▄\u001b[39m\u001b[39m▅\u001b[39m▅\u001b[39m▄\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▁\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m▁\n",
       "  \u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[34m█\u001b[39m\u001b[39m█\u001b[39m█\u001b[39m█\u001b[32m█\u001b[39m\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m▇\u001b[39m▆\u001b[39m▆\u001b[39m▆\u001b[39m█\u001b[39m▆\u001b[39m▁\u001b[39m▄\u001b[39m▄\u001b[39m▆\u001b[39m▄\u001b[39m█\u001b[39m▆\u001b[39m█\u001b[39m▄\u001b[39m▁\u001b[39m▄\u001b[39m▆\u001b[39m▁\u001b[39m▆\u001b[39m▅\u001b[39m▆\u001b[39m▆\u001b[39m▆\u001b[39m▅\u001b[39m█\u001b[39m▆\u001b[39m█\u001b[39m▆\u001b[39m▅\u001b[39m▄\u001b[39m▇\u001b[39m▇\u001b[39m▄\u001b[39m▄\u001b[39m▇\u001b[39m▆\u001b[39m▆\u001b[39m \u001b[39m█\n",
       "  2.52 ms\u001b[90m      \u001b[39m\u001b[90mHistogram: \u001b[39m\u001b[90m\u001b[1mlog(\u001b[22m\u001b[39m\u001b[90mfrequency\u001b[39m\u001b[90m\u001b[1m)\u001b[22m\u001b[39m\u001b[90m by time\u001b[39m     10.2 ms \u001b[0m\u001b[1m<\u001b[22m\n",
       "\n",
       " Memory estimate\u001b[90m: \u001b[39m\u001b[33m12.63 MiB\u001b[39m, allocs estimate\u001b[90m: \u001b[39m\u001b[33m509\u001b[39m."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: 10000 samples with 1 evaluation.\n",
       " Range \u001b[90m(\u001b[39m\u001b[36m\u001b[1mmin\u001b[22m\u001b[39m … \u001b[35mmax\u001b[39m\u001b[90m):  \u001b[39m\u001b[36m\u001b[1m161.460 μs\u001b[22m\u001b[39m … \u001b[35m 13.429 ms\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmin … max\u001b[90m): \u001b[39m0.00% … 88.33%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[34m\u001b[1mmedian\u001b[22m\u001b[39m\u001b[90m):     \u001b[39m\u001b[34m\u001b[1m209.631 μs               \u001b[22m\u001b[39m\u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmedian\u001b[90m):    \u001b[39m0.00%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[32m\u001b[1mmean\u001b[22m\u001b[39m ± \u001b[32mσ\u001b[39m\u001b[90m):   \u001b[39m\u001b[32m\u001b[1m318.901 μs\u001b[22m\u001b[39m ± \u001b[32m324.550 μs\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmean ± σ\u001b[90m):  \u001b[39m1.36% ±  1.81%\n",
       "\n",
       "  \u001b[39m█\u001b[39m▇\u001b[39m▅\u001b[34m▄\u001b[39m\u001b[39m▅\u001b[39m▅\u001b[39m▄\u001b[39m▄\u001b[39m▃\u001b[39m▃\u001b[32m▂\u001b[39m\u001b[39m▂\u001b[39m▂\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m▂\n",
       "  \u001b[39m█\u001b[39m█\u001b[39m█\u001b[34m█\u001b[39m\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[32m█\u001b[39m\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m▇\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m▇\u001b[39m▆\u001b[39m▇\u001b[39m▇\u001b[39m▆\u001b[39m▇\u001b[39m▆\u001b[39m▆\u001b[39m \u001b[39m█\n",
       "  161 μs\u001b[90m        \u001b[39m\u001b[90mHistogram: \u001b[39m\u001b[90m\u001b[1mlog(\u001b[22m\u001b[39m\u001b[90mfrequency\u001b[39m\u001b[90m\u001b[1m)\u001b[22m\u001b[39m\u001b[90m by time\u001b[39m       1.09 ms \u001b[0m\u001b[1m<\u001b[22m\n",
       "\n",
       " Memory estimate\u001b[90m: \u001b[39m\u001b[33m65.66 KiB\u001b[39m, allocs estimate\u001b[90m: \u001b[39m\u001b[33m121\u001b[39m."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 1k parameters\n",
    "display(@benchmark ForwardDiff.gradient(big_function, xs_1k))\n",
    "display(@benchmark ReverseDiff.gradient(big_function, xs_1k))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "a3f0613c",
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: 11 samples with 1 evaluation.\n",
       " Range \u001b[90m(\u001b[39m\u001b[36m\u001b[1mmin\u001b[22m\u001b[39m … \u001b[35mmax\u001b[39m\u001b[90m):  \u001b[39m\u001b[36m\u001b[1m438.218 ms\u001b[22m\u001b[39m … \u001b[35m541.932 ms\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmin … max\u001b[90m): \u001b[39m5.80% … 5.38%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[34m\u001b[1mmedian\u001b[22m\u001b[39m\u001b[90m):     \u001b[39m\u001b[34m\u001b[1m452.138 ms               \u001b[22m\u001b[39m\u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmedian\u001b[90m):    \u001b[39m6.25%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[32m\u001b[1mmean\u001b[22m\u001b[39m ± \u001b[32mσ\u001b[39m\u001b[90m):   \u001b[39m\u001b[32m\u001b[1m462.818 ms\u001b[22m\u001b[39m ± \u001b[32m 28.857 ms\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmean ± σ\u001b[90m):  \u001b[39m6.17% ± 0.78%\n",
       "\n",
       "  \u001b[39m▁\u001b[39m \u001b[39m▁\u001b[39m \u001b[39m \u001b[39m \u001b[39m▁\u001b[39m█\u001b[34m▁\u001b[39m\u001b[39m \u001b[39m█\u001b[39m \u001b[39m \u001b[39m \u001b[32m \u001b[39m\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m▁\u001b[39m \u001b[39m \u001b[39m \u001b[39m▁\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m▁\u001b[39m \u001b[39m \n",
       "  \u001b[39m█\u001b[39m▁\u001b[39m█\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m█\u001b[39m█\u001b[34m█\u001b[39m\u001b[39m▁\u001b[39m█\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[32m▁\u001b[39m\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m█\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m█\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m█\u001b[39m \u001b[39m▁\n",
       "  438 ms\u001b[90m           Histogram: frequency by time\u001b[39m          542 ms \u001b[0m\u001b[1m<\u001b[22m\n",
       "\n",
       " Memory estimate\u001b[90m: \u001b[39m\u001b[33m1.21 GiB\u001b[39m, allocs estimate\u001b[90m: \u001b[39m\u001b[33m5010\u001b[39m."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: 5977 samples with 1 evaluation.\n",
       " Range \u001b[90m(\u001b[39m\u001b[36m\u001b[1mmin\u001b[22m\u001b[39m … \u001b[35mmax\u001b[39m\u001b[90m):  \u001b[39m\u001b[36m\u001b[1m428.516 μs\u001b[22m\u001b[39m … \u001b[35m  8.510 ms\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmin … max\u001b[90m): \u001b[39m0.00% … 85.99%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[34m\u001b[1mmedian\u001b[22m\u001b[39m\u001b[90m):     \u001b[39m\u001b[34m\u001b[1m730.964 μs               \u001b[22m\u001b[39m\u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmedian\u001b[90m):    \u001b[39m0.00%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[32m\u001b[1mmean\u001b[22m\u001b[39m ± \u001b[32mσ\u001b[39m\u001b[90m):   \u001b[39m\u001b[32m\u001b[1m818.725 μs\u001b[22m\u001b[39m ± \u001b[32m484.506 μs\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmean ± σ\u001b[90m):  \u001b[39m2.43% ±  4.85%\n",
       "\n",
       "  \u001b[39m▆\u001b[39m█\u001b[39m▃\u001b[39m \u001b[39m \u001b[39m▁\u001b[39m▁\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[34m \u001b[39m\u001b[39m \u001b[39m \u001b[39m \u001b[32m \u001b[39m\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \n",
       "  \u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m▆\u001b[39m▇\u001b[39m█\u001b[39m█\u001b[39m▇\u001b[39m▆\u001b[39m▄\u001b[39m▃\u001b[39m▅\u001b[39m▅\u001b[39m▅\u001b[34m▅\u001b[39m\u001b[39m▅\u001b[39m▅\u001b[39m▅\u001b[32m▅\u001b[39m\u001b[39m▅\u001b[39m▄\u001b[39m▄\u001b[39m▄\u001b[39m▅\u001b[39m▅\u001b[39m▅\u001b[39m▅\u001b[39m▄\u001b[39m▄\u001b[39m▄\u001b[39m▄\u001b[39m▄\u001b[39m▄\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▃\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▁\u001b[39m▁\u001b[39m▂\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▂\u001b[39m \u001b[39m▃\n",
       "  429 μs\u001b[90m           Histogram: frequency by time\u001b[39m         1.72 ms \u001b[0m\u001b[1m<\u001b[22m\n",
       "\n",
       " Memory estimate\u001b[90m: \u001b[39m\u001b[33m592.31 KiB\u001b[39m, allocs estimate\u001b[90m: \u001b[39m\u001b[33m131\u001b[39m."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 10k parameters\n",
    "display(@benchmark ForwardDiff.gradient(big_function, xs_10k))\n",
    "display(@benchmark ReverseDiff.gradient(big_function, xs_10k))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24595ff7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Conclusion: ReverseDiff for bigger models!\n",
    "\n",
    "Of course, it's not often in economics we have _massive_ parameter spaces -- typically less than thirty or so. If this is the case, definitely consider sticking with `ForwardDiff`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d32f49d5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Finite differences\n",
    "\n",
    "Sometimes you can't do autodiff. In such horrible cases, you can resort to finite differences, which is the brute force way of evaluating derivatives. \n",
    "\n",
    "You essentially calculate function values for slight perturbations around your input. Julia has a pretty good finite differences package called `FiniteDiff.jl`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "1b442f76",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m\u001b[1m   Resolving\u001b[22m\u001b[39m package versions...\n",
      "\u001b[32m\u001b[1m  No Changes\u001b[22m\u001b[39m to `~/code/julia-bootcamp-2022/Project.toml`\n",
      "\u001b[32m\u001b[1m  No Changes\u001b[22m\u001b[39m to `~/code/julia-bootcamp-2022/Manifest.toml`\n"
     ]
    }
   ],
   "source": [
    "import Pkg; Pkg.add(\"FiniteDiff\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "e101bf7c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>2 rows × 3 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>finite</th><th>automatic</th><th>difference</th></tr><tr><th></th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th></tr></thead><tbody><tr><th>1</th><td>4.28909</td><td>4.28909</td><td>-4.66552e-9</td></tr><tr><th>2</th><td>0.707537</td><td>0.707537</td><td>3.92653e-9</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& finite & automatic & difference\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 4.28909 & 4.28909 & -4.66552e-9 \\\\\n",
       "\t2 & 0.707537 & 0.707537 & 3.92653e-9 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m2×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m finite   \u001b[0m\u001b[1m automatic \u001b[0m\u001b[1m difference  \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64  \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64     \u001b[0m\n",
       "─────┼──────────────────────────────────\n",
       "   1 │ 4.28909    4.28909   -4.66552e-9\n",
       "   2 │ 0.707537   0.707537   3.92653e-9"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import it\n",
    "using FiniteDiff, LinearAlgebra, DataFrames\n",
    "\n",
    "xs = rand(MvNormal([1.0, 2.0], I), 100)\n",
    "mean_lp(μ) = sum(logpdf(MvNormal(μ, I), xs))\n",
    "\n",
    "fd = FiniteDiff.finite_difference_gradient(mean_lp, [1.0, 2.0])\n",
    "ad = ForwardDiff.gradient(mean_lp, [1.0, 2.0])\n",
    "\n",
    "DataFrame(finite=fd, automatic=ad, difference=fd-ad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "45300b73",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: 10000 samples with 1 evaluation.\n",
       " Range \u001b[90m(\u001b[39m\u001b[36m\u001b[1mmin\u001b[22m\u001b[39m … \u001b[35mmax\u001b[39m\u001b[90m):  \u001b[39m\u001b[36m\u001b[1m27.357 μs\u001b[22m\u001b[39m … \u001b[35m 34.749 ms\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmin … max\u001b[90m): \u001b[39m 0.00% … 99.84%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[34m\u001b[1mmedian\u001b[22m\u001b[39m\u001b[90m):     \u001b[39m\u001b[34m\u001b[1m47.718 μs               \u001b[22m\u001b[39m\u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmedian\u001b[90m):    \u001b[39m 0.00%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[32m\u001b[1mmean\u001b[22m\u001b[39m ± \u001b[32mσ\u001b[39m\u001b[90m):   \u001b[39m\u001b[32m\u001b[1m62.167 μs\u001b[22m\u001b[39m ± \u001b[32m471.704 μs\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmean ± σ\u001b[90m):  \u001b[39m10.70% ±  1.41%\n",
       "\n",
       "  \u001b[39m█\u001b[39m▅\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m▃\u001b[39m▂\u001b[39m \u001b[34m \u001b[39m\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[32m \u001b[39m\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \n",
       "  \u001b[39m█\u001b[39m█\u001b[39m▅\u001b[39m▄\u001b[39m▃\u001b[39m▃\u001b[39m▂\u001b[39m▂\u001b[39m▃\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m▇\u001b[34m▅\u001b[39m\u001b[39m▄\u001b[39m▄\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[32m▄\u001b[39m\u001b[39m▅\u001b[39m▄\u001b[39m▄\u001b[39m▄\u001b[39m▄\u001b[39m▄\u001b[39m▃\u001b[39m▃\u001b[39m▂\u001b[39m▂\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m \u001b[39m▃\n",
       "  27.4 μs\u001b[90m         Histogram: frequency by time\u001b[39m          121 μs \u001b[0m\u001b[1m<\u001b[22m\n",
       "\n",
       " Memory estimate\u001b[90m: \u001b[39m\u001b[33m41.72 KiB\u001b[39m, allocs estimate\u001b[90m: \u001b[39m\u001b[33m428\u001b[39m."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: 10000 samples with 1 evaluation.\n",
       " Range \u001b[90m(\u001b[39m\u001b[36m\u001b[1mmin\u001b[22m\u001b[39m … \u001b[35mmax\u001b[39m\u001b[90m):  \u001b[39m\u001b[36m\u001b[1m11.141 μs\u001b[22m\u001b[39m … \u001b[35m 26.801 ms\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmin … max\u001b[90m): \u001b[39m 0.00% … 99.73%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[34m\u001b[1mmedian\u001b[22m\u001b[39m\u001b[90m):     \u001b[39m\u001b[34m\u001b[1m17.288 μs               \u001b[22m\u001b[39m\u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmedian\u001b[90m):    \u001b[39m 0.00%\n",
       " Time  \u001b[90m(\u001b[39m\u001b[32m\u001b[1mmean\u001b[22m\u001b[39m ± \u001b[32mσ\u001b[39m\u001b[90m):   \u001b[39m\u001b[32m\u001b[1m23.943 μs\u001b[22m\u001b[39m ± \u001b[32m269.908 μs\u001b[39m  \u001b[90m┊\u001b[39m GC \u001b[90m(\u001b[39mmean ± σ\u001b[90m):  \u001b[39m11.16% ±  1.00%\n",
       "\n",
       "  \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m█\u001b[39m▁\u001b[39m▂\u001b[39m \u001b[34m \u001b[39m\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[32m \u001b[39m\u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \u001b[39m \n",
       "  \u001b[39m▂\u001b[39m▃\u001b[39m▂\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▅\u001b[39m█\u001b[39m█\u001b[39m█\u001b[39m▇\u001b[34m▅\u001b[39m\u001b[39m▅\u001b[39m▃\u001b[39m▃\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[32m▂\u001b[39m\u001b[39m▂\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▃\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▂\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m▁\u001b[39m \u001b[39m▂\n",
       "  11.1 μs\u001b[90m         Histogram: frequency by time\u001b[39m         44.8 μs \u001b[0m\u001b[1m<\u001b[22m\n",
       "\n",
       " Memory estimate\u001b[90m: \u001b[39m\u001b[33m15.48 KiB\u001b[39m, allocs estimate\u001b[90m: \u001b[39m\u001b[33m108\u001b[39m."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bm1 = @benchmark FiniteDiff.finite_difference_gradient(mean_lp, [1.0, 2.0])\n",
    "bm2 = @benchmark ForwardDiff.gradient(mean_lp, [1.0, 2.0])\n",
    "\n",
    "display(bm1)\n",
    "display(bm2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8501d46c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Last note: AD rules\n",
    "\n",
    "Sometimes you want to differentiate a function, but you don't want it to go all the way through your code in a brute force way.\n",
    "\n",
    "Some reasons why:\n",
    "\n",
    "- Your code may call a non-Julia function, and AD can fail.\n",
    "- Evaluating the derivative with AD is costly, and you happen to have the exact gradient handy.\n",
    "- Some expressions in a differentiated function may not impact the return value, so stopping AD from stepping into those expressions can save you some compute.\n",
    "- Some functions you may be using are too type constricted -- `Distributions.jl` is famous for having fairly tight type constraints that don't work with many AD systems.\n",
    "- You are working with custom types and custom algebra on those types -- no derivative rules exist and you may have to right them!\n",
    "\n",
    "These are a little more complex than we have time for, but please check out [ChainRules](/~https://github.com/JuliaDiff/ChainRulesCore.jl) if you are interested or need to work with more complex language constructs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3a46db8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95d3d14a",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Economics is often predicated on some notion of optimization. People maximize utility, businesses maximize profits or minimize costs, etc.\n",
    "\n",
    "Julia has a pretty excellent suite of tools for optimizing functions.\n",
    "\n",
    "- Optim.jl covers nonlinear, possibly noncontinuous problems with continuous parameters.\n",
    "- JuMP.jl is a whole optimization world that solves most types of problems (integer problems, continuous, mixed, etc.).\n",
    "- Nonconvex.jl and Convex.jl, among many others.\n",
    "\n",
    "We'll be sticking with JuMP and Optim for now, but I'd also recommend the wonderful GalacticOptim.jl if you need global search methods."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2497ac49",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Optim vs. JuMP\n",
    "\n",
    "Optim is good for __complicated objective functions__ with no integer variables, and/or very simple constraints. Fantastic for __unconstrained optimization__.\n",
    "\n",
    "JuMP is good for __simple objective functions__ with gnarly variables (integers, binary, continuous) and __lots of constraints__ (or complex constraints)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0154a807",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "Please install the optimization package `Optim.jl` with\n",
    "\n",
    "```julia\n",
    "] add Optim\n",
    "```\n",
    "\n",
    "or \n",
    "\n",
    "```julia\n",
    "import Pkg\n",
    "Pkg.add(\"Optim\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bedfd9d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The main function you'll work with in Optim is `optimize`, which minimizes some function. If you want a max, make sure to flip the sign of your function.\n",
    "\n",
    "Optim supports many gradient-free, gradient-required, and Hessian-required optimization methods.\n",
    "\n",
    "Some common function signatures are\n",
    "\n",
    "```julia\n",
    "optimize(function, initial_points) # use Nelder-Mead\n",
    "optimize(function, initial_points, LBFGS()) # use LBFGS\n",
    "optimize(function, initial_points, Newton()) # use Newton\n",
    "optimize(function, initial_points, Newton(), autodiff=:forward) # use Newton & autodiff\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3db5811f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We are going to start with an extremely simple problem. Given the polynomial\n",
    "\n",
    "$$\n",
    "f(x) = x^2\n",
    "$$\n",
    "\n",
    "Find me the value of $x$ that minimizes $f(x)$. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "c4fe4ad0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip720\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip720)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip721\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip720)\" d=\"\n",
       "M140.696 1486.45 L2352.76 1486.45 L2352.76 47.2441 L140.696 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip722\">\n",
       "    <rect x=\"140\" y=\"47\" width=\"2213\" height=\"1440\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip722)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  411.986,1486.45 411.986,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip722)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  829.356,1486.45 829.356,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip722)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1246.73,1486.45 1246.73,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip722)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1664.1,1486.45 1664.1,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip722)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2081.47,1486.45 2081.47,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  140.696,1486.45 2352.76,1486.45 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  411.986,1486.45 411.986,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  829.356,1486.45 829.356,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1246.73,1486.45 1246.73,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1664.1,1486.45 1664.1,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2081.47,1486.45 2081.47,1467.55 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip720)\" d=\"M380.887 1532.02 L410.562 1532.02 L410.562 1535.95 L380.887 1535.95 L380.887 1532.02 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M433.502 1518.36 L421.697 1536.81 L433.502 1536.81 L433.502 1518.36 M432.275 1514.29 L438.155 1514.29 L438.155 1536.81 L443.085 1536.81 L443.085 1540.7 L438.155 1540.7 L438.155 1548.85 L433.502 1548.85 L433.502 1540.7 L417.9 1540.7 L417.9 1536.19 L432.275 1514.29 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M799.298 1532.02 L828.974 1532.02 L828.974 1535.95 L799.298 1535.95 L799.298 1532.02 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M843.094 1544.91 L859.414 1544.91 L859.414 1548.85 L837.469 1548.85 L837.469 1544.91 Q840.131 1542.16 844.715 1537.53 Q849.321 1532.88 850.502 1531.53 Q852.747 1529.01 853.627 1527.27 Q854.529 1525.51 854.529 1523.82 Q854.529 1521.07 852.585 1519.33 Q850.664 1517.6 847.562 1517.6 Q845.363 1517.6 842.909 1518.36 Q840.478 1519.13 837.701 1520.68 L837.701 1515.95 Q840.525 1514.82 842.978 1514.24 Q845.432 1513.66 847.469 1513.66 Q852.84 1513.66 856.034 1516.35 Q859.228 1519.03 859.228 1523.52 Q859.228 1525.65 858.418 1527.57 Q857.631 1529.47 855.525 1532.07 Q854.946 1532.74 851.844 1535.95 Q848.742 1539.15 843.094 1544.91 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M1246.73 1517.37 Q1243.11 1517.37 1241.29 1520.93 Q1239.48 1524.47 1239.48 1531.6 Q1239.48 1538.71 1241.29 1542.27 Q1243.11 1545.82 1246.73 1545.82 Q1250.36 1545.82 1252.17 1542.27 Q1253.99 1538.71 1253.99 1531.6 Q1253.99 1524.47 1252.17 1520.93 Q1250.36 1517.37 1246.73 1517.37 M1246.73 1513.66 Q1252.54 1513.66 1255.59 1518.27 Q1258.67 1522.85 1258.67 1531.6 Q1258.67 1540.33 1255.59 1544.94 Q1252.54 1549.52 1246.73 1549.52 Q1240.92 1549.52 1237.84 1544.94 Q1234.78 1540.33 1234.78 1531.6 Q1234.78 1522.85 1237.84 1518.27 Q1240.92 1513.66 1246.73 1513.66 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M1658.75 1544.91 L1675.07 1544.91 L1675.07 1548.85 L1653.12 1548.85 L1653.12 1544.91 Q1655.79 1542.16 1660.37 1537.53 Q1664.98 1532.88 1666.16 1531.53 Q1668.4 1529.01 1669.28 1527.27 Q1670.18 1525.51 1670.18 1523.82 Q1670.18 1521.07 1668.24 1519.33 Q1666.32 1517.6 1663.22 1517.6 Q1661.02 1517.6 1658.56 1518.36 Q1656.13 1519.13 1653.35 1520.68 L1653.35 1515.95 Q1656.18 1514.82 1658.63 1514.24 Q1661.09 1513.66 1663.12 1513.66 Q1668.49 1513.66 1671.69 1516.35 Q1674.88 1519.03 1674.88 1523.52 Q1674.88 1525.65 1674.07 1527.57 Q1673.29 1529.47 1671.18 1532.07 Q1670.6 1532.74 1667.5 1535.95 Q1664.4 1539.15 1658.75 1544.91 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M2084.47 1518.36 L2072.67 1536.81 L2084.47 1536.81 L2084.47 1518.36 M2083.25 1514.29 L2089.13 1514.29 L2089.13 1536.81 L2094.06 1536.81 L2094.06 1540.7 L2089.13 1540.7 L2089.13 1548.85 L2084.47 1548.85 L2084.47 1540.7 L2068.87 1540.7 L2068.87 1536.19 L2083.25 1514.29 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip722)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  140.696,1445.72 2352.76,1445.72 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip722)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  140.696,1174.17 2352.76,1174.17 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip722)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  140.696,902.62 2352.76,902.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip722)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  140.696,631.072 2352.76,631.072 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip722)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  140.696,359.524 2352.76,359.524 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip722)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  140.696,87.9763 2352.76,87.9763 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  140.696,1486.45 140.696,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  140.696,1445.72 159.593,1445.72 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  140.696,1174.17 159.593,1174.17 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  140.696,902.62 159.593,902.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  140.696,631.072 159.593,631.072 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  140.696,359.524 159.593,359.524 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  140.696,87.9763 159.593,87.9763 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip720)\" d=\"M92.7512 1431.51 Q89.1401 1431.51 87.3114 1435.08 Q85.5058 1438.62 85.5058 1445.75 Q85.5058 1452.86 87.3114 1456.42 Q89.1401 1459.96 92.7512 1459.96 Q96.3854 1459.96 98.1909 1456.42 Q100.02 1452.86 100.02 1445.75 Q100.02 1438.62 98.1909 1435.08 Q96.3854 1431.51 92.7512 1431.51 M92.7512 1427.81 Q98.5613 1427.81 101.617 1432.42 Q104.696 1437 104.696 1445.75 Q104.696 1454.48 101.617 1459.08 Q98.5613 1463.67 92.7512 1463.67 Q86.941 1463.67 83.8623 1459.08 Q80.8068 1454.48 80.8068 1445.75 Q80.8068 1437 83.8623 1432.42 Q86.941 1427.81 92.7512 1427.81 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M83.7929 1156.89 L102.149 1156.89 L102.149 1160.82 L88.0753 1160.82 L88.0753 1169.3 Q89.0938 1168.95 90.1123 1168.79 Q91.1308 1168.6 92.1493 1168.6 Q97.9363 1168.6 101.316 1171.77 Q104.696 1174.94 104.696 1180.36 Q104.696 1185.94 101.223 1189.04 Q97.7511 1192.12 91.4317 1192.12 Q89.2558 1192.12 86.9873 1191.75 Q84.7419 1191.38 82.3346 1190.64 L82.3346 1185.94 Q84.4179 1187.07 86.6401 1187.63 Q88.8623 1188.18 91.3391 1188.18 Q95.3437 1188.18 97.6817 1186.08 Q100.02 1183.97 100.02 1180.36 Q100.02 1176.75 97.6817 1174.64 Q95.3437 1172.54 91.3391 1172.54 Q89.4641 1172.54 87.5892 1172.95 Q85.7373 1173.37 83.7929 1174.25 L83.7929 1156.89 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M53.3995 915.965 L61.0384 915.965 L61.0384 889.599 L52.7282 891.266 L52.7282 887.007 L60.9921 885.34 L65.668 885.34 L65.668 915.965 L73.3068 915.965 L73.3068 919.9 L53.3995 919.9 L53.3995 915.965 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M92.7512 888.419 Q89.1401 888.419 87.3114 891.983 Q85.5058 895.525 85.5058 902.655 Q85.5058 909.761 87.3114 913.326 Q89.1401 916.868 92.7512 916.868 Q96.3854 916.868 98.1909 913.326 Q100.02 909.761 100.02 902.655 Q100.02 895.525 98.1909 891.983 Q96.3854 888.419 92.7512 888.419 M92.7512 884.715 Q98.5613 884.715 101.617 889.321 Q104.696 893.905 104.696 902.655 Q104.696 911.381 101.617 915.988 Q98.5613 920.571 92.7512 920.571 Q86.941 920.571 83.8623 915.988 Q80.8068 911.381 80.8068 902.655 Q80.8068 893.905 83.8623 889.321 Q86.941 884.715 92.7512 884.715 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M54.3949 644.417 L62.0337 644.417 L62.0337 618.051 L53.7236 619.718 L53.7236 615.459 L61.9874 613.792 L66.6633 613.792 L66.6633 644.417 L74.3022 644.417 L74.3022 648.352 L54.3949 648.352 L54.3949 644.417 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M83.7929 613.792 L102.149 613.792 L102.149 617.727 L88.0753 617.727 L88.0753 626.199 Q89.0938 625.852 90.1123 625.69 Q91.1308 625.505 92.1493 625.505 Q97.9363 625.505 101.316 628.676 Q104.696 631.848 104.696 637.264 Q104.696 642.843 101.223 645.945 Q97.7511 649.023 91.4317 649.023 Q89.2558 649.023 86.9873 648.653 Q84.7419 648.283 82.3346 647.542 L82.3346 642.843 Q84.4179 643.977 86.6401 644.533 Q88.8623 645.088 91.3391 645.088 Q95.3437 645.088 97.6817 642.982 Q100.02 640.875 100.02 637.264 Q100.02 633.653 97.6817 631.547 Q95.3437 629.44 91.3391 629.44 Q89.4641 629.44 87.5892 629.857 Q85.7373 630.273 83.7929 631.153 L83.7929 613.792 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M56.6171 372.869 L72.9365 372.869 L72.9365 376.804 L50.9921 376.804 L50.9921 372.869 Q53.6541 370.114 58.2375 365.485 Q62.8439 360.832 64.0245 359.489 Q66.2698 356.966 67.1494 355.23 Q68.0522 353.471 68.0522 351.781 Q68.0522 349.027 66.1078 347.29 Q64.1865 345.554 61.0847 345.554 Q58.8856 345.554 56.4319 346.318 Q54.0014 347.082 51.2236 348.633 L51.2236 343.911 Q54.0477 342.777 56.5014 342.198 Q58.955 341.619 60.9921 341.619 Q66.3624 341.619 69.5568 344.304 Q72.7513 346.99 72.7513 351.48 Q72.7513 353.61 71.9411 355.531 Q71.1541 357.429 69.0476 360.022 Q68.4689 360.693 65.367 363.911 Q62.2652 367.105 56.6171 372.869 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M92.7512 345.323 Q89.1401 345.323 87.3114 348.888 Q85.5058 352.429 85.5058 359.559 Q85.5058 366.665 87.3114 370.23 Q89.1401 373.772 92.7512 373.772 Q96.3854 373.772 98.1909 370.23 Q100.02 366.665 100.02 359.559 Q100.02 352.429 98.1909 348.888 Q96.3854 345.323 92.7512 345.323 M92.7512 341.619 Q98.5613 341.619 101.617 346.226 Q104.696 350.809 104.696 359.559 Q104.696 368.286 101.617 372.892 Q98.5613 377.475 92.7512 377.475 Q86.941 377.475 83.8623 372.892 Q80.8068 368.286 80.8068 359.559 Q80.8068 350.809 83.8623 346.226 Q86.941 341.619 92.7512 341.619 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M57.6125 101.321 L73.9318 101.321 L73.9318 105.256 L51.9875 105.256 L51.9875 101.321 Q54.6495 98.5665 59.2328 93.9369 Q63.8393 89.2841 65.0198 87.9416 Q67.2652 85.4184 68.1448 83.6823 Q69.0476 81.9231 69.0476 80.2333 Q69.0476 77.4786 67.1032 75.7425 Q65.1819 74.0064 62.08 74.0064 Q59.881 74.0064 57.4273 74.7703 Q54.9967 75.5342 52.219 77.0851 L52.219 72.3629 Q55.043 71.2287 57.4967 70.65 Q59.9504 70.0713 61.9874 70.0713 Q67.3578 70.0713 70.5522 72.7565 Q73.7466 75.4416 73.7466 79.9323 Q73.7466 82.062 72.9365 83.9832 Q72.1494 85.8814 70.0429 88.474 Q69.4642 89.1453 66.3624 92.3628 Q63.2606 95.5573 57.6125 101.321 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M83.7929 70.6963 L102.149 70.6963 L102.149 74.6314 L88.0753 74.6314 L88.0753 83.1036 Q89.0938 82.7564 90.1123 82.5944 Q91.1308 82.4092 92.1493 82.4092 Q97.9363 82.4092 101.316 85.5805 Q104.696 88.7517 104.696 94.1684 Q104.696 99.747 101.223 102.849 Q97.7511 105.928 91.4317 105.928 Q89.2558 105.928 86.9873 105.557 Q84.7419 105.187 82.3346 104.446 L82.3346 99.747 Q84.4179 100.881 86.6401 101.437 Q88.8623 101.992 91.3391 101.992 Q95.3437 101.992 97.6817 99.8859 Q100.02 97.7795 100.02 94.1684 Q100.02 90.5573 97.6817 88.4508 Q95.3437 86.3443 91.3391 86.3443 Q89.4641 86.3443 87.5892 86.761 Q85.7373 87.1777 83.7929 88.0573 L83.7929 70.6963 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip722)\" style=\"stroke:#009af9; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  203.301,87.9763 224.17,141.743 245.038,194.423 265.907,246.017 286.775,296.525 307.643,345.947 328.512,394.282 349.38,441.532 370.249,487.695 391.117,532.772 \n",
       "  411.986,576.762 432.854,619.667 453.723,661.485 474.591,702.218 495.46,741.864 516.328,780.423 537.197,817.897 558.065,854.284 578.934,889.586 599.802,923.801 \n",
       "  620.671,956.93 641.539,988.972 662.408,1019.93 683.276,1049.8 704.145,1078.58 725.013,1106.28 745.882,1132.89 766.75,1158.42 787.619,1182.86 808.487,1206.21 \n",
       "  829.356,1228.48 850.224,1249.66 871.093,1269.75 891.961,1288.76 912.83,1306.68 933.698,1323.52 954.567,1339.27 975.435,1353.93 996.304,1367.51 1017.17,1380 \n",
       "  1038.04,1391.41 1058.91,1401.72 1079.78,1410.96 1100.65,1419.1 1121.51,1426.16 1142.38,1432.14 1163.25,1437.03 1184.12,1440.83 1204.99,1443.54 1225.86,1445.17 \n",
       "  1246.73,1445.72 1267.59,1445.17 1288.46,1443.54 1309.33,1440.83 1330.2,1437.03 1351.07,1432.14 1371.94,1426.16 1392.81,1419.1 1413.67,1410.96 1434.54,1401.72 \n",
       "  1455.41,1391.41 1476.28,1380 1497.15,1367.51 1518.02,1353.93 1538.88,1339.27 1559.75,1323.52 1580.62,1306.68 1601.49,1288.76 1622.36,1269.75 1643.23,1249.66 \n",
       "  1664.1,1228.48 1684.96,1206.21 1705.83,1182.86 1726.7,1158.42 1747.57,1132.89 1768.44,1106.28 1789.31,1078.58 1810.18,1049.8 1831.04,1019.93 1851.91,988.972 \n",
       "  1872.78,956.93 1893.65,923.801 1914.52,889.586 1935.39,854.284 1956.25,817.897 1977.12,780.423 1997.99,741.864 2018.86,702.218 2039.73,661.485 2060.6,619.667 \n",
       "  2081.47,576.762 2102.33,532.772 2123.2,487.695 2144.07,441.532 2164.94,394.282 2185.81,345.947 2206.68,296.525 2227.54,246.017 2248.41,194.423 2269.28,141.743 \n",
       "  2290.15,87.9763 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip720)\" d=\"\n",
       "M1980.81 198.898 L2279.02 198.898 L2279.02 95.2176 L1980.81 95.2176  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1980.81,198.898 2279.02,198.898 2279.02,95.2176 1980.81,95.2176 1980.81,198.898 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip720)\" style=\"stroke:#009af9; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2005.38,147.058 2152.85,147.058 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip720)\" d=\"M2191.28 166.745 Q2189.47 171.375 2187.76 172.787 Q2186.04 174.199 2183.17 174.199 L2179.77 174.199 L2179.77 170.634 L2182.27 170.634 Q2184.03 170.634 2185 169.8 Q2185.98 168.967 2187.16 165.865 L2187.92 163.921 L2177.43 138.412 L2181.95 138.412 L2190.05 158.689 L2198.15 138.412 L2202.66 138.412 L2191.28 166.745 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip720)\" d=\"M2209.96 160.402 L2217.6 160.402 L2217.6 134.037 L2209.29 135.703 L2209.29 131.444 L2217.55 129.778 L2222.22 129.778 L2222.22 160.402 L2229.86 160.402 L2229.86 164.338 L2209.96 164.338 L2209.96 160.402 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /></svg>\n"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot(-5:0.1:5, x-> x^2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5578eae",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Alright, so how do we get the minimum? \n",
    "\n",
    "Let's use Optim to see how. I'm going to use the Nelder-Mead algorithm, which is generally considered a good local optimization algorithm that does not require a gradient. \n",
    "\n",
    "Nelder-Mead is a quick-and-dirty optimization algorithm. If you have a smooth problem with a reasonable gradient, it's probably a little wiser to use an alternative method (more in a bit). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "7b7be090",
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Info: Nelder-Mead optimum\n",
      "│   res.minimizer = [0.00024414062500016914]\n",
      "│   res.minimum = 5.960464477547321e-8\n",
      "│   Optim.converged(res) = true\n",
      "└ @ Main In[105]:11\n"
     ]
    }
   ],
   "source": [
    "using Optim\n",
    "\n",
    "# Define our target function\n",
    "f(x) = x[1]^2\n",
    "\n",
    "# Minimize it with Nelder-Mead, the default\n",
    "res = optimize(f, [1.0])\n",
    "# res = optimize(f, [1.0], NelderMead()) # Or, to explicitly use Nelder-Mead\n",
    "\n",
    "# Get the minimizer info\n",
    "@info \"Nelder-Mead optimum\" res.minimizer res.minimum Optim.converged(res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "c7feaf42",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " * Status: success\n",
      "\n",
      " * Candidate solution\n",
      "    Final objective value:     5.960464e-08\n",
      "\n",
      " * Found with\n",
      "    Algorithm:     Nelder-Mead\n",
      "\n",
      " * Convergence measures\n",
      "    √(Σ(yᵢ-ȳ)²)/n ≤ 1.0e-08\n",
      "\n",
      " * Work counters\n",
      "    Seconds run:   0  (vs limit Inf)\n",
      "    Iterations:    10\n",
      "    f(x) calls:    23\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Or you can just print it out for some summary info:\n",
    "println(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89d8f420",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Concept check!\n",
    "\n",
    "Let's do some maximum likelihood estimation. Suppose we have $n$ data points we believe were drawn from a four-dimensional Normal distribution with an unknown mean. Assume the variance is the identity matrix for now.\n",
    "\n",
    "Then we have\n",
    "\n",
    "$$\n",
    "\\hat \\mu = \\arg \\min_{m} \\sum_{i=1}^n \\log P(X_n \\mid \\mu=m, \\sigma=I)\n",
    "$$\n",
    "\n",
    "How do we find the maximum likelihood estimate? Well, let's use Optim and Nelder-Mead!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34fca0d1",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Concept check!\n",
    "\n",
    "Install the `Distributions` package if you have not already.\n",
    "\n",
    "Use the code below as a starting point to find the maximum likelihood estimates for the means of the data. The likelihood of a distribution is `logpdf(MvNormal(m, I), data)`.\n",
    "\n",
    "```julia\n",
    "# Imports\n",
    "using Distributions, LinearAlgebra\n",
    "\n",
    "# Create the true distribution\n",
    "n = 1000\n",
    "means = [1.0, 0.0, 3.0, 4.0]\n",
    "dist = MvNormal(means, I)\n",
    "xs = rand(dist, n)\n",
    "```\n",
    "\n",
    "If you finish quickly, modify this problem to calculate a homoscedastic variance parameter -- i.e. change `dist = MvNormal(means, sigma*I)` and change the `loglike` function to incorporate `sigma` as well. It can be a little tricky, as I'll demonstrate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "b23ffeda",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "using Distributions, LinearAlgebra\n",
    "\n",
    "# Create the true distribution\n",
    "n = 1000\n",
    "means = [1.0, 0.0, 3.0, 4.0]\n",
    "dist = MvNormal(means, I)\n",
    "xs = rand(dist, n)\n",
    "loglike(theta) = -sum(logpdf(MvNormal(theta, I), xs));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "dce40d6f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4-element Vector{Float64}:\n",
       " 1.0076449294678482\n",
       " 0.006400008096099833\n",
       " 3.021220346225461\n",
       " 3.9949354206614744"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = optimize(loglike, zeros(4), Optim.Options(iterations=10_000))\n",
    "res.minimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "773a575d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Warning: check that you actually solved it first!\n",
    "\n",
    "Optim will let you extract the value `res.minimizer` _even if the problem wasn't solved!_ Make sure to check that you have a solution first!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "c3eff74e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       " * Status: failure (reached maximum number of iterations)\n",
       "\n",
       " * Candidate solution\n",
       "    Final objective value:     1.247511e-01\n",
       "\n",
       " * Found with\n",
       "    Algorithm:     Simulated Annealing\n",
       "\n",
       " * Convergence measures\n",
       "    |x - x'|               = NaN ≰ 0.0e+00\n",
       "    |x - x'|/|x'|          = NaN ≰ 0.0e+00\n",
       "    |f(x) - f(x')|         = NaN ≰ 0.0e+00\n",
       "    |f(x) - f(x')|/|f(x')| = NaN ≰ 0.0e+00\n",
       "    |g(x)|                 = NaN ≰ 1.0e-08\n",
       "\n",
       " * Work counters\n",
       "    Seconds run:   0  (vs limit Inf)\n",
       "    Iterations:    1000\n",
       "    f(x) calls:    1001\n"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rosenbrock(x) =  (1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2\n",
    "result = optimize(\n",
    "    rosenbrock, \n",
    "    zeros(2), \n",
    "    SimulatedAnnealing(), \n",
    "    Optim.Options(iterations=1000)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "9ef17d6f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element Vector{Float64}:\n",
       " 1.1512058571032084\n",
       " 1.2933550450208446"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.minimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4f8cff4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Checking convergence\n",
    "\n",
    "You can check convergence with `converged(res)` or by checking the individual convergence flags:\n",
    "\n",
    "```julia\n",
    "converged(res) # Checks all flags\n",
    "iteration_limit_reached(res)\n",
    "x_converged(res)\n",
    "f_converged(res)\n",
    "g_converged(res)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6403bf9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Optimizing with gradients\n",
    "\n",
    "Oftentimes you have (or can get) gradients of an objective function. Gradients are extraordinarily useful for smooth (or locally smooth) function optimization, because you can often access significantly more efficient optimization algorithms.\n",
    "\n",
    "The most common one of these is LBFGS. LBFGS uses the gradient to approximate a Hessian, and then move each value in the direction of a minima using that implied Hessian.\n",
    "\n",
    "Fortunately, it's super easy to just pick a new algorithm with Optim! Lets try it now returning to our simple quadratic function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "b1cb57f5",
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Info: LBFGS optimum\n",
      "│   res.minimizer = [-6.359357485052897e-13]\n",
      "│   res.minimum = 4.0441427622698303e-25\n",
      "│   Optim.converged(res) = true\n",
      "└ @ Main In[112]:3\n"
     ]
    }
   ],
   "source": [
    "# Minimize it with LBFGS, a common gradient-required method\n",
    "res = optimize(f, [1.0], LBFGS())\n",
    "@info \"LBFGS optimum\" res.minimizer res.minimum Optim.converged(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "687d9aa7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Note that `Optim` will default to using finite differences to calculate gradients. Finite differences are numerical, not exact, and are less accurate and prone to numerical errors.\n",
    "\n",
    "If we wanted to use our shiny new automatic differentiation tools, we could get _exact_ gradients that should speed up our estimation.\n",
    "\n",
    "Let's try this the \"foolproof\" way first, then I'll show you how to provide your own gradient if needed.\n",
    "\n",
    "The foolproof way is to set the keyword `autodiff=:forward`. Only `:finite` or `:forward` are supported -- if you want reverse mode AD, you'll have to wait just a second."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "dede6e68",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Info: Nelder-Mead optimum\n",
      "│   res.minimizer = [0.0]\n",
      "│   res.minimum = 0.0\n",
      "│   Optim.converged(res) = true\n",
      "└ @ Main In[113]:3\n"
     ]
    }
   ],
   "source": [
    "# Minimize it with LBFGS & autodiff\n",
    "res = optimize(f, [1.0], LBFGS(), autodiff=:forward)\n",
    "@info \"Nelder-Mead optimum\" res.minimizer res.minimum Optim.converged(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "368232c6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Using your own gradients\n",
    "\n",
    "You can pass in your own gradient functions if you want to use a different AD-backend, or if you have exact derivatives already coded up.\n",
    "\n",
    "To do so, you need to write a mutating function\n",
    "\n",
    "```julia\n",
    "function g!(G, x)\n",
    "    G[1] = . . .\n",
    "end\n",
    "```\n",
    "\n",
    "that sets each element of `G` to the gradient of your function w.r.t each parameter in `x`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a8d4fde",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall that our target function here, $f(x) = x^2$, has the derivative $2 x$. We can just write this up!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "984316d0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "g! (generic function with 1 method)"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Analytic gradient\n",
    "function g!(G, x)\n",
    "    G[1] = 2*x[1]\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "a1132653",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Info: Nelder-Mead optimum\n",
      "│   res.minimizer = [0.0]\n",
      "│   res.minimum = 0.0\n",
      "│   Optim.converged(res) = true\n",
      "└ @ Main In[116]:3\n"
     ]
    }
   ],
   "source": [
    "# Minimize it with LBFGS & exact gradient\n",
    "res = optimize(f, g!, [1.0], LBFGS())\n",
    "@info \"Nelder-Mead optimum\" res.minimizer res.minimum Optim.converged(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8851b2f4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Suppose now we had a much larger model with a gradient that's too hard (or we are too lazy) to calculate by hand. \n",
    "\n",
    "As per the above, we'd maybe want to use ReverseDiff! Let's write a `g!` function that calls `ReverseDiff` for us.\n",
    "\n",
    "I'm going to use the maximum likelihood example from above, but this time with (a) large dimensionality, good for reverse mode, and (b) a diagonal covariance matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dc1fbc3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Importantly, the __variance parameters cannot be negative!__. You can do this a couple of ways, such as taking the `exp` of variance parameters, but here I will use box minimization to constrain the optimizer to a region where the sigmas have to be positive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "901b1a46",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "using Distributions, LinearAlgebra\n",
    "\n",
    "# Create the true distribution\n",
    "n = 1000\n",
    "K = 100 # dimensionality\n",
    "means = randn(K)\n",
    "variances = rand(InverseGamma(2,3), K)\n",
    "dist = MvNormal(means, Diagonal(variances))\n",
    "xs = rand(dist, n);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "82bce611",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Define our target function\n",
    "function loglike(par)\n",
    "    return -sum(logpdf(MvNormal(par[1:K], Diagonal(par[K+1:end])), xs))\n",
    "end\n",
    "\n",
    "# Define upper/lower bounds so that the variances remain positive\n",
    "lower = vcat(repeat([-Inf], K), repeat([0],K));\n",
    "upper = vcat(repeat([Inf], 2K));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "4902ddba",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "g! (generic function with 1 method)"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Our gradient function simply calls ReverseDiff for us\n",
    "tape = ReverseDiff.GradientTape(loglike, zeros(2*K))\n",
    "g!(G, x) = ReverseDiff.gradient!(G, tape, x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "739b52c7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       " * Status: success\n",
       "\n",
       " * Candidate solution\n",
       "    Final objective value:     1.768179e+05\n",
       "\n",
       " * Found with\n",
       "    Algorithm:     Fminbox with L-BFGS\n",
       "\n",
       " * Convergence measures\n",
       "    |x - x'|               = 2.79e-08 ≰ 0.0e+00\n",
       "    |x - x'|/|x'|          = 7.34e-10 ≰ 0.0e+00\n",
       "    |f(x) - f(x')|         = 0.00e+00 ≤ 0.0e+00\n",
       "    |f(x) - f(x')|/|f(x')| = 0.00e+00 ≤ 0.0e+00\n",
       "    |g(x)|                 = 7.47e-09 ≤ 1.0e-08\n",
       "\n",
       " * Work counters\n",
       "    Seconds run:   94  (vs limit Inf)\n",
       "    Iterations:    7\n",
       "    f(x) calls:    1352\n",
       "    ∇f(x) calls:   1352\n"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Optimize it!\n",
    "res = optimize(loglike, g!, lower, upper, ones(K*2), Fminbox(LBFGS()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e45f09ff",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# JuMP\n",
    "\n",
    "Optim is great for relatively easy problems, but sometimes you need to work with a lot of constraints, integer variables, etc. \n",
    "\n",
    "This is where JuMP comes in."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6209f626",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# JuMP\n",
    "\n",
    "JuMP is technically a _domain specific language_, or DSL. \n",
    "\n",
    "A DSL is essentially an applied programming language for a specific task -- in the case of JuMP, this is defining and solving optimization problems.\n",
    "\n",
    "JuMP allows you to write arbitrary problems, and then hand those problems off to a relatively large library of solvers.\n",
    "\n",
    "JuMP has an _amazing_ community. If you've got optimization problems, the JuMP community can probably help."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c427eca8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## JuMP Solvers\n",
    "\n",
    "JuMP supports lots of solvers, all of which permit you to work on different types of problems. Some solvers are faster than others or may require a license (like Knitro or Gurobi, which are available on the Stanford yens/sherlock).\n",
    "\n",
    "A list of solvers is available [here](https://jump.dev/JuMP.jl/stable/installation/#Supported-solvers)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9c72a01",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You have to install a solver as an additional package -- for example, let's pick the `Clp` solver. It solves linear programs (the simple ones) so it'll be a good demo for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "eb76cd15",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m\u001b[1m   Resolving\u001b[22m\u001b[39m package versions...\n",
      "\u001b[32m\u001b[1m  No Changes\u001b[22m\u001b[39m to `~/code/julia-bootcamp-2022/Project.toml`\n",
      "\u001b[32m\u001b[1m  No Changes\u001b[22m\u001b[39m to `~/code/julia-bootcamp-2022/Manifest.toml`\n"
     ]
    }
   ],
   "source": [
    "import Pkg; Pkg.add([\"JuMP\", \"Clp\", \"HiGHS\", \"Ipopt\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2b50270",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A simple example\n",
    "\n",
    "I'm more or less copying this from JuMP's [spectacular documentation](https://jump.dev/JuMP.jl/stable/tutorials/getting_started/getting_started_with_JuMP/#Getting-started-with-JuMP), so refer there for more information if JuMP is a thing you think you'll end up using.\n",
    "\n",
    "Suppose we have the simple linear program \n",
    "\n",
    "![](problem.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7fbf1d2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's write this as a JuMP model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "15e1f7e3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$$ \\begin{aligned}\n",
       "\\min\\quad & 12 x + 20 y\\\\\n",
       "\\text{Subject to} \\quad & 6 x + 8 y \\geq 100.0\\\\\n",
       " & 7 x + 12 y \\geq 120.0\\\\\n",
       " & x \\geq 0.0\\\\\n",
       " & y \\geq 0.0\\\\\n",
       " & y \\leq 3.0\\\\\n",
       "\\end{aligned} $$"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import our packages.\n",
    "using JuMP\n",
    "using Clp\n",
    "\n",
    "# Set up a 'model', which is a bucket to hold your problem definition.\n",
    "# Use OptimizerPackage.Optimizer to use another solver, i.e. HiGHS.Optimizer\n",
    "model = Model(Clp.Optimizer) \n",
    "\n",
    "# Add our variables\n",
    "@variable(model, x >= 0)\n",
    "@variable(model, 0 <= y <= 3)\n",
    "\n",
    "# Set the objective function\n",
    "@objective(model, Min, 12x + 20y)\n",
    "\n",
    "# Write our constraints\n",
    "@constraint(model, c1, 6x + 8y >= 100)\n",
    "@constraint(model, c2, 7x + 12y >= 120)\n",
    "\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "259a6f29",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coin0506I Presolve 2 (0) rows, 2 (0) columns and 4 (0) elements\n",
      "Clp0006I 0  Obj 0 Primal inf 22.5 (2)\n",
      "Clp0006I 2  Obj 205\n",
      "Clp0000I Optimal - objective value 205\n",
      "Clp0032I Optimal objective 205 - 2 iterations time 0.002\n"
     ]
    }
   ],
   "source": [
    "# Now we solve it!\n",
    "optimize!(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "e8429d5b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "termination_status(model) = MathOptInterface.OPTIMAL\n",
      "primal_status(model) = MathOptInterface.FEASIBLE_POINT\n",
      "dual_status(model) = MathOptInterface.FEASIBLE_POINT\n",
      "objective_value(model) = 205.0\n",
      "value(x) = 14.999999999999995\n",
      "value(y) = 1.2500000000000036\n",
      "shadow_price(c1) = -0.25\n",
      "shadow_price(c2) = -1.5\n"
     ]
    }
   ],
   "source": [
    "# Show some stats on the solution\n",
    "@show termination_status(model)\n",
    "@show primal_status(model)\n",
    "@show dual_status(model)\n",
    "@show objective_value(model)\n",
    "@show value(x) # Extracts the value of x\n",
    "@show value(y) # Extracts the value of y\n",
    "@show shadow_price(c1)\n",
    "@show shadow_price(c2);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "b8eb9bb7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A JuMP Model\n",
       "Minimization problem with:\n",
       "Variables: 2\n",
       "Objective function type: AffExpr\n",
       "`AffExpr`-in-`MathOptInterface.GreaterThan{Float64}`: 2 constraints\n",
       "`VariableRef`-in-`MathOptInterface.GreaterThan{Float64}`: 1 constraint\n",
       "`VariableRef`-in-`MathOptInterface.Interval{Float64}`: 1 constraint\n",
       "Model mode: AUTOMATIC\n",
       "CachingOptimizer state: NO_OPTIMIZER\n",
       "Solver name: No optimizer attached."
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Save our solution to disk\n",
    "write_to_file(model, \"model.mps\")\n",
    "\n",
    "# Read it from disk\n",
    "model = read_from_file(\"model.mps\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c8c5275",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Checking solution status\n",
    "\n",
    "It's generally a good idea to check to make sure your model actually solved. Unlike Optim, JuMP will throw an error if you try to retrieve optimal values if the solver failed, but you shouldn't rely on errors from anything to notify you of estimation failures.\n",
    "\n",
    "Use `termination_status(model)` or `primal_status(model)` to get the status -- you can see all the termination codes in [the docs](https://jump.dev/JuMP.jl/stable/moi/reference/models/#MathOptInterface.ResultStatusCode).\n",
    "\n",
    "JuMP exports all-caps variables following the name of terminations codes, i.e. `INFEASIBLE`, `OPTIMAL`, etc. are all valid variables after calling `Optim.jl`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "63268ee8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coin0507I Presolve determined that the problem was infeasible with tolerance of 1e-08\n",
      "Clp3003W Analysis indicates model infeasible or unbounded\n",
      "Clp0006I 0  Obj 0 Primal inf 4.9999999 (1) Dual inf 0.9999999 (1)\n",
      "Clp0006I 1  Obj -3 Primal inf 2.9999999 (1)\n",
      "Clp0001I Primal infeasible - objective value -3\n",
      "Clp0032I PrimalInfeasible objective -3 - 1 iterations time 0.002\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = Model(Clp.Optimizer)\n",
    "\n",
    "@variable(model, 0 <= x <= 1)\n",
    "@variable(model, 0 <= y <= 1)\n",
    "@constraint(model, x + y >= 5)\n",
    "@objective(model, Max, x - y)\n",
    "optimize!(model)\n",
    "\n",
    "termination_status(model) == INFEASIBLE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "940c1e2d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Solver attributes\n",
    "\n",
    "You can change the behavior of different solvers with `set_solver_attribute(model, \"name_of_attribute\", value)`. This can often help you debug issues if solvers aren't converging.\n",
    "\n",
    "All solvers have different attributes! You'll need to look up the ones specific to your solver.\n",
    "\n",
    "Let's look at [the attributes](/~https://github.com/jump-dev/Clp.jl) for `Clp.jl`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "id": "c84e9fe5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coin0502I Problem is infeasible due to row 0, 5 1.797693134862316e+308\n",
      "Coin0507I Presolve determined that the problem was infeasible with tolerance of 1e-08\n",
      "Clp0033I Presolve took 0.00 seconds (total 0.00)\n",
      "Clp3003W Analysis indicates model infeasible or unbounded\n",
      "Clp1001I Initial range of elements is 1 to 1\n",
      "Clp1004I Not bothering to scale as good enough\n",
      "Clp0022I Absolute values of scaled rhs range from 5 to 5, minimum gap 1e+100\n",
      "Clp0020I Absolute values of scaled objective range from 1 to 1\n",
      "Clp0021I Absolute values of scaled bounds range from 1 to 1, minimum gap 1\n",
      "Clp0060I Primal error 0, dual error 0\n",
      "Clp0060I Primal error 0, dual error 0\n",
      "Clp0006I 0  Obj 0 Primal inf 4.9999999 (1) Dual inf 0.9999999 (1)\n",
      "Clp0060I Primal error 0, dual error 0\n",
      "Clp0060I Primal error 0, dual error 0\n",
      "Clp0006I 1  Obj -3 Primal inf 2.9999999 (1)\n",
      "Clp0001I Primal infeasible - objective value -3\n",
      "Clp0033I Dual took 0.00 seconds (total 0.00)\n",
      "Clp0032I PrimalInfeasible objective -3 - 1 iterations time 0.002\n"
     ]
    }
   ],
   "source": [
    "model = Model(Clp.Optimizer)\n",
    "set_optimizer_attribute(model, \"LogLevel\", 4)\n",
    "\n",
    "@variable(model, 0 <= x <= 1)\n",
    "@variable(model, 0 <= y <= 1)\n",
    "@constraint(model, x + y >= 5)\n",
    "@objective(model, Max, x - y)\n",
    "optimize!(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b50f97a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Variables and domain constraints\n",
    "\n",
    "You can specify most inequality constraints:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "id": "24e1a178",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$$ \\begin{aligned}\n",
       "\\text{feasibility}\\\\\n",
       "\\text{Subject to} \\quad & x\\_fixed = 4.0\\\\\n",
       " & x\\_lower \\geq 0.0\\\\\n",
       " & x\\_interval \\geq 2.0\\\\\n",
       " & x\\_upper \\leq 1.0\\\\\n",
       " & x\\_interval \\leq 3.0\\\\\n",
       "\\end{aligned} $$"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = Model()\n",
    "@variable(model, x_free)\n",
    "@variable(model, x_lower >= 0)\n",
    "@variable(model, x_upper <= 1)\n",
    "@variable(model, 2 <= x_interval <= 3)\n",
    "@variable(model, x_fixed == 4)\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50fd1d95",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Vector/matrix variables\n",
    "\n",
    "Oftentimes you have parameters in a vector format $[x_1, x_2, \\dots, x_k]$. In JuMP, this is very easy to specify:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "id": "ed258cbc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A JuMP Model\n",
       "Feasibility problem with:\n",
       "Variables: 110\n",
       "Model mode: AUTOMATIC\n",
       "CachingOptimizer state: NO_OPTIMIZER\n",
       "Solver name: No optimizer attached.\n",
       "Names registered in the model: x, y"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = Model()\n",
    "\n",
    "n = 10\n",
    "@variable(model, x[1:n]) # Make x a length 5 vector\n",
    "@variable(model, y[1:n, 1:n]) # Make y a 5x5 array\n",
    "\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebba9d9a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Vector constraints\n",
    "\n",
    "You can add vector/matrix constraints by _labelling_ a dimension and using that dimension in the inequalities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "id": "0e5176a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Vector{VariableRef}:\n",
       " y[1]\n",
       " y[2]\n",
       " y[3]"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = Model()\n",
    "bounds = rand(3)\n",
    "@variable(model, sqrt(i) <= x[i = 1:3] <= i^2)\n",
    "@variable(model, y[i = 1:3] <= bounds[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a277b56b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Integer/binary variables\n",
    "\n",
    "Just add the type of your variable at the end of the `@variable` call:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "176923e5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$$ x\\_int\\_limited $$"
      ],
      "text/plain": [
       "x_int_limited"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = Model()\n",
    "@variable(model, x_binary, Bin) # true/false 0/1 only.\n",
    "@variable(model, x_int, Int) # Any integer\n",
    "@variable(model, 0 <= x_int_limited <= 5, Int) # only {0,1,2,3,4,5}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe22dd0b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Positive semidefinite and symmetric matrix variables\n",
    "\n",
    "If you're trying to do something like optimize a maximum likelihood estimate of a covariance matrix, you may want to flag the matrix as PSD (or symmetric in other problems) to avoid errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "974c698f",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model()\n",
    "n = 10\n",
    "@variable(model, Σ[1:n, 1:n], PSD) # For positive semi-definite\n",
    "@variable(model, Ω[1:n, 1:n], Symmetric); # For symmetric"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bf6d7a6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Constraints\n",
    "\n",
    "You can add a _lot_ of constraint types to your problem. Take the following for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "id": "88cb07e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$$ \\begin{aligned}\n",
       "\\max\\quad & q_{1}\\times x_{1} + q_{2}\\times x_{2} + q_{3}\\times x_{3} + q_{4}\\times x_{4} + q_{5}\\times x_{5}\\\\\n",
       "\\text{Subject to} \\quad & q_{1} + q_{2} + q_{3} + q_{4} + q_{5} = 1.0\\\\\n",
       " & q_{1}\\times x_{1} + q_{2}\\times x_{2} + q_{3}\\times x_{3} + q_{4}\\times x_{4} + q_{5}\\times x_{5} \\leq 15.0\\\\\n",
       " & q_{5}\\times x_{1} - x_{5}\\times x_{2} + 2 x_{3} \\leq 0.0\\\\\n",
       " & q_{1} \\geq 0.0\\\\\n",
       " & q_{2} \\geq 0.0\\\\\n",
       " & q_{3} \\geq 0.0\\\\\n",
       " & q_{4} \\geq 0.0\\\\\n",
       " & q_{5} \\geq 0.0\\\\\n",
       " & q_{1} \\leq 1.0\\\\\n",
       " & q_{2} \\leq 1.0\\\\\n",
       " & q_{3} \\leq 1.0\\\\\n",
       " & q_{4} \\leq 1.0\\\\\n",
       " & q_{5} \\leq 1.0\\\\\n",
       " & sqrt(x_{1} * q_{2} + x_{2} * q_{3}) - 100.0 \\leq 0\\\\\n",
       "\\end{aligned} $$"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "using Ipopt\n",
    "model = Model(Ipopt.Optimizer)\n",
    "@variable(model, x[1:5])\n",
    "@variable(model, 0 <= q[1:5] <= 1)\n",
    "@constraint(model, q'x <= 15)\n",
    "@constraint(model, sum(q) == 1)\n",
    "@constraint(model, q[5]*x[1] + 2*x[3] <= x[2] * x[5])\n",
    "@NLconstraint(model, sqrt(sum(x[i]*q[i+1] for i in 1:2)) <= 100)\n",
    "@objective(model, Max, q'x)\n",
    "\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bd01c10",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Programmatic constraints\n",
    "\n",
    "Since this is all just Julia code, you can add constraints programmatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "4d86332f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$$ \\begin{aligned}\n",
       "\\text{feasibility}\\\\\n",
       "\\text{Subject to} \\quad & -z_{1} \\leq 0.0\\\\\n",
       " & -z_{3} \\leq 0.0\\\\\n",
       " & -z_{4} \\leq 0.0\\\\\n",
       " & -z_{6} \\leq 0.0\\\\\n",
       "\\end{aligned} $$"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = Model()\n",
    "\n",
    "@variable(model, z[1:10])\n",
    "x = randn(10)\n",
    "\n",
    "for i in 1:length(x)\n",
    "    if x[i] <= 0\n",
    "        @constraint(model, 0 <= z[i])\n",
    "    end\n",
    "end\n",
    "\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a7031d4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Concept check!\n",
    "\n",
    "Let's look at an example near and dear to my heart, portfolio optimization! Borrowed, of course, from the [JuMP docs](https://jump.dev/JuMP.jl/stable/tutorials/nonlinear/portfolio/).\n",
    "\n",
    "Suppose we had some stocks that followed a known distribution \n",
    "\n",
    "```julia\n",
    "# Set up return distribution\n",
    "using Random; Random.seed!(15)\n",
    "n_stocks = 10\n",
    "μ = rand(MvNormal(zeros(n_stocks), 0.1I))\n",
    "Σ = rand(InverseWishart(n_stocks, diagm(ones(n_stocks))))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a44f4ec",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Concept check!\n",
    "\n",
    "We have an investor who wants to allocate their wealth to each of these stocks so as to minimize their portfolio risk while setting the return to a desired expected return $R$:\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\min_{q_i} \\quad & q'\\Sigma q \\\\\n",
    "    s.t.         & \\sum_i^N q_i = 1   \\\\\n",
    "                 & q'\\mu = R  \\\\\n",
    "                 & q_i \\in [-1,1] \\forall i\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "Use `JuMP` and the solver `Ipopt` to calculate the optimal portfolio for this investor and this economy.\n",
    "\n",
    "__Bonus points:__ Wrap your JuMP code in a function called `opt_portfolio(R)` that returns a tuple `(q'Σ*q, q'μ)`. Use `Plots.jl` to plot the efficient frontier (expected return for a given minimum variance portfolio). This is commonly done with the variance on the x-axis and the return on the y-axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "b068e001",
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Set up return distribution\n",
    "using Random; Random.seed!(15)\n",
    "n_stocks = 10\n",
    "μ = rand(MvNormal(zeros(n_stocks), 0.1I))\n",
    "Σ = rand(InverseWishart(n_stocks, diagm(ones(n_stocks))))\n",
    "\n",
    "function opt_portfolio(R)\n",
    "    portfolio = Model(Ipopt.Optimizer)\n",
    "    set_silent(portfolio)\n",
    "    @variable(portfolio, -1 <= q[1:n_stocks] <= 1)\n",
    "    @objective(portfolio, Min, q' * Σ * q)\n",
    "    @constraint(portfolio, sum(q) == 1)\n",
    "    @constraint(portfolio, q'μ == R)\n",
    "    optimize!(portfolio)\n",
    "    q_star = value.(q)\n",
    "    \n",
    "    return q_star'Σ*q_star, q_star'μ\n",
    "end\n",
    "\n",
    "# Extract the optimal values\n",
    "rs = collect(range(-1, 1.0, length=100))\n",
    "ys = map(opt_portfolio, rs);\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "id": "83e44975",
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip880\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip880)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip881\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip880)\" d=\"\n",
       "M193.936 1486.45 L2352.76 1486.45 L2352.76 47.2441 L193.936 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip882\">\n",
       "    <rect x=\"193\" y=\"47\" width=\"2160\" height=\"1440\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip882)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  465.01,1486.45 465.01,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip882)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1025.24,1486.45 1025.24,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip882)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1585.46,1486.45 1585.46,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip882)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2145.69,1486.45 2145.69,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip880)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  193.936,1486.45 2352.76,1486.45 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip880)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  465.01,1486.45 465.01,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip880)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1025.24,1486.45 1025.24,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip880)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1585.46,1486.45 1585.46,1467.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip880)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2145.69,1486.45 2145.69,1467.55 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip880)\" d=\"M427.812 1517.37 Q424.201 1517.37 422.372 1520.93 Q420.566 1524.47 420.566 1531.6 Q420.566 1538.71 422.372 1542.27 Q424.201 1545.82 427.812 1545.82 Q431.446 1545.82 433.251 1542.27 Q435.08 1538.71 435.08 1531.6 Q435.08 1524.47 433.251 1520.93 Q431.446 1517.37 427.812 1517.37 M427.812 1513.66 Q433.622 1513.66 436.677 1518.27 Q439.756 1522.85 439.756 1531.6 Q439.756 1540.33 436.677 1544.94 Q433.622 1549.52 427.812 1549.52 Q422.001 1549.52 418.923 1544.94 Q415.867 1540.33 415.867 1531.6 Q415.867 1522.85 418.923 1518.27 Q422.001 1513.66 427.812 1513.66 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M447.974 1542.97 L452.858 1542.97 L452.858 1548.85 L447.974 1548.85 L447.974 1542.97 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M473.043 1517.37 Q469.432 1517.37 467.603 1520.93 Q465.798 1524.47 465.798 1531.6 Q465.798 1538.71 467.603 1542.27 Q469.432 1545.82 473.043 1545.82 Q476.677 1545.82 478.483 1542.27 Q480.311 1538.71 480.311 1531.6 Q480.311 1524.47 478.483 1520.93 Q476.677 1517.37 473.043 1517.37 M473.043 1513.66 Q478.853 1513.66 481.909 1518.27 Q484.987 1522.85 484.987 1531.6 Q484.987 1540.33 481.909 1544.94 Q478.853 1549.52 473.043 1549.52 Q467.233 1549.52 464.154 1544.94 Q461.098 1540.33 461.098 1531.6 Q461.098 1522.85 464.154 1518.27 Q467.233 1513.66 473.043 1513.66 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M493.251 1514.29 L511.607 1514.29 L511.607 1518.22 L497.533 1518.22 L497.533 1526.7 Q498.552 1526.35 499.57 1526.19 Q500.589 1526 501.608 1526 Q507.395 1526 510.774 1529.17 Q514.154 1532.34 514.154 1537.76 Q514.154 1543.34 510.682 1546.44 Q507.209 1549.52 500.89 1549.52 Q498.714 1549.52 496.445 1549.15 Q494.2 1548.78 491.793 1548.04 L491.793 1543.34 Q493.876 1544.47 496.098 1545.03 Q498.32 1545.58 500.797 1545.58 Q504.802 1545.58 507.14 1543.48 Q509.478 1541.37 509.478 1537.76 Q509.478 1534.15 507.14 1532.04 Q504.802 1529.94 500.797 1529.94 Q498.922 1529.94 497.047 1530.35 Q495.196 1530.77 493.251 1531.65 L493.251 1514.29 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M987.541 1517.37 Q983.93 1517.37 982.101 1520.93 Q980.296 1524.47 980.296 1531.6 Q980.296 1538.71 982.101 1542.27 Q983.93 1545.82 987.541 1545.82 Q991.175 1545.82 992.981 1542.27 Q994.81 1538.71 994.81 1531.6 Q994.81 1524.47 992.981 1520.93 Q991.175 1517.37 987.541 1517.37 M987.541 1513.66 Q993.351 1513.66 996.407 1518.27 Q999.485 1522.85 999.485 1531.6 Q999.485 1540.33 996.407 1544.94 Q993.351 1549.52 987.541 1549.52 Q981.731 1549.52 978.652 1544.94 Q975.597 1540.33 975.597 1531.6 Q975.597 1522.85 978.652 1518.27 Q981.731 1513.66 987.541 1513.66 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M1007.7 1542.97 L1012.59 1542.97 L1012.59 1548.85 L1007.7 1548.85 L1007.7 1542.97 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M1023.58 1544.91 L1031.22 1544.91 L1031.22 1518.55 L1022.91 1520.21 L1022.91 1515.95 L1031.18 1514.29 L1035.85 1514.29 L1035.85 1544.91 L1043.49 1544.91 L1043.49 1548.85 L1023.58 1548.85 L1023.58 1544.91 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M1062.93 1517.37 Q1059.32 1517.37 1057.49 1520.93 Q1055.69 1524.47 1055.69 1531.6 Q1055.69 1538.71 1057.49 1542.27 Q1059.32 1545.82 1062.93 1545.82 Q1066.57 1545.82 1068.37 1542.27 Q1070.2 1538.71 1070.2 1531.6 Q1070.2 1524.47 1068.37 1520.93 Q1066.57 1517.37 1062.93 1517.37 M1062.93 1513.66 Q1068.74 1513.66 1071.8 1518.27 Q1074.88 1522.85 1074.88 1531.6 Q1074.88 1540.33 1071.8 1544.94 Q1068.74 1549.52 1062.93 1549.52 Q1057.12 1549.52 1054.05 1544.94 Q1050.99 1540.33 1050.99 1531.6 Q1050.99 1522.85 1054.05 1518.27 Q1057.12 1513.66 1062.93 1513.66 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M1548.27 1517.37 Q1544.65 1517.37 1542.83 1520.93 Q1541.02 1524.47 1541.02 1531.6 Q1541.02 1538.71 1542.83 1542.27 Q1544.65 1545.82 1548.27 1545.82 Q1551.9 1545.82 1553.71 1542.27 Q1555.53 1538.71 1555.53 1531.6 Q1555.53 1524.47 1553.71 1520.93 Q1551.9 1517.37 1548.27 1517.37 M1548.27 1513.66 Q1554.08 1513.66 1557.13 1518.27 Q1560.21 1522.85 1560.21 1531.6 Q1560.21 1540.33 1557.13 1544.94 Q1554.08 1549.52 1548.27 1549.52 Q1542.46 1549.52 1539.38 1544.94 Q1536.32 1540.33 1536.32 1531.6 Q1536.32 1522.85 1539.38 1518.27 Q1542.46 1513.66 1548.27 1513.66 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M1568.43 1542.97 L1573.31 1542.97 L1573.31 1548.85 L1568.43 1548.85 L1568.43 1542.97 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M1584.31 1544.91 L1591.95 1544.91 L1591.95 1518.55 L1583.64 1520.21 L1583.64 1515.95 L1591.9 1514.29 L1596.58 1514.29 L1596.58 1544.91 L1604.21 1544.91 L1604.21 1548.85 L1584.31 1548.85 L1584.31 1544.91 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M1613.71 1514.29 L1632.06 1514.29 L1632.06 1518.22 L1617.99 1518.22 L1617.99 1526.7 Q1619.01 1526.35 1620.02 1526.19 Q1621.04 1526 1622.06 1526 Q1627.85 1526 1631.23 1529.17 Q1634.61 1532.34 1634.61 1537.76 Q1634.61 1543.34 1631.14 1546.44 Q1627.66 1549.52 1621.34 1549.52 Q1619.17 1549.52 1616.9 1549.15 Q1614.65 1548.78 1612.25 1548.04 L1612.25 1543.34 Q1614.33 1544.47 1616.55 1545.03 Q1618.77 1545.58 1621.25 1545.58 Q1625.26 1545.58 1627.59 1543.48 Q1629.93 1541.37 1629.93 1537.76 Q1629.93 1534.15 1627.59 1532.04 Q1625.26 1529.94 1621.25 1529.94 Q1619.38 1529.94 1617.5 1530.35 Q1615.65 1530.77 1613.71 1531.65 L1613.71 1514.29 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M2108 1517.37 Q2104.38 1517.37 2102.56 1520.93 Q2100.75 1524.47 2100.75 1531.6 Q2100.75 1538.71 2102.56 1542.27 Q2104.38 1545.82 2108 1545.82 Q2111.63 1545.82 2113.44 1542.27 Q2115.26 1538.71 2115.26 1531.6 Q2115.26 1524.47 2113.44 1520.93 Q2111.63 1517.37 2108 1517.37 M2108 1513.66 Q2113.81 1513.66 2116.86 1518.27 Q2119.94 1522.85 2119.94 1531.6 Q2119.94 1540.33 2116.86 1544.94 Q2113.81 1549.52 2108 1549.52 Q2102.19 1549.52 2099.11 1544.94 Q2096.05 1540.33 2096.05 1531.6 Q2096.05 1522.85 2099.11 1518.27 Q2102.19 1513.66 2108 1513.66 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M2128.16 1542.97 L2133.04 1542.97 L2133.04 1548.85 L2128.16 1548.85 L2128.16 1542.97 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M2147.25 1544.91 L2163.57 1544.91 L2163.57 1548.85 L2141.63 1548.85 L2141.63 1544.91 Q2144.29 1542.16 2148.87 1537.53 Q2153.48 1532.88 2154.66 1531.53 Q2156.91 1529.01 2157.79 1527.27 Q2158.69 1525.51 2158.69 1523.82 Q2158.69 1521.07 2156.75 1519.33 Q2154.82 1517.6 2151.72 1517.6 Q2149.52 1517.6 2147.07 1518.36 Q2144.64 1519.13 2141.86 1520.68 L2141.86 1515.95 Q2144.68 1514.82 2147.14 1514.24 Q2149.59 1513.66 2151.63 1513.66 Q2157 1513.66 2160.19 1516.35 Q2163.39 1519.03 2163.39 1523.52 Q2163.39 1525.65 2162.58 1527.57 Q2161.79 1529.47 2159.68 1532.07 Q2159.11 1532.74 2156 1535.95 Q2152.9 1539.15 2147.25 1544.91 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M2183.39 1517.37 Q2179.78 1517.37 2177.95 1520.93 Q2176.14 1524.47 2176.14 1531.6 Q2176.14 1538.71 2177.95 1542.27 Q2179.78 1545.82 2183.39 1545.82 Q2187.02 1545.82 2188.83 1542.27 Q2190.66 1538.71 2190.66 1531.6 Q2190.66 1524.47 2188.83 1520.93 Q2187.02 1517.37 2183.39 1517.37 M2183.39 1513.66 Q2189.2 1513.66 2192.25 1518.27 Q2195.33 1522.85 2195.33 1531.6 Q2195.33 1540.33 2192.25 1544.94 Q2189.2 1549.52 2183.39 1549.52 Q2177.58 1549.52 2174.5 1544.94 Q2171.44 1540.33 2171.44 1531.6 Q2171.44 1522.85 2174.5 1518.27 Q2177.58 1513.66 2183.39 1513.66 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip882)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  193.936,1445.72 2352.76,1445.72 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip882)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  193.936,1106.28 2352.76,1106.28 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip882)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  193.936,766.846 2352.76,766.846 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip882)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  193.936,427.411 2352.76,427.411 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip882)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  193.936,87.9763 2352.76,87.9763 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip880)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  193.936,1486.45 193.936,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip880)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  193.936,1445.72 212.834,1445.72 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip880)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  193.936,1106.28 212.834,1106.28 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip880)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  193.936,766.846 212.834,766.846 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip880)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  193.936,427.411 212.834,427.411 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip880)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  193.936,87.9763 212.834,87.9763 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip880)\" d=\"M50.9921 1446.17 L80.6679 1446.17 L80.6679 1450.1 L50.9921 1450.1 L50.9921 1446.17 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M91.5706 1459.06 L99.2095 1459.06 L99.2095 1432.69 L90.8993 1434.36 L90.8993 1430.1 L99.1632 1428.44 L103.839 1428.44 L103.839 1459.06 L111.478 1459.06 L111.478 1463 L91.5706 1463 L91.5706 1459.06 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M120.922 1457.12 L125.807 1457.12 L125.807 1463 L120.922 1463 L120.922 1457.12 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M145.992 1431.51 Q142.381 1431.51 140.552 1435.08 Q138.746 1438.62 138.746 1445.75 Q138.746 1452.86 140.552 1456.42 Q142.381 1459.96 145.992 1459.96 Q149.626 1459.96 151.431 1456.42 Q153.26 1452.86 153.26 1445.75 Q153.26 1438.62 151.431 1435.08 Q149.626 1431.51 145.992 1431.51 M145.992 1427.81 Q151.802 1427.81 154.857 1432.42 Q157.936 1437 157.936 1445.75 Q157.936 1454.48 154.857 1459.08 Q151.802 1463.67 145.992 1463.67 Q140.181 1463.67 137.103 1459.08 Q134.047 1454.48 134.047 1445.75 Q134.047 1437 137.103 1432.42 Q140.181 1427.81 145.992 1427.81 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M51.9875 1106.73 L81.6633 1106.73 L81.6633 1110.67 L51.9875 1110.67 L51.9875 1106.73 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M101.756 1092.08 Q98.1447 1092.08 96.316 1095.64 Q94.5104 1099.19 94.5104 1106.32 Q94.5104 1113.42 96.316 1116.99 Q98.1447 1120.53 101.756 1120.53 Q105.39 1120.53 107.196 1116.99 Q109.024 1113.42 109.024 1106.32 Q109.024 1099.19 107.196 1095.64 Q105.39 1092.08 101.756 1092.08 M101.756 1088.38 Q107.566 1088.38 110.621 1092.98 Q113.7 1097.57 113.7 1106.32 Q113.7 1115.04 110.621 1119.65 Q107.566 1124.23 101.756 1124.23 Q95.9456 1124.23 92.8669 1119.65 Q89.8114 1115.04 89.8114 1106.32 Q89.8114 1097.57 92.8669 1092.98 Q95.9456 1088.38 101.756 1088.38 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M121.918 1117.68 L126.802 1117.68 L126.802 1123.56 L121.918 1123.56 L121.918 1117.68 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M137.033 1089 L155.39 1089 L155.39 1092.94 L141.316 1092.94 L141.316 1101.41 Q142.334 1101.06 143.353 1100.9 Q144.371 1100.71 145.39 1100.71 Q151.177 1100.71 154.556 1103.89 Q157.936 1107.06 157.936 1112.47 Q157.936 1118.05 154.464 1121.15 Q150.992 1124.23 144.672 1124.23 Q142.496 1124.23 140.228 1123.86 Q137.982 1123.49 135.575 1122.75 L135.575 1118.05 Q137.658 1119.19 139.881 1119.74 Q142.103 1120.3 144.58 1120.3 Q148.584 1120.3 150.922 1118.19 Q153.26 1116.08 153.26 1112.47 Q153.26 1108.86 150.922 1106.76 Q148.584 1104.65 144.58 1104.65 Q142.705 1104.65 140.83 1105.07 Q138.978 1105.48 137.033 1106.36 L137.033 1089 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M100.76 752.645 Q97.1493 752.645 95.3206 756.209 Q93.515 759.751 93.515 766.881 Q93.515 773.987 95.3206 777.552 Q97.1493 781.094 100.76 781.094 Q104.395 781.094 106.2 777.552 Q108.029 773.987 108.029 766.881 Q108.029 759.751 106.2 756.209 Q104.395 752.645 100.76 752.645 M100.76 748.941 Q106.571 748.941 109.626 753.547 Q112.705 758.131 112.705 766.881 Q112.705 775.608 109.626 780.214 Q106.571 784.797 100.76 784.797 Q94.9502 784.797 91.8715 780.214 Q88.816 775.608 88.816 766.881 Q88.816 758.131 91.8715 753.547 Q94.9502 748.941 100.76 748.941 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M120.922 778.246 L125.807 778.246 L125.807 784.126 L120.922 784.126 L120.922 778.246 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M145.992 752.645 Q142.381 752.645 140.552 756.209 Q138.746 759.751 138.746 766.881 Q138.746 773.987 140.552 777.552 Q142.381 781.094 145.992 781.094 Q149.626 781.094 151.431 777.552 Q153.26 773.987 153.26 766.881 Q153.26 759.751 151.431 756.209 Q149.626 752.645 145.992 752.645 M145.992 748.941 Q151.802 748.941 154.857 753.547 Q157.936 758.131 157.936 766.881 Q157.936 775.608 154.857 780.214 Q151.802 784.797 145.992 784.797 Q140.181 784.797 137.103 780.214 Q134.047 775.608 134.047 766.881 Q134.047 758.131 137.103 753.547 Q140.181 748.941 145.992 748.941 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M101.756 413.21 Q98.1447 413.21 96.316 416.775 Q94.5104 420.316 94.5104 427.446 Q94.5104 434.552 96.316 438.117 Q98.1447 441.659 101.756 441.659 Q105.39 441.659 107.196 438.117 Q109.024 434.552 109.024 427.446 Q109.024 420.316 107.196 416.775 Q105.39 413.21 101.756 413.21 M101.756 409.506 Q107.566 409.506 110.621 414.113 Q113.7 418.696 113.7 427.446 Q113.7 436.173 110.621 440.779 Q107.566 445.362 101.756 445.362 Q95.9456 445.362 92.8669 440.779 Q89.8114 436.173 89.8114 427.446 Q89.8114 418.696 92.8669 414.113 Q95.9456 409.506 101.756 409.506 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M121.918 438.812 L126.802 438.812 L126.802 444.691 L121.918 444.691 L121.918 438.812 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M137.033 410.131 L155.39 410.131 L155.39 414.066 L141.316 414.066 L141.316 422.538 Q142.334 422.191 143.353 422.029 Q144.371 421.844 145.39 421.844 Q151.177 421.844 154.556 425.015 Q157.936 428.187 157.936 433.603 Q157.936 439.182 154.464 442.284 Q150.992 445.362 144.672 445.362 Q142.496 445.362 140.228 444.992 Q137.982 444.622 135.575 443.881 L135.575 439.182 Q137.658 440.316 139.881 440.872 Q142.103 441.427 144.58 441.427 Q148.584 441.427 150.922 439.321 Q153.26 437.214 153.26 433.603 Q153.26 429.992 150.922 427.886 Q148.584 425.779 144.58 425.779 Q142.705 425.779 140.83 426.196 Q138.978 426.613 137.033 427.492 L137.033 410.131 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M91.5706 101.321 L99.2095 101.321 L99.2095 74.9555 L90.8993 76.6222 L90.8993 72.3629 L99.1632 70.6963 L103.839 70.6963 L103.839 101.321 L111.478 101.321 L111.478 105.256 L91.5706 105.256 L91.5706 101.321 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M120.922 99.3767 L125.807 99.3767 L125.807 105.256 L120.922 105.256 L120.922 99.3767 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M145.992 73.775 Q142.381 73.775 140.552 77.3398 Q138.746 80.8814 138.746 88.011 Q138.746 95.1174 140.552 98.6822 Q142.381 102.224 145.992 102.224 Q149.626 102.224 151.431 98.6822 Q153.26 95.1174 153.26 88.011 Q153.26 80.8814 151.431 77.3398 Q149.626 73.775 145.992 73.775 M145.992 70.0713 Q151.802 70.0713 154.857 74.6777 Q157.936 79.261 157.936 88.011 Q157.936 96.7378 154.857 101.344 Q151.802 105.928 145.992 105.928 Q140.181 105.928 137.103 101.344 Q134.047 96.7378 134.047 88.011 Q134.047 79.261 137.103 74.6777 Q140.181 70.0713 145.992 70.0713 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip882)\" style=\"stroke:#009af9; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2291.66,1445.72 2202.16,1432 2116.68,1418.29 2035.24,1404.57 1957.82,1390.86 1884.43,1377.14 1815.05,1363.43 1748.11,1349.71 1682.64,1336 1618.64,1322.28 \n",
       "  1556.11,1308.57 1495.04,1294.86 1435.44,1281.14 1377.31,1267.43 1320.64,1253.71 1265.45,1240 1211.72,1226.28 1159.46,1212.57 1108.67,1198.85 1059.34,1185.14 \n",
       "  1011.48,1171.42 965.095,1157.71 920.173,1144 876.72,1130.28 834.733,1116.57 794.215,1102.85 755.165,1089.14 717.582,1075.42 681.467,1061.71 646.819,1047.99 \n",
       "  613.64,1034.28 581.928,1020.56 551.684,1006.85 522.908,993.136 495.599,979.421 469.759,965.707 445.386,951.992 422.481,938.278 401.043,924.563 381.074,910.849 \n",
       "  362.572,897.134 345.538,883.42 329.971,869.705 315.873,855.99 303.242,842.276 292.079,828.561 282.383,814.847 274.156,801.132 267.396,787.418 262.104,773.703 \n",
       "  258.28,759.989 255.923,746.274 255.035,732.56 255.614,718.845 257.661,705.131 261.175,691.416 266.157,677.701 272.608,663.987 280.525,650.272 289.911,636.558 \n",
       "  300.764,622.843 313.086,609.129 326.874,595.414 342.131,581.7 358.856,567.985 377.048,554.271 396.708,540.556 417.835,526.842 440.431,513.127 464.494,499.412 \n",
       "  490.025,485.698 517.024,471.983 545.491,458.269 575.425,444.554 606.827,430.84 639.697,417.125 674.034,403.411 709.84,389.696 747.113,375.982 785.854,362.267 \n",
       "  826.062,348.553 867.739,334.838 910.883,321.123 955.495,307.409 1001.57,293.694 1049.12,279.98 1098.14,266.265 1148.62,252.551 1200.57,238.836 1253.99,225.122 \n",
       "  1308.88,211.407 1365.23,197.693 1423.05,183.978 1482.34,170.264 1543.48,156.549 1613.38,142.834 1694.61,129.12 1787.17,115.405 1891.06,101.691 2006.29,87.9763 \n",
       "  \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip880)\" d=\"\n",
       "M1988.5 198.898 L2280.8 198.898 L2280.8 95.2176 L1988.5 95.2176  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip880)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1988.5,198.898 2280.8,198.898 2280.8,95.2176 1988.5,95.2176 1988.5,198.898 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip880)\" style=\"stroke:#009af9; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2012.48,147.058 2156.4,147.058 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip880)\" d=\"M2194.23 166.745 Q2192.43 171.375 2190.72 172.787 Q2189 174.199 2186.13 174.199 L2182.73 174.199 L2182.73 170.634 L2185.23 170.634 Q2186.99 170.634 2187.96 169.8 Q2188.93 168.967 2190.11 165.865 L2190.88 163.921 L2180.39 138.412 L2184.91 138.412 L2193.01 158.689 L2201.11 138.412 L2205.62 138.412 L2194.23 166.745 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip880)\" d=\"M2212.91 160.402 L2220.55 160.402 L2220.55 134.037 L2212.24 135.703 L2212.24 131.444 L2220.51 129.778 L2225.18 129.778 L2225.18 160.402 L2232.82 160.402 L2232.82 164.338 L2212.91 164.338 L2212.91 160.402 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /></svg>\n"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Plots\n",
    "plot(ys)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10fac124",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Any closing questions?"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Julia 1.6.5",
   "language": "julia",
   "name": "julia-1.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}