14 Model_persistance_using_SparkML.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMSkillsNetworkBD0231ENCoursera2789-2023-01-01\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\">\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model persistence using SparkML\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **30** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style='color: red'>The purpose of this lab is to show you how to use SparkML to persist a model and to load the persisted model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## __Table of Contents__\n",
    "\n",
    "<ol>\n",
    "  <li>\n",
    "    <a href=\"#Objectives\">Objectives\n",
    "    </a>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Datasets\">Datasets\n",
    "    </a>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Setup\">Setup\n",
    "    </a>\n",
    "    <ol>\n",
    "      <li>\n",
    "        <a href=\"#Installing-Required-Libraries\">Installing Required Libraries\n",
    "        </a>\n",
    "      </li>\n",
    "      <li>\n",
    "        <a href=\"#Importing-Required-Libraries\">Importing Required Libraries\n",
    "        </a>\n",
    "      </li>\n",
    "    </ol>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Examples\">Examples\n",
    "    </a>\n",
    "    <ol>\n",
    "    <li>\n",
    "      <a href=\"#Task-1---Create-a-model\">Task 1 - Create a model\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-2---Save-the-model\">Task 2 - Save the model\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-3---Load-the-model\">Task 3 - Load the model\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Task-4---Predict-using-the-loaded-model\">Task 4 - Predict using the loaded model\n",
    "      </a>\n",
    "    </li>\n",
    "    </ol>\n",
    "  </li>\n",
    "  <li>\n",
    "    <a href=\"#Exercises\">Exercises\n",
    "    </a>\n",
    "  </li>\n",
    "  <ol>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-1---Create-a-model\">Exercise 1 - Create a model\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-2---Save-the-model\">Exercise 2 - Save the model\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-3---Load-the-model\">Exercise 3 - Load the model\n",
    "      </a>\n",
    "    </li>\n",
    "    <li>\n",
    "      <a href=\"#Exercise-4---Predict-using-the-loaded-model\">Exercise 4 - Predict using the loaded model\n",
    "      </a>\n",
    "    </li>\n",
    "  </ol>\n",
    "</ol>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n",
    "\n",
    "After completing this lab you will be able to:\n",
    "\n",
    " - Save a trained model.\n",
    " - Load a saved model.\n",
    " - Make predictions using the loaded model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasets\n",
    "\n",
    "In this lab you will be using dataset(s):\n",
    "\n",
    " - Modified version of car mileage dataset. Original dataset available at https://archive.ics.uci.edu/ml/datasets/auto+mpg \n",
    " - Modified version of diamonds dataset. Original dataset available at https://www.openml.org/search?type=data&sort=runs&id=42225&status=active\n",
    " \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this lab, we will be using the following libraries:\n",
    "\n",
    "*   [`PySpark`](https://spark.apache.org/docs/latest/api/python/index.html?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMSkillsNetworkBD0231ENCoursera2789-2023-01-01) for connecting to the Spark Cluster\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing Required Libraries\n",
    "\n",
    "Spark Cluster is pre-installed in the Skills Network Labs environment. However, you need libraries like pyspark and findspark to connect to this cluster.\n",
    "\n",
    "If you wish to download this jupyter notebook and run on your local computer, follow the instructions mentioned <a href=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-BD0231EN-Coursera/labs/Connecting_to_spark_cluster_using_Skills_Network_labs.ipynb\">here.</a>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following required libraries are __not__ pre-installed in the Skills Network Labs environment. __You will need to run the following cell__ to install them:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pyspark==3.1.2 -q\n",
    "!pip install findspark -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Required Libraries\n",
    "\n",
    "_We recommend you import all required libraries in one place (here):_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# You can also use this section to suppress warnings generated by your code:\n",
    "def warn(*args, **kwargs):\n",
    "    pass\n",
    "import warnings\n",
    "warnings.warn = warn\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# FindSpark simplifies the process of using Apache Spark with Python\n",
    "\n",
    "import findspark\n",
    "findspark.init()\n",
    "\n",
    "#import functions/Classes for sparkml\n",
    "\n",
    "from pyspark.ml.regression import LinearRegression\n",
    "from pyspark.ml.feature import VectorAssembler\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.ml import Pipeline\n",
    "\n",
    "# import functions/Classes for metrics\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examples\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1 - Create a model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/01/08 11:45:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n"
     ]
    }
   ],
   "source": [
    "#Create SparkSession\n",
    "#Ignore any warnings by SparkSession command\n",
    "\n",
    "spark = SparkSession.builder.appName(\"Model Persistence\").getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the data file\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-BD0231EN-SkillsNetwork/datasets/mpg.csv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the dataset into the spark dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "# using the spark.read.csv function we load the data into a dataframe.\n",
    "# the header = True mentions that there is a header row in out csv file\n",
    "# the inferSchema = True, tells spark to automatically find out the data types of the columns.\n",
    "\n",
    "# Load mpg dataset\n",
    "mpg_data = spark.read.csv(\"mpg.csv\", header=True, inferSchema=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the schema of the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- MPG: double (nullable = true)\n",
      " |-- Cylinders: integer (nullable = true)\n",
      " |-- Engine Disp: double (nullable = true)\n",
      " |-- Horsepower: integer (nullable = true)\n",
      " |-- Weight: integer (nullable = true)\n",
      " |-- Accelerate: double (nullable = true)\n",
      " |-- Year: integer (nullable = true)\n",
      " |-- Origin: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "mpg_data.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show top 5 rows from the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+---------+-----------+----------+------+----------+----+--------+\n",
      "| MPG|Cylinders|Engine Disp|Horsepower|Weight|Accelerate|Year|  Origin|\n",
      "+----+---------+-----------+----------+------+----------+----+--------+\n",
      "|15.0|        8|      390.0|       190|  3850|       8.5|  70|American|\n",
      "|21.0|        6|      199.0|        90|  2648|      15.0|  70|American|\n",
      "|18.0|        6|      199.0|        97|  2774|      15.5|  70|American|\n",
      "|16.0|        8|      304.0|       150|  3433|      12.0|  70|American|\n",
      "|14.0|        8|      455.0|       225|  3086|      10.0|  70|American|\n",
      "+----+---------+-----------+----------+------+----------+----+--------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "mpg_data.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We ask the VectorAssembler to group a bunch of inputCols as single column named \"features\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Prepare feature vector\n",
    "assembler = VectorAssembler(inputCols=[\"Cylinders\", \"Engine Disp\", \"Horsepower\", \"Weight\", \"Accelerate\", \"Year\"], outputCol=\"features\")\n",
    "mpg_transformed_data = assembler.transform(mpg_data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the assembled \"features\" and the label column \"MPG\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+----+\n",
      "|            features| MPG|\n",
      "+--------------------+----+\n",
      "|[8.0,390.0,190.0,...|15.0|\n",
      "|[6.0,199.0,90.0,2...|21.0|\n",
      "|[6.0,199.0,97.0,2...|18.0|\n",
      "|[8.0,304.0,150.0,...|16.0|\n",
      "|[8.0,455.0,225.0,...|14.0|\n",
      "|[8.0,350.0,165.0,...|15.0|\n",
      "|[8.0,307.0,130.0,...|18.0|\n",
      "|[8.0,454.0,220.0,...|14.0|\n",
      "|[8.0,400.0,150.0,...|15.0|\n",
      "|[8.0,307.0,200.0,...|10.0|\n",
      "|[8.0,383.0,170.0,...|15.0|\n",
      "|[8.0,318.0,210.0,...|11.0|\n",
      "|[8.0,360.0,215.0,...|10.0|\n",
      "|[8.0,429.0,198.0,...|15.0|\n",
      "|[6.0,200.0,85.0,2...|21.0|\n",
      "|[8.0,302.0,140.0,...|17.0|\n",
      "|[8.0,304.0,193.0,...| 9.0|\n",
      "|[8.0,340.0,160.0,...|14.0|\n",
      "|[6.0,198.0,95.0,2...|22.0|\n",
      "|[8.0,440.0,215.0,...|14.0|\n",
      "+--------------------+----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "mpg_transformed_data.select(\"features\",\"MPG\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split the data set in the ratio of 70:30. 70% training data, 30% testing data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Split data into training and testing sets\n",
    "(training_data, testing_data) = mpg_transformed_data.randomSplit([0.7, 0.3])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a LR model and train the model using the pipeline on training data set\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/01/08 11:47:48 WARN util.Instrumentation: [81255ef0] regParam is zero, which might cause numerical instability and overfitting.\n",
      "24/01/08 11:47:48 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS\n",
      "24/01/08 11:47:48 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS\n",
      "24/01/08 11:47:48 WARN netlib.LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK\n",
      "24/01/08 11:47:48 WARN netlib.LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK\n"
     ]
    }
   ],
   "source": [
    "# Train linear regression model\n",
    "# Ignore any warnings\n",
    "lr = LinearRegression(labelCol=\"MPG\", featuresCol=\"features\")\n",
    "pipeline = Pipeline(stages=[lr])\n",
    "model = pipeline.fit(training_data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2 - Save the model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a folder where the model will to be saved\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!mkdir model_storage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "# Persist the model to the path \"./model_stoarage/\"\n",
    "\n",
    "model.write().overwrite().save(\"./model_storage/\")\n",
    "\n",
    "#The overwrite method is used to overwrite the model if it already exists,\n",
    "#and the save method is used to specify the path where the model should be saved.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3 - Load the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.pipeline import PipelineModel\n",
    "\n",
    "# Load persisted model\n",
    "loaded_model = PipelineModel.load(\"./model_storage/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 4 - Predict using the loaded model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make predictions on test data\n",
    "predictions = loaded_model.transform(testing_data)\n",
    "#In the above example, we use the load method of the PipelineModel object to load the persisted model from disk. We can then use this loaded model to make predictions on new data using the transform method.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your model is now trained. We use the testing data to make predictions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make predictions on testing data\n",
    "predictions = model.transform(testing_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions.select(\"prediction\").show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stop Spark Session\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1 - Create a model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a spark session with appname \"Model Persistence Exercise\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark = #TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Use the SparkSession.builder\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "spark = SparkSession.builder.appName(\"Model Persistence Exercise\").getOrCreate()\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the data set\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-BD0231EN-SkillsNetwork/datasets/diamonds.csv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the dataset into a spark dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "diamond_data = #TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Use the spark.read.csv function\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "diamond_data = spark.read.csv(\"diamonds.csv\", header=True, inferSchema=True)\n",
    "```\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display sample data from dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "diamond_data.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assemble the columns columns carat,depth and table into a single column named \"features\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assembler = #TODO\n",
    "diamond_transformed_data = #TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "Refer to Task1\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "assembler = VectorAssembler(inputCols=[\"carat\", \"depth\", \"table\"], outputCol=\"features\")\n",
    "diamond_transformed_data = assembler.transform(diamond_data)\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the vectorized features and label columns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "diamond_transformed_data.select(\"features\",\"price\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the dataset into training and testing sets in the ratio of 70:30.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(training_data, testing_data) =  #TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "use the randomSplit method\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```\n",
    "(training_data, testing_data) = diamond_transformed_data.randomSplit([0.7, 0.3])\n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a LR model and train the model using the pipeline on training data set\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train linear regression model\n",
    "# Ignore any warnings\n",
    "lr =  #TODO\n",
    "pipeline =  #TODO\n",
    "model =  #TODO\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "use the Pipeline with an lr stage\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```\n",
    "lr = LinearRegression(labelCol=\"price\", featuresCol=\"features\")\n",
    "pipeline = Pipeline(stages=[lr])\n",
    "model = pipeline.fit(training_data)\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2 - Save the model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a folder \"diamond_model\". This is where the model will to be saved\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir diamond_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Persist the model to the folder \"diamond_model\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "use the write method of the model\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```\n",
    "model.write().overwrite().save(\"./diamond_model/\")\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3 - Load the model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the model from the folder \"diamond_model\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.pipeline import PipelineModel\n",
    "\n",
    "# Load persisted model\n",
    "loaded_model = #TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "use the load method of the PipelineModel\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```\n",
    "loaded_model = PipelineModel.load(\"./diamond_model/\")\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4 - Predict using the loaded model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make predictions on test data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "predictions = #TODO\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for a Hint</summary>\n",
    "    \n",
    "use the transform method of the loaded model\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```\n",
    "predictions = loaded_model.transform(testing_data)\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions.select(\"prediction\").show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stop Spark Session\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations you have completed this lab.<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Ramesh Sannareddy](https://www.linkedin.com/in/rsannareddy/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMBD0231ENSkillsNetwork866-2023-01-01)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|Date (YYYY-MM-DD)|Version|Changed By|Change Description|\n",
    "|-|-|-|-|\n",
    "|2023-05-04|0.1|Ramesh Sannareddy|Initial Version Created|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2023 IBM Corporation. All rights reserved.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}