aws/rapids_sagemaker_hpo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f197f7bb",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:22pt\"> **Demo Overview** </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b44cb3da",
   "metadata": {},
   "source": [
    "Automated Model Tuning (AMT) also known as Hyper-Parameter Optimization  (HPO) helps to find the best version of a model by exploring the space of possible configurations. While generally desirable, this search is computationally expensive and can feel prohibitive. \n",
    "\n",
    "\n",
    "In the notebook demo below, we show how SageMaker and RAPIDS working together can tackle model tuning by accelerating compute parallelism within a node's GPUs; and simultaneously accelerating the search by leveraging sets of cloud nodes running parallel experiments. \n",
    "\n",
    "For example, we find a **12x** speedup in wall clock time (6 hours vs 3+ days) and a **4.5x** reduction in cost when comparing between GPU and CPU EC2 Spot instances on 100 XGBoost AMT/HPO trials using 10 parallel workers on 10 years of the Airline Dataset.\n",
    "\n",
    "For more check out our [AWS blog](https://aws.amazon.com/blogs/machine-learning/rapids-and-amazon-sagemaker-scale-up-and-scale-out-to-tackle-ml-challenges/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77a8d2e2",
   "metadata": {},
   "source": [
    "<img src='img/v2_overview.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0a11713",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:22pt\"> **Preamble** </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dde3fac0",
   "metadata": {},
   "source": [
    "To get things rolling let's make sure we can query our AWS SageMaker execution role and session as well as our account ID and AWS region."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00fa4872-c745-4c8c-a665-2f0921d13c5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!docker images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cabb01b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import string\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa9b8b9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "execution_role = sagemaker.get_execution_role()\n",
    "session = sagemaker.Session()\n",
    "\n",
    "account=!(aws sts get-caller-identity --query Account --output text)\n",
    "region=!(aws configure get region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93c7f297",
   "metadata": {},
   "outputs": [],
   "source": [
    "account, region"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a67a7ebe",
   "metadata": {},
   "source": [
    "<span style=\"display: block; color:#8735fb; font-size:22pt\"> 1. RAPIDS Cloud ML Container </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f83f9ac9",
   "metadata": {},
   "source": [
    "Next let us download the latest RAPIDS container pre-integrated with the libraries needed for SageMaker integration (e.g., [sagemaker-training toolkit](/~https://github.com/aws/sagemaker-training-toolkit)).\n",
    "\n",
    "This container also contains the latest code from our [rapidsai/cloud-ml-examples](/~https://github.com/rapidsai/cloud-ml-examples) repository which will run inside the container. We'll go over the code details in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b293e8ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator_info = {\n",
    "    'rapids_container': 'rapidsai/rapidsai-cloud-ml:latest',\n",
    "    'ecr_image': 'sagemaker-rapids-cloud-ml:latest',\n",
    "    'ecr_repository': 'sagemaker-rapids-cloud-ml'\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3da5602c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "!docker pull {estimator_info['rapids_container']}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb531bd8",
   "metadata": {},
   "source": [
    "Once we pull the RAPIDS cloud container we will need to publish it onto the Amazon Elastic Container Registry (ECR) so that it can be used by SageMaker. The full name of the container on ECR will be as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac4a40e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "ECR_container_fullname = f\"{account[0]}.dkr.ecr.{region[0]}.amazonaws.com/{estimator_info['ecr_image']}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20e7c258-fbf3-4847-b87e-c869c8fbb31f",
   "metadata": {},
   "outputs": [],
   "source": [
    "ECR_container_fullname"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2586d3f1",
   "metadata": {},
   "source": [
    "Next we can tag this container with its full ECR name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a37bdea6",
   "metadata": {},
   "outputs": [],
   "source": [
    "!docker tag {estimator_info['rapids_container']} {ECR_container_fullname}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed093bea",
   "metadata": {},
   "outputs": [],
   "source": [
    "print( f\"source      : {estimator_info['rapids_container']}\\n\"\n",
    "       f\"destination : {ECR_container_fullname}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4928814a-09f2-4a8b-b24b-3afe839d8af2",
   "metadata": {},
   "outputs": [],
   "source": [
    "!docker images"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15590cef",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:18pt\"> 1.3 - Publish to Elastic Cloud Registry (ECR) </span>\n",
    "\n",
    "Now that we've built and tagged our container its time to push it to Amazon's container registry (ECR). Once in ECR, AWS SageMaker will be able to leverage our image to build Estimators and run experiments.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04dcab91",
   "metadata": {},
   "source": [
    "Docker Login to ECR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08839bd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "docker_login_str = !(aws ecr get-login --region {region[0]} --no-include-email)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bab9139d",
   "metadata": {},
   "outputs": [],
   "source": [
    "!{docker_login_str[0]}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18bf5afe",
   "metadata": {},
   "source": [
    "Create ECR repository [ if it doesn't already exist]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1f6f1f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "repository_query = !(aws ecr describe-repositories --repository-names {estimator_info['ecr_repository']})\n",
    "if repository_query[0] == '':\n",
    "    !(aws ecr create-repository --repository-name {estimator_info['ecr_repository']})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca7fa721",
   "metadata": {},
   "source": [
    "Let's now actually push the container to ECR\n",
    "> Note the first push to ECR may take some time (hopefully less than 10 minutes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48c77868",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "!docker push {ECR_container_fullname}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07734f8a",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:22pt\"> 2. - DataScience Workflow </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8fe9cee",
   "metadata": {},
   "source": [
    "The data source for this workflow is 3 years of the ([Airline On-Time Statistics](https://www.transtats.bts.gov/ONTIME/) dataset from the US Bureau of Transportation.\n",
    "\n",
    "The machine learning objective is to predict whether flights will be more than 15 minutes late arriving to their destination. \n",
    "\n",
    "The flow of logic can follow two paths:\n",
    "\n",
    "In **training mode**, the code loads the dataset, drops flights with missing values, splits the data, trains an XGBoost model, and evaluates its performance on the hold out values.\n",
    "\n",
    "In **serving or inference mode** there container runs a Flask server which listens for inputs, parses and predicts with the trained model loaded in cache.        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f465de8",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:20pt\"> 2.1 - Default Choices </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d599afe1",
   "metadata": {},
   "source": [
    "There are choices that have been pre-selected in this demo -- we list them below and provide a brief description for each.\n",
    "\n",
    "If you would like to learn more about these choices, and have the ability to change them please check out the [extended demo notebook](/~https://github.com/rapidsai/cloud-ml-examples/blob/main/aws/rapids_sagemaker_hpo_extended.ipynb).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f7e388b",
   "metadata": {},
   "source": [
    "Choice | Demo/Default | Other Option(s)\n",
    "-------|--------|---------\n",
    "Model  | XGBoost | RandomForest, KMeans\n",
    "Datasize  | 3 years | 1, 3, or 10 years\n",
    "Compute/Code | GPU | CPU, GPU/CPU + Dask"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9c82d41",
   "metadata": {},
   "source": [
    "> Note: We host the demo datasets in public S3 demo buckets in both the **us-east-1** (N. Virginia) or **us-west-2** (Oregon) regions (i.e., `sagemaker-rapids-hpo-us-east-1`, and `sagemaker-rapids-hpo-us-west-2`). You should run the SageMaker HPO workflow in either of these two regions if you wish to leverage the demo datasets since SageMaker requires that the S3 dataset and the compute you'll be renting are co-located. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e94c54b7",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:20pt\"> 2.2 - Active Choices </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe5cce37",
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_data_input = f\"s3://sagemaker-rapids-hpo-{region[0]}/1_year\"\n",
    "s3_model_output = f\"s3://{session.default_bucket()}/trained-models\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "074a8056",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:18pt\"> [ Search Ranges and Strategy ] </span>\n",
    "<a id='strategy-and-param-ranges'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23adcede",
   "metadata": {},
   "source": [
    "One of the most important choices when running HPO is to choose the bounds of the hyperparameter search process. Below we've set the ranges of the hyperparameters to allow for interesting variation, you are of course welcome to revise these ranges based on domain knowledge especially if you plan to plug in your own dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dc2bcae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# please choose HPO search ranges\n",
    "hyperparameter_ranges = {\n",
    "    'max_depth'    : sagemaker.parameter.IntegerParameter        ( 5, 15 ),\n",
    "    'num_boost_round' : sagemaker.parameter.IntegerParameter     ( 100, 500 ),\n",
    "    'max_features' : sagemaker.parameter.ContinuousParameter     ( 0.1, 1.0 ),    \n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8badcbcd",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:18pt\"> [ Experiment Scale ] </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "568ff04b",
   "metadata": {},
   "source": [
    "We also need to decide how may total experiments to run, and how many should run in parallel. Below we have a very conservative number of maximum jobs to run so that you don't accidently spawn large computations when starting out, however for meaningful HPO searches this number should be much higher (e.g., in our experiments we often run 100 max_jobs). Note that you may need to request a [quota limit increase](https://docs.aws.amazon.com/general/latest/gr/sagemaker.html) for additional  `max_parallel_jobs` parallel workers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13a9ea9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# please choose total number of HPO experiments[ we have set this number very low to allow for automated CI testing ]\n",
    "max_jobs = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cb20f33",
   "metadata": {},
   "outputs": [],
   "source": [
    "# please choose number of experiments that can run in parallel\n",
    "max_parallel_jobs = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "091417fc",
   "metadata": {},
   "source": [
    "Let's also set the max duration for an individual job to 24 hours so we don't have run-away compute jobs taking too long."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79c46ab7",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_duration_of_experiment_seconds = 60 * 60 * 24"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7c94002",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:18pt\"> [ Compute Platform ] </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e653b5e8",
   "metadata": {},
   "source": [
    "Based on the dataset size and compute choice we will try to recommend an instance choice*, you are of course welcome to select alternate configurations. \n",
    "> e.g., For the 10_year dataset option, we suggest ml.p3.8xlarge instances (4 GPUs) and ml.m5.24xlarge CPU instances ( we will need upwards of 200GB CPU RAM during model training)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "482a3f02",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we will recommend a compute instance type, feel free to modify \n",
    "instance_type = 'ml.p3.2xlarge' #recommend_instance_type(ml_workflow_choice, dataset_directory) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15fa2ec4",
   "metadata": {},
   "source": [
    "In addition to choosing our instance type, we can also enable significant savings by leveraging [AWS EC2 Spot Instances](https://aws.amazon.com/ec2/spot/).\n",
    "\n",
    "We **highly recommend** that you set this flag to `True` as it typically leads to 60-70% cost savings. Note, however that you may need to request a [quota limit increase](https://docs.aws.amazon.com/general/latest/gr/sagemaker.html) to enable Spot instances in SageMaker.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3175c706",
   "metadata": {},
   "outputs": [],
   "source": [
    "# please choose whether spot instances should be used\n",
    "use_spot_instances_flag = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2779cc36",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:20pt\"> 2.2 - Create Estimator </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "511f2708",
   "metadata": {},
   "source": [
    "Having built our container [ +custom logic] and pushed it to ECR, we can finally compile all of efforts into an Estimator instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32c50fc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'volume_size' - EBS volume size in GB, default = 30\n",
    "estimator_params = {\n",
    "    'image_uri': ECR_container_fullname,\n",
    "    'role': execution_role,    \n",
    "    \n",
    "    'instance_type': instance_type,\n",
    "    'instance_count': 1,\n",
    "    \n",
    "    'input_mode': 'File',\n",
    "    'output_path': s3_model_output,\n",
    "    \n",
    "    'use_spot_instances': use_spot_instances_flag,\n",
    "    \n",
    "    'max_run': max_duration_of_experiment_seconds, # 24 hours \n",
    "    'sagemaker_session': session,\n",
    "}\n",
    "\n",
    "if use_spot_instances_flag == True:\n",
    "    estimator_params.update({'max_wait' : max_duration_of_experiment_seconds + 1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79997291",
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator = sagemaker.estimator.Estimator(**estimator_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4d13a6a",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:20pt\"> 2.3 - Test Estimator </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1172f62",
   "metadata": {},
   "source": [
    "Now we are ready to test by asking SageMaker to run the BYOContainer logic inside our Estimator. This is a useful step if you've made changes to your custom logic and are interested in making sure everything works before launching a large HPO search. \n",
    "\n",
    "> Note: This verification step will use the default hyperparameter values declared in our custom train code, as SageMaker HPO will not be orchestrating a search for this single run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "771d1340",
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator.fit(inputs = s3_data_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3fee286",
   "metadata": {},
   "source": [
    "<span style=\"display: block; text-align: center; color:#8735fb; font-size:30pt\"> **3. Run HPO** </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d47b3603",
   "metadata": {},
   "source": [
    "With a working SageMaker Estimator in hand, the hardest part is behind us. In the key choices section we <a href='#strategy-and-param-ranges'>already defined our search strategy and hyperparameter ranges</a>, so all that remains is to choose a metric to evaluate performance on. For more documentation check out the AWS SageMaker [Hyperparameter Tuner documentation](https://sagemaker.readthedocs.io/en/stable/tuner.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "248d4a64",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:20pt\"> 3.1 - Define Metric </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "779f9a6c",
   "metadata": {},
   "source": [
    "We only focus on a single metric, which we call 'final-score', that captures the accuracy of our model on the test data unseen during training. You are of course welcome to add aditional metrics, see [AWS SageMaker documentation on Metrics](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-define-metrics.html). When defining a metric we provide a regular expression (i.e., string parsing rule) to extract the key metric from the output of each Estimator/worker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d200a4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "metric_definitions = [{'Name': 'final-score', 'Regex': 'final-score: (.*);'}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9023ddf9",
   "metadata": {},
   "outputs": [],
   "source": [
    "objective_metric_name = 'final-score'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f837f1b",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:20pt\"> 3.2 - Define Tuner </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23e4427b",
   "metadata": {},
   "source": [
    "Finally we put all of the elements we've been building up together into a HyperparameterTuner declaration. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a15088ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "hpo = sagemaker.tuner.HyperparameterTuner(estimator=estimator,\n",
    "                                          metric_definitions=metric_definitions, \n",
    "                                          objective_metric_name=objective_metric_name,\n",
    "                                          objective_type='Maximize',\n",
    "                                          hyperparameter_ranges=hyperparameter_ranges,\n",
    "                                          strategy='Random',  \n",
    "                                          max_jobs=max_jobs,\n",
    "                                          max_parallel_jobs=max_parallel_jobs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54abb3af",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:20pt\"> 3.3 - Run HPO </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dbc6ef5",
   "metadata": {},
   "source": [
    "Let's be sure we take a moment to confirm before launching all of our HPO experiments. Depending on your configuration options running this cell can kick off a massive amount of computation!\n",
    "> Once this process begins, we recommend that you use the SageMaker UI to keep track of the <a href='../img/gpu_hpo_100x10.png'>health of the HPO process and the individual workers</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29078d34",
   "metadata": {},
   "outputs": [],
   "source": [
    "tuning_job_name = 'unified-hpo-19-' + ''.join(random.choices(string.digits, k = 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "512dc8da",
   "metadata": {},
   "outputs": [],
   "source": [
    "hpo.fit( inputs=s3_data_input,\n",
    "         job_name=tuning_job_name,\n",
    "         wait=True,\n",
    "         logs='All')\n",
    "\n",
    "hpo.wait()  # block until the .fit call above is completed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5a83d51",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:20pt\"> 3.4 - Results and Summary </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75a88927",
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker.HyperparameterTuningJobAnalytics(tuning_job_name).dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ba18c9e",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:30pt\">RAPIDS References</span>\n",
    "\n",
    "> [cloud-ml-examples](http://github.com/rapidsai/cloud-ml-examples)\n",
    "\n",
    "> [RAPIDS HPO](https://rapids.ai/hpo)\n",
    "\n",
    "> [cuML Documentation](https://docs.rapids.ai/api/cuml/stable/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c89bbf2",
   "metadata": {},
   "source": [
    "<span style=\"color:#8735fb; font-size:30pt\">SageMaker References</span>\n",
    "\n",
    "> [SageMaker Training Toolkit](/~https://github.com/aws/sagemaker-training-toolkit)\n",
    "\n",
    "> [Estimator Parameters](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html)\n",
    "\n",
    "> Spot Instances [docs](https://docs.aws.amazon.com/sagemaker/latest/dg/model-managed-spot-training.html), and [blog]()"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}