16. Machine Learning Explainability/exercise/02. exercise-permutation-importance.ipynb

{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**This notebook is an exercise in the [Machine Learning Explainability](https://www.kaggle.com/learn/machine-learning-explainability) course.  You can reference the tutorial at [this link](https://www.kaggle.com/dansbecker/permutation-importance).**\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"## Intro\n\nYou will think about and calculate permutation importance with a sample of data from the [Taxi Fare Prediction](https://www.kaggle.com/c/new-york-city-taxi-fare-prediction) competition.\n\nWe won't focus on data exploration or model building for now. You can just run the cell below to \n- Load the data\n- Divide the data into training and validation\n- Build a model that predicts taxi fares\n- Print a few rows for you to review","metadata":{}},{"cell_type":"code","source":"# Loading data, dividing, modeling and EDA below\nimport pandas as pd\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\n\ndata = pd.read_csv('../input/new-york-city-taxi-fare-prediction/train.csv', nrows=50000)\n\n# Remove data with extreme outlier coordinates or negative fares\ndata = data.query('pickup_latitude > 40.7 and pickup_latitude < 40.8 and ' +\n                  'dropoff_latitude > 40.7 and dropoff_latitude < 40.8 and ' +\n                  'pickup_longitude > -74 and pickup_longitude < -73.9 and ' +\n                  'dropoff_longitude > -74 and dropoff_longitude < -73.9 and ' +\n                  'fare_amount > 0'\n                  )\n\ny = data.fare_amount\n\nbase_features = ['pickup_longitude',\n                 'pickup_latitude',\n                 'dropoff_longitude',\n                 'dropoff_latitude',\n                 'passenger_count']\n\nX = data[base_features]\n\n\ntrain_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)\nfirst_model = RandomForestRegressor(n_estimators=50, random_state=1).fit(train_X, train_y)\n\n# Environment Set-Up for feedback system.\nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.ml_explainability.ex2 import *\nprint(\"Setup Complete\")\n\n# show data\nprint(\"Data sample:\")\ndata.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T06:38:02.553424Z","iopub.execute_input":"2022-07-28T06:38:02.554347Z","iopub.status.idle":"2022-07-28T06:38:11.314981Z","shell.execute_reply.started":"2022-07-28T06:38:02.554246Z","shell.execute_reply":"2022-07-28T06:38:11.313830Z"},"trusted":true},"execution_count":1,"outputs":[]},{"cell_type":"markdown","source":"The following two cells may also be useful to understand the values in the training data:","metadata":{}},{"cell_type":"code","source":"train_X.describe()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T06:38:43.345994Z","iopub.execute_input":"2022-07-28T06:38:43.346901Z","iopub.status.idle":"2022-07-28T06:38:43.383364Z","shell.execute_reply.started":"2022-07-28T06:38:43.346862Z","shell.execute_reply":"2022-07-28T06:38:43.382619Z"},"trusted":true},"execution_count":2,"outputs":[]},{"cell_type":"code","source":"train_y.describe()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T06:38:47.657487Z","iopub.execute_input":"2022-07-28T06:38:47.657858Z","iopub.status.idle":"2022-07-28T06:38:47.668955Z","shell.execute_reply.started":"2022-07-28T06:38:47.657819Z","shell.execute_reply":"2022-07-28T06:38:47.668123Z"},"trusted":true},"execution_count":3,"outputs":[]},{"cell_type":"markdown","source":"## Question 1\n\nThe first model uses the following features\n- pickup_longitude\n- pickup_latitude\n- dropoff_longitude\n- dropoff_latitude\n- passenger_count\n\nBefore running any code... which variables seem potentially useful for predicting taxi fares? Do you think permutation importance will necessarily identify these features as important?\n\nOnce you've thought about it, run `q_1.solution()` below to see how you might think about this before running the code.","metadata":{}},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_1.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T06:38:59.203700Z","iopub.execute_input":"2022-07-28T06:38:59.204100Z","iopub.status.idle":"2022-07-28T06:38:59.213072Z","shell.execute_reply.started":"2022-07-28T06:38:59.204066Z","shell.execute_reply":"2022-07-28T06:38:59.211948Z"},"trusted":true},"execution_count":4,"outputs":[]},{"cell_type":"markdown","source":"## Question 2\n\nCreate a `PermutationImportance` object called `perm` to show the importances from `first_model`.  Fit it with the appropriate data and show the weights.\n\nFor your convenience, the code from the tutorial has been copied into a comment in this code cell.","metadata":{}},{"cell_type":"code","source":"import eli5\nfrom eli5.sklearn import PermutationImportance\n\n# Make a small change to the code below to use in this problem. \nperm = PermutationImportance(first_model, random_state=1).fit(val_X, val_y)\n\n# Check your answer\nq_2.check()\n\n# uncomment the following line to visualize your results\neli5.show_weights(perm, feature_names = base_features)","metadata":{"execution":{"iopub.status.busy":"2022-07-28T06:40:42.103119Z","iopub.execute_input":"2022-07-28T06:40:42.103610Z","iopub.status.idle":"2022-07-28T06:40:45.500206Z","shell.execute_reply.started":"2022-07-28T06:40:42.103573Z","shell.execute_reply":"2022-07-28T06:40:45.499237Z"},"trusted":true},"execution_count":9,"outputs":[]},{"cell_type":"markdown","source":"Uncomment the lines below for a hint or to see the solution.","metadata":{}},{"cell_type":"code","source":"# q_2.hint()\n# q_2.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T06:40:48.861702Z","iopub.execute_input":"2022-07-28T06:40:48.862102Z","iopub.status.idle":"2022-07-28T06:40:48.866716Z","shell.execute_reply.started":"2022-07-28T06:40:48.862068Z","shell.execute_reply":"2022-07-28T06:40:48.865549Z"},"trusted":true},"execution_count":10,"outputs":[]},{"cell_type":"markdown","source":"## Question 3\nBefore seeing these results, we might have expected each of the 4 directional features to be equally important.\n\nBut, on average, the latitude features matter more than the longititude features. Can you come up with any hypotheses for this?\n\nAfter you've thought about it, check here for some possible explanations:","metadata":{}},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_3.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T06:39:48.025323Z","iopub.execute_input":"2022-07-28T06:39:48.025702Z","iopub.status.idle":"2022-07-28T06:39:48.033453Z","shell.execute_reply.started":"2022-07-28T06:39:48.025673Z","shell.execute_reply":"2022-07-28T06:39:48.032647Z"},"trusted":true},"execution_count":7,"outputs":[]},{"cell_type":"markdown","source":"## Question 4\n\nWithout detailed knowledge of New York City, it's difficult to rule out most hypotheses about why latitude features matter more than longitude.\n\nA good next step is to disentangle the effect of being in certain parts of the city from the effect of total distance traveled.  \n\nThe code below creates new features for longitudinal and latitudinal distance. It then builds a model that adds these new features to those you already had.\n\nFill in two lines of code to calculate and show the importance weights with this new set of features. As usual, you can uncomment lines below to check your code, see a hint or get the solution.","metadata":{}},{"cell_type":"code","source":"# create new features\ndata['abs_lon_change'] = abs(data.dropoff_longitude - data.pickup_longitude)\ndata['abs_lat_change'] = abs(data.dropoff_latitude - data.pickup_latitude)\n\nfeatures_2  = ['pickup_longitude',\n               'pickup_latitude',\n               'dropoff_longitude',\n               'dropoff_latitude',\n               'abs_lat_change',\n               'abs_lon_change']\n\nX = data[features_2]\nnew_train_X, new_val_X, new_train_y, new_val_y = train_test_split(X, y, random_state=1)\nsecond_model = RandomForestRegressor(n_estimators=30, random_state=1).fit(new_train_X, new_train_y)\n\n# Create a PermutationImportance object on second_model and fit it to new_val_X and new_val_y\n# Use a random_state of 1 for reproducible results that match the expected solution.\nperm2 = PermutationImportance(second_model, random_state=1).fit(new_val_X, new_val_y)\n\n# show the weights for the permutation importance you just calculated\neli5.show_weights(perm2, feature_names = features_2)\n\n# Check your answer\nq_4.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T06:41:56.203890Z","iopub.execute_input":"2022-07-28T06:41:56.204275Z","iopub.status.idle":"2022-07-28T06:42:03.385046Z","shell.execute_reply.started":"2022-07-28T06:41:56.204244Z","shell.execute_reply":"2022-07-28T06:42:03.383608Z"},"trusted":true},"execution_count":15,"outputs":[]},{"cell_type":"markdown","source":"How would you interpret these importance scores? Distance traveled seems far more important than any location effects. \n\nBut the location still affects model predictions, and dropoff location now matters slightly more than pickup location. Do you have any hypotheses for why this might be? The techniques in the next lessons will help you` dive into this more.","metadata":{}},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\n# q_4.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T06:41:50.158894Z","iopub.execute_input":"2022-07-28T06:41:50.159332Z","iopub.status.idle":"2022-07-28T06:41:50.164420Z","shell.execute_reply.started":"2022-07-28T06:41:50.159292Z","shell.execute_reply":"2022-07-28T06:41:50.163428Z"},"trusted":true},"execution_count":14,"outputs":[]},{"cell_type":"markdown","source":"## Question 5\n\nA colleague observes that the values for `abs_lon_change` and `abs_lat_change` are pretty small (all values are between -0.1 and 0.1), whereas other variables have larger values.  Do you think this could explain why those coordinates had larger permutation importance values in this case?  \n\nConsider an alternative where you created and used a feature that was 100X as large for these features, and used that larger feature for training and importance calculations. Would this change the outputted permutaiton importance values?\n\nWhy or why not?\n\nAfter you have thought about your answer, either try this experiment or look up the answer in the cell below.","metadata":{}},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_5.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T06:42:08.065600Z","iopub.execute_input":"2022-07-28T06:42:08.066011Z","iopub.status.idle":"2022-07-28T06:42:08.074784Z","shell.execute_reply.started":"2022-07-28T06:42:08.065969Z","shell.execute_reply":"2022-07-28T06:42:08.073584Z"},"trusted":true},"execution_count":16,"outputs":[]},{"cell_type":"markdown","source":"## Question 6\n\nYou've seen that the feature importance for latitudinal distance is greater than the importance of longitudinal distance. From this, can we conclude whether travelling a fixed latitudinal distance tends to be more expensive than traveling the same longitudinal distance?\n\nWhy or why not? Check your answer below.","metadata":{}},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_6.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T06:42:09.546676Z","iopub.execute_input":"2022-07-28T06:42:09.547713Z","iopub.status.idle":"2022-07-28T06:42:09.557798Z","shell.execute_reply.started":"2022-07-28T06:42:09.547662Z","shell.execute_reply":"2022-07-28T06:42:09.556494Z"},"trusted":true},"execution_count":17,"outputs":[]},{"cell_type":"markdown","source":"## Keep Going\n\nPermutation importance is useful useful for debugging, understanding your model, and communicating a high-level overview from your model.  \n\nNext, learn about **[partial dependence plots](https://www.kaggle.com/dansbecker/partial-plots)** to see **how** each feature affects predictions.\n","metadata":{}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/machine-learning-explainability/discussion) to chat with other learners.*","metadata":{}}]}