14. Intro to AI Ethics/tutorial/03. identifying-bias-in-ai.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "64524a9a",
   "metadata": {
    "papermill": {
     "duration": 0.005281,
     "end_time": "2021-11-09T00:03:37.342950",
     "exception": false,
     "start_time": "2021-11-09T00:03:37.337669",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Introduction\n",
    "\n",
    "Machine learning (ML) has the potential to improve lives, but it can also be a source of harm. ML applications have discriminated against individuals on the basis of race, sex, religion, socioeconomic status, and other categories.\n",
    "\n",
    "In this tutorial, you’ll learn about **bias**, which refers to negative, unwanted consequences of ML applications, especially if the consequences disproportionately affect certain groups.\n",
    "\n",
    "We’ll cover six different types of bias that can affect any ML application.  Then you’ll put your new knowledge to work in a **[hands-on exercise](https://www.kaggle.com/kernels/fork/15622654)**, where you will identify bias in a real-world scenario.\n",
    "\n",
    "# Bias is complex\n",
    "\n",
    "Many ML practitioners are familiar with “biased data” and the concept of “garbage in, garbage out”. For example, if you’re training a chatbot using a dataset containing anti-Semitic online conversations (“garbage in”), the chatbot will likely make anti-Semitic remarks (“garbage out”).  This example details an important type of bias (called **historial bias**, as you’ll see below) that should be recognized and addressed.\n",
    "\n",
    "This is not the only way that bias can ruin ML applications. \n",
    "\n",
    "Bias in data is complex.  Flawed data can also result in **representation bias** (covered later in this tutorial), if a group is underrepresented in the training data.  For instance, when training a facial detection system, if the training data contains mostly individuals with lighter skin tones, it will fail to perform well for users with darker skin tones.  A third type of bias that can arise from the training data is called **measurement bias**, which you’ll learn about below. \n",
    "\n",
    "And it’s not just biased data that can lead to unfair ML applications: as you’ll learn, bias can also result from the way in which the ML model is defined, from the way the model is compared to other models, and from the way that everyday users interpret the final results of the model.  Harm can come from anywhere in the ML process."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9964031b",
   "metadata": {
    "papermill": {
     "duration": 0.00349,
     "end_time": "2021-11-09T00:03:37.350852",
     "exception": false,
     "start_time": "2021-11-09T00:03:37.347362",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Six types of bias\n",
    "\n",
    "Once we’re aware of the different types of bias, we are more likely to detect them in ML projects.  Furthermore, with a common vocabulary, we can have fruitful conversations about how to mitigate (or reduce) the bias.\n",
    "\n",
    "We will closely follow a [research paper](https://arxiv.org/pdf/1901.10002.pdf) from early 2020 that characterizes six different types of bias.   \n",
    "\n",
    "## Historical bias \n",
    "\n",
    "**Historical bias** occurs when the state of the world in which the data was generated is flawed. \n",
    "\n",
    "> As of 2020, only [7.4%](https://edition.cnn.com/2020/05/20/us/fortune-500-women-ceos-trnd) of Fortune 500 CEOs are women.  Research has shown that companies with female CEOs or CFOs are generally [more profitable](https://edition.cnn.com/2019/10/16/success/women-ceos-and-cfos-outperform/index.html) than companies with men in the same position, suggesting that women are held to higher hiring standards than men.  In order to fix this, we might consider removing human input and using AI to make the hiring process more equitable.  But this can prove unproductive if data from past hiring decisions is used to train a model, because the model will likely learn to demonstrate the same biases that are present in the data.\n",
    "\n",
    "## Representation bias \n",
    "\n",
    "**Representation bias** occurs when building datasets for training a model, if those datasets poorly represent the people that the model will serve.\n",
    "\n",
    "> Data collected through smartphone apps will under-represent groups that are less likely to own smartphones.  For instance, if collecting [data in the USA](https://www.pewresearch.org/internet/fact-sheet/mobile/#:~:text=The%20vast%20majority%20of%20Americans,range%20of%20other%20information%20devices), individuals over the age of 65 will be under-represented.  If the data is used to inform design of a city transportation system, this will be disastrous, since older people have important [needs](https://www.bloomberg.com/news/articles/2017-08-04/why-aging-americans-need-better-transit) to ensure that the system is accessible.\n",
    "\n",
    "## Measurement bias\n",
    "\n",
    "**Measurement bias** occurs when the accuracy of the data varies across groups.  This can happen when working with proxy variables (variables that take the place of a variable that cannot be directly measured), if the quality of the proxy varies in different groups.\n",
    "\n",
    "> Your local hospital uses a model to identify high-risk patients before they develop serious conditions, based on information like past diagnoses, medications, and demographic data.  The model uses this information to predict health care costs, the idea being that patients with higher costs likely correspond to high-risk patients.  Despite the fact that the model specifically excludes race, it seems to demonstrate racial discrimination: the algorithm is less likely to select eligible Black patients.  How can this be the case?  It is because cost was used as a proxy for risk, and the relationship between these variables varies with race: Black patients experience increased barriers to care, have [less trust](https://science.sciencemag.org/content/366/6464/447) in the health care system, and therefore have lower medical costs, on average, when compared to non-Black patients with the same health conditions.\n",
    "\n",
    "## Aggregation bias \n",
    "\n",
    "**Aggregation bias** occurs when groups are inappropriately combined, resulting in a model that does not perform well for any group or only performs well for the majority group.  (This is often not an issue, but most commonly arises in medical applications.)\n",
    "\n",
    "> Hispanics have [higher rates](https://care.diabetesjournals.org/content/31/2/240.short) of diabetes and diabetes-related complications than non-Hispanic whites.  If building AI to diagnose or monitor diabetes, it is important to make the system sensitive to these ethnic differences, by either including ethnicity as a feature in the data, or building separate models for different ethnic groups.\n",
    "\n",
    "## Evaluation bias \n",
    "\n",
    "**Evaluation bias** occurs when evaluating a model, if the benchmark data (used to compare the model to other models that perform similar tasks) does not represent the population that the model will serve.\n",
    "\n",
    "> The [Gender Shades](http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf) paper discovered that two widely used facial analysis benchmark datasets (IJB-A and Adience) were primarily composed of lighter-skinned subjects (79.6% and 86.2%, respectively).  Commercial gender classification AI showed state-of-the-art performance on these benchmarks, but experienced disproportionately [high error rates](http://gendershades.org/overview.html) with people of color. \n",
    "\n",
    "## Deployment bias \n",
    "\n",
    "**Deployment bias** occurs when the problem the model is intended to solve is different from the way it is actually used.  If the end users don’t use the model in the way it is intended, there is no guarantee that the model will perform well.\n",
    "\n",
    "> The criminal justice system uses [tools](https://www.technologyreview.com/2019/01/21/137783/algorithms-criminal-justice-ai/) to predict the likelihood that a convicted criminal will relapse into criminal behavior.  The predictions are [not designed for judges](https://onlinelibrary.wiley.com/doi/full/10.1002/bsl.2456) when deciding appropriate punishments at the time of sentencing. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a416e2b",
   "metadata": {
    "papermill": {
     "duration": 0.003323,
     "end_time": "2021-11-09T00:03:37.357868",
     "exception": false,
     "start_time": "2021-11-09T00:03:37.354545",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can visually represent these different types of bias, which occur at different stages in the ML workflow:\n",
    "\n",
    "[![visual representation of types of bias](https://i.imgur.com/bvArGRY.png)](https://arxiv.org/pdf/1901.10002.pdf)\n",
    "\n",
    "Note that these are *not mutually exclusive*: that is, an ML application can easily suffer from more than one type of bias.  For example, as Rachel Thomas describes in a [recent research talk](https://www.youtube.com/watch?v=1Uyc9SPeYkA&list=PLe6zdIMe5B7IR0oDOobXBDBlYY1eqLYPx&index=10&t=41s), ML applications in wearable fitness devices can suffer from:\n",
    "- **Representation bias** (if the dataset used to train the models exclude darker skin tones), \n",
    "- **Measurement bias** (if the measurement apparatus shows reduced performance with dark skin tones), and \n",
    "- **Evaluation bias** (if the dataset used to benchmark the model excludes darker skin tones).\n",
    "\n",
    "\n",
    "# Your turn\n",
    "\n",
    "In the exercise, you will **[work directly with a model](https://www.kaggle.com/kernels/fork/15622654)** trained on real-world, biased data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "405cd992",
   "metadata": {
    "papermill": {
     "duration": 0.003308,
     "end_time": "2021-11-09T00:03:37.364786",
     "exception": false,
     "start_time": "2021-11-09T00:03:37.361478",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/ai-ethics/discussion) to chat with other learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 10.839957,
   "end_time": "2021-11-09T00:03:38.080951",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2021-11-09T00:03:27.240994",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}