5-AB-Testing/5.1-AB-Test-a-New-Menu-Launch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyzing a Market Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plan Your Analysis\n",
    "\n",
    "#### 1. What is the performance metric you’ll use to evaluate the results of your test?\n",
    "The sum of gross margin will be used as performance metrics to evaluate whether to introduce gourmet sandwiches and limited wine offerings to spur sales growth in Round Roasters\n",
    "\n",
    "#### 2. What is the test period?\n",
    "A period of 12 weeks (29-Apr-16 to 21-Jul-16) is used as test period\n",
    "\n",
    "#### 3. At what level (day, week, month, etc.) should the data be aggregated?\n",
    "The data should be aggregated at weekly level\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean Up Your Data \n",
    "\n",
    "**RoundRoasterTransaction** and **Round-Roaster-Store** datasets are first combined. 76 weeks data (6-Feb-15 to 21-Jul-16) is used as A/B test requires 52 weeks of data in addition to a minimum of 12 weeks needed to calculate seasonality and for the period of testing each.  \n",
    "12 weeks is used instead of 6 weeks in this case as the test period lasted for 12 weeks.\n",
    "\n",
    "_The week_, _week_begin_, _week_end_ and _NewProduct_Flag_ are added to calculate the weekly traffic and sales for each store. **Treatment_Store** dataset is then introduced to create a list of control and treatment stores.\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/14093302/29741452-c96995f6-8a9f-11e7-85b6-b0697069078d.png\" alt=\"Figure 1\" width=\"800\" height=\"600\" />\n",
    "<p style=\"text-align: center;\">Figure 1: Workflow to clean up data</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Match Treatment and Control Units\n",
    "\n",
    "#### 1. What control variables should be considered? Note: Only consider variables in the RoundRoastersStore file.\n",
    "_AvgMonthSales_ should be considered as constant variables while _Square Feet_ should ignored.\n",
    "\n",
    "#### 2. What is the correlation between your each potential control variable and your performance metric?\n",
    "From the Pearson Correlation Analysis, _AvgMonthSales_ has high correlation of 0.99 with the performance metric, i.e. Sum of Gross Margin. On the other hand, _Square Feet_ has a poor correlation of -0.05.\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/14093302/29741451-c9461536-8a9f-11e7-843c-d8b5adfc6702.png\" alt=\"Figure 2\" width=\"500\" height=\"200\" />\n",
    "<p style=\"text-align: center;\">Figure 2: Pearson Correlation Analysis</p>\n",
    "\n",
    "#### 3. What control variables will you use to match treatment and control stores?\n",
    "_AvgMonthSales_ will be used together with Trend and Seasonality when matching treatment and control stores.\n",
    "\n",
    "#### 4. Please fill out the table below with your treatment and control stores pairs:\n",
    "\n",
    "| Treatment Store | Control Store 1 | Control Store 2 |\n",
    "| :-------------: | :-------------: | :-------------: |\n",
    "| 1664            | 1964            | 8562            |\n",
    "| 1675            | 1807            | 7584            |\n",
    "| 1696            | 1863            | 7334            |\n",
    "| 1700            | 7037            | 1508            |\n",
    "| 1712            | 8162            | 7434            |\n",
    "| 2288            | 2568            | 9081            |\n",
    "| 2293            | 12219           | 9639            |\n",
    "| 2301            | 11668           | 12019           |\n",
    "| 2322            | 9238            | 9388            |\n",
    "| 2241            | 2572            | 3102            |\n",
    "\n",
    "<p style=\"text-align: center;\">Table 1: Treatment and Control Stores</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis and Write-up\n",
    "\n",
    "#### 1. What is your recommendation - Should the company roll out the updated menu to all stores? \n",
    "The company should roll out the updated menu to all stores as the sum of profit margin increased by more than 18%, from $17,978.67 per store to $26,687.45 per store during test period.\n",
    "\n",
    "#### 2. What is the lift from the new menu for West and Central regions (include statistical significance)? \n",
    "The lift for West region is 36.6% while the lift for Central region is 43.2% and both have a statistical significance of 99.5% and 100% respectively.\n",
    "\n",
    "#### 3. What is the lift from the new menu overall?\n",
    "The lift for the new menu overall is 43.2% with a statistical significance of 99.6%.\n",
    "\n",
    "### West Region\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/14093302/29741448-c943a94a-8a9f-11e7-8488-ea5eecdd67a4.png\" alt=\"Figure 3-1\" width=\"800\" height=\"400\" />\n",
    "<img src=\"https://user-images.githubusercontent.com/14093302/29741450-c9450150-8a9f-11e7-80d5-d2be9ab50262.png\" alt=\"Figure 3-2\" width=\"800\" height=\"400\" />  \n",
    "<p style=\"text-align: center;\">Figure 3: A/B Analysis for West Region</p>\n",
    "\n",
    "### Central Region\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/14093302/29741449-c94465c4-8a9f-11e7-9a5f-1408f6207fdc.png\" alt=\"Figure 4\" width=\"800\" height=\"400\" />\n",
    "<p style=\"text-align: center;\">Figure 4: A/B Analysis for Central Region</p>\n",
    "\n",
    "### Overall\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/14093302/29741447-c941d3c2-8a9f-11e7-85b3-502c2756be4a.png\" alt=\"Figure 5\" width=\"800\" height=\"400\" />\n",
    "<p style=\"text-align: center;\">Figure 5: A/B Analysis for Overall</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alteryx Workflow\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/14093302/29741446-c93f5386-8a9f-11e7-8603-d37637c9c372.png\" alt=\"Figure 6\" width=\"800\" height=\"500\" />\n",
    "<p style=\"text-align: center;\">Figure 6: Determine treatment and ocntrol store pairing</p>\n",
    "\n",
    "![Figure 7](https://user-images.githubusercontent.com/14093302/29741445-c9162baa-8a9f-11e7-8de9-90098f750fd8.png)\n",
    "<p style=\"text-align: center;\">Figure 7: A.B analysis</p>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}