2020/exercises/Solutions_Exercises_6_7.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Solutions - Exercises 6 - 7.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OES5Wz_Q9pMS"
      },
      "source": [
        "# Exercises 6 - 7"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UmfbDeLu-flW"
      },
      "source": [
        "## Exercise 6 - Slicing and Modifying\r\n",
        "\r\n",
        "There are (at least) two ways in which we can approach this exercise. We can either take a slice of characters (A) or tokenize the string (B).\r\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Be_-HytuBQym"
      },
      "source": [
        "text = 'Python programming can be fun.'"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yHKiEGIzVbYs"
      },
      "source": [
        "### Variant A (Slicing)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GVHgi4L9-vHP"
      },
      "source": [
        "third_word_a = text[19:23]\r\n",
        "third_word_a_upper = third_word_a.upper()"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A08Ly-1R_HCk",
        "outputId": "44b10f53-4d3f-4e5d-ff69-3ae2e9541b4f"
      },
      "source": [
        "print(third_word_a_upper)"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CAN\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kLWassZ__QMw"
      },
      "source": [
        "If we don't want to count the characters, we can also use a combination of `find` and `len`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i263ejbw_Pu8"
      },
      "source": [
        "starting_point = text.find('can')\r\n",
        "end_point = starting_point + len('can')\r\n",
        "\r\n",
        "third_word_a = text[starting_point:end_point]\r\n",
        "third_word_a_upper = third_word_a.upper()"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YAaPEsX8UsAe",
        "outputId": "5a8f8be3-1e73-4525-ac80-87f487865689"
      },
      "source": [
        "print(third_word_a_upper)"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CAN\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_VFR4swuU0Q7"
      },
      "source": [
        "Now that we have this solution, we can generalize even further and create a function that finds and modifies arbitrary substrings. Of course, this function does not make any sense, but it nicely shows how we can gradually approach generalized solutions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c3dwhIa8UwtO"
      },
      "source": [
        "def find_and_uppercase(text, search):\r\n",
        "  starting_point = text.find(search)\r\n",
        "  end_point = starting_point + len(search)\r\n",
        "\r\n",
        "  word = text[starting_point:end_point]\r\n",
        "  word_upper = word.upper()\r\n",
        "\r\n",
        "  return word_upper"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "msr-rLvzVVvV",
        "outputId": "cd678cd7-87cd-4eeb-e1a8-34fef9760996"
      },
      "source": [
        "find_and_uppercase(text, 'can')"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'CAN'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b02QDFFSVfos"
      },
      "source": [
        "### Variant B (Tokenization)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "419IRXkcVhjj",
        "outputId": "a233a998-5690-4695-d868-d09992dc73bc"
      },
      "source": [
        "tokenized = text.split() # Without any arguments, split will use whitespace\r\n",
        "tokenized"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Python', 'programming', 'can', 'be', 'fun.']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LH1cBbsZVkjN"
      },
      "source": [
        "third_word_b = tokenized[2]\r\n",
        "third_word_b_upper = third_word_b.upper()"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "i1CtXgXoVsUs",
        "outputId": "b1686cf3-84f2-4b84-cbc0-49798436d8f2"
      },
      "source": [
        "third_word_b_upper"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'CAN'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GE5BUi2qV2hj"
      },
      "source": [
        "## Exercise 7 - Counting Tokens"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iSxN_weCWDp2"
      },
      "source": [
        "First we will download (`git clone`) the repository. This way, we will have access to the two files (`simple.txt` and `challenge.txt`)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZhNDMR0gV7cY"
      },
      "source": [
        "%%capture\r\n",
        "!git clone /~https://github.com/IngoKl/python-programming-for-linguists"
      ],
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L9kQs41kWyVp"
      },
      "source": [
        "First, we will open and read the file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qXTKWOLOWNdu"
      },
      "source": [
        "with open('python-programming-for-linguists/2020/data/tokenize/simple.txt', 'r') as f:\r\n",
        "  text = f.read()"
      ],
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "IssXg7NKW1ww",
        "outputId": "25df52cc-3df0-445a-98eb-4accf374d80b"
      },
      "source": [
        "text"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'The black cat chased the mouse.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VHaPMEzoW7ga"
      },
      "source": [
        "We can build a very simple tokenizer, just as above, by using the `str.split()` method."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I-lFDl1EW3lP",
        "outputId": "358494d4-a3a6-4d04-e707-6d7b4d64996c"
      },
      "source": [
        "tokenized = text.split()\r\n",
        "\r\n",
        "tokenized"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['The', 'black', 'cat', 'chased', 'the', 'mouse.']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MFHY8M19XDBA"
      },
      "source": [
        "Now we just need to get the length of the resulting list."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D3Udp_EKXCvC",
        "outputId": "58497594-c7f9-46b1-9ab4-22ba1efaf636"
      },
      "source": [
        "len(tokenized)"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "6"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q4yCKfyyXIRG"
      },
      "source": [
        "Now, as is requested in the exercise, we will put all of that into one function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-jhDk9eNXN8i"
      },
      "source": [
        "def count_tokens(file):\r\n",
        "  with open(file, 'r') as f:\r\n",
        "    text = f.read()\r\n",
        "\r\n",
        "  tokenized = text.split()\r\n",
        "\r\n",
        "  return len(tokenized)"
      ],
      "execution_count": 51,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GQVePOM8XUyM",
        "outputId": "5f418242-6146-4afc-84ae-43c1a4c6b7d0"
      },
      "source": [
        "count_tokens('python-programming-for-linguists/2020/data/tokenize/simple.txt')"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "6"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SUeN1RRdXcFf"
      },
      "source": [
        "Great, now let's try to use our function with the more challenging `challenge.txt` example."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5xjsfKUHXjJG",
        "outputId": "81baba78-26cc-4828-d4e7-b5056623182c"
      },
      "source": [
        "count_tokens('python-programming-for-linguists/2020/data/tokenize/challenge.txt')"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "11"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SLfBKsIWXnJP"
      },
      "source": [
        "This does not look good. Let's have a look at both the file and at the output of our tokenizer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GBj2D48aX2Xk",
        "outputId": "15a3d0ef-2300-42b8-8f04-63cf32f959e9"
      },
      "source": [
        "with open('python-programming-for-linguists/2020/data/tokenize/challenge.txt', 'r') as f:\r\n",
        "  text = f.read()\r\n",
        "\r\n",
        "print(text)\r\n",
        "print(text.split())"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sue owed Ms. O'Neil $10. Unfortunately, she  didn't have the money.\n",
            "['Sue', 'owed', 'Ms.', \"O'Neil\", '$10.', 'Unfortunately,', 'she', \"didn't\", 'have', 'the', 'money.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wYpoZjlGYBnb"
      },
      "source": [
        "Alright, there are various problems here.\r\n",
        "\r\n",
        "*   *$* and *10* should arguably be split\r\n",
        "*   *didn't* should also be split into two words or tokens\r\n",
        "*   Due to the fact that we have *O'Neil*, we can't just split at the `'` character.\r\n",
        "*   There's an extra space after *she* that potentially could cause trouble.\r\n",
        "\r\n",
        "Let's try to build a more robust tokenizer that can handle these cases. Our approach will be to modify the text before we do the tokenization.\r\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_YzliU6dYq-a"
      },
      "source": [
        "import re\r\n",
        "\r\n",
        "def count_tokens_optimized(text):\r\n",
        "  # Replace double whitespace\r\n",
        "  text = text.replace('  ', ' ')\r\n",
        "\r\n",
        "  # Add a space between $/€ and numbers\r\n",
        "  text = re.sub(r'(\\$|\\€)([0-9]*)\\b', r'\\1 \\2', text)\r\n",
        "\r\n",
        "  # Add space between words and periods\r\n",
        "  text = re.sub(r'(\\w+)(\\.)', r'\\1 \\2', text)\r\n",
        "\r\n",
        "  # Account for the abbreviation\r\n",
        "  text = text.replace(\"n't\", \" n't\")\r\n",
        "\r\n",
        "  tokenized = text.split()\r\n",
        "\r\n",
        "  return len(tokenized)"
      ],
      "execution_count": 71,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X1j1eBMuZWlA",
        "outputId": "a7b5188e-1659-421d-e59d-f0dc68621d65"
      },
      "source": [
        "count_tokens_optimized(text)"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "16"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gQ-qq1f0dXie"
      },
      "source": [
        "Great, our optimized function works very well. However, it will only work for this particular example and the edge cases (which are really not edge cases) we encountered here. Well, we at least have accounted for not just the *$* sign, but also for *€*."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_82m0dw8dm7A"
      },
      "source": [
        "While state-of-the-art tokenizers use sophisticated language models to solve these problems, there are still good rule-based tokenizers out there. If you want to have a look at some real-word code, have a look at the [NLTKWordTokenizer](/~https://github.com/nltk/nltk/blob/develop/nltk/tokenize/destructive.py)."
      ]
    }
  ]
}