How_to_Fine_tuning_Gemma_Japanese_Haiku.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/bebechien/gemma/blob/main/How_to_Fine_tuning_Gemma_Japanese_Haiku.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Copyright 2024 Google LLC."
      ],
      "metadata": {
        "id": "cSrJYrFrY2aj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "metadata": {
        "id": "i1PHqD-ZY4-c"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# How to Fine-tuning Gemma: Best Practices"
      ],
      "metadata": {
        "id": "YNDq8NbCY7oh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## What is Fine-tuning\n",
        "\n",
        "In the first place, you have to understand what is fine-tuning. It's a specialized form of [transfer learning](https://en.wikipedia.org/wiki/Transfer_learning). It involves taking a pre-trained language model - one that has already been exposed to a vast corpus of text data and learned the general patterns and structures of language - and further training it on a smaller, more specific dataset. This additional training allows the model to adapt and refine its knowledge, making it better suited for a particular task or domain.\n",
        "\n",
        "Imagine you are a skilled gamer who excels at various genres, from action-adventures to strategy games. Fine-tuning is akin to taking you and having you focus intensely on mastering a specific game, like a complex real-time strategy (RTS) title. You already possess a strong foundation of gaming skills and knowledge, but the dedicated practice and study within the RTS genre sharpens your tactics, understanding of game mechanics, and overall proficiency within that particular realm.\n",
        "\n",
        "Similarly, pre-trained language models have a broad understanding of language, but fine-tuning helps them specialize. By exposing them to a curated dataset relevant to your desired application, you guide the model to learn the nuances and intricacies specific to that domain. It's like giving the model a crash course in the language of your chosen field, enabling it to perform tasks with greater accuracy and fluency."
      ],
      "metadata": {
        "id": "HdYG8HAk62oe"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Setup\n",
        "\n",
        "### Select the Colab runtime\n",
        "To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model:\n",
        "\n",
        "1. In the upper-right of the Colab window, select **▾ (Additional connection options)**.\n",
        "2. Select **Change runtime type**.\n",
        "3. Under **Hardware accelerator**, select **T4 GPU**.\n",
        "\n",
        "\n",
        "### Gemma setup on Kaggle\n",
        "To complete this tutorial, you'll first need to complete the setup instructions at [Gemma setup](https://ai.google.dev/gemma/docs/setup). The Gemma setup instructions show you how to do the following:\n",
        "\n",
        "* Get access to Gemma on kaggle.com.\n",
        "* Select a Colab runtime with sufficient resources to run the Gemma 2B model.\n",
        "* Generate and configure a Kaggle username and API key.\n",
        "\n",
        "After you've completed the Gemma setup, move on to the next section, where you'll set environment variables for your Colab environment."
      ],
      "metadata": {
        "id": "3rzH5Ugf5RlJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Set environemnt variables\n",
        "\n",
        "Set environement variables for ```KAGGLE_USERNAME``` and ```KAGGLE_KEY```."
      ],
      "metadata": {
        "id": "URMuBzkMVxpU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "from google.colab import userdata, drive\n",
        "\n",
        "# Note: `userdata.get` is a Colab API. If you're not using Colab, set the env\n",
        "# vars as appropriate for your system.\n",
        "os.environ[\"KAGGLE_USERNAME\"] = userdata.get(\"KAGGLE_USERNAME\")\n",
        "os.environ[\"KAGGLE_KEY\"] = userdata.get(\"KAGGLE_KEY\")\n",
        "\n",
        "# Mounting gDrive for to store artifacts\n",
        "#drive.mount(\"/content/drive\")"
      ],
      "metadata": {
        "id": "IUOX2hqjV7Ku"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Install dependencies\n",
        "\n",
        "Install Keras and KerasNLP"
      ],
      "metadata": {
        "id": "LXfDwRTQVns2"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "zHs7wpZusEML"
      },
      "outputs": [],
      "source": [
        "!pip install -q -U keras-nlp datasets\n",
        "!pip install -q -U keras\n",
        "\n",
        "# Set the backbend before importing Keras\n",
        "os.environ[\"KERAS_BACKEND\"] = \"jax\"\n",
        "# Avoid memory fragmentation on JAX backend.\n",
        "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"1.00\"\n",
        "\n",
        "import keras_nlp\n",
        "import keras\n",
        "\n",
        "# Run at half precision.\n",
        "# NOTE: inference works at half precision while fine-tuning is not.\n",
        "#keras.config.set_floatx(\"bfloat16\")\n",
        "\n",
        "# Training Configurations\n",
        "token_limit = 128\n",
        "num_data_limit = 1000\n",
        "lora_name = \"haiku\"\n",
        "lora_rank = 4\n",
        "lr_value = 1e-4\n",
        "train_epoch = 5\n",
        "model_id = \"gemma2_2b_en\""
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load Model\n",
        "\n",
        "**Why Fine-tuning?**\n",
        "\n",
        "Before embarking on fine-tuning, it's crucial to evaluate if its benefits align with the specific requirements of your application. Fine-tuning involves meticulous data preparation and extensive training, making it an arduous process. Therefore, it's essential to assess whether the potential gains justify the significant effort required.\n",
        "\n",
        "**Try \"Prompt Engineering\" first.**\n",
        "\n",
        "Would you like to enable Gemma's multilingual capabilities?\n",
        "Please note that Gemma 2 already has some multilingual capabilities. Here's the example output from Gemma 2 2B instruction-tuned model.\n",
        "\n",
        "Do you wish to adjust the tone or writing style?\n",
        "Gemma 2 might be familiar with the writing style you have in mind. Here's another output from the same model."
      ],
      "metadata": {
        "id": "kUl0t469YfQY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import keras\n",
        "import keras_nlp\n",
        "\n",
        "import time\n",
        "\n",
        "gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(model_id)\n",
        "gemma_lm.summary()\n",
        "\n",
        "tick_start = 0\n",
        "\n",
        "def tick():\n",
        "    global tick_start\n",
        "    tick_start = time.time()\n",
        "\n",
        "def tock():\n",
        "    print(f\"TOTAL TIME ELAPSED: {time.time() - tick_start:.2f}s\")\n",
        "\n",
        "def text_gen(prompt):\n",
        "    tick()\n",
        "    output = gemma_lm.generate(prompt, max_length=token_limit)\n",
        "    print(\"\\nGemma output:\")\n",
        "    print(output)\n",
        "    tock()\n",
        "\n",
        "# inference before fine-tuning\n",
        "text_gen(\"何となく\")\n",
        "text_gen(\"こんにちは\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 987
        },
        "outputId": "174d9b75-bab6-47b0-f317-938a6a12aefb",
        "id": "Gm4jIEqmYfQY"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"gemma_causal_lm_preprocessor\"</span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                                                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                  Config\u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m)                              │                      Vocab size: \u001b[38;5;34m256,000\u001b[0m │\n",
              "└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                                                  </span>┃<span style=\"font-weight: bold\">                                   Config </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaTokenizer</span>)                              │                      Vocab size: <span style=\"color: #00af00; text-decoration-color: #00af00\">256,000</span> │\n",
              "└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"gemma_causal_lm\"</span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ gemma_backbone                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m)        │   \u001b[38;5;34m2,614,341,888\u001b[0m │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
              "│ (\u001b[38;5;33mGemmaBackbone\u001b[0m)               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m)      │     \u001b[38;5;34m589,824,000\u001b[0m │ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]       │\n",
              "│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ gemma_backbone                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2304</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaBackbone</span>)               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256000</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">589,824,000</span> │ gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]       │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> (9.74 GB)\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> (9.74 GB)\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Gemma output:\n",
            "何となく、このブログのタイトルに「<strong>お笑い芸人</strong>」という言葉が似合う気がして、ついこの前、<strong>お笑い芸人</strong>の<strong>「お笑い芸人」</strong>というタイトルでブログを立ち上げました。\n",
            "\n",
            "しかし、<strong>お笑い芸人</strong>の<strong>「お笑い芸人」</strong>というタイトルでブログを立ち上げたのは、<strong>お笑い芸人</strong>の<strong>「お笑い芸人」</strong>というタイトルでブログを立ち上げたからではありません。\n",
            "\n",
            "<strong>お笑い芸人</strong>の<strong>「お笑い芸人」</strong>というタイトルでブログを立ち上げたのは\n",
            "TOTAL TIME ELAPSED: 21.99s\n",
            "\n",
            "Gemma output:\n",
            "こんにちは。\n",
            "\n",
            "今日は、\n",
            "\n",
            "<strong>「</strong><strong>おうちでできる！</strong><strong>」</strong>\n",
            "\n",
            "<strong>「</strong><strong>おうちでできる！</strong><strong>」</strong>\n",
            "\n",
            "<strong>「</strong><strong>おうちでできる！</strong><strong>」</strong>\n",
            "\n",
            "<strong>「</strong><strong>おうちでできる！</strong><strong>」</strong>\n",
            "\n",
            "<strong>「</strong><strong>おうちでできる！</strong><strong>」</strong>\n",
            "\n",
            "<strong>「</strong><strong>おうちでできる！</strong><strong>」</strong>\n",
            "\n",
            "<strong>「</strong><strong>おうちでできる！</strong><strong>」</strong>\n",
            "\n",
            "<strong>「</strong><strong>おうちでできる！</strong><strong>」</strong>\n",
            "\n",
            "<strong>「</strong><strong>おうちでできる！\n",
            "TOTAL TIME ELAPSED: 5.83s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Tokenizer designed with multilingual in mind\n",
        "\n",
        "Another important thing is the tokenizer.\n",
        "\n",
        "Gemma tokenizer is based on [SentencePiece](/~https://github.com/google/sentencepiece). The size of the vocabulary is predetermined before training. SentencePiece then learns the optimal subword segmentation based on the chosen vocabulary size and the training data. Gemma's large 256k vocabulary allows it to handle diverse text inputs and potentially improve performance on various tasks, e.g. handling multilingual text inputs.\n",
        "\n",
        "(example text: “Hi, Nice to meet you. The weather is really nice today.”)"
      ],
      "metadata": {
        "id": "O6uFSV843L2H"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer = keras_nlp.models.GemmaTokenizer.from_preset(model_id)\n",
        "import jax\n",
        "\n",
        "def detoken(tokens):\n",
        "  print(tokens)\n",
        "  for x in tokens:\n",
        "    word = tokenizer.detokenize(jax.numpy.array([x]))\n",
        "    print(f\"{x:6} -> {word}\")\n",
        "\n",
        "detoken(tokenizer(\"こんにちは。初めまして。今日は本当にいい天気ですね。\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6IF8rZ0LHzVY",
        "outputId": "301cc51f-5936-45f1-b772-dbee532488bb"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[ 32789 235362 147035  60914 235362  40574  35752  10598 106577  14088\n",
            " 235362]\n",
            " 32789 -> こんにちは\n",
            "235362 -> 。\n",
            "147035 -> 初め\n",
            " 60914 -> まして\n",
            "235362 -> 。\n",
            " 40574 -> 今日は\n",
            " 35752 -> 本当に\n",
            " 10598 -> いい\n",
            "106577 -> 天気\n",
            " 14088 -> ですね\n",
            "235362 -> 。\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load Dataset\n",
        "\n",
        "How many datasets do you need? You can start with relatively small number of datasets, approximately 10 to 20, those can have a significant impact on a model's behavior.\n",
        "\n",
        "For improved the output quality, a target of around 200 total examples is recommended. Nevertheless, the amount of data required for tuning really depends on how much you want to influence the model's behavior. Our recommendation is to commence with a limited amount of data and gradually incorporate additional data into the training process until the desired behavior is achieved."
      ],
      "metadata": {
        "id": "hBcmfmX93P81"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer = keras_nlp.models.GemmaTokenizer.from_preset(model_id)\n",
        "\n",
        "from datasets import load_dataset\n",
        "ds = load_dataset(\n",
        "    \"p1atdev/modern_haiku\",\n",
        "    split=\"train\",\n",
        "    trust_remote_code=True\n",
        ").shuffle(1234)\n",
        "print(ds)\n",
        "data = ds.with_format(\"np\", columns=\"haiku\", output_all_columns=False)\n",
        "\n",
        "train = []\n",
        "\n",
        "for x in data:\n",
        "  length = len(tokenizer(x['haiku']))\n",
        "  # skip data if the token length is longer than our limit\n",
        "  if length < token_limit:\n",
        "    train.append(x['haiku'])\n",
        "    if(len(train)>=num_data_limit):\n",
        "      break\n",
        "\n",
        "print(len(train))\n",
        "print(train[0])\n",
        "print(train[1])\n",
        "print(train[2])"
      ],
      "metadata": {
        "id": "ZiS-KU9osh_N",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8635c7e5-cdf9-4a67-ca7d-5fe89b20a1f3"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Dataset({\n",
            "    features: ['id', 'haiku', 'author', 'foreword', 'source', 'comment', 'reviewer', 'note', 'season', 'kigo'],\n",
            "    num_rows: 37158\n",
            "})\n",
            "1000\n",
            "神々のあくびが桜を枯らすのか\n",
            "火種消すように萱草摘み始め\n",
            "舌出して浅蜊本音は明かさずに\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the context of a small dataset, the primary concern is that the model may prioritize memorizing specific examples rather than generalizing well to new and unobserved data. This limitation highlights the importance of utilizing a larger dataset during fine-tuning, as it enhances the model's ability to capture broader patterns and relationships."
      ],
      "metadata": {
        "id": "snjf8IR17wPB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## LoRA Fine-tuning\n",
        "\n",
        "Train your model with lower ranks and evaluate the performance improvemnet on your task. Gradually increase the rank in subsequent trials and see if that further boosts performance."
      ],
      "metadata": {
        "id": "Pt7Nr6a7tItO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Enable LoRA for the model and set the LoRA rank to 4.\n",
        "gemma_lm.backbone.enable_lora(rank=lora_rank)\n",
        "gemma_lm.summary()\n",
        "\n",
        "# Limit the input sequence length (to control memory usage).\n",
        "gemma_lm.preprocessor.sequence_length = token_limit\n",
        "# Use AdamW (a common optimizer for transformer models).\n",
        "optimizer = keras.optimizers.AdamW(\n",
        "    learning_rate=lr_value,\n",
        "    weight_decay=0.01,\n",
        ")\n",
        "# Exclude layernorm and bias terms from decay.\n",
        "optimizer.exclude_from_weight_decay(var_names=[\"bias\", \"scale\"])\n",
        "\n",
        "gemma_lm.compile(\n",
        "    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "    optimizer=optimizer,\n",
        "    weighted_metrics=[keras.metrics.SparseCategoricalAccuracy()],\n",
        ")\n"
      ],
      "metadata": {
        "id": "RCucu6oHz53G",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 400
        },
        "outputId": "d2ce231b-e2c2-4650-a7f9-5d74ae63893e"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"gemma_causal_lm_preprocessor\"</span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                                                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                  Config\u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m)                              │                      Vocab size: \u001b[38;5;34m256,000\u001b[0m │\n",
              "└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                                                  </span>┃<span style=\"font-weight: bold\">                                   Config </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaTokenizer</span>)                              │                      Vocab size: <span style=\"color: #00af00; text-decoration-color: #00af00\">256,000</span> │\n",
              "└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"gemma_causal_lm\"</span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ gemma_backbone                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m)        │   \u001b[38;5;34m2,617,270,528\u001b[0m │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
              "│ (\u001b[38;5;33mGemmaBackbone\u001b[0m)               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m)      │     \u001b[38;5;34m589,824,000\u001b[0m │ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]       │\n",
              "│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ gemma_backbone                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2304</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">2,617,270,528</span> │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaBackbone</span>)               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256000</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">589,824,000</span> │ gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]       │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,617,270,528\u001b[0m (9.75 GB)\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,617,270,528</span> (9.75 GB)\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,928,640\u001b[0m (11.17 MB)\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,928,640</span> (11.17 MB)\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> (9.74 GB)\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that enabling LoRA reduces the number of trainable parameters significantly. In practice, we recommend beginning with a relatively small rank (such as 4, 8, 16). This is computationally efficient for experimentation.\n",
        "\n",
        "To monitor the learning progress, we will evaluate the model at the end of each epoch and save the all lora weights."
      ],
      "metadata": {
        "id": "hQQ47kcdpbZ9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class CustomCallback(keras.callbacks.Callback):\n",
        "  def on_epoch_end(self, epoch, logs=None):\n",
        "    #model_name = f\"/content/drive/MyDrive/{lora_name}_{lora_rank}_epoch{epoch+1}.lora.h5\"\n",
        "    #gemma_lm.backbone.save_lora_weights(model_name)\n",
        "\n",
        "    # Evaluate\n",
        "    text_gen(\"何となく\")\n",
        "    text_gen(\"こんにちは\")\n",
        "\n",
        "history = gemma_lm.fit(train, epochs=train_epoch, batch_size=2, callbacks=[CustomCallback()])\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "plt.plot(history.history['loss'])\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "26d9npFhAOSp",
        "outputId": "ed3bbb15-1ad5-4ed8-baba-2e69a033bd01"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/5\n",
            "\u001b[1m500/500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 920ms/step - loss: 0.6060 - sparse_categorical_accuracy: 0.1491\n",
            "Gemma output:\n",
            "何となくの春を待つ\n",
            "TOTAL TIME ELAPSED: 18.18s\n",
            "\n",
            "Gemma output:\n",
            "こんにちは、おやじのしるし\n",
            "TOTAL TIME ELAPSED: 0.69s\n",
            "\u001b[1m500/500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m522s\u001b[0m 958ms/step - loss: 0.6059 - sparse_categorical_accuracy: 0.1491\n",
            "Epoch 2/5\n",
            "\u001b[1m500/500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 843ms/step - loss: 0.4977 - sparse_categorical_accuracy: 0.2056\n",
            "Gemma output:\n",
            "何となくの春雨の匂い\n",
            "TOTAL TIME ELAPSED: 0.60s\n",
            "\n",
            "Gemma output:\n",
            "こんにちは、秋風吹く墓石の影\n",
            "TOTAL TIME ELAPSED: 0.76s\n",
            "\u001b[1m500/500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m444s\u001b[0m 846ms/step - loss: 0.4977 - sparse_categorical_accuracy: 0.2056\n",
            "Epoch 3/5\n",
            "\u001b[1m500/500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 886ms/step - loss: 0.4677 - sparse_categorical_accuracy: 0.2263\n",
            "Gemma output:\n",
            "何となくの春雨の匂い\n",
            "TOTAL TIME ELAPSED: 0.58s\n",
            "\n",
            "Gemma output:\n",
            "こんにちは、秋風を浴びてゐるこの部屋\n",
            "TOTAL TIME ELAPSED: 0.82s\n",
            "\u001b[1m500/500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m444s\u001b[0m 889ms/step - loss: 0.4677 - sparse_categorical_accuracy: 0.2263\n",
            "Epoch 4/5\n",
            "\u001b[1m500/500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 886ms/step - loss: 0.4381 - sparse_categorical_accuracy: 0.2561\n",
            "Gemma output:\n",
            "何となくの春雨の食う犬\n",
            "TOTAL TIME ELAPSED: 0.64s\n",
            "\n",
            "Gemma output:\n",
            "こんにちは、秋空の空に蝶の羽ばたき\n",
            "TOTAL TIME ELAPSED: 0.94s\n",
            "\u001b[1m500/500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m444s\u001b[0m 889ms/step - loss: 0.4381 - sparse_categorical_accuracy: 0.2561\n",
            "Epoch 5/5\n",
            "\u001b[1m500/500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 886ms/step - loss: 0.4044 - sparse_categorical_accuracy: 0.2900\n",
            "Gemma output:\n",
            "何となく春を思い出す母が居て\n",
            "TOTAL TIME ELAPSED: 0.69s\n",
            "\n",
            "Gemma output:\n",
            "こんにちは、秋晴の街を歩いている\n",
            "TOTAL TIME ELAPSED: 0.70s\n",
            "\u001b[1m500/500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m444s\u001b[0m 889ms/step - loss: 0.4043 - sparse_categorical_accuracy: 0.2900\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAGdCAYAAAAxCSikAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABPK0lEQVR4nO3deVxU9eL/8dcM2wgKLiiIIu4gmqKghLlkUdim1u1q3RKzsuWa5aVNv930m3V/tJqV3jRv5c26abdFW9Ekl0xLBUlyQUVFXFhMBcTYZs7vD2/05eY2CB5meD8fj/OHZz7n+P48TtO8nTnzGYthGAYiIiIiDZjV7AAiIiIi56LCIiIiIg2eCouIiIg0eCosIiIi0uCpsIiIiEiDp8IiIiIiDZ4Ki4iIiDR4KiwiIiLS4HmaHaAuOBwODh06RLNmzbBYLGbHERERkfNgGAYlJSWEhIRgtZ79PRS3KCyHDh0iNDTU7BgiIiJSC7m5ubRv3/6sY9yisDRr1gw4NWF/f3+T04iIiMj5KC4uJjQ0tPp1/GzcorD8+jGQv7+/CouIiIiLOZ/bOWp10+2cOXPo2LEjNpuN2NhYNmzYcMaxCxYswGKx1NhsNtvvxm3fvp0RI0YQEBCAn58f/fv3Z//+/bWJJyIiIm7G6cKyePFikpKSmD59Ounp6fTp04eEhAQKCgrOeIy/vz+HDx+u3nJycmo8np2dzaBBg4iIiGDVqlVs2bKFJ5988rTFRkRERBofi2EYhjMHxMbG0r9/f2bPng2c+oZOaGgokyZNYsqUKb8bv2DBAiZPnszx48fPeM5bbrkFLy8vFi5c6Fz6/yguLiYgIICioiJ9JCQiIuIinHn9duodloqKCtLS0oiPj//tBFYr8fHxrF+//ozHnThxgrCwMEJDQxk5ciRbt26tfszhcPDFF1/QvXt3EhISaNOmDbGxsSxZsuSM5ysvL6e4uLjGJiIiIu7LqcJy5MgR7HY7QUFBNfYHBQWRl5d32mPCw8N56623WLp0Ke+++y4Oh4OBAwdy4MABAAoKCjhx4gTPPvssw4cPZ/ny5dx4443cdNNNrF69+rTnTE5OJiAgoHrTV5pFRETcW71/SyguLo64uLjqPw8cOJAePXowb948nn76aRwOBwAjR47kL3/5CwBRUVGsW7eOuXPnMnTo0N+dc+rUqSQlJVX/+devRYmIiIh7cqqwBAYG4uHhQX5+fo39+fn5BAcHn9c5vLy86Nu3L7t3764+p6enJ5GRkTXG9ejRg7Vr1572HD4+Pvj4+DgTXURERFyYUx8JeXt7Ex0dTWpqavU+h8NBampqjXdRzsZut5OZmUnbtm2rz9m/f3+ysrJqjNu5cydhYWHOxBMRERE35fRHQklJSYwbN46YmBgGDBjArFmzKC0tZfz48QAkJibSrl07kpOTAZgxYwaXXnopXbt25fjx47zwwgvk5ORw9913V5/z0UcfZcyYMQwZMoRhw4aRkpLCZ599xqpVq+pmliIiIuLSnC4sY8aMobCwkGnTppGXl0dUVBQpKSnVN+Lu37+/xg8YHTt2jAkTJpCXl0eLFi2Ijo5m3bp1NT4CuvHGG5k7dy7Jyck8+OCDhIeH89FHHzFo0KA6mKKIiIi4OqfXYWmItA6LiIiI66m3dVhEREREzKDCchaGYTB/zR6e/nyb2VFEREQaNbf4teb6suVAEX/7cjsA4UHNGN1fa72IiIiYQe+wnEWf0OZMju8GwF+X/ERazjGTE4mIiDROKizn8OAV3UjoGUSF3cG9C9M4XPSL2ZFEREQaHRWWc7BaLcwcHUV4UDOOnCjn3oVplFXazY4lIiLSqKiwnAc/H0/mJ8bQ3NeLLQeKmPLRFtzg2+AiIiIuQ4XlPHVo5cvf/9QPD6uFJRmHmP/tHrMjiYiINBoqLE4Y2DWQadefWqH32a92sCqrwOREIiIijYMKi5MS48IYExOKw4BJ729mT+EJsyOJiIi4PRUWJ1ksFmaM6kl0WAtKyqq4+51NFJdVmh1LRETEramw1IKPpwev396PtgE29hSW8tD7m7E7dBOuiIhIfVFhqaU2zWy8MTYGH08rK7MKeXF5ltmRRERE3JYKywW4pH0Az9/cG4DXV2WzNOOgyYlERETckwrLBRoZ1Y57h3YG4LEPt5B5oMjkRCIiIu5HhaUOPJYQweXhrSmvcnDPwk0UlpSbHUlERMStqLDUAQ+rhVdu6Uvn1n4cLirj/nfTKK/S8v0iIiJ1RYWljgQ08WJ+YgzNbJ5syjnG9KVbtXy/iIhIHVFhqUNdWjfl1Vv7YrHAoo25LPw+x+xIIiIibkGFpY4NC2/D48MjAHjqs22sz/7Z5EQiIiKuT4WlHtw7pDMjo0KwOwz+/F4auUdPmh1JRETEpamw1AOLxcJzf+jNJe0COHaykgnvbKK0vMrsWCIiIi5LhaWe2Lw8eCMxmsCmPuzIK+GRf/+IQ8v3i4iI1IoKSz1qG9CEeWP74eVh4auf8njtm91mRxIREXFJKiz1LDqsJc+M6gXAyyt2smxrnsmJREREXI8Ky0Uwpn8H7hjYEYCkxRlk5ZWYG0hERMTFqLBcJE9c14O4zq0orbBz9zsbOVZaYXYkERERl6HCcpF4eVj5+239CG3ZhNyjvzDxX+lU2R1mxxIREXEJKiwXUQs/b+YnxuDr7cG67J955ovtZkcSERFxCSosF1lEsD8zR0cBsGDdPj7YmGtuIBERERegwmKC4b2CmRzfDYC/LvmJtJxjJicSERFp2GpVWObMmUPHjh2x2WzExsayYcOGM45dsGABFoulxmaz2c44/r777sNisTBr1qzaRHMZD17RjYSeQVTYHdy7MI3DRb+YHUlERKTBcrqwLF68mKSkJKZPn056ejp9+vQhISGBgoKCMx7j7+/P4cOHq7ecnNP/ivEnn3zC999/T0hIiLOxXI7VamHm6Cgigptx5EQ59y5Mo6zSbnYsERGRBsnpwjJz5kwmTJjA+PHjiYyMZO7cufj6+vLWW2+d8RiLxUJwcHD1FhQU9LsxBw8eZNKkSbz33nt4eXk5G8sl+fl4Mj8xhua+Xmw5UMSUj7ZgGFq+X0RE5L85VVgqKipIS0sjPj7+txNYrcTHx7N+/fozHnfixAnCwsIIDQ1l5MiRbN26tcbjDoeDsWPH8uijj9KzZ89z5igvL6e4uLjG5qpCW/ry9z/1w8NqYUnGIeZ/u8fsSCIiIg2OU4XlyJEj2O32371DEhQURF7e6ZecDw8P56233mLp0qW8++67OBwOBg4cyIEDB6rHPPfcc3h6evLggw+eV47k5GQCAgKqt9DQUGem0eAM7BrItOsjAXj2qx2syjrzx2siIiKNUb1/SyguLo7ExESioqIYOnQoH3/8Ma1bt2bevHkApKWl8corr1TfnHs+pk6dSlFRUfWWm+v6Xw1OjAtjTEwoDgMmvb+Z7MITZkcSERFpMJwqLIGBgXh4eJCfn19jf35+PsHBwed1Di8vL/r27cvu3ad+ufjbb7+loKCADh064OnpiaenJzk5OTz88MN07NjxtOfw8fHB39+/xubqLBYLM0b1JDqsBSVlVUx4ZxPFZZVmxxIREWkQnCos3t7eREdHk5qaWr3P4XCQmppKXFzceZ3DbreTmZlJ27ZtARg7dixbtmwhIyOjegsJCeHRRx9l2bJlzsRzeT6eHsy9PZq2ATb2FJby0PubsTt0E66IiIinswckJSUxbtw4YmJiGDBgALNmzaK0tJTx48cDkJiYSLt27UhOTgZgxowZXHrppXTt2pXjx4/zwgsvkJOTw9133w1Aq1ataNWqVY2/w8vLi+DgYMLDwy90fi6ndTMf3hgbw81z17Eyq5AXl2fx+PAIs2OJiIiYyunCMmbMGAoLC5k2bRp5eXlERUWRkpJSfSPu/v37sVp/e+Pm2LFjTJgwgby8PFq0aEF0dDTr1q0jMjKy7mbhZi5pH8DzN/fmoUUZvL4qm4jgZoyMamd2LBEREdNYDDdY+KO4uJiAgACKiorc4n6WXyV/tZ15q/fg42nlw/sGckn7ALMjiYiI1BlnXr/1W0IN2GMJEQwLb015lYN7Fm6isKTc7EgiIiKmUGFpwDysFl65tS+dW/txuKiM+99No7xKy/eLiEjjo8LSwPnbvJifGEMzmyebco4xfelWLd8vIiKNjgqLC+jSuimv3toXiwUWbcxl4fen//FIERERd6XC4iKGhbep/nrzU59tY132EZMTiYiIXDwqLC7k3iGdGRUVgt1hMPG9dHKPnjQ7koiIyEWhwuJCLBYLz/6hN5e0C+DYyUomvLOJ0vIqs2OJiIjUOxUWF2Pz8uCNxGgCm/qwI6+ER/79Iw4t3y8iIm5OhcUFtQ1owryx/fDysPDVT3m89s1usyOJiIjUKxUWFxUd1pJnRvUC4OUVO0n5Kc/kRCIiIvVHhcWFjenfgTsGdgQg6YMMduQVmxtIRESknqiwuLgnruvBwC6tOFlhZ8I7mzhWWmF2JBERkTqnwuLivDyszPlTP0JbNiH36C9M/Fc6VXaH2bFERETqlAqLG2jh5838xBh8vT1Yl/0zz3yx3exIIiIidUqFxU1EBPszc3QUAAvW7eODjbnmBhIREalDKixuZHivYCbHdwPgiSWZpOUcNTmRiIhI3VBhcTMPXtGN4T2DqbQb3LswncNFv5gdSURE5IKpsLgZq9XCS6P7EBHcjCMnyrl3YRpllXazY4mIiFwQFRY35OfjyfzEGJr7erHlQBFTPtqCYWj5fhERcV0qLG4qtKUvf/9TPzysFpZkHOKNNXvMjiQiIlJrKixubGDXQKZdHwnAsyk7WJVVYHIiERGR2lFhcXOJcWHc0j8Uw4BJ728mu/CE2ZFEREScpsLi5iwWC0+N7El0WAtKyqqY8M4missqzY4lIiLiFBWWRsDH04O5t0fTNsDGnsJSHnp/M3aHbsIVERHXocLSSLRu5sMbY2Pw8bSyMquQF5ZlmR1JRETkvKmwNCKXtA/g+Zt7AzB3dTZLMw6anEhEROT8qLA0MiOj2nHf0C4APPbhFjIPFJmcSERE5NxUWBqhRxPCGRbemvIqB/cs3ERhSbnZkURERM5KhaUR8rBaeOXWvnRu7cfhojLufzeN8iot3y8iIg2XCksj5W/zYn5iDM1snmzKOcb0pVu1fL+IiDRYKiyNWJfWTXn11r5YLLBoYy4Lv88xO5KIiMhp1aqwzJkzh44dO2Kz2YiNjWXDhg1nHLtgwQIsFkuNzWazVT9eWVnJ448/ziWXXIKfnx8hISEkJiZy6NCh2kQTJw0Lb8OU4REAPPXZNtZlHzE5kYiIyO85XVgWL15MUlIS06dPJz09nT59+pCQkEBBwZl/p8bf35/Dhw9Xbzk5v/1L/uTJk6Snp/Pkk0+Snp7Oxx9/TFZWFiNGjKjdjMRp9wzpzKioEOwOg4nvpZN79KTZkURERGqwGE7euBAbG0v//v2ZPXs2AA6Hg9DQUCZNmsSUKVN+N37BggVMnjyZ48ePn/ffsXHjRgYMGEBOTg4dOnQ45/ji4mICAgIoKirC39//vP8e+U1ZpZ0/zl1P5sEiIoKb8dH9A/Hz8TQ7loiIuDFnXr+deoeloqKCtLQ04uPjfzuB1Up8fDzr168/43EnTpwgLCyM0NBQRo4cydatW8/69xQVFWGxWGjevLkz8eQC2Lw8eCMxmsCmPuzIK+HhD37EoeX7RUSkgXCqsBw5cgS73U5QUFCN/UFBQeTl5Z32mPDwcN566y2WLl3Ku+++i8PhYODAgRw4cOC048vKynj88ce59dZbz9i2ysvLKS4urrHJhWsb0IR5Y/vh7WElZWser32z2+xIIiIiwEX4llBcXByJiYlERUUxdOhQPv74Y1q3bs28efN+N7ayspLRo0djGAavv/76Gc+ZnJxMQEBA9RYaGlqfU2hUosNa8syoXgC8vGInKT+dvoiKiIhcTE4VlsDAQDw8PMjPz6+xPz8/n+Dg4PM6h5eXF3379mX37pr/ev+1rOTk5PD111+f9bOsqVOnUlRUVL3l5uY6Mw05h9H9Q7ljYEcAkj7IYEee3sESERFzOVVYvL29iY6OJjU1tXqfw+EgNTWVuLi48zqH3W4nMzOTtm3bVu/7tazs2rWLFStW0KpVq7Oew8fHB39//xqb1K0nruvBwC6tOFlhZ8I7mzhWWmF2JBERacSc/kgoKSmJ+fPn889//pPt27dz//33U1payvjx4wFITExk6tSp1eNnzJjB8uXL2bNnD+np6dx+++3k5ORw9913A6fKys0338ymTZt47733sNvt5OXlkZeXR0WFXiTN4uVhZc6f+hHasgm5R39h4r/SqbQ7zI4lIiKNlNPfWx0zZgyFhYVMmzaNvLw8oqKiSElJqb4Rd//+/Vitv/WgY8eOMWHCBPLy8mjRogXR0dGsW7eOyMhIAA4ePMinn34KQFRUVI2/a+XKlVx++eW1nJpcqBZ+3sxPjOGmv69jXfbP/O2L7fzviJ5mxxIRkUbI6XVYGiKtw1K/Un7K47530wB4/g+9Gd1fNzmLiMiFq7d1WKRxGt4rmMnx3QB4YkkmaTlHTU4kIiKNjQqLnJcHr+jG8J7BVNoN7l2YzuGiX8yOJCIijYgKi5wXq9XCS6P7EBHcjCMnyrnnnTTKKu1mxxIRkUZChUXOm5+PJ/MTY2jh60XmwSKmfLQFN7gFSkREXIAKizgltKUvc27rh4fVwpKMQ7yxZo/ZkUREpBFQYRGnDewSyLTrT30t/dmUHazKKjA5kYiIuDsVFqmVxLgwbukfimHApPc3k114wuxIIiLixlRYpFYsFgtPjexJdFgLSsqqmPDOJorLKs2OJSIibkqFRWrNx9ODubdH0zbAxp7CUh56fzN2h27CFRGRuqfCIhekdTMf3hgbg4+nlZVZhbywLMvsSCIi4oZUWOSCXdI+gOdv7g3A3NXZLM04aHIiERFxNyosUidGRrXjvqFdAHjswy1kHigyOZGIiLgTFRapM48mhDMsvDXlVQ7uWbiJgpIysyOJiIibUGGROuNhtfDKrX3p3NqPw0Vl3P9uOuVVWr5fREQunAqL1Cl/mxfzE2NoZvMkLecY05du1fL9IiJywVRYpM51ad2UV2/ti8UCizbmsvD7HLMjiYiIi1NhkXoxLLwNU4ZHAPDUZ9tYl33E5EQiIuLKVFik3twzpDOjokKwOwwmvpdO7tGTZkcSEREXpcIi9cZisfDsH3rTu30Ax05WMuGdTZSWV5kdS0REXJAKi9Qrm5cH88ZGE9jUhx15JTz8wY84tHy/iIg4SYVF6l3bgCbMG9sPbw8rKVvzeO2b3WZHEhERF6PCIhdFdFhLnhnVC4CXV+wk5ac8kxOJiIgrUWGRi2Z0/1DuGNgRgKQPMtiRV2xuIBERcRkqLHJRPXFdDwZ2acXJCjsT3tnEsdIKsyOJiIgLUGGRi8rLw8qcP/UjtGUTco/+wsR/pVNpd5gdS0REGjgVFrnoWvh5Mz8xBl9vD9Zl/8zfvthudiQREWngVFjEFBHB/swcHQXAgnX7+GBjrrmBRESkQVNhEdMM7xXM5PhuADyxJJO0nKMmJxIRkYZKhUVM9eAV3bimVzCVdoN7F6ZzuOgXsyOJiEgDpMIiprJaLbz4xz5EBDfjyIly7nknjbJKu9mxRESkgVFhEdP5+XgyPzGGFr5eZB4sYspHWzAMLd8vIiK/qVVhmTNnDh07dsRmsxEbG8uGDRvOOHbBggVYLJYam81mqzHGMAymTZtG27ZtadKkCfHx8ezatas20cRFhbb0Zc5t/fCwWliScYg31uwxO5KIiDQgTheWxYsXk5SUxPTp00lPT6dPnz4kJCRQUFBwxmP8/f05fPhw9ZaTk1Pj8eeff55XX32VuXPn8sMPP+Dn50dCQgJlZWXOz0hc1sAugUy7PhKAZ1N2sDLrzP9NiYhI4+J0YZk5cyYTJkxg/PjxREZGMnfuXHx9fXnrrbfOeIzFYiE4OLh6CwoKqn7MMAxmzZrFX//6V0aOHEnv3r155513OHToEEuWLKnVpMR1JcaFcUv/UAwDHnx/M9mFJ8yOJCIiDYBThaWiooK0tDTi4+N/O4HVSnx8POvXrz/jcSdOnCAsLIzQ0FBGjhzJ1q1bqx/bu3cveXl5Nc4ZEBBAbGzsWc8p7slisTBjZC9iwlpQUlbFhHc2UVxWaXYsERExmVOF5ciRI9jt9hrvkAAEBQWRl3f6X98NDw/nrbfeYunSpbz77rs4HA4GDhzIgQMHAKqPc+ac5eXlFBcX19jEfXh7Wnn99mjaBtjYU1jKQ+9vxu7QTbgiIo1ZvX9LKC4ujsTERKKiohg6dCgff/wxrVu3Zt68ebU+Z3JyMgEBAdVbaGhoHSaWhqB1Mx/eGBuDj6eVlVmFvLAsy+xIIiJiIqcKS2BgIB4eHuTn59fYn5+fT3Bw8Hmdw8vLi759+7J7926A6uOcOefUqVMpKiqq3nJztay7O7qkfQDP39wbgLmrs1macdDkRCIiYhanCou3tzfR0dGkpqZW73M4HKSmphIXF3de57Db7WRmZtK2bVsAOnXqRHBwcI1zFhcX88MPP5zxnD4+Pvj7+9fYxD2NjGrHfUO7APDYh1vIPFBkciIRETGD0x8JJSUlMX/+fP75z3+yfft27r//fkpLSxk/fjwAiYmJTJ06tXr8jBkzWL58OXv27CE9PZ3bb7+dnJwc7r77buDUTZaTJ0/mmWee4dNPPyUzM5PExERCQkIYNWpU3cxSXNqjCeEMC29NeZWDexZuoqBEX3cXEWlsPJ09YMyYMRQWFjJt2jTy8vKIiooiJSWl+qbZ/fv3Y7X+1oOOHTvGhAkTyMvLo0WLFkRHR7Nu3ToiIyOrxzz22GOUlpZyzz33cPz4cQYNGkRKSsrvFpiTxsnDauGVW/syas537Cks5f530/nXhFh8PD3MjiYiIheJxXCDNdCLi4sJCAigqKhIHw+5sezCE4ya8x0lZVXc0j+U5JsuwWKxmB1LRERqyZnXb/2WkLiMLq2b8uqtfbFYYNHGXN5Zn3Pug0RExC2osIhLGRbehinDIwCY8fk21mUfMTmRiIhcDCos4nLuGdKZUVEh2B0GE99LJ/foSbMjiYhIPVNhEZdjsVh49g+96d0+gGMnK5nwziZKy6vMjiUiIvVIhUVcks3Lg3ljowls6sOOvBIe/uBHHFq+X0TEbamwiMtqG9CEeWP74e1hJWVrHq9+s8vsSCIiUk9UWMSlRYe15JlRvQCYtWIXKT+d/gczRUTEtamwiMsb3T+UOwZ2BCDpgwx25OnXu0VE3I0Ki7iFJ67rwcAurThZYWfCO5s4VlphdiQREalDKiziFrw8rMz5Uz9CWzYh9+gvTPxXOpV2h9mxRESkjqiwiNto4efN/MQYfL09WJf9M3/7YrvZkUREpI6osIhbiQj2Z+boKAAWrNvH4o37zQ0kIiJ1QoVF3M7wXsFMju8GwF+X/ERazlGTE4mIyIVSYRG39OAV3bimVzCVdoN7F6Zz6PgvZkcSEZELoMIibslqtfDiH/sQEdyMIyfKueG1tSz4bi/lVXazo4mISC2osIjb8vPxZH5iDJ1b+/FzaQX/+9k2rnxpNZ9sPqBl/EVEXIzFMAyX/z93cXExAQEBFBUV4e/vb3YcaWAq7Q4Wb8zlldRdFJaUAxAR3IzHh0dweXhrLBaLyQlFRBonZ16/VVik0ThZUcXb3+1j7upsSspO/brzgE4teXx4BNFhLUxOJyLS+KiwiJzFsdIKXl+dzYJ1+6ioOrW43FWRQTyWEE63oGYmpxMRaTxUWETOw6Hjv/DKil38Oy0XhwFWC/yhX3v+clV3Qpo3MTueiIjbU2ERccLughJeWJbFsq35AHh7WhkXF8afL+9KCz9vk9OJiLgvFRaRWti8/xjPpezg+z2nFppr5uPJvUM7c+egTvh6e5qcTkTE/aiwiNSSYRis3lnIcylZbD9cDEBgUx8eurIrtwzogJeHVgIQEakrKiwiF8jhMPhsyyFeWr6T/UdPAhDWypeHrw7n+kvaYrXqq9AiIhdKhUWkjlRUOVi0cT+vpu7iyIkKAHqG+PP48AgGdwvUGi4iIhdAhUWkjpWWV/Hm2r28sWYPJ8pPreES17kVj18TQVRoc3PDiYi4KBUWkXpytLSCOSt3s3B9DhX2U2u4XNMrmEcSwunSuqnJ6UREXIsKi0g9O3DsJC9/vYuPNx/AMMDDamF0THseurI7wQE2s+OJiLgEFRaRiyQr79QaLiu2n1rDxcfTyh2XdeTPQ7sS4OtlcjoRkYZNhUXkItu07yjPpexg475jAPjbPLnv8i6MH9iJJt4eJqcTEWmYVFhETGAYBt/sKOD5lCyy8ksACPL34aEruzM6pj2eWsNFRKQGFRYRE9kdBkszDvLS8p0cPP4LAJ0D/XgkIZxregXrq9AiIv/hzOt3rf7JN2fOHDp27IjNZiM2NpYNGzac13GLFi3CYrEwatSoGvtPnDjBAw88QPv27WnSpAmRkZHMnTu3NtFETOdhtXBTv/Z888hQpl0fSUs/b/YcKeXP76Uzcs53fLf7iNkRRURcjtOFZfHixSQlJTF9+nTS09Pp06cPCQkJFBQUnPW4ffv28cgjjzB48ODfPZaUlERKSgrvvvsu27dvZ/LkyTzwwAN8+umnzsYTaTB8PD24c1An1jw2jIeu7IaftwdbDhRx2z9+YOybP/DTwSKzI4qIuAynPxKKjY2lf//+zJ49GwCHw0FoaCiTJk1iypQppz3GbrczZMgQ7rzzTr799luOHz/OkiVLqh/v1asXY8aM4cknn6zeFx0dzTXXXMMzzzxzzkz6SEhcwZET5cz+Zjfv/ZBDpf3U0+763m155OpwOgb6mZxOROTiq7ePhCoqKkhLSyM+Pv63E1itxMfHs379+jMeN2PGDNq0acNdd9112scHDhzIp59+ysGDBzEMg5UrV7Jz506uvvrq044vLy+nuLi4xibS0AU29eF/R/QkNelybuzbDosFPt9ymPiZq3nik0wKisvMjigi0mA5VViOHDmC3W4nKCioxv6goCDy8vJOe8zatWt58803mT9//hnP+9prrxEZGUn79u3x9vZm+PDhzJkzhyFDhpx2fHJyMgEBAdVbaGioM9MQMVWHVr68PCaKLyYNZlh4a6ocBu/9sJ+hL6zihWU7KPql0uyIIiINTr1+z7KkpISxY8cyf/58AgMDzzjutdde4/vvv+fTTz8lLS2Nl156iYkTJ7JixYrTjp86dSpFRUXVW25ubn1NQaTeRIb48/b4ASy+51L6dWjOL5V25qzMZugLK3ljTTZllXazI4qINBhO3cNSUVGBr68vH374YY1v+owbN47jx4+zdOnSGuMzMjLo27cvHh6/LZzlcJz6/RWr1UpWVhYhISEEBATwySefcN1111WPu/vuuzlw4AApKSnnzKV7WMTVGYbB19vyeWFZFrsKTgDQNsDGX+K7c1O/dlrDRUTcUr3dw+Lt7U10dDSpqanV+xwOB6mpqcTFxf1ufEREBJmZmWRkZFRvI0aMYNiwYWRkZBAaGkplZSWVlZVYrTWjeHh4VJcbEXdnsVi4umcwKZOH8PzNvQkJsHG4qIzHPtrC8Fe+JeWnPNxgySQRkVrzdPaApKQkxo0bR0xMDAMGDGDWrFmUlpYyfvx4ABITE2nXrh3JycnYbDZ69epV4/jmzZsDVO/39vZm6NChPProozRp0oSwsDBWr17NO++8w8yZMy9weiKu5dSPKIYyok8I736fw+yVu9ldcIL73k2jb4fmPD48gks7tzI7pojIRed0YRkzZgyFhYVMmzaNvLw8oqKiSElJqb4Rd//+/b97t+RcFi1axNSpU7nttts4evQoYWFh/O1vf+O+++5zNp6IW7B5eXD34M6M7h/KG6v38ObavWzef5xb3viey8Nb81hCBJEh+vhTRBoPLc0v4gIKist49ZtdLNqQS5XDwGKBkX1CSLoqnA6tfM2OJyJSK/otIRE3te9IKS99vZPPfjwEgJeHhT8N6MADV3SjdTMfk9OJiDhHhUXEzf10sIjnl2WxZmchAL7eHtw9qBMThnSmmc3L5HQiIudHhUWkkViXfYTnUrL4Mfc4AC39vJk4rCu3X9oBH0+Psx8sImIyFRaRRsQwDFJ+yuOF5VnsKSwFoF3zJiRd1Z1RfdvhYbWYnFBE5PRUWEQaoSq7gw/TDjBrxS7y/vO7ROFBzXg0IZwre7TBYlFxEZGGRYVFpBErq7SzYN0+/r5yN8VlVQDEhLVgyjURxHRsaXI6EZHfqLCICEUnK5m7Jpu3v9tLWeWpVaPje7ThkYRwIoL1PBER86mwiEi1/OIyZq3YxQebcrH/Zw2XG/u2I+mq7rRvoTVcRMQ8Kiwi8jvZhSeYuXwnX2QeBsDbw8ptl3bggWFdadVUa7iIyMWnwiIiZ/Rj7nGeX7aD73b/DEBTH08mDO7M3YM74efj9K91iIjUmgqLiJzTt7sKeS5lBz8dLAYgsKk3k67oxq0DOuDt6dzvgYmI1IYKi4icF4fD4MufDvPisiz2/XwSgNCWTXj4qnBG9AnBqjVcRKQeqbCIiFMq7Q4Wb8zlldRdFJaUA9CjrT+PDQ/n8u6ttYaLiNQLFRYRqZWTFVW8/d0+5q7OpuQ/a7jEdmrJ49dE0K9DC5PTiYi7UWERkQtyrLSC11dns2DdPiqqTq3hcnVkEI8ND6drm2YmpxMRd6HCIiJ14tDxX5i1Yicfph3AYYDVAn/o156/XNWdkOZNzI4nIi5OhUVE6tTughJeWJbFsq35AHh7WhkXF8afL+9KCz9vk9OJiKtSYRGRepG+/xjPfbWDH/YeBaCZjyf3Xd6F8Zd1xNdba7iIiHNUWESk3hiGweqdhTyXksX2w6fWcGndzIcHr+zGLf1D8fLQGi4icn5UWESk3jkcBp9tOcRLy3ey/+ipNVw6tvLl4avDue6StlrDRUTOSYVFRC6aiioHizbu59XUXRw5UQFAr3b+PJYQweBugVrDRUTOSIVFRC660vIq3ly7lzfW7OFE+ak1XAZ2acXjwyPoE9rc3HAi0iCpsIiIaX4+Uc6cldm8+30OFfZTa7hce0kwD18dTpfWTU1OJyINiQqLiJjuwLGTvPz1Lj7efADDAA+rhdEx7Xnoyu4EB9jMjiciDYAKi4g0GFl5p9ZwWbH91BouPp5Wxl/WifuHdiHA18vkdCJiJhUWEWlwNu07ynMpO9i47xgA/jZP/jysK3cM7IjNy8PkdCJiBhUWEWmQDMPgmx0FPJ+SRVZ+CQBB/j5Mju/OH6Pb46k1XEQaFRUWEWnQ7A6DpRkHeWn5Tg4e/wWAzq39ePTqcIb3CtZXoUUaCRUWEXEJ5VV23vt+P7NX7uZo6ak1XPq0D+Dx4REM7BpocjoRqW8qLCLiUkrKKvnHt3v5x7d7KK2wAzC4WyCPD4+gV7sAk9OJSH1RYRERl3TkRDmzv9nNez/kUGk/9b+m63u35ZGrw+kY6GdyOhGpa868ftfqDrc5c+bQsWNHbDYbsbGxbNiw4byOW7RoERaLhVGjRv3use3btzNixAgCAgLw8/Ojf//+7N+/vzbxRMRFBTb14X9H9CQ16XJu7NsOiwU+33KY+Jmr+d9Pt1J0stLsiCJiEqcLy+LFi0lKSmL69Omkp6fTp08fEhISKCgoOOtx+/bt45FHHmHw4MG/eyw7O5tBgwYRERHBqlWr2LJlC08++SQ2mxaXEmmMOrTy5eUxUXwxaTDDwltT5TBYsG4fQ19cyTvr91H1nxV0RaTxcPojodjYWPr378/s2bMBcDgchIaGMmnSJKZMmXLaY+x2O0OGDOHOO+/k22+/5fjx4yxZsqT68VtuuQUvLy8WLlxYq0noIyER9/bd7iPM+Gxb9Vehuwc15cnrIxncrbXJyUTkQtTbR0IVFRWkpaURHx//2wmsVuLj41m/fv0Zj5sxYwZt2rThrrvu+t1jDoeDL774gu7du5OQkECbNm2IjY2tUWj+W3l5OcXFxTU2EXFfl3UN5IsHB/H0qF608PViZ/4Jxr65gbv/uZG9R0rNjiciF4FTheXIkSPY7XaCgoJq7A8KCiIvL++0x6xdu5Y333yT+fPnn/bxgoICTpw4wbPPPsvw4cNZvnw5N954IzfddBOrV68+7THJyckEBARUb6Ghoc5MQ0RckKeHlbGXhrHqkWHceVknPK0WVmwv4OqXV/O3L7ZRXKb7W0TcWb0uK1lSUsLYsWOZP38+gYGnX1PB4Tj1WfTIkSP5y1/+QlRUFFOmTOH6669n7ty5pz1m6tSpFBUVVW+5ubn1NgcRaVgCfL2YdkMkKZOHMCy8NZV2g/nf7mXYC6v41w/7sTtc/ouPInIans4MDgwMxMPDg/z8/Br78/PzCQ4O/t347Oxs9u3bxw033FC979eC4unpSVZWFqGhoXh6ehIZGVnj2B49erB27drT5vDx8cHHx8eZ6CLiZrq2acrb4wewKquApz/fRnZhKf/zSSYLv89h2vWRxHVpZXZEEalDTr3D4u3tTXR0NKmpqdX7HA4HqampxMXF/W58REQEmZmZZGRkVG8jRoxg2LBhZGRkEBoaire3N/379ycrK6vGsTt37iQsLKyW0xKRxuLy8DakTB7C9Bsi8bd5sv1wMbfO/577Fqax/+eTZscTkTri1DssAElJSYwbN46YmBgGDBjArFmzKC0tZfz48QAkJibSrl07kpOTsdls9OrVq8bxzZs3B6ix/9FHH2XMmDEMGTKEYcOGkZKSwmeffcaqVatqPzMRaTS8PKyMv6wTo6La8fKKnbz3w35StubxzY4C7hrciYnDutLUx+n/3YlIA+L0M3jMmDEUFhYybdo08vLyiIqKIiUlpfpG3P3792O1OndrzI033sjcuXNJTk7mwQcfJDw8nI8++ohBgwY5G09EGrEWft7MGNmL2y8N4+nPt/HtriO8viqbD9MO8GhCODf3a4/Vqh9WFHFFWppfRNySYRikbi/gb19ur/7qc692/ky/oSf9O7Y0OZ2IgH5LyOw4ItKAVFQ5+Oe6fbyauouS8ioAruvdlqnXRNC+ha/J6UQaNxUWEZH/cuREOS8t38nijftxGODjaeWeIZ25//Iu+Hrr/hYRM6iwiIicwbZDxcz4fCvf7zkKQJC/D48Pj2BUVDvd3yJykamwiIichWEYLNuax9++3E7u0V8AiAptzrQbIunXoYXJ6UQaDxUWEZHzUFZp563v9jLnm92UVtgBGBUVwuPXRNA2oInJ6UTcnwqLiIgTCorLeGFZFh+mH8AwoImXB/cN7cI9QzrTxNvD7HgibkuFRUSkFjIPFPHUZ1vZlHMMgJAAG1Ou7cENvdtisej+FpG6psIiIlJLhmHw+ZbDPPvVDg4eP3V/S0xYC6bdEEnv9s3NDSfiZlRYREQuUFmlnTfW7OH1Vdn8UmnHYoE/9GvPYwnhtPG3mR1PxC2osIiI1JG8ojKeT9nBx5sPAuDn7cGfh3XlrkGdsHnp/haRC6HCIiJSxzbvP8aMz7exef9xAEJbNuF/runB8F7Bur9FpJZUWERE6oHDYfDpj4d49qsd5BWXARDbqSXTboikZ0iAyelEXI8Ki4hIPTpZUcXc1XuYtzqb8ioHFgvc0j+Uh68OJ7Cpj9nxRFyGCouIyEVw8PgvPPvVDj778RAAzXw8mXRlV+4Y2AlvT6vJ6UQaPhUWEZGLaOO+o8z4bBuZB4sA6NjKlyeuiyS+Rxvd3yJyFiosIiIXmcNh8FH6AZ5flkVhSTkAg7oG8uT1kYQHNzM5nUjDpMIiImKSE+VV/H3lbv6xdi8VVQ6sFvhTbAeSrgqnpZ+32fFEGhQVFhERk+UePcn/+3I7X/2UB4C/zZOH4ruTGBeGl4fubxEBFRaz44iIVFuf/TMzPt/G9sPFAHRu7ceT10UyLKKNyclEzKfCIiLSgNgdBh9syuXFZVn8XFoBwNDurXny+h50baP7W6TxUmEREWmAissqmf3Nbt7+bi+VdgMPq4Wxl4YxOb4bzX11f4s0PiosIiIN2N4jpfzti+2s2J4PQHNfL5Ku6s6fBnTAU/e3SCOiwiIi4gLW7jrCjM+3sjP/BADdg5ry5PWRDO7W2uRkIheHCouIiIuosjt4f8N+Zn69k2MnKwGI79GGJ66LpFOgn8npROqXCouIiIspOlnJrNSdLFyfQ5XDwMvDwh0DO/LAFd0IaOJldjyReqHCIiLionYXnOCZL7axKqsQgFZ+3iRd3Z1b+nfAw6pl/sW9qLCIiLi4lVkFPPP5NrILSwHo0dafaddHEtellcnJROqOCouIiBuotDtYuD6HWSt2UlxWBcDwnsH8z7U96NDK1+R0IhdOhUVExI0cLa3g5a938t4POTgM8PawctfgTkwc1pWmPp5mxxOpNRUWERE3lJVXwtOfb2Pt7iMAtG7mw6MJ4dzcrz1W3d8iLkiFRUTETRmGQer2Ap75Yhv7fj4JQK92/ky/oSf9O7Y0OZ2Ic5x5/a7Vkopz5syhY8eO2Gw2YmNj2bBhw3kdt2jRIiwWC6NGjTrjmPvuuw+LxcKsWbNqE01ExK1ZLBbiI4NY/pehPHFtD5r5ePLTwWL+OHc9E/+VzoFjJ82OKFIvnC4sixcvJikpienTp5Oenk6fPn1ISEigoKDgrMft27ePRx55hMGDB59xzCeffML3339PSEiIs7FERBoVb08rE4Z0ZuWjl3PrgA5YLPDFlsNc+dJqXlqexcmKKrMjitQppwvLzJkzmTBhAuPHjycyMpK5c+fi6+vLW2+9dcZj7HY7t912G0899RSdO3c+7ZiDBw8yadIk3nvvPby8tEiSiMj5CGzqQ/JNl/DFpMFc2rkl5VUOXvtmN8NeXMXH6QdwOFz+U38RwMnCUlFRQVpaGvHx8b+dwGolPj6e9evXn/G4GTNm0KZNG+66667TPu5wOBg7diyPPvooPXv2PGeO8vJyiouLa2wiIo1ZZIg/70+4lLm39yO0ZRPyi8tJ+uBHbnp9Hen7j5kdT+SCOVVYjhw5gt1uJygoqMb+oKAg8vLyTnvM2rVrefPNN5k/f/4Zz/vcc8/h6enJgw8+eF45kpOTCQgIqN5CQ0PPfxIiIm7KYrEwvFdbvv7LUB4bHo6ftwcZuce56e/rmLxoM4eLfjE7okit1evvmJeUlDB27Fjmz59PYGDgacekpaXxyiuvsGDBAiyW8/ta3tSpUykqKqrecnNz6zK2iIhLs3l58OfLu7Lykcv5Y3R7LBZYknGIK15czSsrdvFLhd3siCJOc+przRUVFfj6+vLhhx/W+KbPuHHjOH78OEuXLq0xPiMjg759++Lh4VG9z+FwAKc+SsrKyuKzzz4jKSkJq/W37mS327FarYSGhrJv375z5tLXmkVEzizzQBFPfbaVTTmnPhoKCbAx5doe3NC77Xn/Q1GkPtTrOiyxsbEMGDCA1157DThVQDp06MADDzzAlClTaowtKytj9+7dNfb99a9/paSkhFdeeYXu3btTUlLC4cOHa4xJSEhg7NixjB8/nvDw8HNmUmERETk7wzD4fMthnv1qBwePn/poKCasBdNuiKR3++bmhpNGy5nXb6fXdE5KSmLcuHHExMQwYMAAZs2aRWlpKePHjwcgMTGRdu3akZycjM1mo1evXjWOb968OUD1/latWtGqVc0f8/Ly8iI4OPi8yoqIiJybxWLhhj4hXBUZxBtr9vD6qmw25RxjxOzvuDm6PY8lhNPG32Z2TJEzcrqwjBkzhsLCQqZNm0ZeXh5RUVGkpKRU34i7f//+Gh/viIhIw2Hz8uDBK7vxx5j2PJ+SxSebD/Jh2gG+zDzMxGFduWtQJ2xeHuc+kchFpqX5RUQasfT9x5jx2TYyco8D0L5FE/7n2h5c0ytY97dIvdNvCYmIyHlzOAyW/niQ577KIq+4DIDYTi2ZdkMkPUMCTE4n7kyFRUREnHayooq5q7KZt2YP5VUOLBYYExPKw1eH07qZj9nxxA2psIiISK0dOHaSZ7/awedbTn2Ds6mPJw9e2ZU7BnbC21P3KErdUWEREZELtnHfUWZ8to3Mg0UAdGzlyxPXRRLfo43ub5E6ocIiIiJ1wuEw+DD9AC8sy6KwpByAQV0DefL6SMKDm5mcTlydCouIiNSpE+VVzFm5mze/3UuF3YHVAn+K7UDSVeG09PM2O564KBUWERGpF/t/Psn/+3I7KVtP/eCtv82Th+K7kxgXhpeH7m8R56iwiIhIvVqf/TMzPt/G9sPFAHRu7ceT10UyLKKNycnElaiwiIhIvbM7DBZvzOWl5Vn8XFoBwNDurXny+h50baP7W+TcVFhEROSiKS6rZPY3u3n7u71U2g08rBbGXhrG5PhuNPfV/S1yZiosIiJy0e09UsrfvtjOiu35ADT39SLpqu78aUAHPHV/i5yGCouIiJjm212FPP35NnbmnwCge1BTnrw+ksHdWpucTBoaFRYRETFVld3B+xv2M/PrnRw7WQnAyKgQpl0fSaumWuZfTnHm9Vvv0YmISJ3z9LAyNq4jqx4Zxh0DO2K1wNKMQ1z18hqWZhzEDf6tLBeZCouIiNSbAF8v/ndETz7+82WEBzXjaGkFDy3K4M4FGzl0/Bez44kLUWEREZF6FxXanM8mDSLpqu54e1hZmVXIVTNX8876fTgcerdFzk2FRURELgpvTysPXtmNLx4cRHRYC0or7ExbupXR89azu+CE2fGkgVNhERGRi6pbUDP+fW8cT43oiZ+3B5tyjnHtK98y+5tdVNodZseTBkqFRURELjqr1cK4gR1ZnjSUy8NbU2F38OLyndzw2lq2HDhudjxpgFRYRETENO2aN+HtO/rz8pg+tPD1YkdeCaPmfMffvtjGLxV2s+NJA6LCIiIiprJYLNzYtz0rkoYyok8IDgPmf7uXhFlrWLf7iNnxpIFQYRERkQahVVMfXr21L2/dEUPbABv7j57kT//4gcc/3ELRfxafk8ZLhUVERBqUKyKCWP6XIdx+aQcAFm/KJf7l1aT8dNjkZGImFRYREWlwmtm8eGbUJXxwbxydA/0oLCnnvnfTuW9hGgXFZWbHExOosIiISIM1oFNLvnxoMBOHdcHTaiFlax7xM1ezeON+Le/fyKiwiIhIg2bz8uDRhAg+fWAQl7QLoLisisc/yuS2f/xAzs+lZseTi0SFRUREXEJkiD+f/Hkg/3NtBDYvK+uyfyZh1hrmr9lDlRacc3sqLCIi4jI8PazcM6QLyyYPIa5zK8oqHfzty+3c9Po6th8uNjue1CMVFhERcTlhrfz414RYnvvDJTSzebLlQBE3vLaWF5dlUVapBefckQqLiIi4JIvFwpj+HViRNJSEnkFUOQxmr9zNda9+y8Z9R82OJ3VMhUVERFxakL+NeWNjeP22fgQ29SG7sJQ/zl3Pk0t+oqRMC865i1oVljlz5tCxY0dsNhuxsbFs2LDhvI5btGgRFouFUaNGVe+rrKzk8ccf55JLLsHPz4+QkBASExM5dOhQbaKJiEgjdc0lbUlNGsromPYALPw+h6tfXsM3O/JNTiZ1wenCsnjxYpKSkpg+fTrp6en06dOHhIQECgoKznrcvn37eOSRRxg8eHCN/SdPniQ9PZ0nn3yS9PR0Pv74Y7KyshgxYoSz0UREpJEL8PXi+Zv78N7dsXRo6cvhojLuXLCJhxZt5ucT5WbHkwtgMZxceSc2Npb+/fsze/ZsABwOB6GhoUyaNIkpU6ac9hi73c6QIUO48847+fbbbzl+/DhLliw549+xceNGBgwYQE5ODh06dDhnpuLiYgICAigqKsLf39+Z6YiIiJs6WVHFzOU7eeu7vTgMaOHrxfQbejIyKgSLxWJ2PMG512+n3mGpqKggLS2N+Pj4305gtRIfH8/69evPeNyMGTNo06YNd91113n9PUVFRVgsFpo3b37ax8vLyykuLq6xiYiI/F++3p789fpIPvnzZUQEN+PYyUomL87gzgUbOXj8F7PjiZOcKixHjhzBbrcTFBRUY39QUBB5eXmnPWbt2rW8+eabzJ8//7z+jrKyMh5//HFuvfXWM7at5ORkAgICqrfQ0FBnpiEiIo1In9DmfPrAIB6+qjveHlZWZhVy9czVvLN+Hw6Hlvd3FfX6LaGSkhLGjh3L/PnzCQwMPOf4yspKRo8ejWEYvP7662ccN3XqVIqKiqq33NzcuowtIiJuxtvTyqQru/HlQ4OICWtBaYWdaUu3MnreenYXnDA7npwHT2cGBwYG4uHhQX5+zTuu8/PzCQ4O/t347Oxs9u3bxw033FC9z+E4tXyyp6cnWVlZdOnSBfitrOTk5PDNN9+c9bMsHx8ffHx8nIkuIiJC1zbN+ODeON79IYfnvtrBppxjXPvKt0y6oiv3Du2Ct6dW+2ionLoy3t7eREdHk5qaWr3P4XCQmppKXFzc78ZHRESQmZlJRkZG9TZixAiGDRtGRkZG9Uc5v5aVXbt2sWLFClq1anWB0xIRETk9q9VCYlxHlicN5fLw1lTYHbz09U5GzF7Lj7nHzY4nZ+DUOywASUlJjBs3jpiYGAYMGMCsWbMoLS1l/PjxACQmJtKuXTuSk5Ox2Wz06tWrxvG/3kj76/7Kykpuvvlm0tPT+fzzz7Hb7dX3w7Rs2RJvb+8LmZ+IiMhptWvehLfv6M/SjEM89dlWduSVcOPfv+POyzqRdHV3fL2dfomUeuT01RgzZgyFhYVMmzaNvLw8oqKiSElJqb4Rd//+/Vit5//GzcGDB/n0008BiIqKqvHYypUrufzyy52NKCIicl4sFguj+rZjcLdAZny+jaUZh/jH2r0s25bHszf15rKu577/Ui4Op9dhaYi0DouIiNSFb3bk89dPfuJQURkAo2Pa88S1kQT4epmczD3V2zosIiIi7uyKiCCWJw0lMS4MgA82HSD+5dV8lXnY5GSiwiIiIvJ/NPXxZMbIXnx4XxxdWvtRWFLO/e+lc+/CTRQUl5kdr9FSYRERETmNmI4t+eLBwTwwrCueVgvLtuZz5czVLNqwHze4m8LlqLCIiIicgc3Lg0cSwvn0gUH0bh9ASVkVUz7O5E/zf2DfkVKz4zUqKiwiIiLnEBniz8f3D+SJa3tg87Kyfs/PDH9lDW+syabK7jA7XqOgwiIiInIePD2sTBjSmWWThzCwSyvKKh38vy93cNPr69h2SD/CW99UWERERJwQ1sqP9+6O5bk/XEIzmydbDhQxYvZaXli2g7JKu9nx3JYKi4iIiJMsFgtj+ncgNWkow3sGU+UwmLMym2tf/ZaN+46aHc8tqbCIiIjUUht/G3PHRjP39n60bubDnsJS/jh3PU8u+YmSskqz47kVFRYREZELNLxXW1b8ZShjYk79qO/C73O4+uU1fLMj3+Rk7kOFRUREpA4E+Hrx3M29ee/uWDq09OVwURl3LtjEg+9v5ucT5WbHc3kqLCIiInXosq6BLJs8hAmDO2G1wKc/HiJ+5mqWbD6oBecugAqLiIhIHWvi7cET10XyyZ8vIyK4GcdOVjJ5cQbjF2zk4PFfzI7nklRYRERE6kmf0OZ8NmkQj1zdHW8PK6uyCrl65mr+uW4fDofebXGGCouIiEg98vKw8sAV3fjyocHEhLWgtMLO9E+38sd569ldUGJ2PJehwiIiInIRdG3TlA/ujePpkT3x8/YgLecY176ylldTd1FRpeX9z0WFRURE5CKxWi2MjevI8qShDAtvTYXdwcyvdzJi9loyco+bHa9BU2ERERG5yNo1b8Jbd/TnlVuiaOnnzY68Em76+3c8/fk2TlZUmR2vQVJhERERMYHFYmFkVDtWJA1lVFQIDgPeXLuXhFlr+G73EbPjNTgqLCIiIiZq6efNrFv68vYd/QkJsJF79Bdu+8cPPPrvHyk6qeX9f6XCIiIi0gAMi2jD8qShJMaFYbHAv9MOcOXM1XyVedjsaA2CCouIiEgD0dTHkxkje/Hve+Po0tqPIyfKuf+9dO5duIn84jKz45lKhUVERKSBienYki8eHMykK7riabWwbGs+8TNXs2jD/ka7vL8Ki4iISANk8/Lg4avD+WzSIPq0D6CkrIopH2fyp/k/sO9IqdnxLjoVFhERkQasR1t/Pv7zZfz1uh7YvKys3/MzCbPWMG91NlX2xrPgnAqLiIhIA+dhtXD34M4snzyUy7q2orzKQfJXO7jx7+vYdqjY7HgXhQqLiIiIi+jQypd374rl+Zt742/zJPNgESNmr+WFZTsoq7SbHa9eqbCIiIi4EIvFwuiYUFYkDeWaXsFUOQzmrMzm2le/ZcPeo2bHqzcqLCIiIi6ojb+N12+PZu7t0bRu5sOewlJGz1vPX5dkUlLmfgvOqbCIiIi4sOG9glmRNJRb+ocC8O73+7n65TWkbs83OVndUmERERFxcQFNvHj2D735192xdGjpy+GiMu765yYmvb+ZIyfKzY5XJ2pVWObMmUPHjh2x2WzExsayYcOG8zpu0aJFWCwWRo0aVWO/YRhMmzaNtm3b0qRJE+Lj49m1a1dtoomIiDRaA7sGsmzyEO4Z0hmrBT778RBXzVzNJ5sPuPyCc04XlsWLF5OUlMT06dNJT0+nT58+JCQkUFBQcNbj9u3bxyOPPMLgwYN/99jzzz/Pq6++yty5c/nhhx/w8/MjISGBsrLGvQyxiIiIs5p4e/A/1/ZgycTLiAhuxrGTlfxl8Y+MX7CRA8dOmh2v1iyGk5UrNjaW/v37M3v2bAAcDgehoaFMmjSJKVOmnPYYu93OkCFDuPPOO/n22285fvw4S5YsAU69uxISEsLDDz/MI488AkBRURFBQUEsWLCAW2655ZyZiouLCQgIoKioCH9/f2emIyIi4rYq7Q7eWLOHV1J3UVHlwNfbg8cSwkmM64jVajE7nlOv3069w1JRUUFaWhrx8fG/ncBqJT4+nvXr15/xuBkzZtCmTRvuuuuu3z22d+9e8vLyapwzICCA2NjYM56zvLyc4uLiGpuIiIjU5OVhZeKwrnz54GD6d2zByQo7//vZNm6eu45d+SVmx3OKU4XlyJEj2O12goKCauwPCgoiLy/vtMesXbuWN998k/nz55/28V+Pc+acycnJBAQEVG+hoaHOTENERKRR6dqmKYvviePpkT3x8/Ygff9xrnt1La/+550XV1Cv3xIqKSlh7NixzJ8/n8DAwDo779SpUykqKqrecnNz6+zcIiIi7shqtTA2riNfJw3liog2VNgdzPx6Jze8tpaM3ONmxzsnT2cGBwYG4uHhQX5+ze925+fnExwc/Lvx2dnZ7Nu3jxtuuKF6n8Nxqsl5enqSlZVVfVx+fj5t27atcc6oqKjT5vDx8cHHx8eZ6CIiIgKENG/Cm+Ni+PTHQzz12Tay8ku46e/fMf6yTjx8dXd8vZ2qBheNU++weHt7Ex0dTWpqavU+h8NBamoqcXFxvxsfERFBZmYmGRkZ1duIESMYNmwYGRkZhIaG0qlTJ4KDg2ucs7i4mB9++OG05xQREZELY7FYGBnVjhVJQ7mxbzscBry5di8Js9awdtcRs+OdltM1KikpiXHjxhETE8OAAQOYNWsWpaWljB8/HoDExETatWtHcnIyNpuNXr161Ti+efPmADX2T548mWeeeYZu3brRqVMnnnzySUJCQn63XouIiIjUnZZ+3rw8JooRUSH89ZOfyD36C7e/+QN/jG7PX6+LJMDXy+yI1ZwuLGPGjKGwsJBp06aRl5dHVFQUKSkp1TfN7t+/H6vVuVtjHnvsMUpLS7nnnns4fvw4gwYNIiUlBZvN5mw8ERERcdKw8DYs+8sQXlyWxT/X7+PfaQdYmVXIjJE9uaZXMBaL+V+BdnodloZI67CIiIjUjbScozz24RayC0sBuDoyiKdH9SLIv+7fRKi3dVhERETEvUWHteTLhwbz4BVd8bRaWL4tn/iZq3l/w35Tl/dXYREREZEafDw9SLo6nM8fHESf9gGUlFXx5JKf2PezeUv7N8zvLomIiIjpIoL9+fjPl/H2d3spr3LQKdDPtCwqLCIiInJGHlYLdw/ubHYMfSQkIiIiDZ8Ki4iIiDR4KiwiIiLS4KmwiIiISIOnwiIiIiINngqLiIiINHgqLCIiItLgqbCIiIhIg6fCIiIiIg2eCouIiIg0eCosIiIi0uCpsIiIiEiDp8IiIiIiDZ5b/FqzYRgAFBcXm5xEREREztevr9u/vo6fjVsUlpKSEgBCQ0NNTiIiIiLOKikpISAg4KxjLMb51JoGzuFwcOjQIZo1a4bFYqnTcxcXFxMaGkpubi7+/v51eu6GwN3nB+4/R83P9bn7HN19fuD+c6yv+RmGQUlJCSEhIVitZ79LxS3eYbFarbRv375e/w5/f3+3/I/wV+4+P3D/OWp+rs/d5+ju8wP3n2N9zO9c76z8SjfdioiISIOnwiIiIiINngrLOfj4+DB9+nR8fHzMjlIv3H1+4P5z1Pxcn7vP0d3nB+4/x4YwP7e46VZERETcm95hERERkQZPhUVEREQaPBUWERERafBUWERERKTBU2EB5syZQ8eOHbHZbMTGxrJhw4azjv/3v/9NREQENpuNSy65hC+//PIiJa0dZ+a3YMECLBZLjc1ms13EtM5Zs2YNN9xwAyEhIVgsFpYsWXLOY1atWkW/fv3w8fGha9euLFiwoN5zXghn57hq1arfXUOLxUJeXt7FCeyk5ORk+vfvT7NmzWjTpg2jRo0iKyvrnMe5yvOwNvNzpefh66+/Tu/evasXFIuLi+Orr7466zGucu1+5ewcXen6nc6zzz6LxWJh8uTJZx13sa9joy8sixcvJikpienTp5Oenk6fPn1ISEigoKDgtOPXrVvHrbfeyl133cXmzZsZNWoUo0aN4qeffrrIyc+Ps/ODUysZHj58uHrLycm5iImdU1paSp8+fZgzZ855jd+7dy/XXXcdw4YNIyMjg8mTJ3P33XezbNmyek5ae87O8VdZWVk1rmObNm3qKeGFWb16NRMnTuT777/n66+/prKykquvvprS0tIzHuNKz8PazA9c53nYvn17nn32WdLS0ti0aRNXXHEFI0eOZOvWracd70rX7lfOzhFc5/r9t40bNzJv3jx69+591nGmXEejkRswYIAxceLE6j/b7XYjJCTESE5OPu340aNHG9ddd12NfbGxsca9995brzlry9n5vf3220ZAQMBFSle3AOOTTz4565jHHnvM6NmzZ419Y8aMMRISEuoxWd05nzmuXLnSAIxjx45dlEx1raCgwACM1atXn3GMqz0P/6/zmZ8rPw8NwzBatGhh/OMf/zjtY6587f6vs83RVa9fSUmJ0a1bN+Prr782hg4dajz00ENnHGvGdWzU77BUVFSQlpZGfHx89T6r1Up8fDzr168/7THr16+vMR4gISHhjOPNVJv5AZw4cYKwsDBCQ0PP+a8IV+NK1+9CRUVF0bZtW6666iq+++47s+Oct6KiIgBatmx5xjGufB3PZ37gms9Du93OokWLKC0tJS4u7rRjXPnawfnNEVzz+k2cOJHrrrvud9fndMy4jo26sBw5cgS73U5QUFCN/UFBQWf8vD8vL8+p8WaqzfzCw8N56623WLp0Ke+++y4Oh4OBAwdy4MCBixG53p3p+hUXF/PLL7+YlKputW3blrlz5/LRRx/x0UcfERoayuWXX056errZ0c7J4XAwefJkLrvsMnr16nXGca70PPy/znd+rvY8zMzMpGnTpvj4+HDffffxySefEBkZedqxrnrtnJmjq10/gEWLFpGenk5ycvJ5jTfjOrrFrzVL3YmLi6vxr4aBAwfSo0cP5s2bx9NPP21iMjlf4eHhhIeHV/954MCBZGdn8/LLL7Nw4UITk53bxIkT+emnn1i7dq3ZUerF+c7P1Z6H4eHhZGRkUFRUxIcffsi4ceNYvXr1GV/QXZEzc3S165ebm8tDDz3E119/3aBvDm7UhSUwMBAPDw/y8/Nr7M/Pzyc4OPi0xwQHBzs13ky1md9/8/Lyom/fvuzevbs+Il50Z7p+/v7+NGnSxKRU9W/AgAENvgQ88MADfP7556xZs4b27dufdawrPQ9/5cz8/ltDfx56e3vTtWtXAKKjo9m4cSOvvPIK8+bN+91YV7x24Nwc/1tDv35paWkUFBTQr1+/6n12u501a9Ywe/ZsysvL8fDwqHGMGdexUX8k5O3tTXR0NKmpqdX7HA4HqampZ/xsMi4ursZ4gK+//vqsn2WapTbz+292u53MzEzatm1bXzEvKle6fnUpIyOjwV5DwzB44IEH+OSTT/jmm2/o1KnTOY9xpetYm/n9N1d7HjocDsrLy0/7mCtdu7M52xz/W0O/fldeeSWZmZlkZGRUbzExMdx2221kZGT8rqyASdex3m7ndRGLFi0yfHx8jAULFhjbtm0z7rnnHqN58+ZGXl6eYRiGMXbsWGPKlCnV47/77jvD09PTePHFF43t27cb06dPN7y8vIzMzEyzpnBWzs7vqaeeMpYtW2ZkZ2cbaWlpxi233GLYbDZj69atZk3hrEpKSozNmzcbmzdvNgBj5syZxubNm42cnBzDMAxjypQpxtixY6vH79mzx/D19TUeffRRY/v27cacOXMMDw8PIyUlxawpnJOzc3z55ZeNJUuWGLt27TIyMzONhx56yLBarcaKFSvMmsJZ3X///UZAQICxatUq4/Dhw9XbyZMnq8e48vOwNvNzpefhlClTjNWrVxt79+41tmzZYkyZMsWwWCzG8uXLDcNw7Wv3K2fn6ErX70z++1tCDeE6NvrCYhiG8dprrxkdOnQwvL29jQEDBhjff/999WNDhw41xo0bV2P8Bx98YHTv3t3w9vY2evbsaXzxxRcXObFznJnf5MmTq8cGBQUZ1157rZGenm5C6vPz61d4/3v7dU7jxo0zhg4d+rtjoqKiDG9vb6Nz587G22+/fdFzO8PZOT733HNGly5dDJvNZrRs2dK4/PLLjW+++cac8OfhdHMDalwXV34e1mZ+rvQ8vPPOO42wsDDD29vbaN26tXHllVdWv5Abhmtfu185O0dXun5n8t+FpSFcR4thGEb9vX8jIiIicuEa9T0sIiIi4hpUWERERKTBU2ERERGRBk+FRURERBo8FRYRERFp8FRYREREpMFTYREREZEGT4VFREREGjwVFhEREWnwVFhERESkwVNhERERkQZPhUVEREQavP8PrMJZJAtNImUAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load LoRA\n",
        "\n",
        "Use the code below if you shared LoRA weights. It's much more lightweight than the model files themselves - for instance, a LoRA rank 4 weights file for a 10gb model might only be on the order of a few megabytes, easily shared over email."
      ],
      "metadata": {
        "id": "nxzSoH2Zbr6c"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Example Code for Load LoRA\n",
        "'''\n",
        "gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(model_id)\n",
        "# Use the same LoRA rank that you trained\n",
        "gemma_lm.backbone.enable_lora(rank=4)\n",
        "\n",
        "# Load pre-trained LoRA weights\n",
        "gemma_lm.backbone.load_lora_weights(f\"/content/drive/MyDrive/{lora_name}_{lora_rank}_epoch{train_epoch}.lora.h5\")\n",
        "'''"
      ],
      "metadata": {
        "id": "qE8I2RXrbt9o"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Try a different sampler\n",
        "\n",
        "The top-K algorithm randomly picks the next token from the tokens of top K probability."
      ],
      "metadata": {
        "id": "ipg1u_wEKTxG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "gemma_lm.compile(sampler=\"top_k\")\n",
        "text_gen(\"何となく\")\n",
        "text_gen(\"何となく\")\n",
        "text_gen(\"何となく\")\n",
        "text_gen(\"何となく\")\n",
        "text_gen(\"何となく\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nV5mD_HqKZRF",
        "outputId": "92e1c451-76bf-40f7-edaf-1f54a44942cd"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Gemma output:\n",
            "何となくの冬夜は母の手にしる\n",
            "TOTAL TIME ELAPSED: 18.12s\n",
            "\n",
            "Gemma output:\n",
            "何となく秋空を吊るす吊り下げ\n",
            "TOTAL TIME ELAPSED: 0.77s\n",
            "\n",
            "Gemma output:\n",
            "何となく秋晴れの町に居てゐる\n",
            "TOTAL TIME ELAPSED: 0.78s\n",
            "\n",
            "Gemma output:\n",
            "何となくの春雨の食すかな\n",
            "TOTAL TIME ELAPSED: 0.66s\n",
            "\n",
            "Gemma output:\n",
            "何となくの春潮に酔ふや蛇\n",
            "TOTAL TIME ELAPSED: 0.72s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Try different prompts"
      ],
      "metadata": {
        "id": "3m1XaCrlMu3Y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text_gen(\"春雨\")\n",
        "text_gen(\"夏の\")\n",
        "text_gen(\"秋風\")\n",
        "text_gen(\"冬空\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qC-MLxYWM1HU",
        "outputId": "c0ab4800-5712-4628-c6f6-74d2721c2f58"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Gemma output:\n",
            "春雨に身を沈む女は千人一首\n",
            "TOTAL TIME ELAPSED: 0.77s\n",
            "\n",
            "Gemma output:\n",
            "夏の影の男の座るテーブルかな\n",
            "TOTAL TIME ELAPSED: 0.70s\n",
            "\n",
            "Gemma output:\n",
            "秋風とけりや野の奥に墓石し\n",
            "TOTAL TIME ELAPSED: 0.90s\n",
            "\n",
            "Gemma output:\n",
            "冬空の白線ひとつも濁らず\n",
            "TOTAL TIME ELAPSED: 0.65s\n"
          ]
        }
      ]
    }
  ]
}