samples/azure-quantum/utilities/storage-data-management.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "source": [
        "# Data Management Sample\n",
        "This sample shows you how to connect and use Azure Quantum notebooks with external datasources such as Azure Blob Storage. First, connect to your Azure Quantum Workspace:"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from azure.quantum import Workspace\n",
        "workspace = Workspace (\n",
        "    subscription_id = \"\",\n",
        "    resource_group = \"\",\n",
        "    name = \"\",\n",
        "    location = \"\"\n",
        ")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "source": [
        "## Work with data in Azure Blob Storage\n",
        "This section will show you how to work with JSON and other file types in Blob Storage.\n",
        "\n",
        "Your [Azure Quantum Workspace](https://docs.microsoft.com/azure/quantum/how-to-create-workspace?tabs=tabid-quick) has a storage account attached to it by default - this is normally used for management of job data but can also be used to store your data.\n",
        "\n",
        "You can find details for this storage account by navigating to your Azure Quantum Workspace in the portal - it is shown in the 'Essentials' section at the top - if you click the name of the storage account it will take you to view it in the portal.\n",
        "\n",
        "You can connect to this storage account directly through the Azure Portal or by using the free [Azure Storage Explorer desktop app](https://azure.microsoft.com/products/storage/storage-explorer/#overview). Both of these methods will allow you to view, upload, download and delete files and containers in your storage account.\n",
        "\n",
        "For further information on the Azure Blob Storage service, please refer to the [docs page](https://docs.microsoft.com/azure/storage/blobs/storage-blobs-introduction)."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Connect to Blob Storage\n",
        "\n",
        "The following code snippet shows how to create a connection to the storage account attached to your Azure Quantum workspace, and work with data stored therein. This snippet works specifically with a container named \"data\", which will not exist by default - the code below will create the container for you if it does not already exist."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "from azure.storage.blob import BlobServiceClient\n",
        "from azure.storage.blob import ContainerClient\n",
        "\n",
        "\n",
        "# Fetch URI for the Storage account attached to your Azure Quantum Workspace\n",
        "# If the blob container 'data' does not exist already, it will be created\n",
        "storage_uri = workspace.get_container_uri(container_name=\"data\")\n",
        "\n",
        "# Use this container client to interact with the \"data\" storage account container\n",
        "container_client = ContainerClient.from_container_url(container_url=storage_uri)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Upload JSON data to Blob Storage\n",
        "\n",
        "The following snippet shows you how to upload JSON data to the newly-created \"data\" container within your storage account."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# Define JSON object for upload\n",
        "json_data_for_upload = {\n",
        "    \"data\": {\n",
        "        \"type\": \"test\",\n",
        "        \"value\": 123\n",
        "    }\n",
        "}\n",
        "\n",
        "# Create blob client for the new file\n",
        "blob_client = container_client.get_blob_client(blob=\"file.json\")\n",
        "\n",
        "# Upload JSON data\n",
        "# Enable/disable overwrite to specify behaviour when a blob already exists with that name\n",
        "try:\n",
        "    blob_client.upload_blob(json.dumps(json_data_for_upload), overwrite=True)\n",
        "    print(\"Upload succeeded!\")\n",
        "except Exception as e:\n",
        "    print(\"Upload failed! Reason: {e}\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Download JSON data from Blob Storage\n",
        "\n",
        "The following snippet shows you how to fetch data from a JSON file stored in the \"data\" container of the Storage Account attached to your Azure Quantum Workspace. For this to work, the file must previously have been uploaded to the blob storage container. For simplicity, this snippet makes use of the file you uploaded in the previous section."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# Create blob client for your file\n",
        "blob_client = container_client.get_blob_client(blob=\"file.json\")\n",
        "\n",
        "# Download file from blob storage & parse JSON into input_data object\n",
        "download_stream = blob_client.download_blob()\n",
        "input_data = json.loads(download_stream.readall())\n",
        "\n",
        "# Display file contents\n",
        "print(json.dumps(input_data, indent=2))"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Upload text/other filetypes to Blob Storage"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define text for upload\n",
        "text_data_for_upload = \"Some text to upload to a file. File could be .csv, .txt or any other format you choose.\"\n",
        "\n",
        "# Create blob client for the new file (make sure to choose the appropriate extension e.g. .txt or .csv)\n",
        "blob_client = container_client.get_blob_client(blob=\"file.txt\")\n",
        "\n",
        "# Upload data\n",
        "# Enable/disable overwrite to specify behaviour when a blob already exists with that name\n",
        "try:\n",
        "    blob_client.upload_blob(text_data_for_upload, overwrite=True)\n",
        "    print(\"Upload succeeded!\")\n",
        "except Exception as e:\n",
        "    print(\"Upload failed! Reason: {e}\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Download text/other filetypes from Blob Storage"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create blob client for your file\n",
        "blob_client = container_client.get_blob_client(blob=\"file.txt\")\n",
        "\n",
        "# Download file from blob storage & assign to input_data bytes object\n",
        "download_stream = blob_client.download_blob()\n",
        "input_data = download_stream.readall().decode(\"utf-8\") \n",
        "\n",
        "print(input_data)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Work with data from an external URL\n",
        "The following code snippet shows you how to load data from a URL (with no authentication) and save the data to a file in Blob Storage."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Download data from URL"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Import requests module\n",
        "import requests\n",
        "\n",
        "# Paste the URL for your file here\n",
        "# This example loads the Broombridge 2.0 schema file from the Microsoft Quantum samples repo\n",
        "url = \"https://raw.githubusercontent.com/microsoft/Quantum/main/Chemistry/Schema/broombridge-0.2.schema.json\"\n",
        "\n",
        "# Get data from URL\n",
        "r = requests.get(url)\n",
        "\n",
        "# Work with the data - e.g. could upload to Blob Storage after processing\n",
        "# See https://www.w3schools.com/PYTHON/ref_requests_response.asp for further properties of the requests.Response object which you can use for processing\n",
        "data_bytes = r.content\n",
        "data_json = r.json()\n",
        "data_text = r.text\n",
        "\n",
        "print(f\"Title: {data_json['title']}\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Work with files in local storage (temporary storage)\n",
        "You can use the following code snippets to write data to local storage. Local storage is treated as a temporary storage location as it is hard to locate local files afterwards and the storage may not persist across sessions. It is therefore recommended that you only use this for temporary file operations and use the Blob Storage access methods provided in this sample to more permanently store your data."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define JSON data to save\n",
        "json_data = {\n",
        "    \"data\": {\n",
        "        \"type\": \"test\",\n",
        "        \"value\": 123\n",
        "    }\n",
        "}\n",
        "\n",
        "# Write JSON to local storage\n",
        "with open(\"file.json\", 'w') as f:\n",
        "    f.write(json.dumps(json_data))\n",
        "\n",
        "# Define text data to save\n",
        "text_data = \"Some text! Hello :)\"\n",
        "\n",
        "# Write text to local storage\n",
        "with open(\"file.txt\", 'w') as f:\n",
        "    f.write(text_data)\n",
        "\n",
        "# Read JSON from local storage\n",
        "with open(\"file.json\") as fp:\n",
        "    input_data = json.load(fp)\n",
        "\n",
        "print(f\"JSON data: \\n{json.dumps(input_data, indent=2)}\")\n",
        "\n",
        "# Read text file from local storage\n",
        "with open(\"file.txt\") as fp:\n",
        "    input_data = fp.read()\n",
        "    \n",
        "print(f\"\\nText data:\\n{input_data}\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## End-to-end example\n",
        "\n",
        "The sample below shows how to load a spin-orbital Hamiltonian from a file containing orbital integrals. Features of the Hamiltonian are then computed and the results are saved to a .csv file in Blob Storage. The input data is loaded from a publicly-accessible GitHub URL.\n",
        "\n",
        "The code for this example is taken directly from the [chemistry section of the samples repo](/~https://github.com/microsoft/Quantum/tree/main/samples/chemistry). This specific example computes the features for Ozone:\n",
        "\n",
        "> Ozone is formed from dioxygen by the action of ultraviolet light (UV) and electrical discharges within the Earth's atmosphere. It is present in very low concentrations throughout the latter, with its highest concentration high in the ozone layer of the stratosphere, which absorbs most of the Sun's ultraviolet (UV) radiation. Quantum mechanical excited-state studies, including localization of avoided crossings and conical intersection play a critical role in understanding its role in Earth atmosphere."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Compute Hamiltonian features for the Ozone molecule\n",
        "You can find further orbital integral data in the samples repo [here](/~https://github.com/microsoft/Quantum/tree/main/samples/chemistry/IntegralData/YAML).\n",
        "\n",
        "To load data from a GitHub URL using the `requests` module, you need to use the 'raw' content URL (should have `raw.githubusercontent.com` as the domain). \n",
        "\n",
        "You can find this URL by navigating to the file you would like to use and selecting 'Raw' from the panel at the top of the document view. This should reload the page - use the URL for the page that just loaded."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests, json\n",
        "import numpy as np\n",
        "from numpy import linalg as LA\n",
        "from qsharp.chemistry import load_broombridge, load_fermion_hamiltonian, IndexConvention\n",
        "from azure.storage.blob import BlobServiceClient\n",
        "from azure.quantum import Workspace\n",
        "\n",
        "# Fetch data from URL\n",
        "# See note above for instructions on how to test with other files\n",
        "r = requests.get(\"https://raw.githubusercontent.com/microsoft/Quantum/main/samples/chemistry/IntegralData/YAML/O3_ccpvtz/o3_13_6_6_90deg_ccvtz.yaml\")\n",
        "\n",
        "ozone_filename = \"o3_13_6_6_90deg_ccvtz.yaml\"\n",
        "\n",
        "# Write file to local storage (temp storage)\n",
        "with open(ozone_filename, 'wb') as f:\n",
        "    f.write(r.content) \n",
        "\n",
        "print(f\"Processing the following file: {ozone_filename}\\n\")\n",
        "\n",
        "# Load Broombridge schema for Ozone\n",
        "broombridge = load_broombridge(ozone_filename)\n",
        "\n",
        "# Load Hamiltonian data\n",
        "general_hamiltonian = broombridge.problem_description[0].load_fermion_hamiltonian(index_convention=IndexConvention.UpDown)\n",
        "\n",
        "# Calculate one-norms and save results to output string\n",
        "output = \"Type,one-norm\\n\"\n",
        "print(\"Computing One-norms:\")\n",
        "for term, matrix in general_hamiltonian.terms:\n",
        "    one_norm = LA.norm(np.asarray([v for k, v in matrix], dtype=np.float32), ord=1)\n",
        "    output += f\"{term},{one_norm}\\n\"\n",
        "    print(f\"\\tOne-norm for term type {term}: {one_norm}\")\n",
        "\n",
        "print()\n",
        "\n",
        "# Fetch URI for the Storage account attached to your Azure Quantum Workspace\n",
        "# If the blob container 'data' does not exist already, it will be created\n",
        "storage_uri = workspace.get_container_uri(container_name=\"data\")\n",
        "\n",
        "# Use this container client to interact with the storage account container\n",
        "container_client = ContainerClient.from_container_url(container_url=storage_uri)\n",
        "\n",
        "# Create blob client for the new .csv output file\n",
        "blob_client = container_client.get_blob_client(blob=\"ozone_results.csv\")\n",
        "\n",
        "# Upload data to Blob Storage\n",
        "# Enable/disable overwrite to specify behaviour when a blob already exists with that name\n",
        "try:\n",
        "    blob_client.upload_blob(output, overwrite=True)\n",
        "    print(\"Results saved to blob storage!\")\n",
        "except Exception as e:\n",
        "    print(\"Upload failed! Reason: {e}\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "source": [
        "If you ran all cells in this sample notebook, you should now be able to see a new container called `data` in the Azure Blob Storage account associated with your Azure Quantum Workspace. It should have three files in it:\n",
        "\n",
        "- file.json\n",
        "- file.txt\n",
        "- ozone_results.csv"
      ],
      "metadata": {}
    }
  ],
  "metadata": {
    "kernel_info": {
      "name": "python3"
    },
    "kernelspec": {
      "name": "python3",
      "language": "python",
      "display_name": "Python 3 (ipykernel)"
    },
    "language_info": {
      "name": "python",
      "version": "3.9.16",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}