Merge pull request #72 from daos-stack/develop

Merge develop branch to main for v0.3.0

.gitignore

@@ -27,6 +27,8 @@ id_rsa*
 hosts*
 *.flag
 keys.txt
+results/
+module.json
 
 # addlicense binary for pre-commit
 tools/autodoc/addlicense

.pre-commit-config.yaml

@@ -26,7 +26,7 @@ repos:
         pass_filenames: true
         require_serial: true
   - repo: /~https://github.com/antonbabenko/pre-commit-terraform
-    rev: v1.64.0
+    rev: v1.76.0
     hooks:
       - id: terraform_fmt
       - id: terraform_tflint
@@ -43,3 +43,7 @@ repos:
         exclude: \.terraform\/.*$
         pass_filenames: true
         require_serial: true
+  - repo: /~https://github.com/jumanjihouse/pre-commit-hooks
+    rev: 3.0.0
+    hooks:
+      - id: shellcheck

README.md

@@ -4,102 +4,82 @@ Distributed Asynchronous Object Storage ([DAOS](https://docs.daos.io/)) on Googl
 
 This repository contains:
 
-- [Packer](https://www.packer.io/) scripts used to build DAOS images with [Google Cloud Build](https://cloud.google.com/build)
-- [Terraform](https://www.terraform.io/) modules that can be used to deploy DAOS Server and Client instances
-- [Terraform](https://www.terraform.io/) examples that demonstrate how to use the Terraform modules
+- [Terraform](https://www.terraform.io/) modules for deploying DAOS Server and Client instances on GCP
+- Scripts used to build DAOS images with [Cloud Build](https://cloud.google.com/build) and [Packer](https://www.packer.io/)
+- Examples that demonstrate how to use the DAOS Terraform modules
+- Documentation for deploying DAOS on GCP
 
-**Directory structure**
+## Pre-Deployment
 
-```
-.
-├── docs            Documentation and Cloud Shell tutorials
-├── images          Cloud Build config files and Packer templates
-│   └── scripts     Scripts that Packer runs to build images
-├── terraform       Terraform content
-│   ├── examples    Examples that demonstrate how to use the DAOS Terraform modules
-│   └── modules     Terraform modules for deploying DAOS server and client instances
-└── tools           Tools used by pre-commit
-```
+In order to deploy DAOS on GCP there are several steps that are required to prepare your project.
 
-## Prerequisites
+For instructions, see the [Pre-Deployment Guide](docs/pre-deployment_guide.md)
 
-In order to deploy DAOS on GCP you will need
+As an alternative to viewing the guide as a standalone document, you can view it as an in-context tutorial in [Cloud Shell](https://cloud.google.com/shell) by clicking the button below.
 
-- **Access to the Google Cloud Platform (GCP)**
+[![Open in Cloud Shell](https://gstatic.com/cloudssh/images/open-btn.png)](https://ssh.cloud.google.com/cloudshell/open?cloudshell_git_repo=/~https://github.com/daos-stack/google-cloud-daos.git&cloudshell_git_branch=main&shellonly=true&cloudshell_tutorial=docs/pre-deployment_guide.md)
 
-   See [Get Started with Google Cloud](https://cloud.google.com/docs/get-started)
 
-- **A GCP Project**
 
-  See [Creating and managing projects](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
+## Deployment
 
-- **Required Software**
+After completing the steps in the [Pre-Deployment Guide](docs/pre-deployment_guide.md) you will be ready to deploy DAOS.
 
-  The documentation in this repository assumes that you will use [Cloud Shell](https://cloud.google.com/shell).
+To deploy DAOS on GCP you may choose one of the following deployment paths.
 
-  With [Cloud Shell](https://cloud.google.com/shell), there is no need to install any software on your system.
+1. **Cloud HPC Toolkit**
 
-  If you do not want to use Cloud Shell, you will need to install
-    - [Git](https://git-scm.com/)
-    - [Google Cloud CLI](https://cloud.google.com/sdk/docs/install)
-    - [Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli)
+   [Cloud HPC Toolkit](https://cloud.google.com/hpc-toolkit) is open-source software offered by Google Cloud which makes it easy for you to deploy high performance computing (HPC) environments. It is designed to be highly customizable and extensible, and intends to address the HPC deployment needs of a broad range of use cases.
 
-## Deploying DAOS on GCP
+   The [community examples](/~https://github.com/GoogleCloudPlatform/hpc-toolkit/tree/main/community/examples/intel) in the [Cloud HPC Toolkit](https://cloud.google.com/hpc-toolkit) use the [DAOS modules](terraform/modules/) in this repository.
 
-### Pre-Deployment Steps
+   For documentation, see the [Cloud HPC Toolkit Overview](https://cloud.google.com/hpc-toolkit/docs/overview)
 
-The following pre-deployment steps are required
+   **Prepare to use the Cloud HPC Toolkit**
 
-1. **Set defaults for Google Cloud CLI (```gcloud```)**
+   - **Dependencies**
 
-   Only needs to be done once in your shell (Cloud Shell or local shell).
+      If you are using Cloud Shell, the dependencies are already installed.
 
-2. **Enable service APIs and grant permissions**
+      If you are not using Cloud Shell, you will need to install a few dependencies.
 
-   Enabling APIs and granting service account permissions only needs to be done once for a GCP project.
+      See [Install dependencies](https://cloud.google.com/hpc-toolkit/docs/setup/install-dependencies)
 
-3. **Create a Packer image in your GCP project**
+   - **Configure Environment**
 
-   In order to build DAOS images with Cloud Build your GCP project must contain a Packer image.
+      Before you can deploy DAOS with the Cloud HPC Toolkit you will need to complete some configuration steps.
 
-   Building the Packer image only needs to be done once for a GCP project.
+      See [Configure your environment](https://cloud.google.com/hpc-toolkit/docs/setup/configure-environment).
 
-4. **Build DAOS Server and Client images**
+   **Deploy DAOS with the Cloud HPC Toolkit**
 
-   DAOS Server and Client instances are deployed using images that have DAOS pre-installed.
+   For instructions on how to deploy the [community examples](/~https://github.com/GoogleCloudPlatform/hpc-toolkit/tree/main/community/examples/intel), see
+   - [DAOS Cluster](/~https://github.com/GoogleCloudPlatform/hpc-toolkit/tree/main/community/examples/intel#daos-cluster)
+   - [DAOS Server with Slurm Cluster](/~https://github.com/GoogleCloudPlatform/hpc-toolkit/tree/main/community/examples/intel#daos-server-with-slurm-cluster)
 
-   Therefore, the images need to be built prior to running Terraform to deploy a DAOS cluster.
+2. **Deploy DAOS with the DAOS cluster example**
 
-Click the button below to open a Cloud Shell tutorial which will guide you through the pre-deployment steps listed above. If you lose your Cloud Shell session you can always come back to this README and click the button again.
+   The [`terraform/examples/daos_cluster`](terraform/examples/daos_cluster/README.md) example demonstrates how to use the [DAOS modules](terraform/modules/) in a Terraform configuration to deploy a DAOS cluster consisting of servers and clients.
 
-[![DAOS on GCP Pre-Deployment](http://gstatic.com/cloudssh/images/open-btn.png)](https://console.cloud.google.com/cloudshell/open?git_repo=/~https://github.com/daos-stack/google-cloud-daos&cloudshell_git_branch=main&shellonly=true&tutorial=docs/tutorials/pre-deployment.md)
+   To deploy the example, see [Deploy the DAOS Cluster Example](docs/deploy_daos_cluster_example.md).
 
-### Deploy a DAOS Cluster with Terraform
+   As an alternative to viewing the instructions in [Deploy the DAOS Cluster Example](docs/deploy_daos_cluster_example.md) as a standalone document, you can view it as an in-context tutorial in [Cloud Shell](https://cloud.google.com/shell) by clicking the button below.
 
-After completing the pre-deployment steps listed above, you will need to write your own Terraform configuration for your particular use case.
+   [![Open in Cloud Shell](https://gstatic.com/cloudssh/images/open-btn.png)](https://ssh.cloud.google.com/cloudshell/open?cloudshell_git_repo=/~https://github.com/daos-stack/google-cloud-daos.git&cloudshell_git_branch=main&shellonly=true&cloudshell_tutorial=docs/deploy_daos_cluster_example.md)
 
-The [terraform/modules](terraform/modules) in this repo can be used in your Terraform configuration to deploy DAOS server and client instances.
+3. **Create your own Terraform Configurations**
 
-The [terraform/examples/daos_cluster](terraform/examples/daos_cluster/README.md) example serves as both a reference and a quick way to deploy a DAOS cluster.
+   While this is the more complex deployment path, there may be reasons why this path is necessary.
 
-Click the button below to open a Cloud Shell tutorial that will walk you through using the [terraform/examples/daos_cluster](terraform/examples/daos_cluster/README.md) example to deploy a DAOS cluster.
+   For example, you may want to add DAOS servers and clients to your existing Terraform configurations.
 
-[![DAOS Cluster Example](http://gstatic.com/cloudssh/images/open-btn.png)](https://console.cloud.google.com/cloudshell/open?git_repo=/~https://github.com/daos-stack/google-cloud-daos&cloudshell_git_branch=main&shellonly=true&tutorial=docs/tutorials/example_daos_cluster.md)
+   When creating your own Terraform configurations refer to the documentation for the modules:
+   - [daos_server module](terraform/modules/daos_server/README.md)
+   - [daos_client module](terraform/modules/daos_client/README.md)
 
-### Deploy a DAOS Cluster with the Google HPC Toolkit
+   You can also use the [`terraform/examples/daos_cluster`](terraform/examples/daos_cluster/README.md) example as a reference.
 
-The [HPC Toolkit](/~https://github.com/GoogleCloudPlatform/hpc-toolkit) is an open-source software offered by Google Cloud which makes it easy for customers to deploy HPC environments on Google Cloud.
-
-The HPC Toolkit allows customers to deploy turnkey HPC environments (compute, networking, storage, etc) following Google Cloud best-practices, in a repeatable manner. It is designed to be highly customizable and extensible, and intends to address the HPC deployment needs of a broad range of customers.
-
-The HPC Toolkit includes the following community examples which use the Terraform modules in this repository.
-
-| HPC Toolkit Community Example | Description |
-| ----------------------------- | ----------- |
-| [DAOS Cluster](/~https://github.com/GoogleCloudPlatform/hpc-toolkit/tree/main/community/examples/intel#daos-cluster) | Use the HPC Toolkit to deploy a standalone DAOS cluster |
-| [DAOS Server with Slurm cluster](/~https://github.com/GoogleCloudPlatform/hpc-toolkit/tree/main/community/examples/intel#daos-server-with-slurm-cluster) | Use the HPC Toolkit to deploy a set of DAOS servers for storage and a Slurm cluster in which the compute nodes are DAOS clients.  The example demonstrates how to use DAOS storage in a Slurm job. |
-
-If you are just getting started with deploying DAOS on GCP, it is highly recommended to use the HPC Toolkit as it can save you a lot of time as opposed to developing your own Terraform configuration.
+   Note that DAOS servers and clients should be deployed in the same Terraform configuration. This is because there are variables in the daos_client module that must get their values from the daos_server module.
 
 ## Support
 
@@ -130,7 +110,7 @@ Intel Corporation provides several ways for the users to get technical support:
 
 - [Distributed Asynchronous Object Storage (DAOS)](https://docs.daos.io/)
 - [Google Cloud Platform (GCP)](https://cloud.google.com/)
-- [Google HPC Toolkit](https://github.com/GoogleCloudPlatform/hpc-toolkit)
+- [Google Cloud HPC Toolkit](https://cloud.google.com/hpc-toolkit)
 - [Google Cloud CLI (gcloud)](https://cloud.google.com/cli)
 - [Google Cloud Build](https://cloud.google.com/build)
 - [Cloud Shell](https://cloud.google.com/shell)