From 37fe28ab1ecc6ed17d19946fa69314fd0a6fecdd Mon Sep 17 00:00:00 2001 From: t-potts Date: Fri, 13 Nov 2020 08:39:27 -0600 Subject: [PATCH] document updates --- docs/docs/step-by-step-walkthrough.md | 150 +++++++------------------- 1 file changed, 38 insertions(+), 112 deletions(-) diff --git a/docs/docs/step-by-step-walkthrough.md b/docs/docs/step-by-step-walkthrough.md index 7ba2f64768..a4c5fac5ac 100644 --- a/docs/docs/step-by-step-walkthrough.md +++ b/docs/docs/step-by-step-walkthrough.md @@ -109,111 +109,37 @@ We're done with the hardest part of deployment! The next step is to run `qhub init` to generate the configuration file `qhub-config.yaml`. This file is where the vast majority of tweaks to the system will be made. There are are several optional (yet highly recommended) flags that deal with automating the deployment: -- `--project`: This is a project name of your choosing IE `test-cluster` +- `--project`: Chose a name that [is a compliant name for S3/Storage buckets]. IE `test-cluster` - `--domain`: This is the base domain for your cluster. After deployment, the DNS will use the base name prepended with `jupyter`. IE if the base name is `test.qhub.dev` then the DNS will be provisioned as `jupyter.test.qhub.dev`. This pattern is also applicable if you are setting your own DNS through a different provider. -- `--ci-provide`: This specifies what provider to use for ci-cd. Currently github-actions is supported. +- `--ci-provider`: This specifies what provider to use for ci-cd. Currently github-actions is supported. - `--oauth-provider`: This will set configuration file to use auth0 for authentication - `--oauth-auto-provision`: This will automatically create and configure an auth0 application - `--repository`: The repository name that will be used to store the infrastructure as code - `--repository-auto-provision`: This sets the secrets for the github repository -An example of the full command is below: +Best practices is to create a new directory and run all the qhub commands inside of it. An example of the full command is below: `qhub init gcp --project test-project --domain test.qhub.dev --ci-provider github-actions --oauth-provider auth0 --oauth-auto-provision --repository github.com/quansight/qhub-test --repository-auto-provision` ### 2.6 QHub render -After initializing, we then need to create all of the other configuration files -qhub deploy -c qhub-config.yaml --dns-provider cloudflare --dns-auto-provision +After initializing, we then need to create all of terraform configuration. This done by running: -With all of the applicable environment variables set, we are ready to get into the wizardry that QHub performs. The first command `qhub init` followed by the abbreviation of your cloud provider. (do: Digital Ocean, aws: Amazon Web Services, gcp: Google Cloud Platform). This will create a `qhub-config.yaml` file in your folder. - - - $ qhub init do # Digital Ocean - $ qhub init aws # Amazon Web Services - $ qhub init gcp # Google Cloud Platform - - -Open the config file `qhub-config.yaml` for editing. - -#### Top section: - -Chose a `project_name` that is a compliant name. E.g. a valid bucket name for AWS. - - project_name: my-jupyterhub - -Set the `domain` field on top of the config file to a domain or sub-domain you own in Cloudflare or your existing DNS, e.g. `testing.qhub.dev`: - - domain: testing.qhub.dev - -#### security section: - -Create an [oauth application] in github and fill in the client_id and client_secret. - - client_id: "7b88..." - client_secret: "8edd7f14..." - -Set the `oauth_callback_url` by prepending your domain with `jupyter` and appending `/hub/oauth_callback`. Example: - - oauth_callback_url: https://jupyter.testing.qhub.dev/hub/oauth_callback - -Add your desired list of authorized github usernames. Set a unique `uid` for each username, and set the `primary_group`. Optional `secondary_groups` may also be set for each user: - - joeuser: - uid: 1000000 - primary_group: users - secondary_groups: - - billing - - admin - janeuser: - uid: 1000001 - primary_group: users - - -#### Cloud provider section: - -Lastly, make the adjusted changes to configure your cluster in he cloud provider section. - - - -**(Digital Ocean only)** - -If your provider is Digital Ocean you will need to install [doctl] and obtain the latest kubernetes version. After installing, run this terminal command: + qhub render -c qhub-config.yaml ./ -f -```bash - $ doctl kubernetes options versions - Slug Kubernetes Version - 1.18.8-do.1 1.18.8 - 1.17.11-do.1 1.17.11 - 1.16.14-do.1 1.16.14 -``` - -Copy the first line under `Slug` which is the latest version. Enter it into the `kubernetes_version` under the `digital_ocean` section of your config file. - - kubernetes_version: 1.18.8-do.1 - - -### 2.4. Render QHub - -The render step will use `qhub-config.yaml` as a template to create an output folder and generate all the necessary files for deployement. - -The below example will create the directory `qhub-deployment` and fill it with the necessary files. - - - $ qhub render -c qhub-config.yaml -o qhub-deployment -f - - -Move the config file into the output directory +You will notice that several directories are created: + - `environments` : conda environments are stored + - `infrastructure` : terraform files that declare state of infrastructure + - `terraform-state` : required by terraform to securly store the state of the terraform deployment + - `image` : docker images used in qhub deployment including: jupyterhub, jupyterlab, and dask-gateway - $ mv qhub-config.yaml qhub-deployment/ - -## 3. Deployment and DNS registry +### 2.7 QHub deploy -The following command will check environment variables, deploy the infrastructure, and prompt for DNS registry +Finally we can deploy with: - $ qhub deploy -c qhub-config.yaml + qhub deploy -c qhub-config.yaml --dns-provider cloudflare --dns-auto-provision -Press `enter` to verify the oauth has been configured. The first stage of deployment will begin and there will be many lines of output text. After a few minutes, you will be prompted to set your DNS. This output will show an "ip" address (DO/GCP) or a CNAME "hostname" (AWS) based on the the cloud service provider: +The terminal will prompt to press `[enter]` to check oauth credentials (which were added by qhub init). After pressing `enter` the deployment will continue and take roughly 10 minutes. Part of the output will show an "ip" address (DO/GCP) or a CNAME "hostname" (AWS) based on the the cloud service provider: Digital Ocean/Google Cloud Platform: @@ -232,40 +158,39 @@ Press `enter` to verify the oauth has been configured. The first stage of deploy "ip" = "" } -Then you will be prompted with + +### 2.8 Push repository + +Add all files to github: - Take IP Address Above and update DNS to point to "jupyter.testing.qhub.dev" [Press Enter when Complete] - -Login to your DNS provider and make the DNS entry with the information above. For AWS add a CNAME, for DO and GCP add a type "A" entry. -When [recording your DNS] on Cloudflare, click on **Proxy Status** and change it to **DNS only**. - -When the DNS entries are made, wait until the DNS has been updated. You can check on your DNS status with the linux command `dig` followed by your url. The ip address or CNAME will show in the output of the command when DNS registry is complete. + git add .github/ .gitignore README.md environments/ image/ infrastructure/ qhub-config.yaml terraform-state/ -Press **Enter** when the DNS is registered to complete the deployment. +Push the changes to your repo: + git push origin master -## 4. **Set up github repository** -Create a github personal access token ([github_access_token]) and check the `repo` and `workflow` options under scopes. +### 3 Post github deployment: -Copy the personal access token Github Secrets with the label `REPOSITORY_ACCESS_TOKEN` +After the files are in github all CICD changes will done via github actions and triggered by a commit to master. To use gitops, make a change to the `qhub-ops.yaml` in a new branch and create pull request into master. When the pull request is merged, it will trigger a deployement of all of those changes to your QHub. -All other environment variables that were created in step **1** also need to be added to github as secrets +The first thing you will want to do is add users to your new QHub. Any type of supported authorization from auth0 can be used as a username. Below is an example configuration of 2 users: -Create a github repo and push all files to it with the following commands: + joeuser@example: + uid: 1000000 + primary_group: users + secondary_groups: + - billing + - admin + janeuser@example.com: + uid: 1000001 + primary_group: users -``` -$ git init -$ git remote add origin -$ git add * -$ git commit -m 'initial commit' -$ git push origin master -``` +As seen above, each username has a unique `uid` and a `primary_group`. Optional `secondary_groups` may also be set for each user. -## 5. **Git ops enabled** -Since the infrastructure state is reflected in the repository, it allows self-documenting of infrastructure and team members to submit pull requests that can be reviewed before modifying the infrastructure. +### 4 Git ops enabled -To use gitops, make a change to the `qhub-ops.yaml` in a new branch and create pull request into master. When the pull request is merged, it will trigger a deployement of all of those changes to your qhub. +Since the infrastructure state is reflected in the repository, it allows self-documenting of infrastructure and team members to submit pull requests that can be reviewed before modifying the infrastructure. Congratulations! You have now completed your QHub cloud deployment! @@ -289,4 +214,5 @@ Congratulations! You have now completed your QHub cloud deployment! [this guide]: https://www.digitalocean.com/community/tutorials/how-to-create-a-digitalocean-space-and-api-key [terraform]: https://www.terraform.io/ [these detailed instructions]: https://cloud.google.com/iam/docs/creating-managing-service-accounts -[these steps]: https://cloud.google.com/iam/docs/creating-managing-service-account-keys#iam-service-account-keys-create-console \ No newline at end of file +[these steps]: https://cloud.google.com/iam/docs/creating-managing-service-account-keys#iam-service-account-keys-create-console +[is a compliant name for S3/Storage buckets]: https://www.google.com/search?q=s3+compliant+name&oq=s3+compliant+name&aqs=chrome..69i57j0.3611j0j7&sourceid=chrome&ie=UTF-8 \ No newline at end of file