Get kubernetes version for all cloud providers + pytest refactor (#927)

* Add k8s version checker for AWS

* Grab latest azure k8s version available by default

* Update initialize with k8s versioning

* Add boto3 to setup.py

* Create _set_kubernetes_version function for init

* Minor update to tests

* Standardize k8s version check for DO and GCP

* Update yaml call based on deprecation warning

* Add setup_fixture to conftest.py

* Use copy of config dicts

* Standardize kubernetes_versions functions for all providers

* Update tests to use new setup_fixture

* fix failing test

* Update docs to reflect changes

* Update usage.md based on linter

* Update usage.md based on linter

* Split test workflow into two seperate workflows

* Update test-provider.yaml

* Test secrets

* Add secrets, remove kubernetes_version

* Add gcloud to test-provider workflow

* Add gcloud to test-provider workflow

* Add gcloud to test-provider workflow

* Add azure to test-provider

* Add azure env vars to test-provider

* Add azure cli login to test-provider

* Remove conditional setup

* Update test-provider.yaml

* Update test-provider.yaml

* Update azure env vars

* Fix issue from merge

* Add azure env vars

* Add azure env vars, part 2

* Include gcloud and az cli conditionally

* Update project-name in tests/conftest based on PR 761

* Minor docs change

* Add QHUB_K8S_VERSION and docs

* doc lint only modified files

* Fix vale complaints

* Fix vale complaints

* Fix vale complaints

* Ignore myqhubsite.com md lint

* Undo fix to login.md

* Update docs/source/installation/usage.md

Co-authored-by: Vinicius D. Cerutti <51954708+viniciusdc@users.noreply.github.com>

* Updates based on code review

* Update digital_ocean.py

Co-authored-by: Vinicius D. Cerutti <51954708+viniciusdc@users.noreply.github.com>

.github/workflows/docs.yml

@@ -20,7 +20,7 @@ jobs:
       - name: documentation quality check
         uses: errata-ai/vale-action@v1.4.0
         with:
-          files: '["docs", "CONTRIBUTING.md", "README.md", "RELEASE.md", "qhub"]'
+          files: __onlyModified
           onlyAnnotateModifiedLines: true
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/markdown.links.config.json

@@ -14,6 +14,9 @@
         },
         {
             "pattern": "127.0.0.1:8000"
+        },
+        {
+            "pattern": "myqhubsite.com"
         }
 	]
 }

.github/workflows/test-provider.yaml

@@ -0,0 +1,73 @@
+name: "Test QHub Provider"
+
+on:
+  pull_request: # Workflow only runs for PR against main anyway
+  push:
+    branches:
+      - '**'
+    tags:
+      - 'v*'
+    paths-ignore:
+      - "docs/**"
+      - "*.md"
+
+env:
+  DIGITALOCEAN_TOKEN: ${{ secrets.DIGITALOCEAN_TOKEN }}
+  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+  ARM_CLIENT_ID: ${{ secrets.ARM_CLIENT_ID }}
+  ARM_CLIENT_SECRET: ${{ secrets.ARM_CLIENT_SECRET }}
+  ARM_SUBSCRIPTION_ID: ${{ secrets.ARM_SUBSCRIPTION_ID }}
+  ARM_TENANT_ID: ${{ secrets.ARM_TENANT_ID }}
+
+jobs:
+  test-render-providers:
+    name: 'Test QHub Provider'
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        provider:
+          - aws
+          - azure
+          - do
+          - gcp
+          - local
+    steps:
+      - name: 'Checkout Infrastructure'
+        uses: actions/checkout@main
+      - name: Set up Python
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.8
+      - name: Set up Cloud SDK
+        uses: google-github-actions/setup-gcloud@master
+        if: ${{ matrix.provider == 'gcp' }}
+        with:
+          project_id: ${{ secrets.GCP_PROJECT_ID }}
+          service_account_key: ${{ secrets.GOOGLE_CREDENTIALS }}
+      - name: Use gcloud CLI
+        if: ${{ matrix.provider == 'gcp' }}
+        run: gcloud info
+      - name: Login to Azure
+        uses: azure/login@v1
+        if: ${{ matrix.provider == 'azure' }}
+        with:
+          creds: ${{ secrets.AZURE_CREDENTIALS }}
+      - name: Use az CLI
+        if: ${{ matrix.provider == 'azure' }}
+        run: az version
+      - name: Install QHub
+        run: |
+          pip install .[dev]
+      - name: QHub Initialize
+        run: |
+          qhub init "${{ matrix.provider }}" --project "test-${{ matrix.provider }}" --domain "${{ matrix.provider }}.qhub.dev" --auth-provider github --disable-prompt
+      - name: QHub Render
+        run: |
+          qhub render -c "qhub-config.yaml" -o "qhub-${{ matrix.provider }}-deployment"
+          cp "qhub-config.yaml" "qhub-${{ matrix.provider }}-deployment/qhub-config.yaml"
+      - name: QHub Render Artifact
+        uses: actions/upload-artifact@master
+        with:
+          name: "qhub-${{ matrix.provider }}-artifact"
+          path: "qhub-${{ matrix.provider }}-deployment"

.github/workflows/test.yaml

@@ -40,37 +40,3 @@ jobs:
         run: |
           pytest --version
           pytest --ignore=tests_deployment
-
-  test-render-providers:
-    name: 'Test QHub Provider'
-    runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        provider:
-          - aws
-          - azure
-          - do
-          - gcp
-          - local
-    steps:
-      - name: 'Checkout Infrastructure'
-        uses: actions/checkout@main
-      - name: Set up Python
-        uses: actions/setup-python@v1
-        with:
-          python-version: 3.8
-      - name: Install QHub
-        run: |
-          pip install .[dev]
-      - name: QHub Initialize
-        run: |
-          qhub init "${{ matrix.provider }}" --project "${{ matrix.provider }}-test" --domain "${{ matrix.provider }}.qhub.dev" --auth-provider github --kubernetes-version "1.18.0" --disable-prompt
-      - name: QHub Render
-        run: |
-          qhub render -c "qhub-config.yaml" -o "qhub-${{ matrix.provider }}-deployment"
-          cp "qhub-config.yaml" "qhub-${{ matrix.provider }}-deployment/qhub-config.yaml"
-      - name: QHub Render Artifact
-        uses: actions/upload-artifact@master
-        with:
-          name: "qhub-${{ matrix.provider }}-artifact"
-          path: "qhub-${{ matrix.provider }}-deployment"

docs/source/dev_guide/testing.md

@@ -31,6 +31,14 @@ component which has not been released. So you may need to manually
 modify the qhub-config.yaml to 'downgrade' the tags to a full release
 version.
 
+### Kubernetes Version Check for Cloud Providers
+
+When `qhub init <cloud provider>` is called, it checks that the `--kubernetes-version` provided is supported by the choosen cloud provider. This flag is optional and if not provided, the `kubernetes_version` is set to the most recent kubernetes version available. This is achieved by using the cloud provider's SDK which thus requires their approiate credentials to be set. To get around this, simply set the `QHUB_K8S_VERSION` environment variable like so:
+
+```
+export QHUB_K8S_VERSION=1.20
+```
+
 ## Modifying Docker Images
 
 All QHub docker images are located in [`qhub/templates/{{
@@ -107,16 +115,16 @@ npm run cypress:open
 ```
 
 The Base URL can point anywhere that should be accessible - it can be the URL of a QHub cloud deployment.
-The QHub Config Path should point to the associated yaml file for that site. Most importantly, the tests will inspect the yaml file to understand
+The QHub Config Path should point to the associated yaml file for that site. Most importantly, the tests inspect the yaml file to understand
 what tests are relevant. To start with, it checks security.authentication.type to determine what should be available on the login page, and
-how to test it. If the login type is 'password' then it uses the value in CYPRESS_EXAMPLE_USER_PASSWORD as the password (default username is
-`example-user` but this can be changed by setting CYPRESS_EXAMPLE_USER_NAME).
+how to test it. If the login type is `password` then it uses the value in `CYPRESS_EXAMPLE_USER_PASSWORD` as the password (default username is
+`example-user`, to change the default username, update `CYPRESS_EXAMPLE_USER_NAME`).
 
-The final command above should open the Cypress UI where you can run the tests manually and see the actions in the browser.
+The final command, in the preceding code-snippet, opens the Cypress UI where you can run the tests manually and see the actions in the browser.
 
 Note that tests are heavily state dependent, so any changes or use of the deployed QHub could affect the results.
 
-## Deployment/Integration Tests
+## Deployment and integration tests
 
 Deployment and Integration testing makes it easier to test various features of deployed QHub
 on minikube such as Dask Gateway, external integrations, state of the kubernetes cluster via

docs/source/installation/configuration.md

@@ -305,6 +305,23 @@ and **Kubernetes versions** will be DIFFERENT. [duplicated info]
 
 ### Providers
 
+To take advantage of the auto-scaling and dask-distributed computing capabilities,
+QHub can be deployed on a handful of the most commonly used cloud providers. QHub
+utilizes many of the resources these cloud providers have to offer, however,
+at it's core, is the Kubernetes engine (or service). Each cloud provider has slightly
+different ways Kubernetes is configured but fear not, all of this is handled by QHub.
+
+Listed below are the cloud providers QHub currently supports.
+
+> NOTE: Many of the cloud providers regularly update their internal Kubernetes
+> versions so if you wish to specify a particular version, please check the following
+> resources. This is *completely optional* as QHub will, by default, select the most
+> recent version available for your choosen cloud provider.
+> [Digital Ocean](https://docs.digitalocean.com/products/kubernetes/changelog/)
+> [Google Cloud Platform](https://cloud.google.com/kubernetes-engine/docs/release-notes-stable)
+> [Amazon Web Services](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html)
+> [Microsoft Azure](https://docs.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli)
+
 #### DigitalOcean
 
 DigitalOcean has a restriction with autoscaling in that the minimum
@@ -314,13 +331,6 @@ Digital Ocean does not have accelerator/gpu support. Digital Ocean is
 a great default choice for tying out QHub. Below is the recommended
 setup.
 
-> Note: DigitalOcean regularly updates Kubernetes versions hence, the
-> field `kubernetes_version` will most likely have to be changed.
-> [See available instance types for
-> DigitalOcean](https://www.digitalocean.com/docs/droplets/).  If you
-> used `qhub init` this version will automatically be compute for you
-> Do not copy the version you see bellow
-
 To see available instance types refer to [Digital Ocean Instance
 Types](https://www.digitalocean.com/docs/droplets/). Additionally the
 Digital Ocean cli `doctl` has [support for listing

docs/source/installation/usage.md

@@ -2,24 +2,26 @@
 
 ## Cloud Deployment
 
-Once all environment variables have been set (in [Configuration](configuration.md)), you will be able to run
-commands on your terminal to initialise the configuration for QHub, and then deploy it.
+Great, you've gone through the `qhub` [Installation](installation.md) and [Setup Initialization](setup.md) steps,
+and have ensured that all the necessary environment variables have been properly set, it's time to deploy QHub
+from your terminal.
 
 ### Initialize configuration
 
-QHub can help you create your configuration YAML file, and you can further edit it as needed.
-
-We advise you to
-start by creating a new project folder. Here, we will name the new
-folder `qhub-test`.
+There are several ways to generate your configuration file, `qhub-config.yaml`. You can
+type the commands when prompted by terminal, or you can set
+it all automatically from the start. In any case, start by creating
+a new project folder. Start by creating a directory `qhub-test`.
 
 On your terminal run:
 
 ```shell
 mkdir qhub-test && cd qhub-test
 ```
 
-To generate a configuration file, on your terminal run something like the following (vary for your own choices):
+#### Fully automated deployment
+
+To generate a fully automated configuration file, on your terminal run:
 
 ```shell
 qhub init aws \
@@ -29,46 +31,45 @@ qhub init aws \
   --auth-provider auth0 --auth-auto-provision \
   --ssl-cert-email admin@test.com
 ```
+There are several **optional** (yet highly recommended) flags that
+allow to configure the deployment:
 
-The command above will generate the `qhub-config.yaml` config file
-with an infrastructure deployed on `aws`, named `projectname`, where
-the domain will be `qhub.dev`.
-
-The deployment
-will use `github-actions` as the continuous integration (CI) provider,
-automatically provisioning a repository on GitHub under the URL `github.com/quansight/projectname`
-
-User authentication will be by `auth0`, and an OAuth2 app will be created on Auth0 automatically.
-
-There are several flags that allow you to configure the deployment:
-
-- `aws` indicates that the project will be deployed on the Amazon AWS Cloud provider.
-    + Other providers are: `gcp`, `do` and `azure`.
+- `aws` indicates the QHub is going to be deployed on the Amazon AWS Cloud provider.
+    + Other optional flags include: `gcp`, `do` and `azure`.
 - `--project`: the name of the project is required to be a string compliant with the Cloud provider recommendations. For
-  more details see official Cloud provider docs on naming policies and check the [note below](#project-naming-convention).
-- `--domain`: domain for your cluster. This pattern is also required if you are setting your own DNS through a provider other than Cloudflare.
-- `--ci-provider`: specifies what provider to use for CI/CD. Currently, supports GitHub Actions, GitLab CI, or none ("github-actions", "gitlab-ci", or "none").
-- `--auth-provider`: This will set configuration file to use the specified provider for user authentication (value can be "auth0", "github", or "password").
-- `--auth-auto-provision`: This will automatically create and configure an application using OAuth (assuming `--auth-provider` is `auth0` or `github`).
-- `--repository`: Repository name that will be used to store the Infrastructure-as-Code on GitHub.
-- `--repository-auto-provision`: Sets the secrets for the GitHub repository used for CI/CD actions.
+  more details see official Cloud provider docs on naming policies and see below on the [project naming convention](#project-naming-convention).
+- `--domain`: base domain for your cluster. This pattern is also applicable if you are setting your own DNS through a different provider.
+  + `qhub.dev` is the domain registered on CloudFlare. If you chose not to use Cloudflare, skip this flag.
+- `--ci-provider`: specifies what provider to use for CI/CD. Currently, supports GitHub Actions, GitLab CI, or none.
+- `--auth-provider`: Used to specified authentication provider, in this case Auth0
+- `--auth-auto-provision`: Whether or not to automatically create and configure the authentication provider.
+- `--repository`: Repository name used to store the Infrastructure-as-Code on GitHub.
+- `--repository-auto-provision`: Sets the secrets for the **GitHub** repository used for CI/CD actions.
 - `--ssl-cert-email`: Provide an admin's email address so that LetsEncrypt can generate a real SSL certificate for your site. If omitted, the site will use a self-signed cert that may cause problems for some browsers but may be sufficient for testing.
 
-You will be prompted to enter values for some of the choices above if they are omitted as command line arguments (e.g. project name and domain).
+For configuration items not set as command line arguments, the script is going to prompt you to enter these values.
 
-The `qhub init` command will also generate an initial password for your root Keycloak user:
+The `qhub init` command also generates an initial password for your root Keycloak user:
 
 ```
 Securely generated default random password=R1E8aWedaQVU6kKv for Keycloak root user stored at path=/tmp/QHUB_DEFAULT_PASSWORD
 ```
 
-The password will also be available in the `qhub-config.yaml` file under the `security.keycloak.initial_root_password` field. It will be needed in the next page of these docs for logging in to your QHub.
+This password is also available in the `qhub-config.yaml` file under the `security.keycloak.initial_root_password field`. It's required in the next page of these docs for logging in to your QHub.
+
+This `qhub init` command generates the `qhub-config.yaml` config file
+with an infrastructure to be deployed on `aws`, named `projectname`, with a
+domain name set to `qhub.dev`. The deployment uses `github-actions` as
+the continuous integration provider,
+automatically provisioned and authenticated by `auth0`. And finally, initialized on
+GitHub under the URL `github.com/quansight/projectname`.
 
-<a href="#" name="project-naming-convention"></a>
-> Note: **Project Naming Convention**
->
-> In order to successfully deploy QHub, there are some project naming conventions which need to be followed. For starters,
-make sure your name is compatible with the specific one for your chosen Cloud provider. In addition, QHub `projectname`
+If employing an infrastructure-as-code approach, this is where you would make the desired infrastructure changes
+including adding users, changing Dask worker instance type and much more. Once you're happy with your changes you would redeploy those changes using GitHub Actions. For more details on the `qhub-config.yaml` please see [Configuration](configuration.md)
+
+##### Project naming convention
+In order to successfully deploy QHub, please follow some project naming conventions. For starters,
+make sure your project name is compatible with the specifics of your chosen cloud provider. In addition, QHub `projectname`
 should also obey to the following format requirements:
 > + letters from A to Z (upper and lower case) and numbers;
 > + Special characters are **NOT** allowed;
@@ -81,26 +82,26 @@ The `qhub init` command may have some side-effects such automatically creating a
 
 This file is the configuration file that will determine how the cloud infrastructure and QHub is built and deployed in the next step.
 
-But at this point it is just a text file! You could edit it manually if you are unhappy with the choices, or delete it and start over again.
+But at this point it's just a text file! You could edit it manually if you are unhappy with the choices, or delete it and start over again.
 
-Or it would be possible to create from scratch or re-use a `qhub-config.yaml` file - it is not essential to use `qhub init` at all, but it is often the easiest way to get started.
+Or it would be possible to create from scratch or re-use a `qhub-config.yaml` file - it's not essential to use `qhub init` at all, but it's often the easiest way to get started.
 
 To understand some ways in which you could decide to edit the YAML file, see [Advanced Configuration](configuration.md).
 
 ## Deploy QHub
 
-Finally, we can deploy QHub with:
+Finally, with the `qhub-config.yaml` created, QHub can be deployed for the first time:
 
 ```shell
 qhub deploy -c qhub-config.yaml --dns-provider cloudflare --dns-auto-provision
 ```
+> Omit `--dns-provider cloudflare --dns-auto-provision` if you are not using Cloudflare and will set up your DNS manually.
 
-Please omit `--dns-provider cloudflare --dns-auto-provision` if you are not using Cloudflare and will set up your DNS manually.
-
-The command will create the following folder structure, which is the Terraform definition language for the QHub platform that will be deployed:
+This creates the following folder structure:
 
 ```
 .
+├── .github             # if "--ci-provider github-actions" was passed in, then an action scripts folder is also generated
 ├── environments        # stores the conda environments
 ├── image               # docker images used on deployment: jupyterhub, jupyterlab, and dask-gateway
 │   ├── dask-worker
@@ -110,11 +111,12 @@ The command will create the following folder structure, which is the Terraform d
 └── terraform-state     # required by terraform to securely store the state of the deployment
 ```
 
-It will also start the deployment of your QHub, which will take around 10 minutes to complete.
+The terminal then prompts you to press `[enter]` to check auth credentials
+(which were added by the `qhub init` command); to disable the prompt, add `--disable-prompt` to the qhub deploy command.
+A first time deeployment can take around 10 minutes to complete.
 
-Part of the output will show an "ip" address (DigitalOcean or GCP), or
-a CNAME "hostname" (for AWS) according to the Cloud service
-provider. Such as:
+During the initial deployment, Digital Ocean, GCP and Azure are going to display an `"ip"` address
+whereas AWS is going to display a CNAME `"hostname"`.
 
 + Digital Ocean/Google Cloud Platform
 ```shell
@@ -154,15 +156,17 @@ Push the changes to the repository (your primary branch may be called
 git push origin main
 ```
 
-Once the files are in GitHub/GitLab, all CI/CD changes will be triggered by commits to main, and deployed via GitHub Actions or GitLab CI.  Since the infrastructure state is reflected in the repository, this workflow
-allows for team members to submit pull requests that can be reviewed
-before modifying the infrastructure, easing the maintenance process.
+Once pushed to GitHub, future commits to `main`  trigger CI/CD to redeploy
+changes the QHub cluster. Since the infrastructure state is reflected in
+the repository, this workflow allows for team members to submit pull requests
+and perform code reviews before modifying the infrastructure, easing the
+overall maintenance process.
 
-To automatically deploy:
+To automatically deploy (and to keep track of changes more effectively):
 - make changes to the `qhub-config.yaml` file on a new branch.
 - create a pull request (PR) to main.
-- Trigger the deployment by merging the PR. All changes will be
-  automatically applied to the new QHub instance.
+- Trigger the deployment by merging the PR and changes are
+  automatically applied to the QHub cluster.
 
 -----