Skip to content

Commit

Permalink
[DOC] Add troubleshooting docs (#1139)
Browse files Browse the repository at this point in the history
* Add troubleshooting docs

* Fix vale errors in login.md

* Vale fixes

* Swap out DNS doc ref

* 📝  Minor improvements to docs

* Update docs/source/installation/login.md

Co-authored-by: Tania Allard <taniar.allard@gmail.com>

* Update docs/source/installation/login.md

Co-authored-by: Tania Allard <taniar.allard@gmail.com>

* Update docs/source/installation/login.md

Co-authored-by: Tania Allard <taniar.allard@gmail.com>

* Update docs/source/installation/login.md

Co-authored-by: Tania Allard <taniar.allard@gmail.com>

* Update docs/source/user_guide/troubleshooting.md

Co-authored-by: Tania Allard <taniar.allard@gmail.com>

* updates from review

* Minor change to link name

* updates from review

* link to conda-store repo

* updates from review

* delete duplicates

Co-authored-by: Tania Allard <taniar.allard@gmail.com>
  • Loading branch information
iameskild and trallard authored Mar 3, 2022
1 parent 3470504 commit 62196c9
Show file tree
Hide file tree
Showing 4 changed files with 66 additions and 24 deletions.
51 changes: 29 additions & 22 deletions docs/source/installation/login.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,24 +8,24 @@ The `root` Keycloak user is only able to login and manage the Keycloak identity

## Change Keycloak root password

After the inital deployment, it is **highly** recommended that you change the Keycloak `root` user password as soon as you can.
After the initial deployment, it is **highly** recommended that you change the Keycloak `root` user password as soon as you can.
> NOTE: From this point on, the `security.keycloak.initial_root_password` field in `qhub-config.yaml` has no effect. If you redeploy QHub it will not reset the password back to the old one (or anything else that might be in the field in your YAML file). We strongly recommend you delete this field.
To change the `root` user password, navigate to `https://myqhubsite.com/auth/admin/`, and login with the pasword provided.
1. To change the `root` user password, your QHub's instance admin dashboard i.e <https://myqhubsite.com/auth/admin/>, and log in with the password provided.

![Root Login to Keycloak](../images/keycloak_master_login.png)
![QhHub admin view - Root Login to Keycloak form](../images/keycloak_master_login.png)

From there, click on the 'Root' dropdown in the top right of the screen, and select 'Manage account'.
2. From there, click on the 'Root' dropdown in the top right of the screen, and select 'Manage account'.

![Keycloak root user, manage account](../images/keycloak_root_user_manage_account.png)
![Keycloak root user page - manage account tab selected](../images/keycloak_root_user_manage_account.png)

Under 'Account Security' click 'Signing In'.
3. Under 'Account Security' click 'Signing In'.

![Keycloak root user, account security](../images/keycloak_root_user_account_security.png)
![Keycloak root user page - account security](../images/keycloak_root_user_account_security.png)

In the Password box, click the 'Update' button. This will guide you through entering your existing root password, and then creating a new password.
4. In the Password section, click the 'Update' button. This will guide you through entering your existing root password, and then creating a new password.

![Keycloak root user, update password](../images/keycloak_root_user_update_password.png)
![Keycloak root user page - account security, update password](../images/keycloak_root_user_update_password.png)


## Adding a QHub user
Expand All @@ -37,41 +37,47 @@ You will need to add a QHub user in order to log in to your QHub platform. If yo
To add a QHub user from the web console for Keycloak, visit <https://myqhubsite.com/auth/admin/>. Log in using the username `root`, as shown above.

All QHub users will be part of the `qhub` realm (a realm is a distinct identity provider and set of users in Keycloak).

> NOTE: The root user alone is a member of the `master` realm.
The `qhub` realm should be selected by default.
The `qhub` realm is selected by default, we strongly recommend leaving it as is.

Click 'Users' along the left-hand side of the page.
1. Click 'Users' along the left-hand side of the page.

Click the 'Add user' button and you will see the new user form:
2. Click the 'Add user' button and you will see the new user form:

![Keycloak add user tab screenshot - new user form ](../images/keycloak_add_users.png)

There are three fields, outlined above, which should be filled out. These are 'Username', 'Email', and 'Groups'.
3. Fill out the three fields, outlined above. These are 'Username', 'Email', and 'Groups'.

Depending on the authentication provider selected ('password', 'GitHub' or 'Auth0'), the values entered into the 'Username' field will differ slightly. The following table outlines those differences:

| | Password | GitHub | Auth0 |
|---|---|---|---|
| Username | *unique username* | *GitHub username* | *Email to login with* |

Once the 'Username' field has been updated, please add a valid email address in the 'Email' field.
> NOTE: Although not required, users may not be able to log into Graphana if this field is not properly set.
Once the 'Username' field is updated, please add a valid email address in the 'Email' field.
> NOTE: Although not required, users may not be able to log into Grafana if this field isn't properly set.
Lastly, associate the user with one or more of the 'Groups'. Out of the box, QHub is deployed with the following groups: 'admin', 'analyst', and 'developer' (see below for more information about 'Groups').
Lastly, associate the user with one or more of the 'Groups'. Out of the box, QHub is deployed with the following groups: 'admin', 'analyst', and 'developer' (see the [Groups](./login.md#groups) section below for more details).

Enter the name you would like for the user then click 'Save'.

Once the user has been created, you can set a password
> NOTE: not needed for GitHub/Auth0 based authentication.
Once the user is created, you can set a password.
> NOTE: Not needed for GitHub/Auth0 based authentication.
![Keycloak add user > credentials tab screenshot - set password](../images/keycloak_user_password.png)

It's best to unset the 'Temporary' on/off button so the user won't be forced to change the password on first login.

### Add user from command line
### Add user from the command line

To make adding users easier for new QHub deployments, there is a QHub command that can help.

> NOTE: If you changed the initial_root_password for Keycloak this method will not work.
> NOTE: This method is primarily used by developers as a quick workaround.
To make adding users easier for new QHub deployments, there is a QHub command that can help. It will only work if the initial_root_password for Keycloak has not yet been changed, so the value available in `qhub-config.yaml` is still correct.

Run:

Expand All @@ -80,13 +86,14 @@ qhub keycloak -c qhub-config.yaml adduser <username> <password>
```

This will create a user `<username>` with the initial password provided. Omit the password completely if you are using GitHub or Auth0.

> NOTE: This will also add the user to 'analyst' group.
## Login to QHub

Your new user can now log into QHub proper (not Keycloak's admin console).
Your new user can now log into the main QHub dashboard (not Keycloak's admin console).

Visit `https://myqhubsite.com/` (or whatever domain you have chosen for your QHub).
Visit <https://myqhubsite.com/> (or whatever domain you have chosen for your QHub).

Click 'Sign in with Keycloak'.

Expand Down
32 changes: 32 additions & 0 deletions docs/source/user_guide/troubleshooting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Troubleshooting guide

## DNS domain=`your_qhub_domain` record does not exist

### Issue
During your initial QHub deployment, at the end of the `04-kubernetes-ingress` stage, you receive an output message stating that the DNS record for `your_qhub_domain` "appears not to exist, has recently been updated, or has yet to fully propagate."

### Reason for observed behavior
As the output message mentions, this is likely the result of the non-deterministic behavior of DNS.

Without going into a deep dive of what DNS is or how it works, the issue encountered here is that the when QHub tries to lookup the IP address associated with the DNS record, `your_qhub_domain`, nothing is returned. Unfortunately, this "lookup" is not as straight-forward as it sounds. To lookup the correct IP associated with this domain, many intermediate servers (root, top level domain, and authoritative nameservers) are checked, each with their own cache which was updated an unknown time ago (usually on the order of minutes but not always).

For those interested to learn more about DNS, [see this interesting comic](https://howdns.works/)).

### Troubleshooting
Again, as the output message mentions, it will ask if you want it to retry this DNS lookup again after another wait period; this wait period keeps increasing after each retry. However, it's still possible that after waiting 15 or more minutes that the DNS still won't resolve.

At this point, feel free to cancel the deployment and rerun the same deployment command again in an hour or two. Although not guaranteed, it's extremely likely that the DNS will resolve correctly after this prolonged wait period.


## A Conda-Store environment fails to build

### Issue
One of the two (`dashboard` or `dask`) [Conda-Store](/~https://github.com/Quansight/conda-store) environments created during the initial QHub deployment fails to appear as options when logged into JupyterHub.

If your user has access to Conda-Store, you can verify this by visiting `<your_qhub_domain>.com/conda-store` and having a look at the build status of the missing environment.

### Reason for observed behavior
The reason for this issue is due to how these environments are simultaneously built. Under the hood, Conda-Store relies on Mamba/Conda to resolve and download the specific packages listed in the environment YAML. If they both environment builds try to download the same package with different versions, the build that started first will have their package overwritten by the second build. This causes the first build to fail.

### Troubleshooting
To resolve this issue, navigate to `<your_qhub_domain>.com/conda-store`, find the environment build that failed and trigger it to re-build.
4 changes: 2 additions & 2 deletions qhub/stages/checks.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ def _attempt_tcp_connect(host, port, num_attempts=NUM_ATTEMPTS, timeout=TIMEOUT)
s.settimeout(5)
result = s.connect_ex((ip, port))
if result == 0:
print(f"Attempt {i+1} succedded to connect to tcp://{ip}:{port}")
print(f"Attempt {i+1} succeded to connect to tcp://{ip}:{port}")
return True
print(f"Attempt {i+1} failed to connect to tcp tcp://{ip}:{port}")
except socket.gaierror:
Expand Down Expand Up @@ -152,7 +152,7 @@ def _attempt_dns_lookup(
if not disable_prompt:
input(
f"After attempting to poll the DNS, the record for domain={domain_name} appears not to exist, "
f"has recently been updated, or has yet to fully propogate. This non-deterministic behavior is likely due to "
f"has recently been updated, or has yet to fully propagate. This non-deterministic behavior is likely due to "
f"DNS caching and will likely resolve itself in a few minutes.\n\n\tTo poll the DNS again in {sleeptime} seconds "
f"[Press Enter].\n\n...otherwise kill the process and run the deployment again later..."
)
Expand Down
3 changes: 3 additions & 0 deletions tests/vale/styles/vocab.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
addons
admin
Anand
api
apis
Expand Down Expand Up @@ -34,6 +35,7 @@ dev
digital
Dockerfile
Dockerfiles
docstrings
doctl
ebs
EBS
Expand All @@ -43,6 +45,7 @@ eks
EKS
elasticsearch
emacs
Email
env
gcp
git
Expand Down

0 comments on commit 62196c9

Please sign in to comment.