Make CDSDashboards.conda_envs dynamically update from function

[costrouc]

Closes #1310
Closes #1293

Changes introduced in this PR:

• cdsdashboards conda environments are now populated by a function. In this case the environments will be updated based on the available environments from conda-store.

Types of changes

What types of changes does your PR introduce?

Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features to not work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe):

Testing

Requires testing

• Yes
• No

In case you checked yes, did you write tests?

• Yes
• No

[costrouc]

Also related #1293

[costrouc]

I will update this issue later today. @viniciusdc will help with testing this PR once the tests pass.

[costrouc]

Will be working on this PR later tomorrow and have complete by Tuesday. Goal is to make use of the conda-store api to get available environments.

[trallard]

• @costrouc will work on setting the service tokens for conda-store
• @HarshCasper and @costrouc will then work on fixing this

---

Will be using the conda-store API

[costrouc]

@HarshCasper some details on how to accomplish this.

In the conda_store_config.py we will need to generate an api token via

c.AuthenticationBackend.predefined_tokens = {
     f"{config["cdsdashboard-secret"]}": {
           "primary_namespace": "cdsdashboards",
           "role_bindings": {
                "*/*": ["viewer"],
           }
     }
}

This is passed inside of here /~https://github.com/Quansight/qhub/blob/main/qhub/template/stages/07-kubernetes-services/modules/kubernetes/services/conda-store/server.tf#L1-L23. The more I think about it these tokens and permissions should be passed in in /~https://github.com/Quansight/qhub/blob/main/qhub/template/stages/07-kubernetes-services/modules/kubernetes/services/conda-store/variables.tf#L70 as a variable with the secret token along with permissions needed.

here you will need to create a function which calls the conda-store kubernetes service api with this endpoint https://conda-store.readthedocs.io/en/latest/api.html#/api/api_list_environments_api_v1_environment__get.

Getting started it might be easiest to hardcode the api token that is added to the conda-store config in order to test the api calls from cdsdashboards.

Later we can talk about how to link them in terraform code. We will likely do tricks similar to how this same pattern is handled in jupyterhub /~https://github.com/Quansight/qhub/blob/main/qhub/template/stages/07-kubernetes-services/modules/kubernetes/services/jupyterhub/variables.tf#L84-L87.

[viniciusdc]

@costrouc Tokens and the above configuration were all completed. I am working on a basic function to call the API.

[viniciusdc]

Removing this line generated the following error:

 Loading extra config: 01-theme.py                                                                                                                                      │
│ Loading extra config: 02-spawner.py                                                                                                                                    │
│ [E 2022-09-05 18:46:15.302 JupyterHub app:2989]                                                                                                                        │
│     Traceback (most recent call last):                                                                                                                                 │
│       File "/opt/conda/lib/python3.7/site-packages/jupyterhub/app.py", line 2986, in launch_instance_async                                                             │
│         await self.initialize(argv)                                                                                                                                    │
│       File "/opt/conda/lib/python3.7/site-packages/jupyterhub/app.py", line 2477, in initialize                                                                        │
│         self.load_config_file(self.config_file)                                                                                                                        │
│       File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 110, in inner                                                                │
│         return method(app, *args, **kwargs)                                                                                                                            │
│       File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 889, in load_config_file                                                     │
│         raise_config_file_errors=self.raise_config_file_errors,                                                                                                        │
│       File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 844, in _load_config_files                                                   │
│         config = loader.load_config()                                                                                                                                  │
│       File "/opt/conda/lib/python3.7/site-packages/traitlets/config/loader.py", line 625, in load_config                                                               │
│         self._read_file_as_dict()                                                                                                                                      │
│       File "/opt/conda/lib/python3.7/site-packages/traitlets/config/loader.py", line 658, in _read_file_as_dict                                                        │
│         exec(compile(f.read(), conf_filename, "exec"), namespace, namespace)                                                                                           │
│       File "/usr/local/etc/jupyterhub/jupyterhub_config.py", line 446, in <module>                                                                                     │
│         exec(config_py)                                                                                                                                                │
│       File "<string>", line 5, in <module>                                                                                                                             │
│       File "/opt/conda/lib/python3.7/site-packages/kubespawner/__init__.py", line 13, in <module>                                                                      │
│         from kubespawner.spawner import KubeSpawner                                                                                                                    │
│       File "/opt/conda/lib/python3.7/site-packages/kubespawner/spawner.py", line 42, in <module>                                                                       │
│         from .objects import make_namespace                                                                                                                            │
│       File "/opt/conda/lib/python3.7/site-packages/kubespawner/objects.py", line 16, in <module>                                                                       │
│         from kubernetes.client.models import V1EndpointPort                                                                                                            │
│     ImportError: cannot import name 'V1EndpointPort' from 'kubernetes.client.models' (/opt/conda/lib/python3.7/site-packages/kubernetes/client/models/__init__.py) 

Which made the Hub deployment enter a crash loop cycle. Returning the block above fixed the problem.

[viniciusdc]

It needs to be tested with QHUB_IMAGE_TAG=sha-563ef5c for the docker images, waiting for 0.4.4 release.

[viniciusdc]

Blocked by ideonate/cdsdashboards#107, I was able to test this locally, though lets-encrypt didn't want to cooperate providing a valid cert... so I was unable to properly "see" the dashboard. Still, it was successfully launched with the mentioned changes in cdsdashboards.

[viniciusdc]

This is ready for review, when testing please be sure to use the latest available image from main

[costrouc]

Will need to wait for a new release of Conda-Store which I will be releasing tonight.

[viniciusdc]

I will do another drive-test on this, then we are set

[viniciusdc]

Tested, and all seems to be working!!!

The new envs are dynamically updated in the dropdown menu once built on conda-store (cdsdashboards is required at least)

with the recent changes made to conda-store 0.4.12, we now have these weird names on the main page, but we can address this later.

[viniciusdc]

Thanks for updating the cypress tests @iameskild, once you approve, this is ready to be merged. Something that we will need to check once v0.4.5rc shows up in beta, is how conda-store old environments will behave with the name changes (we are modifying the exposed env names with this config c.CondaStore.environment_directory)

[iameskild]

@viniciusdc I tested this myself and it works wonderfully :)