-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update conda-store docs #399
Changes from 7 commits
e995e68
40c60cc
4dd432f
987df84
1f5deb8
df9f7b7
4570ba4
8043b67
14e1eef
9f1df5b
5af20c3
27d6bf2
4a376e4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,181 +1,94 @@ | ||
--- | ||
id: creating-new-environments | ||
title: Creating a new environment on Nebari | ||
description: Using conda-store for environment creation | ||
title: Manage packages and environments | ||
description: Using conda-store for environment management | ||
--- | ||
|
||
## Introduction | ||
Nebari uses [`conda-store`][conda-store-docs] for managing and sharing reproducible environments (collection of specific packages and versions) on the platform. | ||
|
||
In this tutorial, you will learn how to create a new environment on Nebari using `conda-store`, | ||
install new `conda` packages in your environment, manage multiple environments, and also share the environment with other Nebari users. | ||
:::warning | ||
conda-store is the most reliable way to manage your packages and environments on Nebari. | ||
|
||
### Why using `conda` environments | ||
|
||
Development environments give developers and Data Scientists a "sandbox" to work in, experiment with new packages, work across multiple projects, and collaborate with their peers. | ||
Environment and package managers such as `conda` enable you to create isolated spaces to separate dependencies on a per-project basis. | ||
They also add other benefits such as: | ||
|
||
- Reduce friction to install and upgrade dependencies without affecting other projects or users | ||
- Reduce the risk of dependency conflicts | ||
- Promote reproducibility and replicability through dependency pinning | ||
- Reduce the "works on my machine" risk - thus working towards reproducible builds across multiple machines | ||
|
||
### What is `conda`? | ||
|
||
[Conda](https://docs.conda.io/projects/conda) is an open source package management system that allows you to create | ||
environments and install packages into them. It allows the creators of a package to specify required dependencies | ||
which `conda` can then solve into an environment. `Conda` can then run an update on the environment to pull all the | ||
latest versions of the packages while still maintaining compatibility. | ||
|
||
While `conda` manages compatibility between the packages in the environment, by default, this does not guarantee reproducible builds over time. | ||
An environment created with a list of packages _today_ can differ from that same environment created with the same | ||
list of packages _tomorrow_. This can happen because package dependencies have changed, new releases have occurred, | ||
or even because a package is no longer available. | ||
|
||
### What is `conda-store`? | ||
|
||
[`conda-store`][conda-store-docs] is a Python package that serves _identical_ `conda` environments by controlling the | ||
environment lifecycle. | ||
It ensures that the management, building, and serving of environments is as identical as possible and seamless for the end users. | ||
|
||
All environments in Nebari are served through `conda-store`. | ||
|
||
Using `conda-store`, Nebari admins can track specific files or directories for changes in environment specifications. | ||
They can manage environments using the web interface, REST API, or the command-line utility (CLI). | ||
This tutorial focuses on using the web interface to interact with our environments. | ||
|
||
## Step 1 - Exploring the `conda-store` web interface | ||
|
||
1. To get started, navigate to `https://<your-nebari-domain>/conda-store` (e.g. <https://quansight.nebari.dev/conda-store>). | ||
|
||
2. You will need to log in to authenticate to the `conda-store` interface - this is required to be able to access most of `conda-store` functionalities. | ||
|
||
![conda-store login UI - before authentication](/img/tutorials/conda_store_login.png) | ||
|
||
3. This will lead you through a series of windows to authorize with Keycloak, after which you will have access to the | ||
`conda-store` dashboard. By default, you will be directed to the user page containing information about your account and the associated permissions. | ||
|
||
![conda-store dashboard UI - after authentication](/img/tutorials/conda_store_dashboard.png) | ||
|
||
Some useful sections to notice in the `user` page are: | ||
|
||
- **User**: this section of the dashboard allows users to explicitly logout of the interface. | ||
- **Namespaces**: `Namespaces` are an important part of the `conda-store` [authorization model][conda-store-docs-auth]. They | ||
control what level of access users are provided to the individual environments. In other words, based on your | ||
permissions in each namespace, your ability to create, read, update, or delete and environment will differ. | ||
- **Permissions**: your current permissions in each namespace. | ||
|
||
That is, unless your admin has configured your namespace differently. You can see here that the `conda-store` | ||
authorization model is able to provide admins with a fine-grained level of control. | ||
|
||
## Step 2 - Creating a new environment | ||
|
||
1. To create a new environment, click on the `Create New Environment` button on the top right of the navigation bar. You will | ||
be presented with an option to upload a | ||
[conda YAML file](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file) | ||
or write your own. | ||
Refrain from installing libraries directly in the notebook or through the terminal (outside a conda-store managed environment). These actions could lead to subtle and unforeseen problems with your environment. | ||
::: | ||
|
||
![conda-store create environment UI](/img/tutorials/conda_store_create_env.png) | ||
## Preliminary reading | ||
|
||
For this tutorial, you can copy and paste the following environment specification: | ||
It's useful to understand basics of conda-store and how it builds on top of the conda ecosystem, to use it effectively in Nebari. | ||
|
||
```yaml title="Sample environment specification" | ||
channels: | ||
- conda-forge | ||
dependencies: | ||
- python=3.9 | ||
- numpy | ||
- matplotlib | ||
- pandas | ||
- panel | ||
- ipykernel | ||
- ipywidgets | ||
name: example-environment | ||
prefix: null | ||
``` | ||
- [Essential conda concepts: Packages, environments, channels, dependencies, etc.](https://conda.store/conda-store/explanations/conda-concepts) | ||
- [conda-store concepts: Reproducibility, namespaces, environment versions, roles, etc.](https://conda.store/conda-store/explanations/conda-store-concepts) | ||
|
||
2. After you copy the above into the UI, go ahead and click the `Submit` button. | ||
Nebari has conda-store integrated, and you can use it through the graphical UI. | ||
|
||
3. You will be redirected to the environments overview page. Find the card with the name of the new environment under your user's namespace (e.g. | ||
`<your-username>/example-environment`). | ||
## Open `conda-store` web interface | ||
|
||
![Newly create environment card](/img/tutorials/conda_store_new_env.png) | ||
There are several options for navigating to conda-store: | ||
|
||
4. Click on the card for your newly created environment. The UI will display the environment specification: | ||
* From Nebari Home, click on **"Environment Management"** under "Services" | ||
* From JupyterLab, click on `Nebari` in the menu bar and go to **"Environments"** | ||
* From anywhere, go to URL: `https://<your-nebari-domain>/conda-store` | ||
|
||
![conda-store - Environment details page](/img/tutorials/conda_store_env_details.png) | ||
If not logged in to conda-store, click on the **"Log in"** button in th left sidebar and authenticate similar to the [Nebari login][login-keycloak]. This is required to be able to access many `conda-store` features. | ||
|
||
If this is the first time visiting this page, the environment may still be in the process of building. | ||
The page will automatically update when the build is complete. | ||
## Create, edit, and manage environments | ||
|
||
From this page, you can `Edit` your YAML specification, or even `Delete` the environment. | ||
Go through the following conda-store (UI) tutorials | ||
on using the graphical interface for various actions: | ||
|
||
:::note | ||
`conda-store` tracks all the environments behind the scenes for the users. Even if a user "deletes" an environment, | ||
it will still exist in the store. This ensures admins always have access to environment history. | ||
::: | ||
* [Create new environments][cs-create-env] | ||
* [Edit & delete existing environments][cs-edit-delete-env] | ||
* [Switch environment versions][version-control] | ||
|
||
5. Now let's take a closer look at the `conda` build detail, to do this, click on the build number link in the `conda-store` interface. | ||
This will display a new page with the metadata about the environment including the time, size and status. | ||
Also included is your original YAML specification and a list of all the installed dependencies, their version and the used `conda` channels. | ||
## Default namespaces in Nebari | ||
|
||
![conda-store UI - Sample environment build details page: showing build details, specification file, and conda packages](/img/tutorials/conda_store_build_details_top.png) | ||
A default Nebari deployment/instance has the following namespaces corresponding to [Nebari groups][configure-keycloak-groups]: | ||
|
||
Scroll down to the bottom of this list, and you'll see a section called `Conda Environment Artifacts`. | ||
This is where you can download your own copy of the YAML file, a [conda-lock](https://conda.github.io/conda-lock/) file, and a `tar.gz` archive of the environment. | ||
<!-- Verify the roles and actions --> | ||
|
||
![conda-store UI - Sample environment build details page: showing the conda environment artifacts](/img/tutorials/conda_store_build_details_bottom.png) | ||
* `analyst` namespace - Users in the `analyst` group can view and `admin` group can view+edit the environments in this namespace | ||
* `developer` namespace - Users in the `developer` and `admin` groups can view+edit the environments in this namespace | ||
* `nebari-git` namespace - Everyone can view and `admin`s can edit | ||
|
||
6. Lastly, click on `Full Logs` to view the full output from the `conda` build. | ||
As an individual user, you also have a personal namespace with the same name as your Nebari username. | ||
|
||
:::note | ||
If you want to use your new environment in a Jupyter Notebook, don't forget to include `ipykernel` and `ipywidgets` in | ||
your environment’s `yaml` file, or it may not be visible in the list of available kernels. | ||
If you can "view" an environment, you can use it. | ||
::: | ||
|
||
## Step 3 - Installing packages via `conda` or `pip` | ||
## Select environments in editors | ||
|
||
To install new packages through either `conda` or `pip` you'll need to navigate back to the `Edit` page of your environment | ||
in the `conda-store` web interface. | ||
Instructions to select any environment you have access to in the following editing spaces: | ||
|
||
Adding `conda` packages to a `conda-store` environment via the command line, is not possible since the files are | ||
read-only in that context. | ||
* **JupyterLab** - In a Jupyter Notebook, click on the "Select Kernel" dropdown in the top-left corner, and select the environment. | ||
|
||
Additionally, adding `pip` packages via the command line is strongly discouraged. Not only do `conda` and `pip` not | ||
always play nice together, but it can lead to some inconsistencies across users' dependencies. | ||
This happens because when you `pip` install a package from the command line this is installed in your `.local` folder, so this changes will not be applied to other Nebari users. | ||
* **VS Code** - Click on the ⚙️ icon in the bottom-right to open `Settings` -> `Command Pallette`, and type "Python: Select Interpreter" and press <kbd>Enter</kbd> to get the list of environments to select from. | ||
|
||
:::note | ||
One exception to this rule is packages that you are actively developing. | ||
As you are building a package, you will likely want to install it as a _dev_ package. | ||
This can be done using: | ||
|
||
```bash | ||
pip install --no-build-isolation --user -e . | ||
``` | ||
* **Terminal** - In the terminal window, you can use `conda` CLI commands like `conda activate <namespace>-<environment_name>` to activate the relevant environment and `conda env list` to view the list of available environments. | ||
|
||
Or, if you’re using `flit`, you can install with: | ||
## Special requirements | ||
|
||
```bash | ||
flit install -s | ||
``` | ||
### Dask | ||
|
||
Please keep in mind that these are _NOT_ available to Dask workers! | ||
::: | ||
Include the [`nebari-dask` metapackage](https://anaconda.org/conda-forge/nebari-dask) in your environment to use Dask. This ensures you have the correct version of `dask-gateway` and the latest versions of `dask` and `distributed` libraries. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Add note that the nebari-dask metapackage version should match nebari version There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. also note again here that you have to be in the develop group to be able to access dask. |
||
|
||
## Troubleshooting | ||
By default, the `nebari-git-nebari-git-dask` environment (available to everyone) can be used for basic Dask workflows. | ||
|
||
1. If you have an environment that fails to build properly, you'll be able to see this failure on the build status page. | ||
### JHub App Launcher | ||
|
||
Navigate to the `Full Logs` to investigate in more detail. Also, from the build status page you can trigger re-build in | ||
case you hit issues with intermittent outages, etc. | ||
Include the `jhub-apps` package in your environment to create apps using the JHub App Launcher. You will also need the relevant app development framework and other necessary packages in the environment. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You don't need jhub-apps itself. You need For panel apps you'll also need There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The app launcher now has its own docs which explain the environment requirements. I think its safe to remove this here. |
||
|
||
2. If you need to use Dask. | ||
<!-- External links --> | ||
|
||
We highly recommend you include the [Nebari Dask metapackage](https://anaconda.org/conda-forge/nebari-dask) to maintain version compatibility between the Dask client and server. | ||
This replaces `distributed`, `dask`, and `dask-gateway` with the correctly pinned versions. | ||
[conda-docs]: https://docs.conda.io/projects/conda | ||
[conda-store-docs]: https://conda.store/ | ||
[conda-store-docs-auth]: https://conda.store/conda-store/references/auth#authorization-model | ||
[cs-create-env]: https://conda.store/conda-store-ui/tutorials/create-envs | ||
[cs-edit-delete-env]: https://conda.store/conda-store-ui/tutorials/edit-delete-envs | ||
[version-control]: https://conda.store/conda-store-ui/tutorials/version-control | ||
|
||
<!-- reusable links --> | ||
<!-- Internal links --> | ||
|
||
[conda-store-docs]: https://conda-store.readthedocs.io/ | ||
[conda-store-docs-auth]: https://conda-store.readthedocs.io/en/latest/contributing.html#authorization-model | ||
<!--Update when PR#397 is merged --> | ||
[login-keycloak]: /docs/how-tos/login-thru-keycloak-howto.md | ||
[configure-keycloak-groups]: /docs/how-tos/configuring-keycloak#in-depth-look-at-roles-and-groups |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm pretty sure developer is required for access to the dask workers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was about to suggest some changes, but noticed this will be confusing with the keycloak roles and the concepts of namespaces intertwined, so I will think about how to rewrite this a bit