Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build pyodide wheels in CI #2433

Merged
merged 6 commits into from
Mar 6, 2023
+172 −15
Merged

Build pyodide wheels in CI #2433

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
761c13a
try out pyodide wheels
luizirber Jan 5, 2023
e5193e8
delay loading screed
luizirber Jan 8, 2023
62d939d
revert delay loading screed
luizirber Jan 11, 2023
21acbd6
update nix actions
luizirber Mar 5, 2023
d62909b
revert screed import at toplevel in minhash.py
luizirber Mar 5, 2023
0248eff
fix cargo lock
luizirber Mar 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 28 additions & 1 deletion .github/workflows/build_wheel.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,11 +55,38 @@ jobs:
with:
path: './wheelhouse/sourmash*.whl'

build_wasm:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: 3.10.2
- run: |
pip install 'pyodide-build>=0.22.0'
pyodide config get emscripten_version # trigger setup
echo EMSCRIPTEN_VERSION=$(pyodide config get emscripten_version) >> $GITHUB_ENV
- uses: mymindstorm/setup-emsdk@v11
with:
version: ${{ env.EMSCRIPTEN_VERSION }}
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
target: wasm32-unknown-emscripten
- run: |
export RUSTC_BOOTSTRAP=1
pyodide build

- uses: actions/upload-artifact@v3
with:
path: './dist/sourmash*.whl'


release:
name: Publish wheels
runs-on: ubuntu-20.04
if: startsWith(github.ref, 'refs/tags/v')
needs: build_wheels
needs: [build_wheels, build_wasm]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fast enough (3 minutes) to be in the default wheels to be built per PR.


steps:
- name: Fetch wheels from artifacts
Expand Down
9 changes: 7 additions & 2 deletions .github/workflows/dev_envs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,19 @@ jobs:
with:
fetch-depth: 0

- uses: cachix/install-nix-action@v19
- uses: cachix/install-nix-action@v20
with:
extra_nix_config: |
access-tokens = github.com=${{ secrets.GITHUB_TOKEN }}

- uses: cachix/cachix-action@v12
with:
name: sourmash-bio
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'

- run: nix run .# -- --version

- run: nix-shell --command "tox -e py39"
- run: nix-shell --command "tox -e py310"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run with the same Python version for the nix package, so we don't need to download/build two Python versions


mamba:
runs-on: ubuntu-latest
Expand Down
130 changes: 125 additions & 5 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 8 additions & 4 deletions src/core/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,19 +78,23 @@ harness = false

## Wasm section. Crates only used for WASM, as well as specific configurations

[target.'cfg(all(target_arch = "wasm32", target_vendor="unknown"))'.dependencies.wasm-bindgen]
[target.'cfg(all(target_arch = "wasm32", target_os="unknown"))'.dependencies.wasm-bindgen]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • the pure wasm target is wasm32-unknown-unknown (used for the NPM package)
  • the pyodide wasm target is wasm32-unknown-emscripten
  • wasi is target wasm32-wasi

so setting os instead of vendor here matches wasm32-*-unknown instead of wasm32-unknown-* and avoid restricting POSIX/libc-like targets like emscripten and wasi

version = "0.2.84"
features = ["serde-serialize"]

[target.'cfg(all(target_arch = "wasm32", target_vendor="unknown"))'.dependencies.web-sys]
[target.'cfg(all(target_arch = "wasm32", target_os="unknown"))'.dependencies.web-sys]
version = "0.3.61"
features = ["console", "File"]

[target.'cfg(all(target_arch = "wasm32", target_vendor="unknown"))'.dev-dependencies]
[target.'cfg(all(target_arch = "wasm32"))'.dependencies.chrono]
version = "0.4.23"
features = ["wasmbind"]
Comment on lines +89 to +91
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't have a direct dependency on chrono, but setting this feature here solves problems in upstream deps


[target.'cfg(all(target_arch = "wasm32", target_os="unknown"))'.dev-dependencies]
wasm-bindgen-test = "0.3.34"

### These crates don't compile on wasm
[target.'cfg(not(all(target_arch = "wasm32", target_vendor="unknown")))'.dependencies]
[target.'cfg(not(all(target_arch = "wasm32", target_os="unknown")))'.dependencies]

[package.metadata.maturin]
name = "sourmash._lowlevel"
4 changes: 2 additions & 2 deletions src/core/src/errors.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ pub enum SourmashError {
#[error(transparent)]
IOError(#[from] std::io::Error),

#[cfg(not(all(target_arch = "wasm32", target_vendor = "unknown")))]
#[cfg(not(all(target_arch = "wasm32", target_os = "unknown")))]
#[error(transparent)]
Panic(#[from] crate::ffi::utils::Panic),
}
Expand Down Expand Up @@ -110,7 +110,7 @@ pub enum SourmashErrorCode {
NifflerError = 100_005,
}

#[cfg(not(all(target_arch = "wasm32", target_vendor = "unknown")))]
#[cfg(not(all(target_arch = "wasm32", target_os = "unknown")))]
impl SourmashErrorCode {
pub fn from_error(error: &SourmashError) -> SourmashErrorCode {
match error {
Expand Down
3 changes: 2 additions & 1 deletion src/core/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,8 @@ use cfg_if::cfg_if;
use murmurhash3::murmurhash3_x64_128;

cfg_if! {
if #[cfg(all(target_arch = "wasm32", target_vendor = "unknown"))] {
if #[cfg(all(target_arch = "wasm32", target_os = "unknown"))] {
// Explicitly keeping emscripten and wasi out of this
pub mod wasm;
} else {
pub mod ffi;
Expand Down