pyo3 crate recompiled on (every?) build #1708
Comments
|
See also #1551 This might be fixed on main / upcoming 0.14 release depending on your build environment. I think that rust-analyzer runs things like This is worthy of a topic in the guide, and probably there could be more discussion whether we need to make further changes. |
|
Ah good to know! I tried setting |
|
Thanks. I've also been considering reaching out to the rust-analyzer folks who might be more familiar with all the cases when pyo3's build.rs is invoked, however haven't gotten around to that yet. |
|
I have pyo3 as an optional dependency for this reason. |
|
I have this happening semi-randomly (happens "almost all the time", but, not always). I am running rust-analyzer (in emacs via lsp-mode), and mostly notice this when switching to terminal to Running build with I get the following output (can probably change 'trace' to 'info') Basically rust-analyzer uses the same [..snip..] {"RerunIfEnvChanged":{"var":"PATH","val":"/home/dbr/.cargo/bin:/home/dbr/.cargo/bin:[..snip..]"}}],"rustflags":[],"metadata":0,"config":0,"compile_kind":0} [..snip..]So I guess this issue basically becomes: why is |
|
After much poking around, I think the problem lies in For now I think configuring |
|
Thanks for the investigation! If the rustup bug getting fixed does indeed remove this frequently-encountered papercut, I'll be very happy 😄 |
…build-config rebuilds See PyO3/pyo3#1708 (comment)
|
The PR that fixes it in |
When I run "scripts": {
"build": "cargo build"
}It recompiles pyo3 every time. But when I run I think it's because yarn adds temp path, which changes every time. Tho running via |
|
Unfortunately, problem still remains. In my case it is only manifest itself when rust-analyzer is running in IDE. |
|
I can confirm too. I use rust-analyzer (extension v0.3.1402) with vscode and the py03 gets recompiled every time. use pyo3::{types::PyDict, types::PyModule, PyResult, Python}; |
|
Can you try running your builds with |
|
I managed to solve a problem. Or at least found a decent workaround.
Seems like the same issue: Full log: Looking at [env]
PYO3_PYTHON = "/usr/local/bin/python3"And now everything works like a charm 🍾. Hope this helps. |
|
Thanks for the repro and suggestion @bazhenov !
Would there be an approach that generalizes to others who have different locations for their Python executable? |
It's kinda is. Cargo config can be located not in project but home directory – https://doc.rust-lang.org/cargo/reference/config.html |
|
We see:
on basically every CI build. I believe I can explain what's happening: We use Because we are a PEP517 package, when This leads to invalidating the |
|
Disregard this, it was entirely wrong 😭. Turns out we were sharing caches between different |
|
Is there any update on this? I still experience PyO3 recompiling every time when using |
|
They seem to have found a workaround. The documentation in ipetkov/crane#486 specifies setting a path variable for the python binary in NIX. It can also just be put into the [env] section of ...
[env]
PYO3_PYTHON = "path/to/python" |
|
@Jgfrausing This workaround was already described in a previous comment. |
|
Thanks. Maybe also mark this as solved, so that others know to look for a solution within this issue. EDIT: I see, you've also covered that. Thanks, mate. |
|
I tried the workaround with [env]
PYO3_PYTHON = "./venv/bin/python"but it does not change anything for me. When |
# .cargo/config.toml
[env]
PYO3_PYTHON = "./.venv/bin/python"Worked for me. Had to make it ( |
|
FWIW, I'm having the same problem when using RustRover. Thanks to everyone for the workaround suggestions. |
|
For any other Emacs users out there, what I ultimately settled on was, in my .emacs And I set the same variables with the same values in my |
|
I could only stop the invalidation by using a different target subdirectory for RA, setting |
Running `cargo build` in the project root directory (not necessarily
useful, I did it by accident) means you also build `pumpkin-py`.
However, I have a new laptop so I didn't yet have need to install the
development headers for Python (`python3-dev` package on APT). This
causes the the build to fail with the below linker error:
```
= note: /usr/bin/ld: cannot find -lpython3.13: No such file or directory
collect2: error: ld returned 1 exit status
```
This confused me (I've used PyO3 before), it's just an extension module
(you're calling Rust from Python so you always have an interpreter) so
why would you need the dev headers, which you usually only need if
you're calling Python from a Rust executable
(https://pyo3.rs/v0.22.6/index.html?highlight=python3-dev#using-python-from-rust).
I then noticed the `extension-module` feature is not enabled in the
`Cargo.toml`. It is enabled in the `pyproject.toml` so it works fine if
you build from `maturin`. However I think it's useful (to prevent
confusion) to have `cargo build` when invoked on the whole workspace to
still not fail. Plus, I think it's nice you can still just run `cargo
build` even in `pumpkin-py` without needing a global package that isn't
really required under these circumstances. It also means the
dependencies in the `Cargo.toml` don't really match what you are
actually building when you use `maturin`.
There's a tiny wrinkle, though. Maybe it's the reason you removed the
feature from the `Cargo.toml` in the first place. When using workspaces
and an IDE that runs `cargo check` on save, probably the Python
interpreter will not match the one in the virtualenv that you use for
maturin. This causes PyO3 to recompile on most builds. Thankfully I've
had this problem and this can be fixed, see
PyO3/pyo3#1708. This does mean that by default
someone who will develop the python interface will need to add something
to their config manually. The alternative solution however means that
cargo build will also fail without some manual work by the user
(installing a venv to the right place).
When testing, I found that the current version of the Python script
actually fails, so I fixed that and updated the README with the correct
path to the file to run. So even if you disagree with the main change
that part is definitely useful.
I don't have a minimal repro yet, but the pyo3 crate seems to be recompiled on most builds which significantly increases compilation times. I'm on pyo3 version 0.13.2, rustc version 1.53.0.
The text was updated successfully, but these errors were encountered: