Sharing Rust structs in foreign libraries / Writing complex wrapper classes

[warriorstar-orion]

I apologize if this has been answered somewhere already, but I've tried to understand the documentation and code search GitHub, and can't really find a solution that seems to apply for my issue. There seems to be a few related discussions which I've mentioned below. I also apologize as I'm pretty unfamiliar with Rust and this question might actually be super easy.

Most examples in the docs focus on rust structs that can be annotated and have primitive members. I am attempting to wrap an external crate, with various nested structs and containers. I've created a minimal example at /~https://github.com/warriorstar-orion/pyo3_complex_wrapper.

You can see the "foreign" structs are:

pub struct Foo {
    pub mapping: HashMap<String, Vec<Bar>>,
}

pub struct Bar {
    pub num: i32,
    pub word: String,
}

and my wrapper pyclasses are

#[pyclass(module = "pyo3_complex_wrapper")]
#[pyo3(name = "Foo")]
pub struct FooWrapper(foreign::Foo);

#[pyclass(module = "pyo3_complex_wrapper")]
#[pyo3(name = "Bar")]
pub struct BarWrapper(foreign::Bar);

foreign::Foo has a demonstration method compute which iterates over its Bars and does something with them. So far, so good:

>>> from pyo3_complex_wrapper import Foo
>>> foo = Foo()
>>> print(foo.compute())
hellohello
therethere
worldworldworld
>>> bars = foo.pop("greeting") # obvious ownership change
>>> bars
[<Bar 2, 'hello'>, <Bar 2, 'there'>]
>>> print(foo.compute())
worldworldworld
>>> bars[0].word = 'goodbye'
[<Bar 2, 'goodbye'>, <Bar 2, 'there'>]

The issue arises when I attempt to implement something like this (ignore how terrible the indirection is):

>>> from pyo3_complex_wrapper import Foo
>>> foo = Foo()
>>> bar = foo["greeting"][0]
>>> bar.word = "goodbye"
>>> print(foo.compute())
goodbyegoodbye
therethere
worldworld

My understanding is that by assigning a bar to a python local, we're crossing this boundary where the native Bar struct needs to be owned by both the vec inside the Foo's IndexMap, as well as by Python, because the BarWrapper could exist forever and be assigned to as many locals as we want. It seems like PyCell and PyRef/Mut were introduced for handling these cases, but they seem meant for structs we can annotate as #[pyclass], or for which traits can be written.

I've attempted several solutions:

• This discussion: simple reference example #3473 seems to get close to what I want but I'm having trouble applying it to this case due to the layers of indirection.
• Using std::mem::transmute seems like heresy but does work e.g. if the BarWrapper instead contains a reference to the FooWrapper and the key name, but I couldn't find the right incantations to get it to work with elements in a pure Rust vec. I also want to be able to create Bars independent of the Foo and insert them, so it would need to function in reverse as well, which I can't seem to pin down.
• Ivan Kud's article How I Design And Develop Real-World Python Extensions In Rust suggests using an Arc<Mutex<RustStruct>> pattern, but I think the issue here is that this requires ownership that conflicts with the ownership of the Foo's vec which contains the Bars. I wasn't able to find a way to say "keep this struct in this Vec but let me give out mutable, safe references that are wrapped in a #[pyclass] somewhere along the way".
• The discussion on this issue: How to make Rust struct in other crate accessible from Python? #1581 suggests that in a complex case a better approach would be to write a layer between Rust and Python that does the bookkeeping of translating from PyObjects to structs but it sounds like it would end up something like this, which would become pretty onerous for a struct that does a lot of work:

#[pyclass]
struct FooWrapper {
    foo: foreign::Foo,
    bars: PyDict<PyList<BarWrapper>>,
}

#[pymethods]
impl FooWrapper {
    #[new]
    fn new() -> Self {
        // Remove all the elements from foo.mapping so they can be owned by the wrapper/pylist/pydict
    }

    fn compute() -> String {
        // We gotta put all the objects back into foo.mapping so that the foreign struct can do the work
    }
}

Again, sorry if one of the above solutions applies and it's just a matter of implementation; the documentation might benefit from a section specifically on "wrapper" libraries with a fleshed out example similar to this minimal case, or I just may need to learn more Rust before tackling this.

[adamreichold]

You can see the "foreign" structs are:

Those structures do not support shared ownership (not even in Rust) and hence you cannot use that on language level. You would would have to introduce something like Arc or our own Py smart pointer into the mix so that you would store e.g. HashMap<String, Vec<Arc<Bar>> or HashMap<String, Vec<Py<Bar>>.

If you really cannot modify these structures because they are defined by a foreign crate, I would suggest to use shared ownership only conceptually but not actually, for example do not store Bar within BarWrapper, but the necessary data to access the bar, e.g.

struct BarWrapper {
  foo: Py<FooWrapper>,
  key: String,
  index: usize,
}

impl BarWrapper {
   fn value(&self, py: Python<'_>) -> &Bar {
      &self.foo.borrow(py).0[&self.key][self.index]
   }
}

(I have not compiled this, so it will most likely trigger some compiler errors, but I think it can be made to work.)

[warriorstar-orion]

Thanks for the response! That seems to be a good way forward for the rust-owned Bars. If I want to create Bars in Python and add them to the Rust struct, it seems like I might need to have one more layer of indirection, something like:

// BarWrapper from above but renamed
struct LookupBar {
  foo: Py<FooWrapper>,
  key: String,
  index: usize,
}

enum BarInstance {
  Shared(LookupBar),
  Unique(Bar),
}

#[pyclass]
struct BarWrapper {
  inner: BarInstance,
}

#[pymethods]
impl BarWrapper {
    pub fn __str__(&self, py: Python<'_>) -> String {
        match self.inner {
            BarInstance::Shared(s) => {
                let bar = s.foo.borrow(py).0.mapping[&s.key][s.index];
                return format!("<Bar {}, '{}'>", bar.num, bar.word);
            }
            BarInstance::Unique(b) => {
                return format!("<Bar {}, '{}'>", b.num, b.word);
            }
        }
    }

    /* ... */
}

That way I can switch between one or the other as I add or remove them from the Foo, and manage the 'conceptual sharing' under the hood, possibly with most of the implementation on the BarInstance enum itself. Does that seem right?

[adamreichold]

Yes, if the Bar are created in standalone manner, the above seems like a good approach. Alternatively, you could limit creation of Bar from Python to methods on FooWrapper which would take care of creation and insertion as a single operation (from Python's point of view).