Add cargo builder for rust extensions

[ianks]

After a couple years of stagnation, and thanks to the help from @qubitrenegade and @briankung finally got around to building a Rust gem extension builder for Rubygems.

How it works

tl;dr here is a fully working example of a Rust gem, and an excellent blog post from @briankung

1. If a Cargo.toml file is detected, the new Gem::Ext::CargoBuilder is invoked.
2. We invoke cargo rustc and handle the linking, and all of the gymnastics around that
3. Move the compiled dylibs to their expected locations for Ruby (i.e. my_rust_extension.{so,bundle,dll})

Why the extconf?

My reason is this: In order for ruby to successfully load a ruby extension (via dlopen), its first check to see that ruby_xmalloc matches on the built library. This ensures that the extension is only loaded if it was compiled with the same version of Ruby.

We no longer need to do this since we are correctly handling the linking process using Cargo.

Hopefully this is on the right track, would love to get some feedback and get this merged at some point!

closes #4903

[welcome]

Thanks for opening a pull request and helping make RubyGems and Bundler better! Someone from the RubyGems team will take a look at your pull request shortly and leave any feedback. Please make sure that your pull request has tests for any changes or added functionality.

We use GitHub Actions to test and make sure your change works functionally and uses acceptable conventions, you can review the current progress of GitHub Actions in the PR status window below.

If you have any questions or concerns that you wish to ask, feel free to leave a comment in this PR or join our #rubygems or #bundler channel on Slack.

For more information about contributing to the RubyGems project feel free to review our CONTRIBUTING guide

[ianks]

Though: It would be nice if this thing “just worked” with rake-compiler. That would enable cross-compilation and what not. I think in order to do that, it might be best to do the whole cargo compilation logic extconf.rn. Although I’m not sure if that would work…

[bjorn3]

First, the crate is built (either staticlib or cdylib crates will work`)

Linking two rust staticlibs together will cause symbol conflicts because each contains a copy of the standard library. If you are linking only a single rust staticlib into a dylib you should probably ensure that all mangled symbols and all symbols used by rust itself are not exported from the dylib (rustc normally handles this for you when compiling cdylibs). Using two rust cdylibs in the same program on the other hand is fine.

[ianks]

So do you think it's best to raise an error when staticlib is used @bjorn3?

[bjorn3]

I think that would be the best option indeed. I presume it would be easier to forbid staticlibs for now and maybe allow them in the future with appropriate safety guards than to allow them now and possibly have to backtrack if it causes issues.

[ianks]

I think that would be the best option indeed. I presume it would be easier to forbid staticlibs for now and maybe allow them in the future with appropriate safety guards than to allow them now and possibly have to backtrack if it causes issues.

Another option might be to add -C metadata=GEM_NAME,GEM_VERSION to prevent symbol conflicts. Also, it seems the rust docs recommend staticlib for this type of use case:

This format is recommended for use in situations such as linking Rust code into an existing non-Rust application because it will not have dynamic dependencies on other Rust code.

Thoughts?

[bjorn3]

That is unfortunately not enough. The standard library doesn't get recompiled when specifying -Cmetadata. In addition there are a couple of functions in the standard library that aren't mangled at all. (for example __rust_alloc, rust_eh_personality or __rust_foreign_exception)

[bjorn3]

Also, it seems the rust docs recommend staticlib for this type of use case:

This format is recommended for use in situations such as linking Rust code into an existing non-Rust application because it will not have dynamic dependencies on other Rust code.

Thoughts?

This recommendation makes sense when only a single rust library is linked. Unfortunately it breaks down as soon as multiple rust libraries are linked.

Another argument against staticlibs is that they don't have an easy way to specify the required dynamic libraries to link against. To get this list you need to pass --print native-static-libs to the rustc invocation producing the staticlib. This will result in a note like note: native-static-libs: -lgcc_s -lutil -lrt -lpthread -lm -ldl -lc listing all arguments you need to pass to the linker.

[ianks]

Also, it seems the rust docs recommend staticlib for this type of use case:

This format is recommended for use in situations such as linking Rust code into an existing non-Rust application because it will not have dynamic dependencies on other Rust code.

Thoughts?

This recommendation makes sense when only a single rust library is linked. Unfortunately it breaks down as soon as multiple rust libraries are linked.

Another argument against staticlibs is that they don't have an easy way to specify the required dynamic libraries to link against. To get this list you need to pass --print native-static-libs to the rustc invocation producing the staticlib. This will result in a note like note: native-static-libs: -lgcc_s -lutil -lrt -lpthread -lm -ldl -lc listing all arguments you need to pass to the linker.

Thank you so much for this info! Going to raise unless we detect a cdylib.

[ianks]

@bjorn3 Added a patch to raise unless a dylib is found

Also added full integration spec, and things seem to be working well. Hopefully we can get some more eyes on this 👀

[simi]

This is on my radar for whole time. I'm just waiting for proper time to start reviewing this. Few initial ideas, questions.

Are there any gems we can check using this already? (edit: I see there is some list at https://bugs.ruby-lang.org/issues/15759, I'll check on the approach in there.)

Would it be possible to integrate this into bundle gem template as new argument? Currently it is possible to generate C extension skeleton with bundle gem example --ext. Maybe we can extend --ext to support argument (C or RUST, and fallback to C when nothing is present) or add another argument like --rust-ext. It would be super useful for testing. But we can also add this later.

Can we make Rust extension testing optional (enabled only when Rust tools are around) so our test suite doesn't rely on Rust being set on the dev. machine?

[iliabylich]

I'm so glad to see this being (hopefully) a part of rubygems. I spent enormous amount of time building it manually here. In short: I followed a very similar path. Rust library (actually, its C bindings, but it doesn't matter) is compiled statically and included with -Lpath -lname with a a bunch of glue code that is based on RbConfig data.

Compiling Rust lib to dynamic library seems fine, but actually I don't really understand the reason behind it. What is the conflict of standard libraries here? There's a single crate that (once compiled) includes a single version of Rust stdlib, and dylibs are allowed to share code and symbols. IIRC it is even possible to dlopen the same library multiple times (I think I've seen toy implementation of a singleton in a global variable that is different for every loaded dylib because each has its own.bss). There's nothing bad about it, but this extra dlopen feels a bit redundant. Rust stdlib cannot be extracted to a separate shared library anyway (like libstdc++), because, well, ABI is unstable.

But actually maybe I'm wrong and maybe it's easier to build it this way: Rust dylib doesn't know about Ruby dependencies and it's 100% valid on its own, it's isolated and self-contained. Ruby dylib that wraps it is also linked with libruby.[so,bundle] and other things that Ruby needs on different platforms. It is going to be small and transparent, and so things like Rust <-> C LTO that could be used with static lib don't bring that much value.

Quick question (sorry if it doesn't belong to this PR directly): is there a way to customize generated Makefile? Or will it run cargo build every time you run rake compile? I think it's ok for now to rely on cargo dependency tracking, just wondering what's the plan.

Also, what about targets? I think it's irrelevant for linux/mac, but on windows x86_64-pc-windows-gnu should be used, I'm not even sure if dlopen from mingw (that is used for Ruby on windows builds) can load .dll compiled with msvc backend. Because of that, I think it's worth passing --target explicitly.

[ianks]

Update: following in the footsteps of pgx, I've setup a macro for rb-sys named ruby_extension!() which does the setup needed to export the ruby_abi_version function needed for Ruby 3.2+ (thanks for the guidance @peterzhu2118!). Gem authors will do something like this:

#[macro_use]
extern crate rb_sys;

ruby_extension!()!;

#[allow(non_snake_case)]
#[no_mangle]
pub extern "C" fn Init_my_gem() {
    // ...
}

This will set everything up needed to load the extension at runtime.

[ianks]

🎊

I think some of the jobs just timed out @simi, could you trigger a re-run?

[simi]

done 🙏

[simi]

Let me please check the code once more time. I think there is some chance to DRY some stuff shared across builders.

[ianks]

Leaving this here as a note if anyone wants to give cross-compilation a shot after this is merged... I think the mapping to target triples looks something like this:

    MAPPING = Hash[
      "x86-mingw32" => "i686-pc-windows-gnu",
      "x64-mingw-ucrt"  => "x86_64-pc-windows-gnu",
      "x64-mingw32"  => "x86_64-pc-windows-gnu",
      "x86-linux"  => "i686-unknown-linux-gnu",
      "x86_64-linux" => "x86_64-unknown-linux-gnu",
      "arm-linux"  => "arm-unknown-linux-gnueabihf",
      "aarch64-linux" => "aarch64-unknown-linux-gnu",
      "x86_64-darwin" => "x86_64-apple-darwin",
      "arm64-darwin" => "aarch64-apple-darwin",
    ].freeze

[hsbt]

@simi I hope to merge this before Ruby 3.2.0-preview1 release. Can we refactor/improve builders after merging this?

[simi]

Great job everyone. I'll open additional PR with bundle gem --ext=rust support and various improvements.

[simi]

Btw. I would consider this still experimental feature for now.

[hsbt]

@simi Thanks!

[ianks]

Awesome!! Thanks for the help everyone ❤️

Now onto making cross-compilation with rake-compiler as breezy as possible, so rubygems users don’t have to have a cargo tool chain installed!

[pynixwang]

how about dsl

#[ruby_module(BigDecimal)]
mod bigdecimal {
}

[ilyazub]

how about dsl

#[ruby_module(BigDecimal)]
mod bigdecimal {
}

@pynixwang Looks like the rutie and macros from the wasmer-ruby. (Usage example of wasmer-ruby macros)

#[rubyclass(module = "Wasmer")]
pub struct Baz {
    inner: i32,
};

#[rubymethods]
impl Foo {
    pub fn bar(&self, baz: &Baz) -> RubyResult<…> {
        let inner = &baz.inner;

        …
    }
}

While rutie is ready for use, macros from wasmer-ruby are not exported. Help is much appreciated: danielpclark/rutie#145.