Add custom toolchains for powerpc* and s390x

[cuviper]

Using Ubuntu's cross-toolchains meant these targets were depending on glibc symbols from Ubuntu 16.04. And if that host is ever updated to a new release, the toolchains would raise the bar too.

This switches powerpc* and s390x to use crosstool-ng toolchains, with the kernel and glibc configured approximately like RHEL7. That should also be compatible with Debian 8 (jessie) and Ubuntu 14.04 (trusty).

Incidentally, this also updates the netbsd and llvm URLs to fix the docker build.

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @brson (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[cuviper]

Note, however, there is one hiccup preventing powerpc64 from working. In Rust mk/cfg/ and in gcc-rs, powerpc64 is configured to use the powerpc-linux-gnu- prefix, which only works because it's also using -m64 and the toolchain supports multilib. Here, crosstool-ng does not provide multilib by default, and describes that option as being experimental. But I'm not sure why powerpc64 is being compiled this way. @alexcrichton It looks like you set this -- do you recall why?

[cuviper]

Also, my original motivation for this was to get RHEL6-compatible binaries, glibc-2.12, but I wasn't able to wrangle crosstool-ng to provide that old of a glibc along with a new enough gcc for LLVM. So I think it's still worthwhile for Rust to compile these arches at a lower target than Ubuntu 16.04, but I'll have to figure out our own cross-compilation to get RHEL6 going.

Actually, powerpc64le in this PR isn't even low enough for RHEL7, but that's because upstream support wasn't added until glibc-2.19, and RHEL7's glibc has it backported. It would almost be close enough anyway, but __cxa_thread_atexit_impl@GLIBC_2.18 sneaks in and ruins it. 😢

[cuviper]

It's also not clear to me how this all relates to rust/src/ci/docker/... is this repo now obsolete?

[alexcrichton]

Thanks for the PR @cuviper! Things are definitely in a bit of a state of flux, but we can accept this PR anyway for now. To answer some of your questions:

Note, however, there is one hiccup preventing powerpc64 from working. In Rust mk/cfg/ and in gcc-rs, powerpc64 is configured to use the powerpc-linux-gnu- prefix, which only works because it's also using -m64 and the toolchain supports multilib.

Yeah this is fine, and also one of the many reasons to jettison the makefiles! Rustbuild handles this case w/ the various env vars being set, so the defaults in the makefiles should become irrelevant quickly!

So I think it's still worthwhile for Rust to compile these arches at a lower target than Ubuntu 16.04, but I'll have to figure out our own cross-compilation to get RHEL6 going.

Aw that's a bummer :(. I've had tons of difficulty in the past getting various versions to match up w/ crosstool-ng. If you can get glibc/gcc builds from source though that'd also work great, we don't mind too much vendoring tons of shell scripts here.

It's also not clear to me how this all relates to rust/src/ci/docker/... is this repo now obsolete?

Yes and no! Eventually we're going to deprecate this repo in favor of the support in the rust-lang/rust repository (tracked here: rust-lang/rust#38531). For now though this repo is still the source of truth for producing releases.

If you're ok with a month-ish time frame to get this out to nightlies, then we may want to move all this over to the rust-lang/rust repo rather than here. If you'd like to get it out sooner though we can do it here, build a new image, and update the current release bots to use that.

I just sent a PR actually (rust-lang/rust#39198) to do what you're doing here for the existing compilers we've got in this repo for the Travis builds. Eventually we'll want to send this up to rust-lang/rust as well, but that's up to you in terms of time frame when you'd like to do so.

Does that help clear things up?

[alexcrichton]

Oh and this PR looks great to me! I'm fine to merge whenever. Curious on your thoughts from my previous comment though.

[cuviper]

Yeah this is fine, and also one of the many reasons to jettison the makefiles! Rustbuild handles this case w/ the various env vars being set, so the defaults in the makefiles should become irrelevant quickly!

Ah, I knew that the makefiles are on the way out, but I missed that gcc-rs checks the env first. It's still a weird default to have "powerpc64-unknown-linux-gnu" => Some("powerpc-linux-gnu"), but whatever. :)

If you can get glibc/gcc builds from source though that'd also work great, we don't mind too much vendoring tons of shell scripts here.

I think ultimately to handle RHEL6 from source, we'd have to use two compilers - the old gcc to compile that old glibc, and then a new gcc to use with Rust/LLVM, still linking to that old glibc. I have no idea how to convince crosstool-ng to work like this, nor am I really keen to duplicate its procedure manually. But it's my first time with the tool, so maybe there's a way...

At least if I'm cross-compiling for RHEL6 on my own, I can just use the actual glibc binaries of the target. And it's a one-time pain, then I can chase Rust releases with native builds thereafter. But it doesn't help anyone outside Red Hat either. We'll see, maybe after I try it my estimates of the difficulty will change.

If you're ok with a month-ish time frame to get this out to nightlies, then we may want to move all this over to the rust-lang/rust repo rather than here. If you'd like to get it out sooner though we can do it here, build a new image, and update the current release bots to use that.

Given that I still have to figure out RHEL6, and having done so can reuse that for RHEL7 too, this leaves me in no rush for merging this PR. I just wanted to bear some fruit from my efforts with crosstool-ng. :)

So the long-term approach is fine with me. We can avoid rocking the boat on the current infrastructure, and as soon as rust-lang/rust#39198 lands I can try to carry these changes over there too.

[alexcrichton]

Ok that all sounds good to me!

So I'll also note that the minimum gcc for LLVM is 4.7, so it's not too recent. I'll also note that crosstool-ng is just the "first thing that worked", and I'd be totally down for using anything else that worked. For example the netbsd/freebsd toolchains have their own shell scripts to compile gcc (just a standard configure-then-make). This rarely works out in practice though unless everything is just right, so your mileage may vary...

If we can get precompiled binaries of glibc, though, then perhaps we can take the strategy of avoiding compiling it and going straight to glibc? This tends to be much easier than also having to compile glibc.

[cuviper]

So I'll also note that the minimum gcc for LLVM is 4.7, so it's not too recent.

Well, LLVM's guide says GCC 4.8, but yeah, not too recent. That's even what RHEL7 ships with - how shiny! :)

I wonder though, maybe I can use two compilers - just use crosstool-ng for gcc-4.4/glibc-2.12/binutils, then manually run a build script for newer gcc to live next to that. Hmm...

That still leaves a question for RHEL7 ppc64le, with its odd backported glibc-2.17. If I could just find a way to hack out __cxa_thread_atexit_impl from 2.19, I think it would be ok. The crosstool-ng page mentions it can do custom patches, so maybe I can cut it out. Or applying the same backport patches to 2.17 would be cleaner.

If we can get precompiled binaries of glibc, though

AIUI, RHEL binary redistribution is still restricted, even though source is freely available. Maybe it would be fine, but I'd rather not involve legal question if we can avoid it. :) But for ppc64le in particular, it might work to grab binaries from centos7.

Anyway, there are a few more things for me to try here.

[alexcrichton]

Oh dear looks like we may be in for a gcc upgrade soon ...

Also yeah binaries from a random centos disk image or something like that would be fine by me, even some historical release of ubuntu might work. All we really need is a basic "chroot" kinda

[cuviper]

OK, pushed an update -- I managed to get everything back as far as I need, yay!

Quoting the commit:

For crosstool-ng with powerpc, powerpc64, and s390x, a few small patches
are now applied to glibc-2.12 to get it to compile with newer GCC. This
should now be compatible with RHEL6, Debian 7 (wheezy), and Ubuntu 12.04
(precise).

For powerpc64le, the challenge was that only glibc-2.19 officially added
support, but el7 backported those changes to glibc-2.17. The backport
patches are complex and numerous, so instead of trying to push those
into crosstool-ng, just use glibc binaries directly from centos7 and
build the toolchain manually.

Now, I've only checked it briefly, to see that the full docker build succeeds and compiling C/C++ hello-world produces binaries for the right target. I can try an actual rust build tomorrow...

[alexcrichton]

Looks good! I'll merge here and we'll pick it up on the next update (if ever, not sure if we'll proactively do this). Want to submit this upstream to the docker images in the main repo as well? It should actually be pretty soon that we have full releases built there.

[cuviper]

Well, further testing says I'm not quite there. I cross-compiled 1.14 successfully, but trying to use it:

• s390x ends up infinitely recursing in __clzdi2 from compiler-rt.
  • This appears to be a problem with the default -march=z900, because that function looks better when I rebuild it forcing z9-109 or z10.
  • RHEL6's gcc defaults to -march=z9-109 -mtune=z10
  • Rust's s390x target is configured for z10, citing LLVM's minimum support.
• powerpc and powerpc64 are failing to link, and the hashes on missing symbols are different than what's actually in the libraries.
  • This may be related to the recent blake2 endianness PRs in Rust, but AFAIK that was a beta regression, and 1.14 should be fine. I'll try git master just in case.
  • Also comparing CPU tuning, RHEL6 uses -mcpu=power4 -mtune=power6. Perhaps I should apply that just in case.
• My build for powerpc64le appears to be working fine on RHEL7, yay!
  • But also for completeness, RHEL7 usually forces -mcpu=power8.

There were also a lot of errors like this in the build, so maybe I'm cross-compiling wrong:

warning: ../compiler-rt/lib/builtins/divdc3.c:21:1: warning: conflicting types for built-in function '__divdc3'
warning:  __divdc3(double __a, double __b, double __c, double __d)
warning:  ^
warning: ../compiler-rt/lib/builtins/divsc3.c:21:1: warning: conflicting types for built-in function '__divsc3'
warning:  __divsc3(float __a, float __b, float __c, float __d)
warning:  ^
warning: ../compiler-rt/lib/builtins/muldc3.c:21:1: warning: conflicting types for built-in function '__muldc3'
warning:  __muldc3(double __a, double __b, double __c, double __d)
warning:  ^
warning: ../compiler-rt/lib/builtins/mulsc3.c:21:1: warning: conflicting types for built-in function '__mulsc3'
warning:  __mulsc3(float __a, float __b, float __c, float __d)
warning:  ^

Want to submit this upstream to the docker images in the main repo as well?

Sure, once I'm actually confident in it. 😄

[cuviper]

I'm reasonably confident that these cross-compilers are working now, but I'm running into other issues trying to use their result for a native rebuild on el6, like:

rustc: powerpc64-unknown-linux-gnu/stage1/lib/rustlib/powerpc64-unknown-linux-gnu/lib/libcore
error[E0081]: discriminant value `0isize` already exists
    --> src/libcore/slice.rs:1178:5
     |
1176 |     Continue,
     |     -------- first use of `0isize`
1177 |     // Fold is complete and will return this value
1178 |     Done(T),
     |     ^^^^^^^ enum already has `0isize`

error[E0081]: discriminant value `0isize` already exists
  --> src/libcore/num/flt2dec/decoder.rs:46:5
   |
44 |     Nan,
   |     --- first use of `0isize`
45 |     /// Infinities, either positive or negative.
46 |     Infinite,
   |     ^^^^^^^^ enum already has `0isize`

[many similar errors later...]

error: aborting due to 57 previous errors

Same on s390x, but my ppc64le rebuild on el7 was fine, so it could be an endianness issue again, or maybe just an el6 issue. I'm trying to reproduce elsewhere to isolate this now, especially on Fedora where I don't need these new cross-compilers, and if so I'll file a bug in the Rust repo.

[cuviper]

This is now in rust-lang/rust#39382.

And for reference, I found the enum discriminant problem was independent of these cross-toolchains, so I filed rust-lang/rust#39331, and when I get a new nightly I'll verify if rust-lang/rust#39332 fixed it.

[alexcrichton]

Awesome, thanks so much for the investigation and digging here @cuviper! Sounds like this is working, so I'll merge.

[cuviper]

@alexcrichton Will I need to do anything to get Cargo using this too?

[alexcrichton]

Oh right good point! Right now Cargo uses this same docker image. I'll build/publish this image (working on that now) and then we'll need to send a PR to update that container to use the freshly minted builds.

Eventually this won't be necessary as we'll compile Cargo as part of the normal release process, but alas!

[alexcrichton]

@cuviper ok I'm now in the process of pushing up alexcrichton/rust-slave-linux-cross:2016-02-02, want to send a PR for Cargo to use that?

[cuviper]

Sure, will do.