Identifying ARM platforms

[mattspencer-arm]

There are a number of threads running on this - and after a very productive chat with a number of people at DockerCon last week and an internal review with the ARM Kernel/Distro and Toolchain team, I thought it would be useful to open the discussion here:

Firstly, the metadata in the image defines what the image contains, not what it will run on (as I have tried to express in #660). with this in mind, the baseline metadata requirements for ARM would be:

{
  'arch': ['arm'|'arm64']  /* at some point in the future 'arm64ilp32' */
  'variant': ['6' | '7' | '8']
  'features': '...'
}

This means that for ARM, arch ~ ABI, variant ~ ISA.

We would ignore point releases of the architecture - 8.1, 8.2 etc as the extensions found can be expressed through HWCAPS and enabled in platform libraries. Any features that are present in the binary that are not detectable through HWCAPS will need to be expressed in the platform.features field. Neon for example is not assumed to be present in the platform, but should be runtime detectable. If however this is not possible, the usage of Neon would have to be expressed in the platform.features field.

There would be some baseline assumptions for all platforms, as we don't believe there is a need to enable all existing ARM platforms though this mechanism.

Baseline assumptions would be:

arch = arm

arm means everything is built to run little-endian, using A32 instructions, for AArch32 execution mode, using the EABI spec, with 16 floating point registers used by the ABI, so VFP hardware must be present.

This is the same default used by debian/ubuntu 'armhf', fedora 'armv7hf', rasbian, suse and Linaro as the 32-bit ARM base ABI. And is thus the default output of gcc targetting arm-linux-gnueabihf on those platforms [I hope - is this always true?]

It is expressed explicitly with -mfloat-abi=hard in gcc targeting arm-linux-gnueabihf (maybe some more options?)

Note that neon is not assumed to be present, and would need to be declared in features if used without runtime detection (it should always be used with runtime detection to make this moot).

https://wiki.debian.org/ArmHardFloatPort#Background_information is a useful page if you wish to understand the details of this.

Valid variants are currently '6' '7' '8'

arch = arm64

arm64 means everything is built to run little endian, using A64 instructions, for AArch64 execution mode, using the LP64 ABI. Documentation on the ARM ABI's can be found here http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.subset.swdev.abi/index.html

This is the default arm64 ABI with 64-bit longs and pointers.

Assume use of base v8 instructions only. v8.1, 8.2 etc extensions must be declared in features if not runtime detected (runtime detection should be used to avoid having to do this).

This is the same default used by debian/ubuntu 'arm64', fedora/centos/redhat 'aarch64', Suse. And is thus the default output of gcc targetting aarch64-linux-gnu on these platforms.

Valid variants are currently '8'

arch = arm64ilp32

arm64ilp32 means everything is built to run little endian, using A64 instructions, for the AArch64 execution mode using the ilp32 ABI.

This is the rarely-used arm64 ABI using 32-bit ints, longs and pointers.

It is the default output of gcc targeting aarch64-linux-gnu_ilp32 or gcc targetting aarch64-linux-gnu with option -mabi=ilp32

Valid variants are currently '8'

[wking]

Image-spec is currently punting to runtime-spec for architecture information and punting to Go for variant information (with the in-flight #650). Runtime-spec is punting to Go for architecture information. Is there a reason for having this discussion in image-spec instead of further upstream (e.g. in Go)? Image-spec seems like an odd choice for fixing “nobody identifies ARM appropriately”. Maybe the ARM folks could publish a stand-alone ARM-identification spec, and the OCI specs could punt to that instead of punting to Go?

[mattspencer-arm]

The reason I bring it up here is that I spoke with a number of people at DockerCon who suggested that ARM get involved in helping to define the image-spec so that ARM platforms can be expressed properly.

I am probably missing a lot of context here, so please excuse me getting this completely wrong!

If the images are expressing what is contained within them, not the platform that they will run on, how can the runtime spec expect to explain what goes in the image metadata? If this is the case, then an image could be marked {arch: arm, variant 8} and it could run on a platform {arch:arm64, variant: 8}. If this is expected to be a strict pattern match, then this would not be possible?

And I am not sure I understand the logic of Go expressing the architecture information either? Are all consumers of this spec going to build their platforms on Go? And what if the go community decide to change the meaning of attributes.

[wking]

On Fri, Apr 28, 2017 at 10:27:17AM -0700, mattspencer-arm wrote: The reason I bring it up here is that I spoke with a number of people at DockerCon who suggested that ARM get involved in helping to define the image-spec so that ARM platforms can be expressed properly.

I'm in favor of getting ARM expressed properly, both inside and outside of image-spec. If we have an external spec for expressing ARM, both image-spec and other consumers (e.g. runtime-spec, Go, package managers, …) can lean on that ARM-expression spec. If, on the other hand, the best ARM-expression spec was built into image-spec, I think it's less likely that those other consumers would use it, and we end up with https://xkcd.com/927/.

[stevvooe]

arch = arm64ilp32

This is not in the Go toolchain. They will likely add something to GOARM. It cannot appear in the GOARCH field if it is not supported by the Go toolchain.

Valid variants are currently '6' '7' '8'

The "variant" field was not at all meant to map to the GOARM field. This should contain values like "armhf" or "arm64ipl32" or whatever is required to resolve the ISA+ABI.

Rather than suggesting a mapping, it might be good for ARM to provide a topology of their requirements, describing both the ISA and ABI variation. This would allow us to design an appropriate methodology that maps to the existing fields or perhaps define new ones.

[wookey]

HI, I helped Matt write the doc at the top of this thread.

My understanding on this issue was that what is required is some way of expressing the contents of containers that makes long-term sense, so that builds can describe what they built, and such that clients can ask for something that will run on them. Bear in mind that whilst I have made years of messing with chroots and qemu, and architecture bootstraps in Debian, I know absolutely nothing about the world of docker, so could have erroneous assumptions.

The responses here so far talk about Go fields and toolchains. I understand that docker is written in Go, but I thought that we were talking about containers in a wider context here and thus the implementation language on one particular container image system is not the main consideration. Is there some reason why this spec will only ever be used for docker and that all possible relevant implementations will be in Go?

The "variant" field was not at all meant to map to the GOARM field.

I don't understand this comment. What 'GOARM field'?

This should contain values like "armhf" or "arm64ipl32" or whatever is required to resolve the ISA+ABI.

I understood that there are 3 fields we can use and that we need to try and fit a sensible description into those: ARCH, VARIANT, FEATURES. Is that wrong?

You suggest putting ISA+ABI into the 'variant' field, but in that case the 'arch' field is essentially wasted and you end up concatenating two things into the one VARIANT field, which makes a mess as there a lot of valid ABI+ISA variants. IMHO it makes a lot more sense to use the arch field for ARCH+ABI (as Debian does), and put the ISA level into the VARIANT field. You will have to explain to me why this is not appropriate here, and why concatenating both bits of info into one field is better.

The point being that saying that some binaries are 'x86_64' or 'i386' or 'arm' doesn't really tell you much useful unless you also take that to imply an ABI, which it is built to use.

Remember that the base ISA changes over time, unlike the ABI. An 'i386' image once used 386 instructions, but later on 486, 586, 686. All of these will run on a sufficiently-recent machine, and can interwork. This is one reason why it makes sense to specify the base ISA used in a separate field from the arch/ABI.

Rather than suggesting a mapping, it might be good for ARM to provide a topology of their requirements, describing both the ISA and ABI variation. This would allow us to design an appropriate methodology that maps to the existing fields or perhaps define new ones.

Well, I thought that's what we tried to do in the above doc, but sure, lets run at this again. We (ARM) don't have requirements as such - the requirements come from the ways people use containers and build images for them. My understanding is that clients want to be able to say 'I am a foo machine - send me an image that will run here'. People who build images want to be able to say 'this image contains stuff built like this'.

One question we had when trying to design the spec was 'do people expect to be able to download a container image and then run binaries built elsewhere on it?', to which we assumed the answer was 'no', or at least that they would use mechanisms inside the container (like distro package managers, or language ecosystem package managers) to get extra binaries, or that they would build them in-situ and thus automatically get stuff that matched.

There is lots of possible variation in ISA and ABI on ARM systems, especially older ones where all sorts of craziness went on because it was embedded-world. Fortunately if we can restrict the context to Linux images, on vaguely modern hardware (i.e. no older than the original RPi (arm v6 ISA)), then that cuts the space down to something sane, which is described at top of thread. 3 ABIs (only 2 currently in use anywhere real), and 3 base ISA levels. Not all combinations valid. Then there are various optional ISA extensions (like SSE, MMX, 386, 486, 686 in x86 world) which we assume a base level of in the default arch/ABI spec, and anything beyond that is either described in FEATURES or (much better) runtime-detected before use.

Does that all make sense?

[wking]

On Wed, May 03, 2017 at 09:59:58AM -0700, Wookey wrote: The responses here so far talk about Go fields and toolchains. I understand that docker is written in Go, but I thought that we were talking about containers in a wider context here and thus the implementation language on one particular container image system is not the main consideration. Is there some reason why this spec will only ever be used for docker and that all possible relevant implementations will be in Go?

The spec is intended for wider use than Go, but Go already has a way to identify ARM platforms (using GOARCH and GOARM), and I don't see the point of defining a *new*, image-spec-specific way to specify ARM platforms. If GOARCH and GOARM are insufficient, it would seem like the Go folks would also be interested in adjusting their characterization to allow more reliable compilation for ARM targets. Instead of standardizing around Go's scheme, I'd bee fine standardizing around Debian's scheme (or whatever). But if every toolset around ARM defines its own identification scheme, mapping between the various identification schemes sounds like a lot of trouble for no gain.

One question we had when trying to design the spec was 'do people expect to be able to download a container image and then run binaries built elsewhere on it?', to which we assumed the answer was 'no', or at least that they would use mechanisms inside the container (like distro package managers, or language ecosystem package managers) to get extra binaries, or that they would build them in-situ and thus automatically get stuff that matched.

If the arch/variant/features tuple is sufficient to know that the original image's binaries will run on a given host, won't it be sufficient to specify a complication target for new binaries so a new image (including the new binaries) will run on other hosts that support that arch/variant/features tuple?

Fortunately if we can restrict the context to Linux images, on vaguely modern hardware (i.e. no older than the original RPi (arm v6 ISA)), then that cuts the space down to something sane, which is described at top of thread. 3 ABIs (only 2 currently in use anywhere real), and 3 base ISA levels. Not all combinations valid. Then there are various optional ISA extensions (like SSE, MMX, 386, 486, 686 in x86 world) which we assume a base level of in the default arch/ABI spec, and anything beyond that is either described in FEATURES or (much better) runtime-detected before use.

Runtime-detected before use works when you have coded in a fallback for a missing instruction. That's great when you have one, and when you don't you can use ‘features’. And the ABI/ISA/extensions mapping to arch/variant/features sounds good to me, but isn't that already what Go has [1]? * GOARCH: arm, arm64, (presumably eventually arm64ilp32) * GOARM: 5, 6, 7 (they don't see a need for 8 [2]) And then we punt to the Linux kernel for features like #631. The only difference to your proposal seems to be whether or not we need GOARM/variant for 8. [1]: https://golang.org/doc/install/source#environment [2]: /~https://github.com/golang/go/wiki/GoArm#supported-architectures

[stevvooe]

I must reiterate again, the arch field MUST map to what is available in GOARCH. Please, please stop re-litigating this as it is not changing. It is making the conversation wildly confusing.

For the most part, we need to decide what goes in the variant field, as that is the place that is "controlled" by the platform. It should effectively encode the ABI and minimum ISA requirements. Other things, like "features", should be detected at runtime.

We can simplify this into a table. On the left, we have the name of the ABI+ISA and on the right, we have the contents of the platform fields:

ISA/ABI	Arch	Variant
Arm 32-bit, v6	arm	armv6
Arm 32-bit, v7+	arm	armhf
Arm 64-bit, v8	arm64	arm64
Arm 64-bit, v8, ilp32	arm64	arm64ilp32

What are we missing from the above?

[stevvooe]

I've put together a complete proposal here. Please feel free to comment.

We have the following table for the use of the variant field in ARM:

ISA/ABI	architecture	variant
Arm 32-bit, v6 (ie Raspberry Pi 1)	arm	v6
Arm 32-bit, v7	arm	v7
Arm 32-bit, v8	arm	v8
Arm 64-bit, v8	arm64	v8

[AkihiroSuda]

PR ready: #650

[AkihiroSuda]

Should we update runtime-spec as well to remove support for pre-v6?

[stevvooe]

Should we update runtime-spec as well to remove support for pre-v6?

Where is that support specified in the runtime spec?

[AkihiroSuda]

@stevvooe

Here: /~https://github.com/opencontainers/runtime-spec/blob/3908f2846586831a5b6eb38aaf6f9a4300230c71/config.md

Values for arch SHOULD use, and runtimes SHOULD understand, arch entries listed in the Go Language document for GOARCH.

But probably no need to remove pre-v6 explicitly, as you mentioned in #650 (comment)