Multi platform container

[sladage]

From what I understood from the specs is that a container targets a single platform? If so, wouldn't it be better to allow for different configurations depending on the platform, all packaged into one container?

[wking]

On Thu, Jul 23, 2015 at 10:12:43AM -0700, Stefan Ladage wrote:

From what I understood from the specs is that a container targets a
single platform? If so, wouldn't it be better to allow for different
configurations depending on the platform, all packaged into one
container?

I was recently confused about this too, but I think the current spec
almost allows this already. We just need to remove the requirement
for the config.json name 1. The ‘runc’ command already supports
launching a container from an alternative spec 2, although I haven't
been able to find clear docs describing this functionality.

However, this still requires the launcher to figure out which of
several configs to use for their platform. I think it might make more
sense to replace the default config.json with a config/ directory.
The launcher (e.g. runC) would walk that directory to find potential
config files (anything ending in ‘.json’?) and then figure out which
of the found spec files was the best platform match (erroring out if
there were no compatible specs).

To keep the config DRY, it would be nice if there was also a way to
share code between configs. For example, inheritance:

{
…
"parents": [
"../base.json",
"./amd64.json"
],
…
}

or embedding (e.g. with JSON Schema's $ref 3:

{
…
"linux": {
"$ref": "./amd64.json#linux"
}
…
}

Since inheritance and embedding are potentially complicated, I'm fine
punting on those for now, but I do think that a walkable config
directory is a clear win for multi-platform fat containers.

Of course, keeping the option to manually specify a config file when
invoking runC is good too. For example, you may have one config to
launch your indended application, and another with something like:

{
"version": "pre-draft",
"parents": ["./main.json"],
"process": {
"args": ["bash"]
}
}

that's identical to the app config but just launches a shell. For the
default platform-matching logic, you'd want a way to break ties like
this. I'd suggest either C-locale sorts of the relative path from
config/ (earlier paths win) or an explicit “"priority": ”
setting (higher priorities win).

[sladage]

Why not change the platform section into something like this:

"platforms":[
{
    "os": "linux",
    "arch": "amd64",
    "config":"linux64.json"
},
{
    "os": "windows",
    "arch": "amd64",
    "config":"win64.json"
}
]

The main config file serves as a base and the platform specific configs may override properties from the base config or add platform specific sections.

[wking]

On Fri, Jul 24, 2015 at 04:27:55AM -0700, Stefan Ladage wrote:

Why not change the platform section into something like this:

"platforms":[
{
    "os": "linux",
    "arch": "amd64",
    "config":"linux64.json"
},
{
    "os": "windows",
    "arch": "amd64",
    "config":"win64.json"
},
]

The main config file serves as a base and the platform specific
configs may override properties from the base config or add platform
specific sections.

That saves a directory walk vs 1, and it shifts the platform
information from linux64.json to the parent config's platform entry.
But it doesn't give you $ref to share arbitrary snippets [1,2].

Do you expect it to be recursive? For example, could linux.json have
(following #39):

"platforms":[
{
"os": "linux",
"architecture": "arm64",
"variant": "aarch64"
"endian": "big",
"abi": "ipl32"
"config": "aarch64-be-ipl32.json",
},
{
"os": "linux",
"architecture": "arm64",
"variant": "aarch64"
"endian": "little",
"abi": "ipl32"
"config": "aarch64-le-ipl32.json",
},
…
]

I'm not sure which way (this or 1) would be easier to use. Say you
have N platforms, M entry-points, …. How do we avoid forcing authors
to write N⋅M⋅… configs, while still giving them the freedom to
customize a given combination if they need to? With your explicit
child approach, you could do things like a single shell config for all
platforms:

{
"version": "pre-draft",
"platforms": [ {"os": "all"} ],
"process": {"args": ["bash"]}
}

(os no need for N⋅M⋅… configs), but I'm not sure how you'd tweak the
arguments for aarch64-le-ipl32 (where maybe you couldn't get Bash to
compile and wanted to use Dash).

With my explicit parent approach 1, you need an override config for
every leaf in the platform tree
(e.g. config/linux/aarch64/le/ipl32-shell.json, …). You could use
$ref to link to the override itself (in case it was more involved than
a single setting):

{
"version": "pre-draft",
"parents": ["./ipl32-main.json"],
"$ref": "../../../dash.json"
}

but it would still be tedious.

One approach would be to add if/then logic and such to the config, but
dynamic configs are no fun (which is why setup.py can be such a pain
in Python).

I think the best choice at the moment is:

1. If config.json exists (or you were passed a file path), use that.
2. If config/ exists (or you were passed a directory path), walk it
   looking for the best platform match and use that.

And then just punting on fancy inheritance ("$ref", "config",
"parents", …) until someone has a better idea than we've had so far
;).

[philips]

Can someone write-out a use case for this? What is an example use case for a bundle that supports multiple archs? Why is having two bundles too complicated?

[crosbymichael]

I don't think this is wise. A config + rootfs probably cannot be amd64 and win64 at the same time. You will have to have two different configs.

[glevand]

This single multi platform container idea adds complexity. I can't think of how it would have an advantage over multiple single platform containers. Is there a real need?

[wking]

On Tue, Jul 28, 2015 at 12:30:16PM -0700, Geoff Levand wrote:

This single multi platform container idea adds complexity. I can't
think of how it would have an advantage over multiple single
platform containers.

Somebody is going to have to decide which of a family of $X is the
best match for a given platform. Whether that's picking the best
config from a given directory (#76) or picking the best bundle from a
set of related bundles doesn't really matter to me. But I'll want a
way to say “I'd like to launch an Nginx container” and have a tooling
layer underneath me that setup a container appropriate to whatever
hardware I was on. I'm fine pushing this to a higher-level than this
spec, but I'm not sure where that higher level is. See also 1,
where I wonder about how we expect coordination with higher (or
parallel) levels to work.

[sladage]

Having multi-platform containers allow you to deploy the same containers on multiple machines with different platforms, greatly simplifying the deployment process. Isn't the whole idea of using containers that you don't have to worry about what your deployment target looks like?

As for the added complexity, the complexity of having to deal with multiple platforms is there whether we deal with it here, in this spec, or make it someone else's problem. At least if we add it to this spec, we have standardized way of deploying to multiple platforms.

Personally, I can see a lot of benefits using a single container with shared resources which I can move between linux, osx and windows.

[stevvooe]

While this is a real use case, for the current level of runc, this is out of scope.

We can imagine a distribution system that accomplishes this task, coordinating the cross-platform creation of bundles. This is not part of the near term goals.

[wking]

On Thu, Jul 30, 2015 at 11:11:29AM -0700, Stephen Day wrote:

While this is a real use case, for the current level of runc, this
is out of scope. We can imagine a distribution system that
accomplishes this task, coordinating the cross-platform creation of
bundles.

That's a reasonable approach. Does anyone know of a spec for that
higher-level system? One approach is 1, but that's just me ;). I
imagine the CNCF folks are interested in this area, but I can't find
links to a spec on their site 2. Does anyone here know anyone
involved with the CNCF or Kubernetes? Do we need a new project to
discuss these higher levels?

[vbatts]

@wking I'll check with some folks, regarding CNCF.
And this is not to say this is out of scope of OCI, just the current level of runc.

[wking]

On Thu, Jul 30, 2015 at 11:31:19AM -0700, Vincent Batts wrote:

@wking I'll check with some folks, regarding CNCF.

Any word from the CNCF folks?

[vbatts]

Hey hey. That is still getting underway and while cncf is a direction this
could go, it ought not block any primitive plumbing here. There are some
service level specs being worked on like
/~https://github.com/projectatomic/atomicapp, though working with OCF as a
back end, ans across multiple platforms is not yet a possibility.
On Aug 12, 2015 18:07, "W. Trevor King" notifications@github.com wrote:

On Thu, Jul 30, 2015 at 11:31:19AM -0700, Vincent Batts wrote:

@wking I'll check with some folks, regarding CNCF.

Any word from the CNCF folks?

—
Reply to this email directly or view it on GitHub
#73 (comment)
.

[wking]

On Wed, Aug 12, 2015 at 04:34:39PM -0700, Vincent Batts wrote:

Hey hey. That is still getting underway and while cncf is a
direction this could go, it ought not block any primitive plumbing
here.

Yeah, it's not a hurry. But asking how things are going didn't seem
like it would cause too much trouble ;). I just want to find a place
for out-of-scope issues to go, so we don't get tempted to roll them
into the runtime spec just to get them handled ;).

There are some service level specs being worked on like
/~https://github.com/projectatomic/atomicapp, though working with OCF
as a back end, ans across multiple platforms is not yet a
possibility.

The spec is in [1](atomicapp is the reference implementation), but
that looks like a good candidate to me. It looks like some of the
main contributors are Red Hat folks. Would you or @mrunalp be good
contact points for negotiating the boundary between this spec and
theirs?

[goern]

@wking Hi there, I am happy to volunteer @aweiteka and myself to be your contacts for Nulecule Specification and Atomic App. By now we most of the communication is done via IRC #nulecule (On Freenode), pls see all communication channels at http://www.projectatomic.io/docs/nulecule/

[vbatts]

@wking there a likely folks from those projects that can pipe in, but there will likely be some boundaries. And definitely some definition of requirements from a project like a service-level/mutli-platform spec, and the OCI

[aweiteka]

@wking I'm pretty deep into nulecule. We're trying to keep it simple. We don't want to duplicate any platform functionality but rather provide a lean template that can be an abstraction to the various platforms.

[conqerAtapple]

Why not adopt linux's device tree model for cross-platform support?

TL;DR -

1. Have common and basic configuration at the root
2. Have overlay (platform specific) configuration in pre-determined directories
3. Have overlays "include" any other configuration data

[wking]

On Mon, Sep 28, 2015 at 10:49:19AM -0700, Jojy George Varghese wrote:

Why not adopt linux's device tree model for cross-platform support?

Can you spell this out in more detail? Skimming over [1,2] didn't
turn up an obvious parallel. Are you suggesting:

{
…platform agnostic stuff…,
"linux": {
…Linux-specific overrides…,
"x86" {
…x86-specific overrides…,
}
},
"amd64" {
…x86-64-specific overrides…,
}
}

For example:

{
…
"mounts": [{"name": "data", "path": "/data"}],
"windows": {
…
"mounts": [{"name": "data", "path": "c:\data"}],
}
}

That still doesn't cover:

1. Have overlays "include" any other configuration data

Maybe there you're imagining JSON Schema's $ref 3?

[conqerAtapple]

I am suggesting a structure where the top directory would have the base
configuration.

./
config.json
./config_overlays
/windows
windows.inc
config_win.json

 /linux
        centos.inc
        amd64.inc
        config_linux.json

The "configs" directory is where "overlay"/"override" configuration will be
searched.

The ".inc" files could have common information for a platform for example.

-jojy

On Mon, Sep 28, 2015 at 11:20 AM, W. Trevor King notifications@github.com
wrote:

On Mon, Sep 28, 2015 at 10:49:19AM -0700, Jojy George Varghese wrote:

Why not adopt linux's device tree model for cross-platform support?

Can you spell this out in more detail? Skimming over [1,2] didn't
turn up an obvious parallel. Are you suggesting:

{
…platform agnostic stuff…,
"linux": {
…Linux-specific overrides…,
"x86" {
…x86-specific overrides…,
}
},
"amd64" {
…x86-64-specific overrides…,
}
}

For example:

{
…
"mounts": [{"name": "data", "path": "/data"}],
"windows": {
…
"mounts": [{"name": "data", "path": "c:\data"}],
}
}

That still doesn't cover:

1. Have overlays "include" any other configuration data

Maybe there you're imagining JSON Schema's $ref 3?

—
Reply to this email directly or view it on GitHub
#73 (comment)
.

Jojy G Varghese

[wking]

On Mon, Sep 28, 2015 at 12:27:16PM -0700, Jojy George Varghese wrote:

The "configs" directory is where "overlay"/"override" configuration
will be searched.

Ok, so like the JSON I sketched out in 1, but with directories
instead of entries in the JSON.

The ".inc" files could have common information for a platform for
example.

How would you reference these inclusions? With something like JSON
Schema's $ref 2?

This approach seems to support both inheritance (the overlays inherit
from the more generic configs) and embedding (the *.inc) files, so I
expect it will be flexible enough to efficiently handle the
multi-platform cases I can imagine. I still think that it would be
nicer to keep in an external tool [3,4], at least until we see how
well it catches on.

[conqerAtapple]

Yes you captured the idea right. This model has been tested by linux device
tree system.

On Mon, Sep 28, 2015 at 2:17 PM, W. Trevor King notifications@github.com
wrote:

On Mon, Sep 28, 2015 at 12:27:16PM -0700, Jojy George Varghese wrote:

The "configs" directory is where "overlay"/"override" configuration
will be searched.

Ok, so like the JSON I sketched out in 1, but with directories
instead of entries in the JSON.

The ".inc" files could have common information for a platform for
example.

How would you reference these inclusions? With something like JSON
Schema's $ref 2?

This approach seems to support both inheritance (the overlays inherit
from the more generic configs) and embedding (the *.inc) files, so I
expect it will be flexible enough to efficiently handle the
multi-platform cases I can imagine. I still think that it would be
nicer to keep in an external tool [3,4], at least until we see how
well it catches on.

—
Reply to this email directly or view it on GitHub
#73 (comment)
.

Jojy G Varghese

[conqerAtapple]

Maybe have the proposed layout 1 in a PR? I didnt see anything along those lines.

[wking]

On Wed, Sep 30, 2015 at 08:59:21AM -0700, Jojy George Varghese wrote:

Maybe have the proposed layout [1] in a PR? I didnt see anything
alone those lines.

Yeah, I don't know of any PRs pushing for that format. But I'd
probably just write up an external tool that reads that format and
spits out the current config files. Then folks can test your approach
without needing to land it here first.

[duglin]

we're going to be looking at multi-arch/os archive formats (see #302) so that will probably cover this request.

[vbatts]

like @duglin said, the consensus is that the on disk bundle is specific to the runtime that is going to run it, which is completely platform (os/arch) specific.
This use-case is intended to be solved at the distribution archive level.