Storing/retrieving container state

[mrunalp]

We need to standardize on how to store/retrieve the state of a running container. runc today allows one to specify an id during invocation and then stores the state of that container under /var/run/ocf/<id>.

I think we need to agree on a directory and the mechanism if there are any alternate suggestions/ideas.

@crosbymichael @vmarmol @LK4D4 @philips

[kunalkushwaha]

Hi,

Just one thought, Instead of using directory structures can a lightweight DB like SQLite be used to store such information.

Problems of directory based layout for containers information

Typically, most of containers running on one system do not have very large life, But one single host can have large numbers of containers running.

Though there is no problem in extracting information for particular container, but for offline monitoring tools, it becomes difficult to scan through whole list of containers, and read files and make analysis.

Drawback of DB:

• I am not sure, how much it will become bottleneck, since write operations will be done by container runtime only.
• DB corruption, in case host shutdown abruptly.

[wking]

On Wed, Jul 22, 2015 at 04:43:43PM -0700, Mrunal Patel wrote:

We need to standardize on how to store/retrieve the state of a
running container.

I think we need to standardize on the state content and syntax
(e.g. so we can write hooks 1), but I don't see a need to
standardize on how / where the launcher stores that information unless
we want to expose it to the running container (like nsinit used to
2). Personally, I see no need to leak that information to the
container (although some of the code is still in runC 3).

It would make portability between controllers easier if there was a
consistent way to retrieve this information, list running containers,
etc., but that seems orthogonal to the focus of this repository, which
is about authoring containers (and not about managing them). Maybe
move this issue to wherever the Cloud Native Computing Foundation
(CNCF) [4,5] is spec'ing the controller layer? That may be 6.

[wking]

On Wed, Jul 22, 2015 at 08:12:46PM -0700, W. Trevor King wrote:

I think we need to standardize on the state content and syntax
(e.g. so we can write hooks [1])…

And this seems to be #41.

[vmarmol]

To have various implementations of the spec be able to function over the same containers, I think we need to agree on a common state storage.

A DB is probably much higher abstraction than we need and arguably overkill. If you're looking to have something that queries containers that frequently, having some daemon as a front makes sense in that case I think.

[LK4D4]

@vmarmol Yes, if we want one implementation to work with containers spawned by another, then we need standard. We should decide if it's real case, like spawn container with runc and exec to it with rkt. Because letting implementation to decide how to store and discover state would be much simpler.

[vmarmol]

My usecase for this sorta thing has always been something like cAdvisor :) Docker (runc) creates the containers, but cAdvisor discovers and monitors them.

[wking]

On Wed, Jul 22, 2015 at 10:03:02PM -0700, Alexander Morozov wrote:

… if we want one implementation to work with containers spawned by
another, then we need standard. We should decide if it's real case…

Even if it is a real case, it's still a container-consumer issue, and
not a container-author issue. I think container-consumer issues are
out of scope for this repository (but they are certainly in scope for
CNCF).

[kunalkushwaha]

Ideally, runc will enable different container orchestration tools to work seamlessly.
Though runc will not be about managing the containers, but it (container-author) should expose all such information, so orchestration tools (container-consumer) can utilize without any hacks.

Regarding DB, (Just of reference) LXD too have adopted this approach.

[LK4D4]

@vmarmol I think now about adding command to binary, which just returns you state to stdout. This will definitely solve tooling problem. Not sure if all would agree about this.

[wking]

On Wed, Jul 22, 2015 at 10:48:10PM -0700, Kunal Kushwaha wrote:

Ideally, runc will enable different container orchestration tools to
work seamlessly.

I agree that the runC API should provide a consistent way to get the
state (and a command-line argument sounds good to me for that purpose
1), but I think the spec in this repository, is for
container-authors, and not for runC-users (container-consumers). I
would love to have a container-consumer spec (maybe just runC, maybe
manager-independent?), but I think that's an orthogonal issue best
kept to a separate repository.

… but it (container-author) should expose all such information…

Can you spell out why container-author's would need to know about how
the management tool (e.g. runC) decides to store the state? They just
need to know how the hooks they configure will get that state, and
that's already clear (read from stdin 2).

[kunalkushwaha]

@wking I Agree, as far as, hooks or clean method to get state information is available no need to go into storage part.

[crosbymichael]

I think storing the state information in a state.json file located in /run/ocf/<containerid>/state.json seems like a simple and clean way. It's the way we do it today and gives us a directory to store other runtime information for that container. To find all containers on a systemd you just listdir(/run/ocf)

[philips]

Having a well-known directory layout is fine with me. However, is the idea that /run/ocf//rootfs a bind mount to somewhere with actual storage?

[crosbymichael]

I don't htink we should have the rootfs bind mounted into anywhere like /run The config and rootfs should be decoupled from the runtime state in my opinion