Move init-container feature from alpha to beta. #31026

erictune · 2016-08-19T20:51:46Z

Moved init-container feature from alpha to beta.

Security Action Required: 

This only applies to you if you use the PodSecurityPolicy feature.  You are using that feature if `kubectl get podsecuritypolicy` returns one or more objects.  If it returns an error, you are not using it.  

If there are any pods with the key `pods.beta.kubernetes.io/init-containers`, then that pod may not have been filtered by the PodSecurityPolicy.  You should find such pods and either delete them or audit them to ensure they do not use features that you intend to be blocked by PodSecurityPolicy.  

Explanation of Feature


In 1.3, an init container is specified with this annotation key
on the pod or pod template: `pods.alpha.kubernetes.io/init-containers`.

In 1.4, either that key or this key: `pods.beta.kubernetes.io/init-containers`,
can be used.

When you GET an object, you will see both annotation keys with the same values.

You can safely roll back from 1.4 to 1.3, and things with init-containers
will still work (pods, deployments, etc).

If you are running 1.3, only use the alpha annotation, or it may be lost when
rolling forward.

The status has moved from annotation key
`pods.beta.kubernetes.io/init-container-statuses` to
`pods.beta.kubernetes.io/init-container-statuses`.
Any code that inspects this annotation should be changed to use the new key.

State of Initialization will continue to be reported in both pods.alpha.kubernetes.io/initialized
and in `podStatus.conditions.{status: "True", type: Initialized}`

Mini-design for this change:

Goals:

A user can create an object with the beta annotation
on 1.4, and it works. The fact that the annotation has beta
in it communicates to the user that the feature is beta,
and so the user should have confidence in using it. Preferably,
when the user gets the annotation back, he see the beta
annotation.

If someone had an existing alpha object in their apiserver,
such as a RS with a pod template with an init-containers
annotation on it, it should continue to work (init containers
run) when stack upgraded to 1.4.
If someone is using a chart or blog post that has alpha
annotation on it and they create it on a 1.4 cluster, it should
work.
If someone had something with an init container in 1.4
and they roll back stack to 1.3, it should not silently stop
working (init containers don't run anymore).

To meet all these, we mirror an absent beta label from the alpha
key and vice versa. If they are out of sync, we use the alpha
one. We do this in conversion since there was already logic there.

In 1.3 code, all annotations are preserved across a round trip
(v1 -> api -> v1), and the alpha annotation turns into the internal
field that kubelet uses.

In 1.4 code, the alpha annotation is always preserved across
a round trip, and a beta annotation is always set equal to
the alpha one, after a round trip.

Currently, the kubelet always sees the object after a round trip
when it GETs it. But, we don't want to rely on that behavior,
since it will break when fastpath is implemented.

So, we rely on this:
all objects either are created with an alpha annotation (1.3 or 1.4
code) or are created with a beta annotation under 1.4. In the later
case, they are round tripped at creation time, and so get both
annotations. So all subsequent GETs see both labels.

This change is

```relnote Moved init-container feature from alpha to beta. In 1.3, an init container is specified with this annotation key on the pod or pod template: `pods.alpha.kubernetes.io/init-containers`. In 1.4, either that key or this key: pods.beta.kubernetes.io/init-containers`, can be used. When you GET an object, you will see both annotation keys with the same values. You can safely roll back from 1.4 to 1.3, and things with init-containers will still work (pods, deployments, etc). If you are running 1.3, only use the alpha annotation, or it may be lost when rolling forward. The status has moved from annotation key `pods.beta.kubernetes.io/init-container-statuses` to `pods.beta.kubernetes.io/init-container-statuses`. Any code that inspects this annotation should be changed to use the new key. State of Initialization will continue to be reported in both pods.alpha.kubernetes.io/initialized and in `podStatus.conditions.{status: "True", type: Initialized}` ``` Mini-design for this change: Goals: 1. A user can create an object with the beta annotation on 1.4, and it works. The fact that the annotation has beta in it communicates to the user that the feature is beta, and so the user should have confidence in using it. Preferably, when the user gets the annotation back, he see the beta annotation. 1) If someone had an existing alpha object in their apiserver, such as a RS with a pod template with an init-containers annotation on it, it should continue to work (init containers run) when stack upgraded to 1.4. 2) If someone is using a chart or blog post that has alpha annotation on it and they create it on a 1.4 cluster, it should work. 3) If someone had something with an init container in 1.4 and they roll back stack to 1.3, it should not silently stop working (init containers don't run anymore). To meet all these, we mirror an absent beta label from the alpha key and vice versa. If they are out of sync, we use the alpha one. We do this in conversion since there was already logic there. In 1.3 code, all annotations are preserved across a round trip (v1 -> api -> v1), and the alpha annotation turns into the internal field that kubelet uses. In 1.4 code, the alpha annotation is always preserved across a round trip, and a beta annotation is always set equal to the alpha one, after a round trip. Currently, the kubelet always sees the object after a round trip when it GETs it. But, we don't want to rely on that behavior, since it will break when fastpath is implemented. So, we rely on this: all objects either are created with an alpha annotation (1.3 or 1.4 code) or are created with a beta annotation under 1.4. In the later case, they are round tripped at creation time, and so get both annotations. So all subsequent GETs see both labels.

dchen1107 · 2016-08-20T04:11:54Z

pkg/api/v1/conversion.go

+	if valueBeta, okBeta := in.Annotations[PodInitContainersBetaAnnotationKey]; okBeta {
+		in.Annotations[PodInitContainersAnnotationKey] = valueBeta
+	}
+	// Move the annotation to the internal repr. field


nit: s/repr./representative?

dchen1107 · 2016-08-20T04:20:59Z

In 1.4 code, the alpha annotation is always preserved across
a round trip, and a beta annotation is always set equal to
the alpha one, after a round trip.

When we talked yesterday, we only thought other way around by setting alpha one to beta after a round trip. But that has rollback issue. Your new way resolved that rollback issue and much conciser than the old one. Clever!

LGTM

dchen1107 · 2016-08-20T04:21:35Z

I believe you already did upgrade test for this. :-)

liggitt · 2016-08-21T01:27:04Z

@smarterclayton @pweil- fyi

k8s-bot · 2016-08-21T12:03:43Z

GCE e2e build/test passed for commit 6e5a7f9.

k8s-github-robot · 2016-08-21T12:04:11Z

Automatic merge from submit-queue

smarterclayton · 2016-08-21T17:38:34Z

Why are we using an annotation for beta? We say we preserve beta functionality over an upgrade - which means we have to support beta annotation -> final field. Why wouldn't we just add a field here?

Also, please cc me on init container changes so I can review them in the future

The way we are versioning annotations seems more and more arbitrary every time we apply it.

smarterclayton · 2016-08-21T17:39:49Z

pkg/api/v1/conversion.go

-	// TODO: when we move init container to beta, remove these conversions
+	// If there is a beta annotation, copy to alpha key.
+	// See commit log for PR #31026 for why we do this.
+	if valueBeta, okBeta := in.Annotations[PodInitContainersBetaAnnotationKey]; okBeta {


I really can't see how this provides value to end users. Why not drop the alpha annotation? Why do I need to support alpha and beta at the same time?

The value is that a user who learns about the feature knows how stable it is.
For example, on GKE, we don't typically expose users to alpha features, but we do expose beta ones.

erictune · 2016-08-22T18:29:51Z

@smarterclayton: /~https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api_changes.md#alpha-beta-and-stable-versions says:

Beta level
- Upgradeability: the object schema and semantics may change in a later software release; when this happens, an upgrade path will be documented; in some cases, objects will be automatically converted to the new version; in other cases, a manual upgrade may be necessary; a manual upgrade may require downtime for anything relying on the new feature, and may require manual conversion of objects to the new version; when manual conversion is necessary, the project will provide documentation on the process (for an example, see v1 conversion tips)

erictune · 2016-08-22T18:34:18Z

As far as I know, this it the first time an alpha annotation feature has moved to beta, so there is no precedent, right?

smarterclayton · 2016-08-22T21:36:32Z

There is no precedence. I was not aware that we weasel worded beta that
much (earlier definitions were more 'we won't break you if we can possibly
avoid it') where this is 'still could break you'. That's fine - forewarned
is ok for me.

Something we need to do for this is to add a release note instructing
admins who are running pod security policy on how to not have a security
gap on upgrade. Since a user can set these annotations prior to a cluster
upgrade, it is possible for a malicious user to pre-fill the annotation
before the policy is enforced, allowing them to bypass PSP protection once
it goes on. Admins will need to upgrade the masters (to get the new
enforcing PSP) and delete all all pre-upgrade pods that have the new "beta"
annotation set on them.

Here's the doc I wrote up for OpenShift 3.3 (where SCC policy was already
GA)
openshift/openshift-docs#2507 (comment)
openshift/openshift-docs#2507 (comment)

On Mon, Aug 22, 2016 at 2:34 PM, Eric Tune notifications@github.com wrote:

As far as I know, this it the first time an alpha annotation feature has
moved to beta, so there is no precedence, right?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#31026 (comment),
or mute the thread
/~https://github.com/notifications/unsubscribe-auth/ABG_pzTeRdJv_SeK4BZM3roTPhhqjOfuks5qieu5gaJpZM4Jo2A9
.

erictune · 2016-08-24T00:20:36Z

The procedure you referenced appears to be solid for going 1.2 to 1.3. It cannot be used as is in a race-free manner for 1.3 to 1.4. Need to think about it a little more.

erictune · 2016-08-24T00:20:48Z

Updated the relnote to reflect action required.

Automatic merge from submit-queue Fix backward compatibility issue caused by promoting initcontainers f… #31026 moves init-container feature from alpha to beta, but only took care the backward compatibility for pod specification, not deal with status. For status, it simply moved from `pods.beta.kubernetes.io/init-container-statuses` to `pods.beta.kubernetes.io/init-container-statuses` instead of introducing one more pods.beta.kubernetes.io/init-container-statuses. This breaks when the cluster is running with 1.4, but the user is still running with kubectl 1.3.x. Fixed #32711

googlebot added the cla: yes label Aug 19, 2016

erictune added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Aug 19, 2016

erictune assigned dchen1107 Aug 19, 2016

erictune added this to the v1.4 milestone Aug 19, 2016

k8s-github-robot added kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 19, 2016

erictune force-pushed the betainit branch from b8b1f9f to 6e5a7f9 Compare August 19, 2016 21:00

k8s-github-robot mentioned this pull request Aug 19, 2016

[k8s.io] Probing container should *not* be restarted with a /healthz http liveness probe [Conformance] {E2eNode Suite} #30714

Closed

dchen1107 reviewed Aug 20, 2016
View reviewed changes

dchen1107 added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 20, 2016

k8s-github-robot merged commit e9bd805 into kubernetes:master Aug 21, 2016

smarterclayton reviewed Aug 21, 2016
View reviewed changes

erictune added release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Aug 23, 2016

dchen1107 mentioned this pull request Sep 19, 2016

Fix backward compatibility issue caused by promoting initcontainers f… #33055

Merged

k82cn mentioned this pull request Feb 6, 2017

Enhancement for init containers annotation. #41004

Closed

3 tasks

erictune deleted the betainit branch August 8, 2017 18:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move init-container feature from alpha to beta. #31026

Move init-container feature from alpha to beta. #31026

erictune commented Aug 19, 2016 •

edited

Loading

dchen1107 Aug 20, 2016

dchen1107 commented Aug 20, 2016

dchen1107 commented Aug 20, 2016

liggitt commented Aug 21, 2016

k8s-bot commented Aug 21, 2016

k8s-github-robot commented Aug 21, 2016

smarterclayton commented Aug 21, 2016

smarterclayton Aug 21, 2016

erictune Aug 22, 2016

erictune commented Aug 22, 2016

erictune commented Aug 22, 2016 •

edited

Loading

smarterclayton commented Aug 22, 2016

erictune commented Aug 24, 2016

erictune commented Aug 24, 2016

Move init-container feature from alpha to beta. #31026

Move init-container feature from alpha to beta. #31026

Conversation

erictune commented Aug 19, 2016 • edited Loading

dchen1107 Aug 20, 2016

Choose a reason for hiding this comment

dchen1107 commented Aug 20, 2016

dchen1107 commented Aug 20, 2016

liggitt commented Aug 21, 2016

k8s-bot commented Aug 21, 2016

k8s-github-robot commented Aug 21, 2016

smarterclayton commented Aug 21, 2016

smarterclayton Aug 21, 2016

Choose a reason for hiding this comment

erictune Aug 22, 2016

Choose a reason for hiding this comment

erictune commented Aug 22, 2016

erictune commented Aug 22, 2016 • edited Loading

smarterclayton commented Aug 22, 2016

erictune commented Aug 24, 2016

erictune commented Aug 24, 2016

erictune commented Aug 19, 2016 •

edited

Loading

erictune commented Aug 22, 2016 •

edited

Loading