aws.profile not respected in ecr-credential-provider template.

[nairb774]

Image I'm using:

Bottlerocket OS 1.12.0 (aws-k8s-1.22)

What I expected to happen:

Using settings like the following should result in the AWS_PROFILE environment variable being set to kubelet within the /etc/kubernetes/kubelet/credential-provider-config.yaml file on the host. Additionally the desired role isn't being used for pulling images.

[settings.aws]
config = "W3Byb2ZpbGUga3ViZWxldF0KY3JlZGVudGlhbF9zb3VyY2UgPSBFYzJJbnN0YW5jZU1ldGFkYXRhCnJvbGVfYXJuID0gYXJuOmF3czppYW06OjEyMzQ1Njc4OTAxMjpyb2xlL015RXh0cmFSb2xlCg=="
profile = "kubelet"
region = "us-east-1"

[settings.kubernetes.credential-providers.ecr-credential-provider]
enabled = true
image-patterns = ["*.dkr.ecr.*.amazonaws.com"]

The expanded settings.aws.config is as follows:

[profile kubelet]
credential_source = Ec2InstanceMetadata
role_arn = arn:aws:iam::123456789012:role/MyExtraRole

What actually happened:

The credential helper file is missing the needed environment variable. This is the output as seen with the above configuration:

[root@admin]# cat /.bottlerocket/rootfs/etc/kubernetes/kubelet/credential-provider-config.yaml
apiVersion: kubelet.config.k8s.io/v1alpha1
kind: CredentialProviderConfig
providers:
  - name: ecr-credential-provider
    matchImages:
      - "*.dkr.ecr.*.amazonaws.com"
    defaultCacheDuration: "12h"
    apiVersion: credentialprovider.kubelet.k8s.io/v1alpha1
    env:
      - name: HOME
        value: /root

How to reproduce the problem:

Use the EKS BottlerocketOS image with the additional settings shown above to observe the incorrect configuration generation. Obviously the role referenced in the kubelet profile should exist with sufficient permissions to allow the instance role to assume it, and be useful for container image pulling. Given this looks to, at a minimum, just be a templating issue the whole setup might not be entirely necessary.

[stmcginnis]

Hi @nairb774 - did you see this documented somewhere? If so, that needs to be corrected. The credential provider uses the default profile.

Do you have a use case in mind for wanting a separate profile?

[nairb774]

Few things make me think that the settings.aws.profile setting should work. The documentation for settings.kubernetes.credential-providers has the following note:

Note: ecr-credential-provider is currently the only supported provider. To manage its AWS credentials, see the settings.aws.config and settings.aws.credentials settings.

In that section has the settings.aws.profile documented right under it: /~https://github.com/bottlerocket-os/bottlerocket#aws-specific-settings which says:

settings.aws.profile: The profile name to use from the provided config and credentials settings.
Which points at being able to control the profile configured in settings.aws.config.

The documentation wasn't very clear, but I was able to locate #2377 which added the CredentialProviderConfig and that template seems to explicitly try to set the AWS_PROFILE environment variable if the relevant config setting is set:

bottlerocket/packages/kubernetes-1.22/credential-provider-config-yaml

Lines 18 to 21 in 6ef1139

	{{#if settings.aws.profile}}
	- name: AWS_PROFILE
	value: {{settings.aws.profile}}
	{{/if}}

Do you have a use case in mind for wanting a separate profile?

I was attempting to see if this might be a partial path for implementing something like described in #1624. Basically a way to "hide" the IAM permissions needed by the kubelet from other programs running on the box. Partially for security, and partially for trying to prevent crafty engineers from depending on policies being attached to the instance.

Hope that helps some.

Edit: Using default as the profile name seems to work, but feels a little messy.

[stmcginnis]

Sorry, you are absolutely right. Let me look in to what's happening here...

[stmcginnis]

I believe this may be an order of operations issue. When the credential-providers settings are updated, that triggers the re-rendering of the credential-provider-config.yaml file. But that is tied to the settings.kubernetes values. So when the settings.aws.config settings are changed, I think right now that does not cause the template to be re-rendered.

So my hunch at this point is, if you set the aws config settings first, then the kubernetes credential provider settings, when the template is rendered it will pick up the aws.profile value. If you do it in the other order, setting the credential provider settings, then the AWS profile, it will miss the value and not include it rendered template.

Will do more testing to verify if this is true or not, and if so, see what can be done to have updates to either setting trigger the re-rendering.

[nairb774]

I attempted forcing a rerendering by updating the creds provider settings (add image pattern) and it didn't seem to pickup the profile name. Knowing nothing about the implementation, I'd guess the available config was being pruned or something before rendering, but the code didn't seem to have any implementation for that.

[stmcginnis]

I tested applying the AWS config settings first, then the credential provider settings. Profile was not picked up.

I then tested applying credential provider settings, then AWS config. Same results.

So it looks like the issue is regardless of order, when the credential-provider-config template is rendered, it never has a value for the settings.aws.profile value. For some reason it's just never populated.

And ordering-wise, I believe it is an issue that applying the setting for aws.profile does not cause the credential provider config file to be re-rendered, and does not cause the kubelet service to be restarted.

So either those issues need to be sorted out, or the profile setting deprecated. It was only added as a convenience so someone could take their own ~/.aws/config file, base64 encode it, and push it into the instance settings. There currently isn't any other reason to use a different profile than default on the Bottlerocket instance.

Removal, or deprecation with a big note in the settings documentation stating it is unused, might be the most straightforward thing to do, but would love to hear arguments for or against that idea. :)

[stmcginnis]

Also tested rebooting an instance to see if the settings would get rendered correctly on fresh initialization. No luck.