Able to set --cluster-dns

[mzupan]

What I'd like:

Being able to set --cluster-dns flag for kubelet. This allows the kubelet to set the dns search to something other then coredns like node-local-dns then fail back to coredns

[gthao313]

Hi @mzupan Thanks for opening this issue. Bottlerocket team will research on this and reach back to you soon. Thanks.

[mzupan]

@gthao313 thanks how it was used with just a linux ami in eks was like

 --cluster-dns=169.254.20.10,172.20.0.10

The first IP being the IP node-local sets on each node and the 2nd being coredns.

That set any pod running on that node to have a resolv.conf that looked like

❯ k exec -it debug-6ff4966bc5-dn6tq -- cat /etc/resolv.conf
nameserver 169.254.20.10
nameserver 172.20.0.10
search default.svc.cluster.local svc.cluster.local cluster.local ec2.internal
options ndots:5

[bcressey]

Does settings.kubernetes.cluster-dns-ip not work for this? We generate a value for the setting automatically, but if you specify a value that should override the default.

[mzupan]

@bcressey that doesn't seem to allow you to specify multiple values like --cluster-dns in the kubelet does. Really there for if node-local somehow fails it uses coredns as a failback

[bcressey]

Now that you point it out, it's obvious that it can be a list of IPs in the kubelet config template, but it never crossed my mind to make it a list in our API. 😄

@etungsten seems like we need some of the serde magic here to migrate cluster-dns-ip into a list shape on update.

[cbgbt]

This is resolved in the latest Bottlerocket release, v1.8.0 -- settings.kubernetes.cluster-dns-ip now supports being set to an array of IP addresses. Check the documentation for an example on setting this.

Let us know if you have any questions!