Error creating instance: Expected Array or Hash, not Nil

[Funzinator]

Hi,
I have the feeling it's a problem with the Hetzner API or something, but I am getting this since yesterday night. I was assuming some sort of rate-limiting because I played around with cluster creation and created many clusters in short time.
However, since it happens still, I wanted to raise this here. Is there a way to see the API responses?

longer log:

| |__   ___| |_ _____ __   ___ _ __     | | _|___ / ___ 
| '_ \ / _ \ __|_  / '_ \ / _ \ '__|____| |/ / |_ \/ __|
| | | |  __/ |_ / /| | | |  __/ | |_____|   < ___) \__ \
|_| |_|\___|\__/___|_| |_|\___|_|       |_|\_\____/|___/

Version: 2.2.6

[Configuration] Validating configuration...
[Configuration] ...configuration seems valid.
[Placement groups] Deleting unused placement group test-masters...
[Placement groups] ...placement group test-masters deleted
[Placement groups] Creating placement group test-masters...
[Placement groups] ...placement group test-masters created
Error creating instance: Expected Array or Hash, not Nil
Instance creation for test-master1 failed. Try rerunning the create command.

And in case it's required, my config:

cluster_name: test
kubeconfig_path: "./kubeconfig"
k3s_version: v1.30.3+k3s1

networking:
  ssh:
    port: 22
    use_agent: false
    public_key_path: "/root/.ssh/id_rsa.pub"
    private_key_path: "/root/.ssh/id_rsa"
  allowed_networks:
    ssh:
      - 0.0.0.0/0
    api:
      - 0.0.0.0/0
  public_network:
    ipv4: false
    ipv6: false
  private_network:
    enabled: true
    subnet: 10.63.0.0/16
    existing_network_name: "test"
  cni:
    enabled: true
    encryption: false
    mode: flannel

datastore:
  mode: etcd
  external_datastore_endpoint: postgres://....

schedule_workloads_on_masters: false

image: debian-12
autoscaling_image: debian-12

masters_pool:
  instance_type: cax11
  instance_count: 1
  locations:
    - fsn1

worker_node_pools:
- name: medium-autoscaled
  instance_type: cax11
  location: fsn1
  autoscaling:
    enabled: true
    min_instances: 0
    max_instances: 3

embedded_registry_mirror:
  enabled: false

protect_against_deletion: true

create_load_balancer_for_the_kubernetes_api: false

[Funzinator]

OK, I have seen #418 and then realized that deleting .kubeconfig solves the issue. Next time I encounter this issue, I will provide a copy of the file for further investigation. I have deleted it now already.

[Funzinator]

Alright, it happened again (it's related to private-network only setups, I am pretty sure...)

 _          _                            _    _____     
| |__   ___| |_ _____ __   ___ _ __     | | _|___ / ___ 
| '_ \ / _ \ __|_  / '_ \ / _ \ '__|____| |/ / |_ \/ __|
| | | |  __/ |_ / /| | | |  __/ | |_____|   < ___) \__ \
|_| |_|\___|\__/___|_| |_|\___|_|       |_|\_\____/|___/

Version: 2.2.6

[Configuration] Validating configuration...
[Configuration] ...configuration seems valid.
[Placement groups] Deleting unused placement group test-masters...
[Placement groups] ...placement group test-masters deleted
[Placement groups] Creating placement group test-masters...
[Placement groups] ...placement group test-masters created
[Instance test-master1] Creating instance test-master1 (attempt 1)...
[Instance test-master1] Instance status: off
[Instance test-master1] Powering on instance (attempt 1)
[Instance test-master1] Waiting for instance to be powered on...
[Instance test-master1] Instance status: running
[Instance test-master1] ...instance test-master1 created
[Firewall] Updating firewall...
[Firewall] ...firewall updated
[Instance test-master1] 🕒 Awaiting cloud config (may take a minute...)
[Instance test-master1] Cloud init finished: 14.74 - Thu, 13 Feb 2025 11:27:09 +0000 - v. 22.4.2
[Instance test-master1] Private network IP in subnet 10.63.0.0/16 is up
[Instance test-master1] Waiting for the control plane to be ready...
[Control plane] Generating the kubeconfig file to /root/kubeconfig...
error: no context exists with the name: "test-master1"
[Control plane] : error: no context exists with the name: "test-master1"

/root/kubeconfig then looks like this, so I guess it's because clusters is null:

apiVersion: v1
clusters: null
contexts: null
current-context: ""
kind: Config
preferences: {}
users: null

And the next run yields Error creating instance: Expected Array or Hash, not Nil.

[vitobotta]

I just reproduced the issue with the public network turned off. In this setup, when the tool attempts to install k3s, it tries to fetch the right version from https://get.k3s.io. However, since the node isn't connected to the internet via a public interface, it can’t access the web and thus fails to install. Could you give it another go using a NAT gateway setup? I’ll do the same test on my end.

I will also add some exception handling to hetzner-k3s to spot when the nodes can't access the internet.

[Funzinator]

Is there a special way to indicate a NAT setup in the cluster config? I have NAT setup properly. The node I execute hetzner-k3s is also the NAT gateway. If I ssh into the master after the failure I can access get.k3s.io without issues.

[vitobotta]

I got it working nicely. I will write a doc page on this in a bit.

[Funzinator]

Can't wait ... (to spend my evening with my new shiny cluster)

[vitobotta]

Here's the page: /~https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Private_clusters_with_public_network_interface_disabled.md

I've tested this configuration several times, and every time the cluster is set up properly. I'll update the page once I find some time to test it with other Linux versions.

[Funzinator]

Thank you so much! I finally got it working properly.
For autoscaling to work, I had to kubectl edit deployment -n kube-system cluster-autoscaler and move the post_create_commands that are base64 encoded in HCLOUD_CLOUD_INIT before the k3s install part. Basically what @mertcangokgoz wrote in #466.
But this is a one-off change I can live with.

[vitobotta]

I have reopened this as another reminder to address the autoscaler as well.

[mertcangokgoz]

Thank you so much! I finally got it working properly. For autoscaling to work, I had to kubectl edit deployment -n kube-system cluster-autoscaler and move the post_create_commands that are base64 encoded in HCLOUD_CLOUD_INIT before the k3s install part. Basically what @mertcangokgoz wrote in #466. But this is a one-off change I can live with.

I am glad that your problem is solved and I should also remind you that in a cluster configured as nat-gateway, especially the hc-utils package should be removed from the nodes.

apt autoremove -y hc-utils && apt purge -y hc-utils 

Do not forget to add the cloud-init command. If not uninstalled, the hc-utils package cuts network access directly and we may have to restart the machine. (hetzner has already mentioned this in the FAQ. https://docs.hetzner.com/cloud/networks/faq)

[vitobotta]

Thanks for the reminder @mertcangokgoz ! I updated the reference page with sample config.