TLS errors after failed plugin initialization

[mafos]

Hi, I'm seeing persistent login failures with 403 responses after leadership failover in Vault v1.12.1.

Server log error message:

login unauthorized: err="Post \"https://kube-apiserver.example/apis/authentication.k8s.io/v1/tokenreviews\": x509: certificate signed by unknown authority"

This is with unchanged mount & role configuration that was previously functional. The configured kubernetes_ca_cert value served via /config also aligns with the live Kubernetes API.

It appears to be related to intermittent storage errors at plugin initialization and a change in HTTP client initialization with #142.

Server logs near unseal time (azure storage backend in this case):

error=
| -> github.com/Azure/azure-pipeline-go/pipeline.NewError, /home/runner/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.3/pipeline/error.go:157
| HTTP request failed
| 
| Get "https://examplecontainer.blob.core.windows.net/vault-data/auth/kubernetes/config?timeout=61": context canceled

Executing vault plugin reload is necessary to fix the 403 responses and re-enable successful TokenReview requests to the Kubernetes API. The plugin mount does not recover on its own otherwise.

Steps to reproduce

1. Provision a local Kubernetes cluster using kind and serviceaccount tokens

   # create k8s cluster via kind 
   kind create cluster --name demo
   
   # create vault serviceaccount & binding
   kubectl create sa vault
   kubectl create clusterrolebinding \
           system:auth-delegator:vault \
           --clusterrole=system:auth-delegator \
           --serviceaccount=default:vault
   
   # create vault reviewer token
   VAULT_REVIEWER_JWT=$(kubectl create token vault)

2. Start a vault server using file storage

   cat <<- EOF > vault.hcl
   listener "tcp" {
     tls_disable = true
     address = "127.0.0.1:8200"
   }
   
   storage "file" {
     path = "./vault-data"
   }
   
   log_level = "Debug"
   EOF
   
   vault server -config=vault.hcl

3. Test login after forced plugin config storage read failure on initialization

   # initialize & unseal vault
   vault operator init -key-shares=1 -key-threshold=1
   vault operator unseal $VAULT_UNSEAL_KEY
   
   # configure kubernetes auth backend
   vault auth enable kubernetes
   vault write auth/kubernetes/config \
           kubernetes_host="$(kubectl config view -ojson | jq -r '.clusters[] | select(.name == "kind-demo") | .cluster.server')" \
           kubernetes_ca_cert="$(kubectl get cm kube-root-ca.crt -ojson | jq -r '.data["ca.crt"]')" \
           token_reviewer_jwt=$VAULT_REVIEWER_JWT
   
   # create kubernetes "default" auth role
   vault write auth/kubernetes/role/default \
           bound_service_account_names=default \
           bound_service_account_namespaces=default \
           policies=default
   
   # generate test serviceaccount token
   DEFAULT_JWT=$(kubectl create token default)
   
   # attempt login => success
   vault write auth/kubernetes/login role=default jwt=$DEFAULT_JWT
   
   # stop vault
   pkill vault
   
   # disable reads of underlying backend config to fail plug initialize
   chmod 200 vault-data/auth/*/_config
   
   # restart vault separately & unseal
   vault operator unseal $VAULT_UNSEAL_KEY
   
   # re-enable reads of underlying backend config 
   chmod 600 vault-data/auth/*/_config
   
   # attempt login => failure (x509: “kube-apiserver” certificate is not trusted)
   vault write auth/kubernetes/login role=default jwt=$DEFAULT_JWT

[heatherezell]

Thank you for reporting this! Our engineers are currently discussing a fix. It might not be able to go out in 1.12.2, but we will do our best to get this fixed quickly. Thanks again!

[nsimons]

@hsimon-hashicorp, we bumped into a similar issue with the cached HTTP client improvement in #142 not reloading the CA certificate from local disk. I have a fix proposal about 90% complete that solves both that issue and this one. Would you be open to a contribution or are you handling it internally?

Should I also create a separate issue about that, since it's a different use case?

[anthonyralston]

@hsimon-hashicorp We have also seen this issue in our Vault clusters--We initially contacted HashiCorp support about it 15 days ago. +1 for a fix ASAP.