How many flux kustomizations are too many kustomizations? Using multiple flux kustomizations in the same repo.

[ikatolkar]

First of all thank you for Flux !! Its an amazing tool !!!

We have a gitops repository for a tenant (product team) which contains helmreleases for multiple users working in the same team (each user has a kustomization sub directory in the repo). Initially I had setup the repo by creating a flux kustomization that pointed to the top level kustomization in the repo, thus recursively working on each users kustomization.

The problem here was that if any user pushed a bad yaml, that would stop the overall reconciliation for the entire repo affecting other users.

To solve this, I created multiple flux kustomization objects that would point to the user's subdirectory, instead of pointing to the top level kustomization. So that a user's broken kustomization wouldn't affect overall repo's kustomization.

My question is, would this have an adverse affect on the reconciliation time/performance and also would it start increasing resource requirements? We have around 10 tenants on the cluster with max 30 users in each tenant repo. (max 3 helmreleases for each user)

Any suggestions on better tackling this problem would also be much appreciated.

[repo structure]
tenantrepo
|-->user1-dir <------- flux kustomization 1 for user1
| |-->helmrelease1.yaml
| |-->helmrelease2.yaml
| |-->kustomization.yaml
|-->user2-dir <------- flux kustomization 2 for user2
| |-->helmrelease1.yaml
| |-->kustomization.yaml
|--->kustomization.yaml <------- top level kustomization for entire repo (earlier used, but not anymore)

PS. We can't have a separate repo for each user inside a team due to IT management limitations.

[stefanprodan]

My question is, would this have an adverse affect on the reconciliation time/performance and also would it start increasing resource requirements? We have around 10 tenants on the cluster with max 30 users in each tenant repo.

Flux was designed in this way, each tenant/team/app should be reconciled by its own Kustomization. I see no issues running hundreds of Kustomization per cluster. If you get to thousands of Flux objects then you’ll need to increase the memory limits and the max parallel processing settings of kustomize-controller.

[ikatolkar]

Thanks Stefan, that answers my question for now.

[pedro-n-rocha]

Hi, just to clarify—does not Flux track the entire Git repository’s revision hash, meaning that any commit anywhere in the repo will trigger a new revision artifact and a reconciliation of all Kustomizations inside it?

For example, if a commit is made inside user1-dir resources , will it also trigger a reapply for user2-dir , userx-dir, etc , even if nothing changed inside those directories?

[stefanprodan]

if a commit is made inside user1-dir resources , will it also trigger a reapply for user2-dir , userx-dir, etc , even if nothing changed inside those directories?

Yes, but only user1 changes will be applied, for all the others the apply will be skipped as the drift detection returns nothing.