simulator: synchronically replicate cluster state from a real cluster to a fake one for the scheduler testing

[sanposhiho]

/assign
/kind feature

---

This issue proposes a new simple component to keep replicating the state from a prod cluster to a fake cluster.

background

Testing the scheduler is a complex challenge. There are countless patterns of operations executed within a cluster, making it impractical to anticipate every scenario with a finite number of tests. More often than not, bugs are discovered only when the scheduler is deployed in an actual cluster.

Having a development or sandbox environment for testing the scheduler—or, indeed, any Kubernetes controllers—is a common practice. However, this approach falls short of capturing all the potential scenarios that might arise in a production cluster. It’s an inevitable truth that a development cluster never sees the exact same use or exhibits the same behavior as its production counterpart, with notable differences in workload sizes and scaling dynamics.

User story

We have a custom scheduler which has a co-scheduling feature.
We want to test it in a cluster that gets similar resources as our production cluster. But, our production cluster is much bigger than our development cluster and it's unrealistic to catch all bugs there.

Resources to sync

We shouldn't simply do that, we have to think about what to sync and what not to.

All resources involved in the scheduling should be synced.
And, we should make it configurable to select which resources to sync, given everyone could have a different scheduler plugin which schedules Pods based on anything.

By default, we should sync:

• Pods
• Nodes
• PVs
• PVCs
• SC

Scheduled Pods

We cannot simply sync all changes to Pods, because the real cluster has the scheduler, and it schedules all Pods in the cluster.
If we simply synced all changes to Pods, the scheduling result would also be synced. (and may conflicted with the decision of another scheduler which is in a fake cluster.)

So, we don't sync any of updated events to scheduled Pods.
Pods are synced like:

1. In a real cluster, Pod-a is created
2. In a fake cluster, Pod-a is created. (synced)
3. In a real cluster, the scheduler schedules Pod-a to Node-a. We don't copy this change to a fake cluster.
4. In a fake cluster, the scheduler, which is different one from (3), schedules Pod-a to Node-x.

It means that the scheduling results may be different between a real cluster and a fake cluster. But, it's OK.
Our purpose is to create a fake cluster for testing the scheduling, which gets the same load as the production cluster.

[utam0k]

I'm also interested in this feature.

[sanposhiho]

/retitle simulator: synchronically replicate cluster state from a real cluster to a fake one for the scheduler testing
/area simulator

I'll make it in the simulator, on second thought.

[sanposhiho]

/priority next-release

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[sanposhiho]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[sanposhiho]

/remove-lifecycle rotten

[utam0k]

Our company is interested in giving it a try. May I ask you to assign me?

[sanposhiho]

/assign @utam0k

[saza-ku]

Hi! I'd like to take over this issue. Could you assign me?

[saza-ku]

First, I would start with adding a pseudo real cluster needed for debugging syncer to docker-compose-local.yml.

Then, I'd incrementally implement syncer replacing #335. Is it okay to close #335?

[sanposhiho]

/assign @saza-ku

Is it okay to close #335?

Yup, I'll close that one myself.

[sanposhiho]

/unassign

[saza-ku]

So, we don't sync any of updated events to scheduled Pods.

When it starts to sync, it won't sync pods that are already exist in the real cluster. But is it okay because they can be imported by enabling externalImportEnabled?

[saza-ku]

Anyway I'll proceed to implement in the following steps.

• Add a psuedo real cluster required for debugging syncer: Add fake-source-cluster for testing the sync service #364
• Rename ImportClusterResourceService to OneshotClusterResourceService: Rename ImportClusterResourceService to OneshotClusterResourceImporter #365
• Implement the syncing service
• Add the syncing service to the DI container, and call it in the entry point of the simulator server

[sanposhiho]

When it starts to sync, it won't sync pods that are already exist in the real cluster.

It will. The syncer's event handler propagates existing Pods to the fake cluster when it starts.

[saza-ku]

It will. The syncer's event handler propagates existing Pods to the fake cluster when it starts.

I see. So the syncer will add all the pods regardless of whether they are scheduled or not, right?

If so, it might be a problem when already scheduled pods (users manually specify nodeName of them) are added in the real cluster. For example, the specified node in the simulator cluster can have no capacity while the one in the real cluster has.

But it's an edge case that only occurs occasionally. Let's discuss the problem in a following issue. I'll do the first implementation in the way you say.

[sanposhiho]

We don't sync update event of scheduled Pods, but we can sync addition events of them.