SchedulerCache uses global mutex across cache.go and event_handlers.go causing contention/race-risk in read-heavy paths

[hzxuzhonghu]

Problem

SchedulerCache embeds a single global sync.Mutex which is locked in both pkg/scheduler/cache/cache.go and pkg/scheduler/cache/event_handlers.go for all operations, including read-heavy code paths (e.g., Snapshot(), String(), IsJobTerminated(), parseErrTaskKey()). Informer callbacks also take the same lock in event handlers. This coarse locking increases contention and makes it easy to accidentally introduce unsafe access as the cache grows.

Why it matters

• Read-only operations block each other and block writes, increasing latency.
• Snapshot() and similar methods hold the lock while doing heavy work (clone/iterate), which amplifies contention.
• Global lock across handlers and cache logic makes correctness harder to reason about and increases risk of subtle race bugs.

Suggested improvements (incremental)

1. Switch to sync.RWMutex and use RLock for clearly read-only paths (String(), IsJobTerminated(), parseErrTaskKey(), etc.).
2. Audit Snapshot() and avoid mutating data under shared lock; if needed, keep write lock there or refactor to avoid mutation.
3. (Optional) Consider per-domain locks for Jobs, Nodes, Queues, CSINodesStatus, etc., or serialize mutations via a worker queue.

References

• pkg/scheduler/cache/cache.go
• pkg/scheduler/cache/event_handlers.go

Expected outcome

Reduced contention and clearer separation of read vs write paths, lowering race-condition risk and improving performance under load.

[hzxuzhonghu]

cc @JesseStutler @hajnalmt

I believe sync.RWMutex may not improve much, we need other fine grained lock. This avoids unrelated operations blocking each other, but needs consistent lock ordering, can be a little bit risky.

[JesseStutler]

It's a good catch! The single lock may also downgrade our scheduling performance

[hajnalmt]

/priority high

Yeah we need this.

[JesseStutler]

/assign
I will take a look how we can improve performance in this version