[bug]: The worker component takes up too much memory on multi-core CPUs

[TheresaQWQ]

Is there an existing issue for this?

• I have searched the existing issues

Current behavior

Using a server with enough cores (e.g. 32 cores), start Plane using docker or Kubernetes, and observe the logs to see celery Killed, etc. This is due to the fact that celery uses the number of CPU cores as the number of threads by default, which leads to the memory usage exceeding the default memory limits

I hope it's possible to limit the number of threads by adding, for example, a parameter like "--concurrency=8".

Steps to reproduce

Using a server with enough cores (e.g. 32 cores), start Plane using docker or Kubernetes, then, boom!

Environment

Production

Browser

Google Chrome

Variant

Self-hosted

Version

v1.8.1 (helm)