How to specify requests and limits for main container of orchestrator pod

[ogusak]

• We are running Airbyte open source on EKS cluster with container orchestrator enabled (CONTAINER_ORCHESTRATOR_ENABLED=true).
• We know we can control CPU and memory limits of main container of all job types (orchestor, source/destination connector) with worker's environment variable JOB_MAIN_CONTAINER_CPU_LIMIT (for CPU limit).
• We'd like to control cpu and memory limits for the orchestrator container separately from other job pods as we noticed its main container requires a lot more CPU than any other job container (evident by monitoring CPU throttling metric for the container).
• We could not find any documentation on this issue and in the code could only find this parameter REPLICATION_ORCHESTRATOR_CPU_LIMIT
• We provided REPLICATION_ORCHESTRATOR_CPU_LIMIT environment variable to worker pod/container but it didn't change the value of the CPU limit of the newly spawned orchestrator containers.

Please let us know if it is possible to control orchestrator's main container CPU limit.
Thanks!

[Vadym5]

the same questions..

[LarsHanegraaf]

I'm also interested in the answer, as it seems it is the performance bottleneck for our connectors built with the Connector Builder as well.

[Vadym5]

we resolved this via k8 admission controller with providing custom resources for pods we need

[LarsHanegraaf]

In the end I managed to get the result by altering the values.yaml of our Airbyte Kubernetes deployment:

global:
  workload-launcher:
    resources:
      limits:
        cpu: 2