Kubernetes API Load Balancer

[RFE] Add option to configure worker_connections / worker_rlimit_nofile"

Bug #1948019 reported by Pedro Victor Lourenço Fragola
24
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Kubernetes API Load Balancer
Fix Committed
Medium
Mateo Florido
Kubernetes API Load Balancer 1.29

Bug Description

I was working with cluster k8s and facing the error "worker_connections are not enough" because the environment had more than a thousand connections and by default the configuration of charm/nginx is 768:

/etc/nginx/nginx.conf
worker_connections 768;

After manually changing the parameter "work_connections" the environment was healthy.

To avoid manual configuration the charm could have the option to configure the parameters "worker_connections" and "worker_rlimit_nofile".

Tags: sts
Revision history for this message
Navdeep (navdeep-bjn) wrote :

Any Update on this?

Revision history for this message
Sebastian Kosterczka (skosteczka) wrote (last edit ):

I recently encountered this same error ("worker_connections are not enough"). It causes a "NotReady" status for nodes. Any updates on this?

Kevin W Monroe (kwmonroe)
Changed in charm-kubeapi-load-balancer:
status: New → Triaged
importance: Undecided → Low
importance: Low → Medium
assignee: nobody → Kevin W Monroe (kwmonroe)
milestone: none → 1.26
George Kraft (cynerva)
Changed in charm-kubeapi-load-balancer:
milestone: 1.26 → 1.26+ck1
Adam Dyess (addyess)
Changed in charm-kubeapi-load-balancer:
milestone: 1.26+ck1 → 1.26+ck2
Revision history for this message
John Puskar (jpuskar-amtrust) wrote :

Just got hit by this too. Our prod cluster had failures due to node readiness, due to this:

2023/02/12 14:28:47 [alert] 3006059#3006059: *11584397 socket() failed (24: Too many open files) while connecting to upstream, client: 10.88.0.62, server: server_443, request: "GET /api/v1/nodes/juju-ed4383-16?timeout=10s HTTP/2.0", upstream: "https://10.88.0.42:6443/api/v1/nodes/juju-ed4383-16?timeout=10s", host: "10.88.0.41:443"

2023/02/12 14:28:47 [alert] 3006059#3006059: *11584397 socket() failed (24: Too many open files) while connecting to upstream, client: 10.88.0.62, server: server_443, request: "GET /api/v1/nodes/juju-ed4383-16?timeout=10s HTTP/2.0", upstream: "https://10.88.0.39:6443/api/v1/nodes/juju-ed4383-16?timeout=10s", host: "10.88.0.41:443"

Etc.

I'm aware that it's my bad for not having HA on the kubeapi-load-balancer component to share the load, but it'd be great to have this be tunable.

Revision history for this message
John Puskar (jpuskar-amtrust) wrote :

I forgot to mention that my comment above was in regards to worker_rlimit_nofile. Although, we also got hit with OP's original example of needing to increase worker_connections.

Kevin W Monroe (kwmonroe)
Changed in charm-kubeapi-load-balancer:
milestone: 1.26+ck2 → 1.27
Revision history for this message
Kevin W Monroe (kwmonroe) wrote :

The requested options are in different config contexts (events and main, respectively). Rather than exposing individual configs, it would be better to expose each context as a config option. Perhaps even allowing the whole `nginx.conf` to be replaced via config.

In any case, this should to be done at the layer-nginx level. It won't make ck8s 1.27 GA so i'm re-targeting for the next maintenance release.

Changed in charm-kubeapi-load-balancer:
milestone: 1.27 → 1.27+ck1
Revision history for this message
Kevin W Monroe (kwmonroe) wrote :

This won't make the upcoming maint release - re-targeting for the next ck8s ga.

Changed in charm-kubeapi-load-balancer:
milestone: 1.27+ck1 → 1.28
Adam Dyess (addyess)
Changed in charm-kubeapi-load-balancer:
milestone: 1.28 → 1.28+ck1
Mateo Florido (mateoflorido)
Changed in charm-kubeapi-load-balancer:
assignee: Kevin W Monroe (kwmonroe) → Mateo Florido (mateoflorido)
status: Triaged → In Progress
Mateo Florido (mateoflorido)
Changed in charm-kubeapi-load-balancer:
status: In Progress → Fix Committed
Pedro Victor Lourenço Fragola (pedrovlf)
tags: added: sts
Revision history for this message
Kevin W Monroe (kwmonroe) wrote :

PR for reference:

https://github.com/charmed-kubernetes/charm-kubeapi-load-balancer/pull/31

Adam Dyess (addyess)
tags: added: backport-needed
Revision history for this message
Adam Dyess (addyess) wrote :

Backport completed by pulling commits in sync
https://github.com/charmed-kubernetes/charm-kubeapi-load-balancer/pull/33

tags: removed: backport-needed
Adam Dyess (addyess)
Changed in charm-kubeapi-load-balancer:
status: Fix Committed → Fix Released
Revision history for this message
George Kraft (cynerva) wrote :

Due to https://bugs.launchpad.net/bugs/2038347 and https://bugs.launchpad.net/bugs/2038348, we had to roll back kubeapi-load-balancer to a version prior to this fix. This is now targeted for 1.29.

Changed in charm-kubeapi-load-balancer:
status: Fix Released → Fix Committed
milestone: 1.28+ck1 → 1.29
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.