To summarize, based on my investigations, the issue should be at twisted/uvloop level. In particular, if libuv raises an exception (out of memory/too many files open/IO error/whatever), uvloop might silently close the connection and the `connection_lost` callback is not invoked.
MAAS keeps track of the connections to the rack controllers using the `connection_made` and `connection_lost` callbacks. In particular, if `connection_lost` is NOT called, we don't remove this connection from the queue.
You can find the investigations I've made here https:/ /docs.google. com/presentatio n/d/1_YuUw7Dlfg hQxBakmnU4Fv_ o8kp6PdnYEn4MxA cti5I/edit# slide=id. p .
To summarize, based on my investigations, the issue should be at twisted/uvloop level. In particular, if libuv raises an exception (out of memory/too many files open/IO error/whatever), uvloop might silently close the connection and the `connection_lost` callback is not invoked.
MAAS keeps track of the connections to the rack controllers using the `connection_made` and `connection_lost` callbacks. In particular, if `connection_lost` is NOT called, we don't remove this connection from the queue.
I've opened https:/ /github. com/MagicStack/ uvloop/ issues/ 552.
If the uvloop maintainers don't confirm this is a bug, we'll have to investigate on twisted side (for example, the exception should be handled here https:/ /github. com/twisted/ twisted/ blob/a9ee8e59a5 cd1950bf1a18c4b 6ca813e5f56ad08 /src/twisted/ internet/ asyncioreactor. py#L171 - apart from that, twisted should anyway call the `connection_lost` accordingly).