Comment 0 for bug 2032949

Revision history for this message
Jacopo Rota (r00ta) wrote : Rack keeps opening idle connections over the limits

As per title, the rack is opening idle connections without limits under no load.

Steps to reproduce:

1) add a log statement like
```
        log.info("THERE ARE " + str(len(self.getAllClients())) + " CONNECTIONS!")
```
in provisioningserver/rpc/clusterservice.py
2) install maas from sources
3) maas init region+rack
4) wait **some minutes** and take a look at the logs:

```
Aug 24 12:35:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 0 CONNECTIONS!
Aug 24 12:35:20 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 3
...
CONNECTIONS!
Aug 24 12:35:45 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:46 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:47 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:48 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:36:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 6 CONNECTIONS!
Aug 24 12:36:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 8 CONNECTIONS!
Aug 24 12:37:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 10 CONNECTIONS!
Aug 24 12:37:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 12 CONNECTIONS!
Aug 24 12:38:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 14 CONNECTIONS!
```

and they keep increasing every 30 seconds. In addition to this strange behaviour, it is actually exceeding the configuration limits which are

```
    max_idle_rpc_connections = ConfigurationOption(
        "max_idle_rpc_connections",
        "The nominal number of connections to have per endpoint",
        Number(min=1, max=1024, if_missing=1),
    )
    max_rpc_connections = ConfigurationOption(
        "max_rpc_connections",
        "The maximum number of connections to scale to when under load",
        Number(min=1, max=1024, if_missing=4),
    )
```

If the region has 4 workers, it reaches 20 connections and it stops. If I specify the max max_rpc_connections=1000, then it keeps opening connections.