More debugging today came up with some additional theories. Openstack uses mysql, which ends up getting deployed across three units. In our case, keystone and nova end up on the same physical machines. Mysql is deployed with a default setting for innodb_buffer_pool_size which causes it to think it can allocate up to 50% of memory for it internal buffer pool. This ends up being 50% of the host machines memory, even though it's smooshed together with 6 other services.
I'm going to attempt to tune innodb_buffer_pool_size to a lower value (say 10%) and see what the outcome is.
More debugging today came up with some additional theories. Openstack uses mysql, which ends up getting deployed across three units. In our case, keystone and nova end up on the same physical machines. Mysql is deployed with a default setting for innodb_ buffer_ pool_size which causes it to think it can allocate up to 50% of memory for it internal buffer pool. This ends up being 50% of the host machines memory, even though it's smooshed together with 6 other services.
I'm going to attempt to tune innodb_ buffer_ pool_size to a lower value (say 10%) and see what the outcome is.