After looking at this some more it seems that the neutron agents report their state frequently. Each of those updates requires database queries. These queries are what check out the stale connections from the pool. Each of those stale connections has to wait until mysql kills it. I think if we had more compute nodes in our environment (generating more heartbeat messages) we would see the same issue with nova-conductor. Disposing of the connection pool when we encounter this error seems to be the right thing to do.
After looking at this some more it seems that the neutron agents report their state frequently. Each of those updates requires database queries. These queries are what check out the stale connections from the pool. Each of those stale connections has to wait until mysql kills it. I think if we had more compute nodes in our environment (generating more heartbeat messages) we would see the same issue with nova-conductor. Disposing of the connection pool when we encounter this error seems to be the right thing to do.