all controllers node hang in MOS 9.2 env which have 300 nodes

Bug #1744689 reported by Xiwen Deng
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Incomplete
Undecided
Xiwen Deng

Bug Description

We face a grievous issue that all controller nodes hang in MOS 9.2 env. Cann't ssh and all openstack service block.

Env describe is below:
1. Three controller nodes、three network nodes、three rabbitmq nodes. We separate network and rabbitmq service from controller nodes.
2. We have 300 openstack nodes and about 2700 VMs.

Issue describe is below:
1. all controller、network、rabbitmq node hang and cann't ssh. ALL services include openstack service and ubuntu service block.
2. use cmd "strace" ssh service at the controller node, we find ssh service block at "connect(7, {sa_family=AF_LOCAL, such_path="/dev/log"}, 110) = 0"
3. if we stop rsyslog service all nodes can access and all openstack service re-work.

So what is the root cause? Do anyone face the same issue?

Xiwen Deng (deng-xiwen)
summary: - all controllers node hang in MOS 9.2 env
+ all controllers node hang in MOS 9.2 env that have 300 nodes
summary: - all controllers node hang in MOS 9.2 env that have 300 nodes
+ all controllers node hang in MOS 9.2 env which have more than 300 nodes
summary: - all controllers node hang in MOS 9.2 env which have more than 300 nodes
+ all controllers node hang in MOS 9.2 env which have 300 nodes
Revision history for this message
Denis Meltsaykin (dmeltsaykin) wrote :

Is there enough free disk space on nodes? On Fuel Master node?

Changed in fuel:
status: New → Incomplete
assignee: nobody → Xiwen Deng (deng-xiwen)
Revision history for this message
Xiwen Deng (deng-xiwen) wrote :

@Denis
    Yes, all nodes have enough free disk space. We find the issue key is that all nodes will send some service logs to Fuel node. If env have too many nodes, Fuel node can not process so many service logs.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.