Large amount of tftp processes on the Fuel master
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Fuel for OpenStack |
In Progress
|
Medium
|
Alexey Shtokolov | ||
| Mitaka |
Confirmed
|
Medium
|
Fuel Sustaining | ||
Bug Description
During Rally test suite run on big scale.
On Fuel master node where discovered large number of tftp processes:
[root@fuel ~]# ps auxwwwwf | grep in.tftp | wc -l
1721
[root@fuel ~]# w
16:47:00 up 4 days, 22:58, 8 users, load average: 119.04, 139.14, 92.04
which causes loading system and slowing down of "Nova_create_
It was found that two nodes which weren't included in the cluster and weren't discovered by Fuel, but resided in the lab environment constantly generating error messages: "Could not find kernel images" trying to boot.
After that nodes were powered off the tftp processes on the Fuel master were released.
Conclusion: expected behavior - cluster shouldn't be affected by some nodes which aren't included into the cluster. Undiscovered nodes shouldn't create pefromance impact on the Fuel master node.
| description: | updated |
| description: | updated |
| description: | updated |
| Changed in mos: | |
| milestone: | none → 9.2 |
| Michael Semenov (msemenov) wrote | #1 |
| no longer affects: | mos |
| tags: | added: area-library |
| no longer affects: | fuel/newton |
| Changed in fuel: | |
| importance: | High → Medium |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to fuel-library (master) | #2 |
| Changed in fuel: | |
| assignee: | Fuel Sustaining (fuel-sustaining-team) → Richard Berwald (rberwald) |
| status: | Confirmed → In Progress |
| Changed in fuel: | |
| assignee: | Richard Berwald (rberwald) → Alexey Shtokolov (ashtokolov) |
| OpenStack Infra (hudson-openstack) wrote Change abandoned on fuel-library (master) | #3 |
Very hard to reproduce such status of nodes, but it's definitely needs attention that not properly deployed 2 nodes can slow down 200-nodes cluster. It seems that on 500, 1000, 2000 nodes it can be reproduced much easier, and potential consequences might be much worse. So, let's try to fix it in 9.2.