Fuel for OpenStack

20 nodes, not all are able to get provisioning info from cobbler

Bug #1355347 reported by Aleksandr Shaposhnikov on 2014-08-11

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Committed	High	Matthew Mosesohn	Fuel for OpenStack 5.1
	5.0.x	Won't Fix	High	Meg McRoberts	Fuel for OpenStack 5.0.2

Bug Description

When a lot of nodes trying to provisioning fuel not able to provide them with preseed info. At least it true for debian installation.

Steps to reproduce:

1. Add 20 nodes to the fuel (3 controllers + 17 computes). All of them also with ceph-osd's.
2. Start provisioning.

Observed behavior:
At least 2 nodes failed to retrieve preseed from fuel(cobbler).

Screenshot from one of the nodes in attachment.

Tags:

Revision history for this message

Aleksandr Shaposhnikov (alashai8) wrote on 2014-08-11:

Screen Shot 2014-08-11 at 11.04.15.png Edit (97.2 KiB, image/png)

Revision history for this message

Aleksandr Shaposhnikov (alashai8) wrote on 2014-08-11:

This bug only affect generation of preseed for ubuntu. CentOS kickstart generation is fine (at least for 20 nodes)

Alexander Ignatov (aignatov) on 2014-08-12

Changed in fuel:
importance:	Undecided → High
milestone:	none → 5.1

Dima Shulyak (dshulyak) on 2014-08-12

Changed in fuel:
assignee:	nobody → Fuel Library Team (fuel-library)

Andrew Woodward (xarses) on 2014-08-12

summary:

- Node not able to get provisioning info from cobbler
+ 20 nodes, not all are able to get provisioning info from cobbler

Vladimir Sharshov (vsharshov) on 2014-08-13

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Vladimir Sharshov (vsharshov)
status:	New → Confirmed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-13: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/113867

Changed in fuel:
assignee:	Vladimir Sharshov (vsharshov) → Matthew Mosesohn (raytrac3r)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-13: Fix proposed to fuel-main (master)

Fix proposed to branch: master
Review: https://review.openstack.org/113920

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-13: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/113924

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-13: Fix proposed to fuel-astute (master)

Fix proposed to branch: master
Review: https://review.openstack.org/113942

Changed in fuel:
assignee:	Matthew Mosesohn (raytrac3r) → Vladimir Kuklin (vkuklin)

OpenStack Infra (hudson-openstack) on 2014-08-13

Changed in fuel:
assignee:	Vladimir Kuklin (vkuklin) → Matthew Mosesohn (raytrac3r)

Nastya Urlapova (aurlapova) on 2014-08-13

Changed in fuel:
importance:	High → Medium
tags:	added: release-notes

Roman Alekseenkov (ralekseenkov) on 2014-08-13

Changed in fuel:
importance:	Medium → High

Revision history for this message

Aleksandr Shaposhnikov (alashai8) wrote on 2014-08-13:

3 controllers + 7 computes deployment and 10 computes adds works fine so this workaround (10 and 10) works.

Revision history for this message

Aleksandr Shaposhnikov (alashai8) wrote on 2014-08-14:

Checked fix with 20 nodes on Ubuntu and couldn't reproduce this bug anymore. Checked 5.0.1rc + fix for ubuntu preseed + fix for rsync.

OpenStack Infra (hudson-openstack) on 2014-08-14

Changed in fuel:
assignee:	Matthew Mosesohn (raytrac3r) → Vladimir Sharshov (vsharshov)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-14: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/113924
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=26b55bc341cb99bf9bfb9e1888693dded8663760
Submitter: Jenkins
Branch: master

commit 26b55bc341cb99bf9bfb9e1888693dded8663760
Author: Matthew Mosesohn <email address hidden>
Date: Wed Aug 13 19:04:14 2014 +0400

Add iops fact and add it to astute

Astute needs to know how many nodes can
be provisioned at a time, based on IOPS.

Change-Id: If2cf35ac8195baf37cda50564b70ba3db836ccea
Partial-Bug: #1355347

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-14: Fix merged to fuel-main (master)

#10

Reviewed: https://review.openstack.org/113920
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=50cde45c09d3c0228fb3d357f412d6024964ea15
Submitter: Jenkins
Branch: master

commit 50cde45c09d3c0228fb3d357f412d6024964ea15
Author: Matthew Mosesohn <email address hidden>
Date: Wed Aug 13 19:01:24 2014 +0400

Add sysstat to Fuel Master and astute

sysstat provides iostat which is useful
for calculating HDD throughput.

Change-Id: I19bbee60b9ff509064843a468f47c0061736b370
Partial-Bug: #1355347

OpenStack Infra (hudson-openstack) on 2014-08-14

Changed in fuel:
assignee:	Vladimir Sharshov (vsharshov) → Matthew Mosesohn (raytrac3r)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-15: Fix merged to fuel-library (master)

#11

Reviewed: https://review.openstack.org/113867
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=4ae17af4bf2cf775987de036b4fd09fb18b44767
Submitter: Jenkins
Branch: master

commit 4ae17af4bf2cf775987de036b4fd09fb18b44767
Author: Matthew Mosesohn <email address hidden>
Date: Wed Aug 13 16:07:07 2014 +0400

Enable retry for Ubuntu installs

    This is a proactive solution to bug
    1355347, which allows retries for failed
    downloads of packages from apt mirror
    during installation.

Change-Id: I848a86ef27f9ad8235024996b7417a30689daa4f
Partial-Bug: #1355347

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-15: Fix merged to fuel-astute (master)

#12

Reviewed: https://review.openstack.org/113942
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=35c4403067c550cfbd556afcdde251d732d7ed4f
Submitter: Jenkins
Branch: master

commit 35c4403067c550cfbd556afcdde251d732d7ed4f
Author: Vladimir Sharshov <email address hidden>
Date: Wed Aug 13 19:46:20 2014 +0400

Add splay sleep before reboot nodes

    Bug:
     Resolves concurrency issue when deploying
     large clusters. Uses formula of nodes /
     IOPS * 180 to determine how large the
     splay size should be.

Refactoring:
- speed up tests

Co-Authored-By: Matthew Mosesohn <email address hidden>

    Depends on:
    - https://review.openstack.org/#/c/113924/
    - https://review.openstack.org/#/c/113920/

Change-Id: Ieefc843ccba0cdb0441679c73d2e39d6729a4fb0
Closes-Bug: #1355347