OS hosts not resolving after env update

Bug #1366875 reported by Igor Yozhikov
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Critical
Matthew Mosesohn

Bug Description

Strange behavior was investigated during bug-fix process of the https://bugs.launchpad.net/mos/+bug/1364026 .
Short follow-up:
release 2014.1 (5.0) -> Upload and unpack tarball with fuel-5.1 release (build #499) -> update cluster to release 2014.1.1-5.0.2

Result:
...
[root@nailgun ~]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online
---|----------|---------------------|---------|---------------|-------------------|------------|---------------|-------
2 | discover | slave-01_controller | 1 | 10.108.20.134 | 64:df:89:aa:ad:7a | controller | | True
1 | ready | slave-04_compute | 1 | 10.108.20.3 | 72:fe:42:d2:cc:4c | compute | | True
4 | ready | slave-05_compute | 1 | 10.108.20.6 | 42:e4:79:46:bd:47 | compute | | True
3 | ready | slave-03_controller | 1 | 10.108.20.5 | 6a:58:24:b4:35:41 | controller | | True
5 | ready | slave-02_controller | 1 | 10.108.20.7 | 16:4a:1f:28:a2:41 | controller | | True
[root@nailgun ~]# ping node-1
ping: unknown host node-1
...
Also, I have rebooted server instance with id=2 and it boots into bootstrap instead of normal boot from HDD. You can see the result above in the fuel node output table.

[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "499"
  build_id: "2014-09-05_00-01-17"
  astute_sha: "b622d9b36dbdd1e03b282b9ee5b7435ba649e711"
  fuellib_sha: "6ebadd157c5153e9bb3ed53fa73507a5c73572c3"
  ostf_sha: "f7b5d9d0d1cfaba5f1fe1e2c634493e92bce11db"
  nailgun_sha: "b36dccbc28584532ca1bdb679b096cc3f5f6f80a"
  fuelmain_sha: "b3afef75675ae3ab251a4f990f3432968eb746eb"

Revision history for this message
Igor Yozhikov (iyozhikov) wrote :
description: updated
description: updated
Changed in fuel:
status: New → Confirmed
summary: - OS hosts does not resolving after env update
+ OS hosts not resolving after env update
Changed in fuel:
importance: Undecided → Critical
assignee: nobody → Fuel Library Team (fuel-library)
milestone: none → 5.1
Revision history for this message
Artem Panchenko (apanchenko-8) wrote :

Reproduced the same issue using upgrade archive # 503. Here is cobbler's systems configuration after upgrade (looks good):

http://paste.openstack.org/show/108763/

and dnsmasq config (incomplete):

http://paste.openstack.org/show/108764/

Also I discovered that /var/lib/cobbler/cobbler_hosts file is empty. After I restarted cobbler container configuration file was generated correctly:

http://paste.openstack.org/show/108765/

resolving worked fine and deployed nodes weren't bootstrapped after restart.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/120047

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Matthew Mosesohn (raytrac3r)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-web (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/120066

Evgeniy L (rustyrobot)
tags: added: upgrade
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/120047
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=e3f947cc4142210499a282bc5f183c333552fa23
Submitter: Jenkins
Branch: master

commit e3f947cc4142210499a282bc5f183c333552fa23
Author: Matthew Mosesohn <email address hidden>
Date: Tue Sep 9 12:55:03 2014 +0400

    Trigger cobbler_sync after profile creation

    Set cobbler_sync to always run every puppet
    run just in case. Nailgun cobbler sync should
    trigger after profile creation.

    Closes-Bug: #1366875

    Change-Id: I9107e42374e0013e63a40856af7224e3811b6063

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/120066
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=82091e0d61f252619a0842d0f8debb6b602a61fe
Submitter: Jenkins
Branch: master

commit 82091e0d61f252619a0842d0f8debb6b602a61fe
Author: Evgeniy L <email address hidden>
Date: Tue Sep 9 15:31:41 2014 +0400

    Upgrades, copy cobbler files right after container is created

    Supervisor start is too slow, puppet run (and cobbler sync)
    passes before we copy the data for cobbler.

    The solution is to copy all required files right
    after container is created.

    Change-Id: I6b4b0ab879d31e43a70e5c9e72c83b1de47ddfb2
    Related-bug: #1366875

tags: added: release-notes
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.