Wrong listen-address for dnsmasq service after Updating to 9.1

Bug #1616087 reported by Alexander Kurenyshev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Kyrylo Galanov
Mitaka
Fix Released
High
Kyrylo Galanov
Newton
Fix Committed
High
Kyrylo Galanov

Bug Description

There are two ways how to update Fuel from 9.0 to 9.1:
1) Install 9.0 (fuel-devops framework with test group prepare_release could be used)
2) Create a new yum repo file with appropriate repository (for ex. [1])
3) Run yum update
4) Run `fuel-bootstrap build --verbose --debug --activate --label 'new-bootstrap'` on the master ndoe to rebuild bootstrap.

Result: on the each step I see right ip address at the /etc/dnsmasq.d/default.conf, it is the master node ip address, for me it is 10.109.0.2

The second way:
1) Export an environment variables UPDATE_MASTER, UPDATE_FUEL_MIRROR, EXTRA_DEB_REPOS, MIRROR_UBUNTU (you can see my deploy.sh script in attachments)
2) Deploy an environment (group prepare_release is used again)

In this way we also run yum update and a bootstrap rebuild under tests.

Result: as I see at the dnsmasq.conf listen-address is wrong:

[root@nailgun ~]# cat /etc/dnsmasq.d/default.conf

listen-address=172.17.0.1
dhcp-range=default,10.109.0.3,10.109.0.254,255.255.255.0,120m
dhcp-option=net:default,option:router,10.109.0.1
dhcp-boot=net:default,pxelinux.0,boothost,10.109.0.2
dhcp-match=set:ipxe,175
dhcp-option-force=tag:ipxe,210,http://10.109.0.2/cobbler/boot/

This ip address belongs to docker0 (?!) interface. After some manipulations this interface has disappeared from the system.

And as a result:

[root@nailgun ~]# systemctl status dnsmasq.service
● dnsmasq.service - DNS caching server.
   Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/dnsmasq.service.d
           └─restart.conf
   Active: activating (auto-restart) (Result: exit-code) since Tue 2016-08-23 13:32:52 UTC; 1s ago
  Process: 3498 ExecStart=/usr/sbin/dnsmasq -k (code=exited, status=2)
 Main PID: 3498 (code=exited, status=2)

Aug 23 13:32:52 nailgun.test.domain.local systemd[1]: Child 3498 belongs to dnsmasq.service
Aug 23 13:32:52 nailgun.test.domain.local systemd[1]: dnsmasq.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 23 13:32:52 nailgun.test.domain.local systemd[1]: dnsmasq.service changed running -> failed
Aug 23 13:32:52 nailgun.test.domain.local systemd[1]: Unit dnsmasq.service entered failed state.
Aug 23 13:32:52 nailgun.test.domain.local systemd[1]: dnsmasq.service failed.
Aug 23 13:32:52 nailgun.test.domain.local systemd[1]: dnsmasq.service changed failed -> auto-restart
Aug 23 13:32:52 nailgun.test.domain.local systemd[1]: dnsmasq.service: cgroup is empty
Aug 23 13:32:52 nailgun.test.domain.local dnsmasq[3498]: dnsmasq: failed to create listening socket for 172.17.0.1: Cannot assign requested address

[root@nailgun ~]# netstat -nap| grep masq
[root@nailgun ~]#

If I tried to boot some node, it couldn't obtain an ip address and boot from pxe.
So, I think that we need check that config for a valid ip address and re-configure it if it needed.
Or to bind to the interface.

[1] http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-latest/x86_64/

Revision history for this message
Alexander Kurenyshev (akurenyshev) wrote :
Changed in fuel:
status: New → Confirmed
assignee: Fuel Sustaining (fuel-sustaining-team) → Maksim Malchuk (mmalchuk)
tags: added: area-library
tags: added: team-bugfix
Revision history for this message
Alex Schultz (alex-schultz) wrote :

If you don't have any ip addresses at time of setup it'll only default to the docker ip as it's the only interface that'll have an address. The bug seems related to https://review.openstack.org/#/c/349721/ since that is the code that adds the listen addresses to the configuration. It seems that you did not have any ip addresses configured at the time of the cobbler setup except the docker interface. Anytime you change the network interfaces and their ip addresses on the server you'll need to rerun the fuel master setup (or at least the cobbler task).

Changed in fuel:
assignee: Maksim Malchuk (mmalchuk) → Kyrylo Galanov (kgalanov)
Revision history for this message
Maksim Malchuk (mmalchuk) wrote :

the related (not duplicate) issue: https://bugs.launchpad.net/fuel/+bug/1615250

Changed in fuel:
milestone: 10.0 → 9.1
Revision history for this message
Kyrylo Galanov (kgalanov) wrote :
Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/362063
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=86f44e72be9d8dab987f0f3e07bcd188bab3b49b
Submitter: Jenkins
Branch: stable/mitaka

commit 86f44e72be9d8dab987f0f3e07bcd188bab3b49b
Author: Kyrylo Galanov <email address hidden>
Date: Mon Aug 29 14:12:58 2016 +0200

    Bind dnsmasq to specific interface

    Bind dnsmasq to interface specified in astute.yaml instead of IP
    provided by facter. Facter may provide IP of docker interface.

    Change-Id: Ib1b2c1b23909e1a90061eb0aad0434cab50c3043
    Closes-bug: #1616087

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/364058

Changed in fuel:
status: In Progress → Fix Committed
tags: added: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/364058
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=5f64e7f36eb7807862568d8f442c9a17a79ae08a
Submitter: Jenkins
Branch: master

commit 5f64e7f36eb7807862568d8f442c9a17a79ae08a
Author: Kyrylo Galanov <email address hidden>
Date: Mon Aug 29 14:12:58 2016 +0200

    Bind dnsmasq to specific interface

    Bind dnsmasq to interface specified in astute.yaml instead of IP
    provided by facter. Facter may provide IP of docker interface.

    Change-Id: Ib1b2c1b23909e1a90061eb0aad0434cab50c3043
    Closes-bug: #1616087
    (cherry picked from commit 86f44e72be9d8dab987f0f3e07bcd188bab3b49b)

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

Verified on 9.1 snapshot #226.

tags: removed: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 10.0.0rc1

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 10.0.0

This issue was fixed in the openstack/fuel-library 10.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.