dnsmasq is down after too often restarts by cobbler

Bug #1669828 reported by Roman Podoliaka
This bug report is a duplicate of:  Bug #1667006: Provisioning as a graph is unstable. Edit Remove
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
New
Undecided
Unassigned

Bug Description

master-pkg-mos-systest-ubuntu-xenial (e.g. https://packaging-ci.infra.mirantis.net/view/MOS%20master/job/master-pkg-mos-systest-ubuntu-xenial/356/console) is currently failing with the following error:

AssertionError: Task 'deploy' has incorrect status. error != ready, 'All nodes are finished. Failed tasks: Task[setup_repositories/1], Task[setup_repositories/3], Task[setup_repositories/2], Task[setup_repositories/5], Task[setup_repositories/4], Task[setup_repositories/6] Stopping the deployment process!'

This is due to the fact that dnsmasq on the master node is down:

[root@nailgun ~]# systemctl status dnsmasq.service
● dnsmasq.service - DNS caching server.
   Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/dnsmasq.service.d
           └─restart.conf
   Active: failed (Result: start-limit) since Fri 2017-03-03 13:40:22 UTC; 40min ago
  Process: 4348 ExecStart=/usr/sbin/dnsmasq -k (code=exited, status=0/SUCCESS)
 Main PID: 4348 (code=exited, status=0/SUCCESS)

Mar 03 13:40:25 nailgun.test.domain.local systemd[1]: dnsmasq.service failed.
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: start request repeated too quickly for dnsmasq.service
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: Failed to start DNS caching server..
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: dnsmasq.service failed.
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: start request repeated too quickly for dnsmasq.service
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: Failed to start DNS caching server..
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: dnsmasq.service failed.
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: start request repeated too quickly for dnsmasq.service
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: Failed to start DNS caching server..
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: dnsmasq.service failed.

systemd gives up restarting the dnsmasq after subsequent terminations:

[root@nailgun ~]# grep SIGTERM /var/log/dnsmasq.log
...
Mar 3 13:39:17 dnsmasq[3517]: exiting on receipt of SIGTERM
Mar 3 13:39:17 dnsmasq[3525]: exiting on receipt of SIGTERM
Mar 3 13:39:17 dnsmasq[3545]: exiting on receipt of SIGTERM
Mar 3 13:39:19 dnsmasq[3565]: exiting on receipt of SIGTERM
Mar 3 13:39:20 dnsmasq[3655]: exiting on receipt of SIGTERM
Mar 3 13:40:21 dnsmasq[4001]: exiting on receipt of SIGTERM
Mar 3 13:40:21 dnsmasq[4182]: exiting on receipt of SIGTERM
Mar 3 13:40:21 dnsmasq[4240]: exiting on receipt of SIGTERM
Mar 3 13:40:22 dnsmasq[4281]: exiting on receipt of SIGTERM
Mar 3 13:40:22 dnsmasq[4307]: exiting on receipt of SIGTERM
Mar 3 13:40:22 dnsmasq[4348]: exiting on receipt of SIGTERM

It's not clear why cobbler restarts dnsmasq so often:

[root@nailgun ~]# grep "running: service dnsmasq restart" /var/log/cobbler/cobbler.log
Fri Mar 3 13:39:16 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:16 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:16 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:16 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:17 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:17 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:19 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:20 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:20 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:20 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:20 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:20 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:24 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:26 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:38 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:40:21 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:40:21 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:40:21 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:40:22 2017 - INFO | running: service dnsmasq restart

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.