MAAS

2.1 DHCP reported as enabled but not running

Bug #1629142 reported by John George
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Blake Rouse
MAAS 2.1.0

Bug Description

After installing and configuring 2.1.0~beta1+bzr5411-0ubuntu1~16.04.1, DHCP is reported by the GUI as enabled but it's really not.
Please see attached logs.

$ systemctl -a | grep maas
  maas-dhcpd.service loaded inactive dead MAAS instance of ISC DHCP server for IPv4
  maas-dhcpd6.service loaded inactive dead MAAS instance of ISC DHCP server for IPv6
  maas-proxy.service loaded active running MAAS Proxy
  maas-rackd.service loaded active running MAAS Rack Controller
  maas-regiond-worker@1.service loaded active running MAAS Region Controller (Worker 1)
  maas-regiond-worker@2.service loaded active running MAAS Region Controller (Worker 2)
  maas-regiond-worker@3.service loaded active running MAAS Region Controller (Worker 3)
  maas-regiond-worker@4.service loaded active running MAAS Region Controller (Worker 4)
  maas-regiond.service loaded active exited MAAS Region Controller
  system-maas\x2dregiond\x2dworker.slice loaded active active system-maas\x2dregiond\x2dworker.slice

Related branches

lp://staging/~blake-rouse/maas/fix-1629142
Andres Rodriguez (community): Approve
Diff: 33 lines (+12/-0)
2 files modified
src/maasserver/dhcp.py (+1/-0)
src/maasserver/tests/test_dhcp.py (+11/-0)
Revision history for this message
John George (jog) wrote :
Revision history for this message
Andres Rodriguez (andreserl) wrote :

John,

Can you provide an step by step to reproduce? The only thing I see is that the rack failed to import images, but that may be causing your rack to not work.

Also, can you confirm:

1. DHCP was enabled for the VLAN
2. The UI shows DHCP is active
3. Have you waited a few to see if the rack start dhcp ? there's some situations in which the rack will wait a few seconds to start DHCP.

Changed in maas:
status: New → Incomplete
milestone: none → 2.1.0
Revision history for this message
John George (jog) wrote :

>1. DHCP was enabled for the VLAN
Here is evidence from the our Jenkins job, that DHCP was enabled:
2016-09-29-21:23:54 INFO 'maas maas vlan update fabric-0 untagged dhcp_on=true primary_rack=gckmmf'
2016-09-29-21:23:56 INFO {
    "mtu": 1500,
    "id": 5001,
    "dhcp_on": true,
    "vid": 0,
    "name": "untagged",
    "secondary_rack": null,
    "external_dhcp": null,
    "fabric": "fabric-0",
    "primary_rack": "gckmmf",
    "resource_uri": "/MAAS/api/2.0/vlans/5001/"
}
2016-09-29-21:23:56 INFO
2016-09-29-21:23:56 DEBUG {
    "name": "untagged",
    "vid": 0,
    "dhcp_on": true,
    "mtu": 1500,
    "primary_rack": "gckmmf",
    "external_dhcp": null,
    "resource_uri": "/MAAS/api/2.0/vlans/5001/",
    "id": 5001,
    "secondary_rack": null,
    "fabric": "fabric-0"
}
>2. The UI shows DHCP is active
Yes, in the UI DHCP was listed as enabled. I also tried disabling and then re-enabling DHCP.
>3. Have you waited a few to see if the rack start dhcp ? there's some situations in which the rack will wait a few seconds to start DHCP.
Yes, the server had been running for many minutes, while we investigated. I also manually tried restarting maas-rackd.service and then maas-dhcpd.service (e.g. sudo systemctl restart maas-rackd)

Revision history for this message
Andres Rodriguez (andreserl) wrote : Re: [Bug 1629142] [NEW] 2.1 DHCP reported as enabled but not running

John,

Can you please attach the systemd journal for Maas-dhcp? Also confirm you
are using 2.1a4?

On Thursday, September 29, 2016, John George <email address hidden>
wrote:

> >1. DHCP was enabled for the VLAN
> Here is evidence from the our Jenkins job, that DHCP was enabled:
> 2016-09-29-21:23:54 INFO 'maas maas vlan update fabric-0 untagged
> dhcp_on=true primary_rack=gckmmf'
> 2016-09-29-21:23:56 INFO {
> "mtu": 1500,
> "id": 5001,
> "dhcp_on": true,
> "vid": 0,
> "name": "untagged",
> "secondary_rack": null,
> "external_dhcp": null,
> "fabric": "fabric-0",
> "primary_rack": "gckmmf",
> "resource_uri": "/MAAS/api/2.0/vlans/5001/"
> }
> 2016-09-29-21:23:56 INFO
> 2016-09-29-21:23:56 DEBUG {
> "name": "untagged",
> "vid": 0,
> "dhcp_on": true,
> "mtu": 1500,
> "primary_rack": "gckmmf",
> "external_dhcp": null,
> "resource_uri": "/MAAS/api/2.0/vlans/5001/",
> "id": 5001,
> "secondary_rack": null,
> "fabric": "fabric-0"
> }
> >2. The UI shows DHCP is active
> Yes, in the UI DHCP was listed as enabled. I also tried disabling and then
> re-enabling DHCP.
> >3. Have you waited a few to see if the rack start dhcp ? there's some
> situations in which the rack will wait a few seconds to start DHCP.
> Yes, the server had been running for many minutes, while we investigated.
> I also manually tried restarting maas-rackd.service and then
> maas-dhcpd.service (e.g. sudo systemctl restart maas-rackd)
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1629142
>
> Title:
> 2.1 DHCP reported as enabled but not running
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1629142/+subscriptions
>

--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

Revision history for this message
Blake Rouse (blake-rouse) wrote :
Download full text (3.4 KiB)

2016-09-29 21:20:47 [-] Failed configuring DHCP on rack controller 'id:1'.
 Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 434, in errback
     self._startRunCallbacks(fail)
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
     self._runCallbacks()
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult
     _inlineCallbacks(r, g, deferred)
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
     result = result.throwExceptionIntoGenerator(g)
   File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
     return g.throw(self.type, self.value, self.tb)
   File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 640, in configure_dhcp
     config = yield deferToDatabase(get_dhcp_configuration, rack_controller)
   File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
     result = inContext.theWork()
   File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
     return func(*args,**kw)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 834, in callInContext
     return func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 211, in wrapper
     return func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 603, in call_within_transaction
     return func_outside_txn(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 422, in retrier
     return func(*args, **kwargs)
   File "/usr/lib/python3.5/contextlib.py", line 30, in inner
     return func(*args, **kwds)
   File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 556, in get_dhcp_configuration
     ntp_servers = get_ntp_server_addresses_for_rack(rack_controller)
   File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 266, in get_ntp_server_addresses_for_rack
     rack_addresses, get_space_id_and_family)
   File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 265, in <dictcomp>
     for space_id_and_family, group in groupby(
   File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 264, in <genexpr>
     (ip for _, _, ip in group), key=IPAddress)
   File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 260, in get_space_id_and_family
     return space_id, IPNetwork(cidr).version
   File "/usr/lib/python3/dist-packages/netaddr/ip/__init__.py", line 923, in __init__
   ...

Read more...

Revision history for this message
Blake Rouse (blake-rouse) wrote :

Looks like the cidr for the subnet is None, which is not possible, so something really strange is going on here.

Blake Rouse (blake-rouse)
Changed in maas:
status: Incomplete → In Progress
importance: Undecided → Critical
assignee: nobody → Blake Rouse (blake-rouse)
MAAS Lander (maas-lander)
Changed in maas:
status: In Progress → Fix Committed
Andres Rodriguez (andreserl)
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.