virtual nodes don't always PXE boot on the same NIC

Everything looks fine from a MAAS pov;
 c0:3f:d5:63:ff:41 has got IP 10.14.100.2
 c0:3f:d5:60:4e:f2 has got IP 10.14.100.11

DNS is linking node1.maas to 10.14.100.2

Why do you think it's handing out incorrect addresses? Are you talking about juju not being able to contact the node on the network interface which is in the DNS? (it's always the first/oldest MAC on the node)

On Tue, Sep 09, 2014 at 11:42:21PM -0000, Julian Edwards wrote:
> Everything looks fine from a MAAS pov;
> c0:3f:d5:63:ff:41 has got IP 10.14.100.2
> c0:3f:d5:60:4e:f2 has got IP 10.14.100.11
>
> DNS is linking node1.maas to 10.14.100.2

I think maybe I missed telling a key piece of information:

node1.maas (c0:3f:d5:60:4e:f2) should be 10.14.100.11

but DNS thinks it's 10.14.100.2

node9.maas c0:3f:d5:60:4e:f2 I have no idea about, but it probably
wasn't turned on at the time.

Anyway, in the leases file, *:4e:f2 is 10.14.100.11 and is the only
entry in there, yet the dns mapping was wrong.

This is the bug.

Maas version: 1.7.0~beta2+bzr2916-0ubuntu1~ppa1

--
David Britton <email address hidden>

On Tue, Sep 09, 2014 at 11:42:21PM -0000, Julian Edwards wrote:
> about juju not being able to contact the node on the network interface
> which is in the DNS? (it's always the first/oldest MAC on the node)

Just so it's also clear, these nodes have just one NIC/MAC.

Thanks for looking into this.

--
David Britton <email address hidden>

On Wednesday 10 September 2014 04:42:57 you wrote:
> On Tue, Sep 09, 2014 at 11:42:21PM -0000, Julian Edwards wrote:
> > about juju not being able to contact the node on the network interface
> > which is in the DNS? (it's always the first/oldest MAC on the node)
>
> Just so it's also clear, these nodes have just one NIC/MAC.
>
> Thanks for looking into this.

Ok that makes more sense, thanks for clarifying.

Does the zone file get rewritten correctly if you restart apache? (restarting
it forces a start up task that rewrites the dns config)

Thanks.

I don't have this exact config up any longer, so if you can't get enough information from what I posted, I guess we'll have to drop it.

I don't have enough information to validate this right now so I'm marking it incomplete again; if you have a way to consistently re-create this that would be fantastic. Otherwise if you see it happen again, please grab a core maas team member *right away* so we can try to track this down. Thank you.

This happened again in RC1. Sorry, couldn't grab anyone since it was on an Orange Box that was about to leave the metaphorical door.

node0vm2 booted and MAAS thought it had IP 10.14.100.3 but it actually got a 10.14.66.x address. Note the duplicate entries in the leases file (attached).

We worked around it by truncating the leases file and restarting maas-dhcpd

maas logs: /var/log/maas/**, /var/log/syslog, fubar leases file

15:01 < roaksoax-us-holiday> dpb: weird, only 134
15:01 < roaksoax-us-holiday> dpb: ok, i think i've seen this bug before
15:01 < dpb> not sure how it got there, but from my understanding it was a pretty vanilla orangebox
15:02 < roaksoax-us-holiday> dpb: i think something else crashes on maas-clusterd(pserv.log), and this causes the dhcp stuff to break
15:02 < dpb> roaksoax-us-holiday: only other weird symptom I saw was it wasn't consistent. we allocated/started the machine to check, came up fine on the 100.3 address. Then, the next bootstrap, same split IP problem.
15:03 < dpb> but ya, could have been something as you say, not sure
15:03 < roaksoax-us-holiday> dpb: at a certain point in time, this happened: http://pastebin.ubuntu.com/8946932/
15:03 < roaksoax-us-holiday> dpb: and i'mn thinking that's the cause
15:04 < roaksoax-us-holiday> dpb: either cluster/region connection broke or something, that caused the DHCP stuff to go funky
15:04 < dpb> ahh, interesting
15:04 < roaksoax-us-holiday> dpb: so, restarting apache2, maas-clusterd and maas-dhcpd should fix this
15:04 < roaksoax-us-holiday> dpb: but ensuring that maas region/cluster *can* communicate
15:04 < dpb> well, IIUC, we had to truncate the file first
15:04 < roaksoax-us-holiday> but I think we definitely need to investigate further
15:04 < sparkiegeek> there was a tick in the cluster page

Can you guys try this with the latest RC3 please? An important IP fix went in for IP allocation in RC2.

Next time this happens, it would be great to get the content of the DHCPLease and the StaticIPAddress tables.

A node's IP addresses are: the static addresses allocated to it if it has some (StaticIPAddress: static IP allocated during deployment), the dynamic IP addresses otherwise (DHCPLease: dynamic IP gathered from the leases file).

Here is how to collect that info:

$ sudo maas-region-admin shell

>>> from maasserver.models.staticipaddress import StaticIPAddress
>>> from maasserver.models.dhcplease import DHCPLease
>>> from maasserver.models.nodegroup import NodeGroup

>>> ng = NodeGroup.objects.all()[0]
>>> DHCPLease.objects.get_hostname_ip_mapping(ng)
>>> StaticIPAddress.objects.get_hostname_ip_mapping(ng)

This will help us diagnose the problem: it will allow us to make sure a static address was assigned to the node.

This happened again

>>> DHCPLease.objects.get_hostname_ip_mapping(ng)
{}
>>> StaticIPAddress.objects.get_hostname_ip_mapping(ng)
defaultdict(<type 'list'>, {})

(That last comment is not useful at this time as juju had released the node)

Ok we've got to the bottom of this.

When a node makes a PXE request, we record the NIC that did that as pxe_mac on the Node. This is used as the NIC for which we allocate a static IP.

If it then boots from a different NIC, the previous NIC will get the static IP! The pxe_mac is reset, but by then it's too late.

So, it turns out that you cannot pre-set the NIC that should netboot in virt-manager. The solution suggested is to have:

 1. a global setting that says whether to assign IPs to all NICs (on managed networks) or just the PXE boot NIC
 2. a per-node setting that overrides this

We will arrange for the Landscape guys to carry a custom patch until this fix is available in a package.

So, after some hacking, we came up with a partial fix — but only a
partial one — for this. We can assign both NICs static IPs and write the
hostmaps out just fine. DNS is a bit trickier — it needs a refactoring
since the DNS Zone config uses `hostname` as a dict key, so you can only
have one entry per host at the moment — but not a blocker.

However, this uncovers several other very problematic failure states:

 - We don't currently (AIUI) bring up all the interfaces on a node on
   boot; only one is auto-configured, so even if we assign IPs to all
   the NICs on a node, only one will come up. However, Juju may try the
   IP address for which the NIC hasn't come up, in which case it will
   fail.
 - Similarly, we don't guarantee that the networks of all the NICs will
   be routable from Juju's point of view (this doesn't fit the immediate
   case, since both NICs are on the same network here). The upshot is
   that having DNS entries for the node that point at IPs on different
   subnets could lead to Juju using the wrong address, and so failing.

We discussed moving the allocation of static IP and the writing of the
DHCP host map until after the node has finished deploying (i.e. once it
turns of netboot). However, this doesn't solve the problem, because we
can't guarantee that the node will use the correct interface when it
reboots, and we're back in the same state as we were before.

Realistically, the only correct way for us to fix this is to get the
udev rules right to force the correct NIC to be the primary NIC *every*
time the node boots. That's not quick work, however.

@raharper, what, if anything, could we do from the libvirt/qemu side of things, to predictably a pxe device?

I spent quite a bit of time to see if there was something to do. The current state of pxe control in QEMU isn't ideal. You can disable loading of the option rom which prevents any nic of that *type* from pxe booting. However, there is no control over a per-nic basis.

The other effect of loading a pxe rom, is that *all* nics of a type (virtio, e1000, rtl8139, etc) will be tried. They will be tried in the order they appear on the PCI bus. The order is static.

From what I understand, the unpredictable nature doesn't stem from a QEMU/libvirt side directly, as I said, QEMU will pxe attempt exactly the same every single time. Rather the relatively short time the netboot runs is the cause of the variability as seen by MAAS PXE/DHCP. The first nic issues a bootp/dhcp broadcast request and after a few seconds with no response, it will try the next nic. In the above scenario then it's likely that the first nic times out before it can get a response from MAAS and QEMU switches to the next nic.
The performance/timeout of the bridge reacting to a new device (when KVM launches it adds the guest nic to the software bridge, it has a timeout before packets get forwarded to the devices on the bridge) could be tuned to reduce/delay this, but the problem could still show up on a heavily loaded MAAS when PXE/DHCP is slow to respond.

I've not looked at adjusting the PXE rom timeouts; there are no command line tunables, so it would likely involve generating custom PXE roms (or updating the default, and large default timeouts punish everyone).

For Orange box specifically, I suggest making the following changes to the KVM guest VM to handle this issue:

http://paste.ubuntu.com/9349582/

The changes included switch the first nic to use virtio (instead of rtl8139, unrelated but a better, faster choice), and then switch the second nic to be e1000, and then include the <rom file=''> directive which disables pxe booting on all e1000 nics. The VM now will pxe boot only from the virtio nic. If the bridge is slow or maas is busy, then the VM may not successfully pxe boot. This may or may not be more desirable from an Orange Box perspective.

On Tue, Dec 16, 2014 at 04:50:18PM -0000, Ryan Harper wrote:
> I spent quite a bit of time to see if there was something to do. The
> current state of pxe control in QEMU isn't ideal. You can disable
> loading of the option rom which prevents any nic of that *type* from pxe
> booting. However, there is no control over a per-nic basis.

@rharper -- didn't you also say something about a bridge timeout?

>
> The changes included switch the first nic to use virtio (instead of
> rtl8139, unrelated but a better, faster choice), and then switch the
> second nic to be e1000, and then include the <rom file=''> directive
> which disables pxe booting on all e1000 nics. The VM now will pxe boot
> only from the virtio nic. If the bridge is slow or maas is busy, then
> the VM may not successfully pxe boot. This may or may not be more
> desirable from an Orange Box perspective.
>

It should put us into a 'fast-fail' sitation, and remove a
false-positive (i.e., the node looks fine in MAAS, but is not reachable
via name or IP listed in maas).

If we are using juju -- the problem gets detected as a timeout
eventually (also not ideal). After switching to this, it would get
detected earlier as a 409, and a 'failed deployment' in the MAAS GUI.

I think all-in-all it's a good change for the orange box as it will
allow us to more precisely detect when the problem occurrs.

Thanks for this nice write-up, btw.

--
David Britton <email address hidden>

On Tue, Dec 16, 2014 at 11:08 AM, David Britton <<email address hidden>
> wrote:

> On Tue, Dec 16, 2014 at 04:50:18PM -0000, Ryan Harper wrote:
> > I spent quite a bit of time to see if there was something to do. The
> > current state of pxe control in QEMU isn't ideal. You can disable
> > loading of the option rom which prevents any nic of that *type* from pxe
> > booting. However, there is no control over a per-nic basis.
>
> @rharper -- didn't you also say something about a bridge timeout?
>

Yes, I've not done the benchmarking to determine if this resolves the nic
timeout with MaaS,
but here's some info on bridge forward_delay, defaults and how to modify.

The virbr0 bridge on systems with libvirt installed set the forward_delay
value to 2.0 (seconds).

% brctl showstp virbr0
virbr0
 bridge id 8000.000000000000
 designated root 8000.000000000000
 root port 0 path cost 0
 max age 20.00 bridge max age 20.00
 hello time 2.00 bridge hello time 2.00
 forward delay 2.00 bridge forward delay 2.00
 ageing time 300.00
 hello timer 0.73 tcn timer 0.00
 topology change timer 0.00 gc timer 243.26
 flags

However, the default for bridges are much higher:

% sudo brctl addbr testbr
(foudres) ~ % sudo brctl showstp testbr
testbr
 bridge id 8000.000000000000
 designated root 8000.000000000000
 root port 0 path cost 0
 max age 20.00 bridge max age 20.00
 hello time 2.00 bridge hello time 2.00
 forward delay 15.00 bridge forward delay 15.00
 ageing time 300.00
 hello timer 0.00 tcn timer 0.00
 topology change timer 0.00 gc timer 0.00
 flags

You can lower this value to 2 or 0. For reference, fowarding delay[1]
is the time spent in each of the Listening and Learning states before
the Forwarding state is entered. This delay is so that when a new bridge
comes onto a busy network it looks at some traffic before participating.

% sudo brctl setfd testbr 0
(foudres) ~ % sudo brctl showstp testbr
testbr
 bridge id 8000.000000000000
 designated root 8000.000000000000
 root port 0 path cost 0
 max age 20.00 bridge max age 20.00
 hello time 2.00 bridge hello time 2.00
 forward delay 0.00 bridge forward delay 0.00
 ageing time 300.00
 hello timer 0.00 tcn timer 0.00
 topology change timer 0.00 gc timer 0.00
 flags

If the bridge the KVM VM is on is also on public networks, it's possible
that lowering this value could cause issues[2].

1.
http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge#Forwarding_delay
2.
http://www.microhowto.info/troubleshooting/troubleshooting_ethernet_bridging_on_linux.html#idp212224

> >
> > The changes included switch the first nic to use virtio (instead of
> > rtl8139, unrelated but a better, faster choice), and then switch the
> > second nic to be e1000, and then include the <rom file=''> directive
> > which disables pxe booting on all e1000 nics. The VM now will pxe boot
> > only from the virtio nic. If the bridge is slow or maas is busy, then
> > the VM may not successfully pxe boot. This may or may not be more
> > desirable from an Orange Box perspective.
> >
>
> It should put us into a 'fast-fail' sitation, and remove a
> false-positive (i...

This is something that is really out of control of MAAS. The paste that rharper provided will work for virtual machines that have multiple NICs.

Noting 1.7.2 for release-note purposes.

For reference, this is the libvirt page which covers bridge timeouts affecting PXE booting: http://wiki.libvirt.org/page/PXE_boot_%28or_dhcp%29_on_guest_failed#Long_Forward_Delay_Time_on_Bridge