poor random seed generation resulting in duplicate random MAC generation for virbr0

It was encountered in production that two systems bootstrapped at the same time with identical configurations generated the same random MAC address for virbr0-nic/virbr0

Looking at libvirt, the MAC address is created in src/util/virmacaddr.c:virMacAddrGenerate() using virRandomBits()

src/util/virrandom.c:virRandomOnceInit seeds the random number generator using this formula:
  unsigned int seed = time(NULL) ^ getpid();

This seems to be a popular method after a quick google but it's easy to see how this can be problematic. The time is only in seconds, and during boot of a relatively identical system these numbers are both likely to be relatively similar across multiple systems which is quite likely in cloud-like environments. Secondly, by using bitwise OR only a small difference is created and if the 1st or 2nd MSB of the pid or time are 0 then it would be easy to have colliding values.

Though problematic from basic logic, I also tested this with a small test program trying 67,921 unique combinations of time() and pid() which produced only 5,693 random seeds using PID range 6799-6810 and time() range 1502484340 to 1502489999.

During the actual incident in question, the 4 systems were all booted within 1-2 seconds of each other. We can see from dmesg that the two systems that generated the same MAC did in fact boot during the same second and the other two did not

[kvm01] Aug 11 16:45:47 unassigned kernel: [ 3.643820] rtc_cmos 00:00: setting system clock to 2017-08-11 20:45:42 UTC (1502484342)
[kvm02] Aug 11 16:45:45 unassigned kernel: [ 3.629268] rtc_cmos 00:00: setting system clock to 2017-08-11 20:45:40 UTC (1502484340)
[kvm03] Aug 11 16:45:45 unassigned kernel: [ 3.623670] rtc_cmos 00:00: setting system clock to 2017-08-11 20:45:40 UTC (1502484340)
[kvm04] Aug 11 16:45:48 unassigned kernel: [ 3.631698] rtc_cmos 00:00: setting system clock to 2017-08-11 20:45:43 UTC (1502484343)

We can also see the libvirt pid happened to be identical on the 2 systems
Aug 11 18:11:15 kvm01 libvirtd[13209]: libvirt version: 1.3.1
Aug 11 18:11:15 kvm02 libvirtd[6409]: libvirt version: 1.3.1
Aug 11 18:11:15 kvm03 libvirtd[6409]: libvirt version: 1.3.1
Aug 11 18:11:17 kvm04 libvirtd[6472]: libvirt version: 1.3.1

Leading to the same MAC being generated. The practical fall out of this was that the virbr0-nic MAC was used as a unique identifier by MAAS which caused problems with it thinking the two hosts were the same.

Given deployments of libvirt-bin in bulk at a similar time are not uncommon, and that having similar PIDs at bootup is also not all that unlikely - I think the randomness of the libvirt seed needs to be improved to prevent this happening on another occasion.

I think there are also potentially wider affects. The same random number generator is used to generate among other things MACs for virtual machines, machine UUIDs, qemuDomainSecretAESSetup (if GNUTLS is not in use, which it maybe is not), etc. Brute forcing the initial random seed in an attempt to generate duplicate or predict UUIDs or MACs may be difficult to target in practice but probably not impossible - perhaps if you could force reboot two systems at the same time, etc.

It seems this has been observed at least once before, where many VMs got the same MAC. But the current upstream source is the same as that of xenial:libvirt where the issue was observed.
https://www.redhat.com/archives/libvirt-users/2014-November/msg00012.html

OpenStack generates it's own MACs so likely isn't affected by that side of things but in any case a more thorough check of the various code paths and implications seems prudent.