[fan-network] Race-condition between "apt update" and dhcp request causes cloud-init error

Bug #1944080 reported by Pedro Guimarães
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
High
Joseph Phillips
OpenStack RabbitMQ Server Charm
Invalid
Undecided
Unassigned
ubuntu-fan (Ubuntu)
New
Undecided
Unassigned

Bug Description

Hi,

Using manual provider on top of VMWare.

Juju: 2.9.14
VM: Focal
Containers: Bionic

I've noticed that, in a given Focal host, 2x Bionic containers may have different behavior: one successfully completes cloud-init whereas the other fails. The failed container errors at "apt update" in cloud-init with:

Err:1 http://archive.ubuntu.com/ubuntu bionic InRelease
  Something wicked happened resolving 'PROXY_IP:<PORT>' (-9 - Address family for hostname not supported)
(Likely this is the line where it breaks: https://github.com/Debian/apt/blob/766b24b7f7484751950c76bc66d3d6cdeaf949a5/methods/connect.cc#L436)

Failed container, full cloud-init-output.log: https://pastebin.canonical.com/p/hKNw49mvJP/
Successful container, full cloud-init-output.log: https://pastebin.canonical.com/p/P6MZdQPrWd/
Host syslog: https://pastebin.canonical.com/p/jjMmWRFd9y/

I can see that, on the host, dnsmasq served the FAN IP to the broken container after the "apt-update" was called.

----- FAILED CONTAINER -----:
Sep 19 13:02:14 lma-stack-2 dnsmasq-dhcp[1243]: DHCPOFFER(fan-252) 252.252.15.198 00:16:3e:da:dd:c4
apt-update on cloud-init was ran:
Cloud-init v. 21.2-3-g899bfaa9-0ubuntu2~18.04.1 running 'modules:config' at Sun, 19 Sep 2021 13:02:13 +0000. Up 3.90 seconds.

----- SUCCESSFUL CONTAINER -----:
Sep 19 13:09:40 lma-stack-2 dnsmasq-dhcp[1243]: DHCPOFFER(fan-252) 252.252.15.253 00:16:3e:d7:6b:ef
apt-update on cloud-init was ran:
Cloud-init v. 21.2-3-g899bfaa9-0ubuntu2~18.04.1 running 'modules:config' at Sun, 19 Sep 2021 13:10:10 +0000. Up 35.99 seconds.

In the case of the Failed Container, the DHCPOFFER arrived 1s after the "apt update" was requested, whereas the Successful Container logged a DHCPOFFER 30s before the "apt update".

Revision history for this message
Pedro Guimarães (pguimaraes) wrote :

Also added charm-rabbitmq-server, where this error is more clear. Rabbitmq-server fails on: https://github.com/openstack/charm-rabbitmq-server/blob/master/hooks/install#L9 because it runs a "dpkg" command before a successful execution of "apt update" on the container. Install eventually fails with:

# ./hooks/install
dpkg-query: package 'python3-pip' is not installed and no information is available
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package python3-pip

John A Meinel (jameinel)
Changed in juju:
assignee: nobody → Joseph Phillips (manadart)
importance: Undecided → High
milestone: none → 2.9.16
status: New → In Progress
Changed in juju:
milestone: 2.9.16 → 2.9.17
Changed in juju:
milestone: 2.9.17 → 2.9.18
Changed in juju:
milestone: 2.9.18 → 2.9.19
Changed in juju:
milestone: 2.9.19 → 2.9.20
Changed in juju:
milestone: 2.9.20 → 2.9.21
Changed in juju:
milestone: 2.9.21 → 2.9.22
Changed in juju:
milestone: 2.9.22 → 2.9.23
Revision history for this message
Liam Young (gnuoy) wrote :

Perhaps I'm missing something but this does not seem to be a bug in the rabbitmq-server charm. It may be easier to observe there but the root cause is elsewhere.

Changed in charm-rabbitmq-server:
status: New → Invalid
Changed in juju:
milestone: 2.9.23 → 2.9.24
Changed in juju:
milestone: 2.9.24 → 2.9.25
Changed in juju:
status: In Progress → Triaged
Changed in juju:
milestone: 2.9.25 → 2.9.26
Changed in juju:
milestone: 2.9.26 → 2.9.27
Changed in juju:
milestone: 2.9.27 → 2.9.28
Changed in juju:
milestone: 2.9.28 → 2.9.29
Changed in juju:
milestone: 2.9.29 → 2.9.30
Ian Booth (wallyworld)
Changed in juju:
status: Triaged → In Progress
Revision history for this message
Harry Pidcock (hpidcock) wrote :
Changed in juju:
status: In Progress → Fix Committed
Changed in juju:
milestone: 2.9.30 → 2.9.31
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.