Bug #1956981 “Single lxd machine stuck on pending, Container st...” : Bugs : Canonical Juju

Bas de Bruijne (basdbruijne) on 2022-01-10

description:

updated

Revision history for this message

Bas de Bruijne (basdbruijne) wrote on 2022-10-03:

#1

This issue seems to be very active on jammy as well

summary:

- Single lxd machine stuck on pending, creating container
+ Single lxd machine stuck on pending, Container started

Revision history for this message

Bas de Bruijne (basdbruijne) wrote on 2022-10-20:

#2

We see this issue on juju 3.0rc2 as well: https://solutions.qa.canonical.com/testruns/testRun/285a9db6-3f91-4c23-926b-a0fc8c7ba389

Revision history for this message

Heather Lanigan (hmlanigan) wrote on 2022-10-24:

#3

@basdbruijne, the solqa results in #2 do not show the same bug as this.

In the case of #2, the container starts, as does the juju agent on the container, however it has errors.

$ grep pending juju-crashdump-openstack-2022-10-20-07.32.21/juju_status.txt
5/lxd/5 pending 10.246.167.51 juju-dd3b16-5-lxd-5 ubuntu:20.04 zone3 Container started

The machine agent shutdown down starting with?
2022-10-20 03:46:29 DEBUG juju.worker.dependency engine.go:616 "unconverted-api-workers" manifold worker stopped: agent should be terminated

Revision history for this message

John A Meinel (jameinel) wrote on 2022-11-09:

#4

Solutions QA is saying that they are also seeing failures to start containers on Jammy, and this is preventing them from doing their Openstack Jammy testing.

Changed in juju:
importance:	Undecided → High
milestone:	none → 2.9.38
status:	New → Triaged

Revision history for this message

Bas de Bruijne (basdbruijne) wrote on 2022-11-11:

#5

Download full text (8.2 KiB)

In testrun https://solutions.qa.canonical.com/testruns/testRun/fd79805c-8f0c-4965-af14-e01017439fe9 I looked around on the life env. Here, 2 machines are in this state:

```
Machine State Address Inst id Series AZ Message
0 started 10.246.167.190 solqa-lab1-server-07 jammy zone1 Deployed
0/lxd/0 started 10.246.167.149 juju-6ba804-0-lxd-0 jammy zone1 Container started
0/lxd/1 pending juju-6ba804-0-lxd-1 jammy zone1 Container started
0/lxd/2 started 10.246.164.253 juju-6ba804-0-lxd-2 jammy zone1 Container started
0/lxd/3 pending juju-6ba804-0-lxd-3 jammy zone1 Container started
0/lxd/4 started 10.246.166.148 juju-6ba804-0-lxd-4 jammy zone1 Container started
0/lxd/5 started 10.246.165.82 juju-6ba804-0-lxd-5 jammy zone1 Container started
0/lxd/6 started 10.246.167.96 juju-6ba804-0-lxd-6 jammy zone1 Container started
0/lxd/7 started 10.246.167.159 juju-6ba804-0-lxd-7 jammy zone1 Container started
0/lxd/8 started 10.246.166.215 juju-6ba804-0-lxd-8 jammy zone1 Container started
0/lxd/9 started 10.246.165.72 juju-6ba804-0-lxd-9 jammy zone1 Container started
0/lxd/10 started 10.246.164.203 juju-6ba804-0-lxd-10 jammy zone1 Container started
```

But logging on to the machines themselves shows no problem:
```
ubuntu@solqa-lab1-server-07:~$ sudo lxc list
To start your first container, try: lxc launch ubuntu:22.04
Or for a virtual machine: lxc launch ubuntu:22.04 --vm

+----------------------+---------+-----------------------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-0 | RUNNING | 10.246.173.8 (eth1) | | CONTAINER | 0 |
| | | 10.246.172.111 (eth1) | | | |
| | | 10.246.169.47 (eth0) | | | |
| | | 10.246.168.111 (eth0) | | | |
| | | 10.246.167.149 (eth2) | | | |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-1 | RUNNING | 10.246.176.28 (eth1) | | CONTAINER | 0 |
| | | 10.246.172.62 (eth0) | | | |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-2 | RUNNING | 10.246.172.251 (eth1) | | CONTAINER | 0 |
| | | 10.246.168.255 (eth0) | | | |
| | | 10.246.164.253 (eth2) | | | |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-3 | RUNNING | 10.246.169.48 (eth0) | | CONTAINER | 0 |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-4 | ...

In testrun https://solutions.qa.canonical.com/testruns/testRun/fd79805c-8f0c-4965-af14-e01017439fe9 I looked around on the life env. Here, 2 machines are in this state:

```
Machine   State    Address         Inst id               Series  AZ     Message
0         started  10.246.167.190  solqa-lab1-server-07  jammy   zone1  Deployed
0/lxd/0   started  10.246.167.149  juju-6ba804-0-lxd-0   jammy   zone1  Container started
0/lxd/1   pending                  juju-6ba804-0-lxd-1   jammy   zone1  Container started
0/lxd/2   started  10.246.164.253  juju-6ba804-0-lxd-2   jammy   zone1  Container started
0/lxd/3   pending                  juju-6ba804-0-lxd-3   jammy   zone1  Container started
0/lxd/4   started  10.246.166.148  juju-6ba804-0-lxd-4   jammy   zone1  Container started
0/lxd/5   started  10.246.165.82   juju-6ba804-0-lxd-5   jammy   zone1  Container started
0/lxd/6   started  10.246.167.96   juju-6ba804-0-lxd-6   jammy   zone1  Container started
0/lxd/7   started  10.246.167.159  juju-6ba804-0-lxd-7   jammy   zone1  Container started
0/lxd/8   started  10.246.166.215  juju-6ba804-0-lxd-8   jammy   zone1  Container started
0/lxd/9   started  10.246.165.72   juju-6ba804-0-lxd-9   jammy   zone1  Container started
0/lxd/10  started  10.246.164.203  juju-6ba804-0-lxd-10  jammy   zone1  Container started
```

But logging on to the machines themselves shows no problem:
```
ubuntu@solqa-lab1-server-07:~$ sudo lxc list
To start your first container, try: lxc launch ubuntu:22.04
Or for a virtual machine: lxc launch ubuntu:22.04 --vm

+----------------------+---------+-----------------------+------+-----------+-----------+
|         NAME         |  STATE  |         IPV4          | IPV6 |   TYPE    | SNAPSHOTS |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-0  | RUNNING | 10.246.173.8 (eth1)   |      | CONTAINER | 0         |
|                      |         | 10.246.172.111 (eth1) |      |           |           |
|                      |         | 10.246.169.47 (eth0)  |      |           |           |
|                      |         | 10.246.168.111 (eth0) |      |           |           |
|                      |         | 10.246.167.149 (eth2) |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-1  | RUNNING | 10.246.176.28 (eth1)  |      | CONTAINER | 0         |
|                      |         | 10.246.172.62 (eth0)  |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-2  | RUNNING | 10.246.172.251 (eth1) |      | CONTAINER | 0         |
|                      |         | 10.246.168.255 (eth0) |      |           |           |
|                      |         | 10.246.164.253 (eth2) |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-3  | RUNNING | 10.246.169.48 (eth0)  |      | CONTAINER | 0         |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-4  | RUNNING | 10.246.173.9 (eth1)   |      | CONTAINER | 0         |
|                      |         | 10.246.172.117 (eth1) |      |           |           |
|                      |         | 10.246.169.49 (eth0)  |      |           |           |
|                      |         | 10.246.168.117 (eth0) |      |           |           |
|                      |         | 10.246.166.148 (eth2) |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-5  | RUNNING | 10.246.169.0 (eth0)   |      | CONTAINER | 0         |
|                      |         | 10.246.165.82 (eth1)  |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-6  | RUNNING | 10.246.172.250 (eth1) |      | CONTAINER | 0         |
|                      |         | 10.246.168.254 (eth0) |      |           |           |
|                      |         | 10.246.167.96 (eth2)  |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-7  | RUNNING | 10.246.172.55 (eth1)  |      | CONTAINER | 0         |
|                      |         | 10.246.172.123 (eth1) |      |           |           |
|                      |         | 10.246.169.176 (eth0) |      |           |           |
|                      |         | 10.246.168.123 (eth0) |      |           |           |
|                      |         | 10.246.167.159 (eth2) |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-8  | RUNNING | 10.246.172.57 (eth1)  |      | CONTAINER | 0         |
|                      |         | 10.246.172.132 (eth1) |      |           |           |
|                      |         | 10.246.169.178 (eth0) |      |           |           |
|                      |         | 10.246.168.132 (eth0) |      |           |           |
|                      |         | 10.246.166.215 (eth2) |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-9  | RUNNING | 10.246.169.46 (eth0)  |      | CONTAINER | 0         |
|                      |         | 10.246.165.72 (eth1)  |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-10 | RUNNING | 10.246.172.252 (eth1) |      | CONTAINER | 0         |
|                      |         | 10.246.172.124 (eth1) |      |           |           |
|                      |         | 10.246.169.1 (eth0)   |      |           |           |
|                      |         | 10.246.168.124 (eth0) |      |           |           |
|                      |         | 10.246.164.203 (eth2) |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
```

The spaces in this deployment are:
```
ubuntu@lab1-silo2-cpe-fd79805c-8f0c-4965-af14-e01017439fe9:/root$ juju spaces
Name                Space ID  Subnets
alpha               0
oam-space           1         10.246.164.0/22
public-space        2         10.246.172.0/22
internal-space      3         10.246.168.0/22
external-space      4         10.246.180.0/22
ceph-replica-space  5         10.246.176.0/22
undefined           6         10.36.87.0/24
```
So both these machines didn't get an address in the oam-space.

I tried adding another unit of the charm on 0/lxd/1:
```
ubuntu@lab1-silo2-cpe-fd79805c-8f0c-4965-af14-e01017439fe9:/root$ juju add-unit ceph-mon --to lxd:0
```

And the new unit that came up did have an address on the oam-space (even though they should be the same):
```
ubuntu@solqa-lab1-server-07:~$ sudo lxc list
+----------------------+---------+-----------------------+------+-----------+-----------+
|         NAME         |  STATE  |         IPV4          | IPV6 |   TYPE    | SNAPSHOTS |
+----------------------+---------+-----------------------+------+-----------+-----------+
...
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-1  | RUNNING | 10.246.176.28 (eth1)  |      | CONTAINER | 0         |
|                      |         | 10.246.172.62 (eth0)  |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
...
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-6ba804-0-lxd-11 | RUNNING | 10.246.176.66 (eth1)  |      | CONTAINER | 0         |
|                      |         | 10.246.173.183 (eth0) |      |           |           |
|                      |         | 10.246.166.53 (eth2)  |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
```

Nothing stands out to me in the logs on why this is happening.

The crashdumps should appear here when the testrun finished:
https://oil-jenkins.canonical.com/artifacts/fd79805c-8f0c-4965-af14-e01017439fe9/index.html

Jeffrey Chang (modern911) on 2022-11-16

tags:

added: cdo-qa

Canonical Juju QA Bot (juju-qa-bot) on 2023-01-06

Changed in juju:
milestone:	2.9.38 → 2.9.39

Revision history for this message

Moises Emilio Benzan Mora (moisesbenzan) wrote on 2023-01-11:

#6

Seen once more on a juju release run for version 2.9.38 here:

https://solutions.qa.canonical.com/v2/testruns/205c62cc-272b-4384-82c9-69a94db534a3/

Artifacts: https://oil-jenkins.canonical.com/artifacts/205c62cc-272b-4384-82c9-69a94db534a3/index.html

John A Meinel (jameinel) on 2023-01-11

description:

updated

Canonical Juju QA Bot (juju-qa-bot) on 2023-01-23

Changed in juju:
milestone:	2.9.39 → 2.9.40

Canonical Juju QA Bot (juju-qa-bot) on 2023-02-17

Changed in juju:
milestone:	2.9.40 → 2.9.41

Revision history for this message

Cristovao Cordeiro (cjdc) wrote on 2023-02-17 (last edit on 2023-02-17):

#7

I confirm this happens too, consistently, when running the tutorial from https://juju.is/docs/sdk/build-and-deploy-minimal-machine-charm

$ juju version
2.9.38-ubuntu-amd64

Canonical Juju QA Bot (juju-qa-bot) on 2023-02-27

Changed in juju:
milestone:	2.9.41 → 2.9.42

Canonical Juju QA Bot (juju-qa-bot) on 2023-03-01

Changed in juju:
milestone:	2.9.42 → 2.9.43

Canonical Juju QA Bot (juju-qa-bot) on 2023-06-02

Changed in juju:
milestone:	2.9.43 → 2.9.44

Canonical Juju QA Bot (juju-qa-bot) on 2023-07-10

Changed in juju:
milestone:	2.9.44 → 2.9.45

Canonical Juju QA Bot (juju-qa-bot) on 2023-09-18

Changed in juju:
milestone:	2.9.45 → 2.9.46

Revision history for this message

Ian Booth (wallyworld) wrote on 2023-11-29:

#8

The next 2.9.46 candidate release will not include a fix for this bug and we don't plan on any more 2.9 releases. As such it is being removed from its 2.9 milestone.

If the bug is still important to you, let us know and we can consider it for inclusion on a 3.x milestone.

Changed in juju:
milestone:	2.9.46 → none

Revision history for this message

Bas de Bruijne (basdbruijne) wrote on 2024-02-06:

#9

Marking as invalid due to inactivity

Changed in juju:
status:	Triaged → Invalid

Revision history for this message

Jeffrey Chang (modern911) wrote on 2024-11-19:

#10

Checked the crashdump, there's no log available for the lxd instance at all, since it blocked during boot up.
We need to find a live environment, and see what block the boot up.

Revision history for this message

Jeffrey Chang (modern911) wrote on 2024-11-27:

#11

Download full text (9.4 KiB)

bumped into https://solutions.qa.canonical.com/testruns/8ac03f3d-fae7-40e9-ae4d-2092566947eb

2 started 10.241.128.93 meowth ubuntu@22.04 zone2 Deployed
2/lxd/0 started 10.241.128.127 juju-16f545-2-lxd-0 ubuntu@22.04 zone2 Container started
2/lxd/1 started 10.241.128.128 juju-16f545-2-lxd-1 ubuntu@22.04 zone2 Container started
2/lxd/2 pending juju-16f545-2-lxd-2 ubuntu@22.04 zone2 Container started
2/lxd/3 started 10.241.128.105 juju-16f545-2-lxd-3 ubuntu@22.04 zone2 Container started

root@meowth:~# lxc list
+------------- | NAME +------------- | juju-16f545-2-lxd-0 | | | | +------------- | juju-16f545-2-lxd-1 | | +------------- | juju-16f545-2-lxd-2 | | +------------- | juju-16f545-2-lxd-3 | | | | | />---------+---------+-----------------------+------+-----------+-----------+
| STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
/>---------+---------+-----------------------+------+-----------+-----------+
| RUNNING | 192.168.108.61 (eth2) | | CONTAINER | 0 |
| | 192.168.108.20 (eth2) | | | |
| | 10.242.108.66 (eth1) | | | |
| | 10.242.108.20 (eth1) | | | |
| | 10.241.128.127 (eth0) | | | |
/>---------+---------+-----------------------+------+-----------+-----------+
| RUNNING | 192.168.108.62 (eth2) | | CONTAINER | 0 |
| | 10.242.108.67 (eth1) | | | |
| | 10.241.128.128 (eth0) | | | |
/>---------+---------+-----------------------+------+-----------+-----------+
| RUNNING | 192.168.113.9 (eth1) | | CONTAINER | 0 |
| | 192.168.109.10 (eth0) | | | |
| | 10.241.128.111 (eth2) | | | |
/>---------+---------+-----------------------+------+-----------+-----------+
| RUNNING | 192.168.109.9 (eth0) | | CONTAINER | 0 |
| | 192.168.108.40 (eth3) | | | |
| | 192.168.108.34 (eth3) | | | |
| | 10.242.108.9 (eth2) | | | |
| | 10.242.108.34 (eth2) | | | |
| | 10.241.128.105 (eth1) | | | |

syslog from juju-16f545-2-lxd-2
Nov 26 19:39:12 juju-16f545-2-lxd-2 systemd[1]: Starting Update APT News...
Nov 26 19:39:12 juju-16f545-2-lxd-2 systemd[1]: Starting Update the local ESM caches...
Nov 26 19:39:14 juju-16f545-2-lxd-2 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 192.168.108.4.
Nov 26 19:39:17 juju-16f545-2-lxd-2 systemd-resolved[298]: Using degraded feature set TCP instead of UDP for DNS server 192.168.108.3.
Nov 26 19:39:27 juju-16f545-2-lxd-2 systemd-resolved[298]: Using d...

bumped into https://solutions.qa.canonical.com/testruns/8ac03f3d-fae7-40e9-ae4d-2092566947eb

2         started  10.241.128.93   meowth                ubuntu@22.04  zone2  Deployed
2/lxd/0   started  10.241.128.127  juju-16f545-2-lxd-0   ubuntu@22.04  zone2  Container started
2/lxd/1   started  10.241.128.128  juju-16f545-2-lxd-1   ubuntu@22.04  zone2  Container started
2/lxd/2   pending                  juju-16f545-2-lxd-2   ubuntu@22.04  zone2  Container started
2/lxd/3   started  10.241.128.105  juju-16f545-2-lxd-3   ubuntu@22.04  zone2  Container started

root@meowth:~# lxc list
+----------------------+---------+-----------------------+------+-----------+-----------+
|         NAME         |  STATE  |         IPV4          | IPV6 |   TYPE    | SNAPSHOTS |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-16f545-2-lxd-0  | RUNNING | 192.168.108.61 (eth2) |      | CONTAINER | 0         |
|                      |         | 192.168.108.20 (eth2) |      |           |           |
|                      |         | 10.242.108.66 (eth1)  |      |           |           |
|                      |         | 10.242.108.20 (eth1)  |      |           |           |
|                      |         | 10.241.128.127 (eth0) |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-16f545-2-lxd-1  | RUNNING | 192.168.108.62 (eth2) |      | CONTAINER | 0         |
|                      |         | 10.242.108.67 (eth1)  |      |           |           |
|                      |         | 10.241.128.128 (eth0) |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-16f545-2-lxd-2  | RUNNING | 192.168.113.9 (eth1)  |      | CONTAINER | 0         |
|                      |         | 192.168.109.10 (eth0) |      |           |           |
|                      |         | 10.241.128.111 (eth2) |      |           |           |
+----------------------+---------+-----------------------+------+-----------+-----------+
| juju-16f545-2-lxd-3  | RUNNING | 192.168.109.9 (eth0)  |      | CONTAINER | 0         |
|                      |         | 192.168.108.40 (eth3) |      |           |           |
|                      |         | 192.168.108.34 (eth3) |      |           |           |
|                      |         | 10.242.108.9 (eth2)   |      |           |           |
|                      |         | 10.242.108.34 (eth2)  |      |           |           |
|                      |         | 10.241.128.105 (eth1) |      |           |           |

syslog from juju-16f545-2-lxd-2
Nov 26 19:39:12 juju-16f545-2-lxd-2 systemd[1]: Starting Update APT News...
Nov 26 19:39:12 juju-16f545-2-lxd-2 systemd[1]: Starting Update the local ESM caches...
Nov 26 19:39:14 juju-16f545-2-lxd-2 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 192.168.108.4.
Nov 26 19:39:17 juju-16f545-2-lxd-2 systemd-resolved[298]: Using degraded feature set TCP instead of UDP for DNS server 192.168.108.3.
Nov 26 19:39:27 juju-16f545-2-lxd-2 systemd-resolved[298]: Using degraded feature set TCP instead of UDP for DNS server 192.168.108.2.
Nov 26 19:39:33 juju-16f545-2-lxd-2 python3[1112]: ["2024-11-26T19:39:33.470", "ERROR", "ubuntupro.http", "_readurl_urllib", 171, "[Errno -3] Temporary failure in name resolution", {"exc_info": "Traceback (most recent call last):\n  File \"/usr/lib/python3.10/urllib/request.py\"
, line 1348, in do_open\n    h.request(req.get_method(), req.selector, req.data, headers,\n  File \"/usr/lib/python3.10/http/client.py\", line 1283, in request\n    self._send_request(method, url, body, headers, encode_chunked)\n  File \"/usr/lib/python3.10/http/client.py\", lin
e 1329, in _send_request\n    self.endheaders(body, encode_chunked=encode_chunked)\n  File \"/usr/lib/python3.10/http/client.py\", line 1278, in endheaders\n    self._send_output(message_body, encode_chunked=encode_chunked)\n  File \"/usr/lib/python3.10/http/client.py\", line 10
38, in _send_output\n    self.send(msg)\n  File \"/usr/lib/python3.10/http/client.py\", line 976, in send\n    self.connect()\n  File \"/usr/lib/python3.10/http/client.py\", line 1448, in connect\n    super().connect()\n  File \"/usr/lib/python3.10/http/client.py\", line 942, in
 connect\n    self.sock = self._create_connection(\n  File \"/usr/lib/python3.10/socket.py\", line 824, in create_connection\n    for res in getaddrinfo(host, port, 0, SOCK_STREAM):\n  File \"/usr/lib/python3.10/socket.py\", line 955, in getaddrinfo\n    for res in _socket.getad
drinfo(host, port, family, type, proto, flags):\nsocket.gaierror: [Errno -3] Temporary failure in name resolution\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/lib/python3/dist-packages/uaclient/http/_
_init__.py\", line 167, in _readurl_urllib\n    resp = request.urlopen(req, timeout=timeout)  # nosec B310\n  File \"/usr/lib/python3.10/urllib/request.py\", line 216, in urlopen\n    return opener.open(url, data, timeout)\n  File \"/usr/lib/python3.10/urllib/request.py\", line 
519, in open\n    response = self._open(req, data)\n  File \"/usr/lib/python3.10/urllib/request.py\", line 536, in _open\n    result = self._call_chain(self.handle_open, protocol, protocol +\n  File \"/usr/lib/python3.10/urllib/request.py\", line 496, in _call_chain\n    result 
= func(*args)\n  File \"/usr/lib/python3.10/urllib/request.py\", line 1391, in https_open\n    return self.do_open(http.client.HTTPSConnection, req,\n  File \"/usr/lib/python3.10/urllib/request.py\", line 1351, in do_open\n    raise URLError(err)\nurllib.error.URLError: <urlopen
 error [Errno -3] Temporary failure in name resolution>"}]
Nov 26 19:39:33 juju-16f545-2-lxd-2 python3[1112]: ["2024-11-26T19:39:33.472", "ERROR", "ubuntupro.lib.esm_cache", "main", 17, "Error updating the cache: Failed to connect to https://contracts.canonical.com/v1/resources?architecture=amd64&kernel=5.15.0-126-generic&series=jammy&v
irt=lxc\n[Errno -3] Temporary failure in name resolution\n", {}]
Nov 26 19:39:33 juju-16f545-2-lxd-2 systemd[1]: esm-cache.service: Deactivated successfully.
Nov 26 19:39:33 juju-16f545-2-lxd-2 systemd[1]: Finished Update the local ESM caches.
Nov 26 19:39:37 juju-16f545-2-lxd-2 systemd-resolved[298]: Using degraded feature set TCP instead of UDP for DNS server 10.241.128.5.
Nov 26 19:39:37 juju-16f545-2-lxd-2 cloud-init[1101]: Ign:1 http://security.ubuntu.com/ubuntu jammy-security InRelease
Nov 26 19:39:37 juju-16f545-2-lxd-2 cloud-init[1101]: Ign:2 http://archive.ubuntu.com/ubuntu jammy InRelease
Nov 26 19:39:37 juju-16f545-2-lxd-2 cloud-init[1101]: Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
Nov 26 19:39:37 juju-16f545-2-lxd-2 cloud-init[1101]: Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB]
Nov 26 19:39:38 juju-16f545-2-lxd-2 cloud-init[1101]: Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]
Nov 26 19:39:39 juju-16f545-2-lxd-2 cloud-init[1101]: Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1954 kB]
Nov 26 19:39:39 juju-16f545-2-lxd-2 cloud-init[1101]: Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [312 kB]
Nov 26 19:39:39 juju-16f545-2-lxd-2 cloud-init[1101]: Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB]
Nov 26 19:39:39 juju-16f545-2-lxd-2 cloud-init[1101]: Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2611 kB]
Nov 26 19:39:39 juju-16f545-2-lxd-2 cloud-init[1101]: Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [453 kB]
Nov 26 19:39:39 juju-16f545-2-lxd-2 cloud-init[1101]: Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [580 B]
Nov 26 19:39:39 juju-16f545-2-lxd-2 cloud-init[1101]: Get:11 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [957 kB]
Nov 26 19:39:39 juju-16f545-2-lxd-2 cloud-init[1101]: Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [203 kB]
Nov 26 19:39:39 juju-16f545-2-lxd-2 cloud-init[1101]: Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.5 kB]
Nov 26 19:39:39 juju-16f545-2-lxd-2 cloud-init[1101]: Get:14 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.6 kB]
Nov 26 19:39:39 juju-16f545-2-lxd-2 cloud-init[1101]: Get:15 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [8260 B]
Nov 26 19:39:39 juju-16f545-2-lxd-2 cloud-init[1101]: Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [224 B]
Nov 26 19:39:46 juju-16f545-2-lxd-2 systemd[1]: apt-news.service: Deactivated successfully.
Nov 26 19:39:46 juju-16f545-2-lxd-2 systemd[1]: Finished Update APT News.
Nov 26 19:41:10 juju-16f545-2-lxd-2 systemd[1]: Starting Download data for packages that failed at package install time...
Nov 26 19:41:10 juju-16f545-2-lxd-2 systemd[1]: update-notifier-download.service: Deactivated successfully.
Nov 26 19:41:10 juju-16f545-2-lxd-2 systemd[1]: Finished Download data for packages that failed at package install time.
Nov 26 19:51:10 juju-16f545-2-lxd-2 systemd[1]: Starting Cleanup of Temporary Directories...
Nov 26 19:51:10 juju-16f545-2-lxd-2 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Nov 26 19:51:10 juju-16f545-2-lxd-2 systemd[1]: Finished Cleanup of Temporary Directories.

Revision history for this message

Jeffrey Chang (modern911) wrote on 2024-11-28 (last edit on 2024-11-29):

#12

from one of the pending lxd unit, I feel network is not smooth here.

Nov 28 23:19:13 juju-c14165-2-lxd-0 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 192.168.110.4.
Nov 28 23:19:13 juju-c14165-2-lxd-0 systemd-journald[59]: Forwarding to syslog missed 444 messages.
Nov 28 23:19:17 juju-c14165-2-lxd-0 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 192.168.110.2.
Nov 28 23:19:20 juju-c14165-2-lxd-0 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 10.241.144.5.
Nov 28 23:19:21 juju-c14165-2-lxd-0 systemd[1]: systemd-timedated.service: Deactivated successfully.
Nov 28 23:19:23 juju-c14165-2-lxd-0 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 10.241.144.2.
Nov 28 23:19:26 juju-c14165-2-lxd-0 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 10.241.144.3.
Nov 28 23:19:29 juju-c14165-2-lxd-0 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 10.241.144.4.
Nov 28 23:19:32 juju-c14165-2-lxd-0 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 10.241.144.6.

root@juju-c14165-2-lxd-0:~# ip route
default via 10.241.144.1 dev eth1 proto static
10.241.144.0/21 dev eth1 proto kernel scope link src 10.241.144.102
10.242.10.0/24 dev eth2 proto kernel scope link src 10.242.10.8
192.168.112.0/24 dev eth0 proto kernel scope link src 192.168.112.7

root@juju-c14165-2-lxd-0:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
19: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:12:49:f0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.112.7/24 brd 192.168.112.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe12:49f0/64 scope link
       valid_lft forever preferred_lft forever
21: eth1@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:b4:e6:15 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.241.144.102/21 brd 10.241.151.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:feb4:e615/64 scope link
       valid_lft forever preferred_lft forever
23: eth2@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:e3:3f:7a brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.242.10.8/24 brd 10.242.10.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fee3:3f7a/64 scope link
       valid_lft forever preferred_lft forever

from one of the pending lxd unit, I feel network is not smooth here.

Nov 28 23:19:13 juju-c14165-2-lxd-0 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 192.168.110.4.
Nov 28 23:19:13 juju-c14165-2-lxd-0 systemd-journald[59]: Forwarding to syslog missed 444 messages.
Nov 28 23:19:17 juju-c14165-2-lxd-0 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 192.168.110.2.
Nov 28 23:19:20 juju-c14165-2-lxd-0 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 10.241.144.5.
Nov 28 23:19:21 juju-c14165-2-lxd-0 systemd[1]: systemd-timedated.service: Deactivated successfully.
Nov 28 23:19:23 juju-c14165-2-lxd-0 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 10.241.144.2.
Nov 28 23:19:26 juju-c14165-2-lxd-0 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 10.241.144.3.
Nov 28 23:19:29 juju-c14165-2-lxd-0 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 10.241.144.4.
Nov 28 23:19:32 juju-c14165-2-lxd-0 systemd-resolved[298]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 10.241.144.6.

root@juju-c14165-2-lxd-0:~# ip route
default via 10.241.144.1 dev eth1 proto static 
10.241.144.0/21 dev eth1 proto kernel scope link src 10.241.144.102 
10.242.10.0/24 dev eth2 proto kernel scope link src 10.242.10.8 
192.168.112.0/24 dev eth0 proto kernel scope link src 192.168.112.7

root@juju-c14165-2-lxd-0:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
19: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:12:49:f0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.112.7/24 brd 192.168.112.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe12:49f0/64 scope link 
       valid_lft forever preferred_lft forever
21: eth1@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:b4:e6:15 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.241.144.102/21 brd 10.241.151.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:feb4:e615/64 scope link 
       valid_lft forever preferred_lft forever
23: eth2@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:e3:3f:7a brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.242.10.8/24 brd 10.242.10.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fee3:3f7a/64 scope link 
       valid_lft forever preferred_lft forever

Revision history for this message

Jeffrey Chang (modern911) wrote on 2024-11-29 (last edit on 2024-11-29):

#13

root@juju-690314-0-lxd-0:/var/log# ps -efw
UID PID PPID C STIME TTY TIME CMD
root 1095 1 0 03:33 ? 00:00:00 /usr/bin/python3 /usr/bin/cloud-init modules --mode=final
root 1097 1095 0 03:33 ? 00:00:00 /bin/sh -c tee -a /var/log/cloud-init-output.log
root 1098 1097 0 03:33 ? 00:00:00 tee -a /var/log/cloud-init-output.log
root 1099 1095 0 03:33 ? 00:00:00 /usr/bin/apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet update
_apt 1110 1099 0 03:33 ? 00:00:00 /usr/lib/apt/methods/http
_apt 1111 1099 0 03:33 ? 00:00:00 /usr/lib/apt/methods/http
_apt 1123 1099 0 03:33 ? 00:00:00 /usr/lib/apt/methods/gpgv
_apt 1278 1099 0 03:33 ? 00:00:01 /usr/lib/apt/methods/store
root 1493 1344 0 04:18 pts/1 00:00:00 ps -efw

confirmed apt-get update got stuck, from /var/log/cloud-init-output.log.

Jeffrey Chang (modern911) on 2025-06-16

tags:

added: severity-high

Canonical Juju

Single lxd machine stuck on pending, Container started

Bug Description

Other bug subscribers

Remote bug watches