Canonical Juju

socket.timeout as part of container.exec inside a pebble-ready hook

Bug #1989004 reported by Tom Haddon
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
High
Unassigned

Bug Description

This pastebin illustrates the problem best: https://paste.ubuntu.com/p/KzS4r2PVV2/

We have an indico-pebble-ready hook that's firing. We run `indico_env_config = self._get_indico_env_config(container)` per this line of code https://github.com/canonical/indico-operator/blob/main/src/charm.py#L221. That then runs `process = container.exec(["indico", "setup", "list-plugins"])` per https://github.com/canonical/indico-operator/blob/main/src/charm.py#L338 but the `process.wait_output()` on the following line is resulting in a socket.timeout per the paste.

Tom Haddon (mthaddon)
tags: added: canonical-is
Revision history for this message
Tom Haddon (mthaddon) wrote :

Also just to mention that paste was from indico-pebble-ready, but here's a similar one just after from indico-celery-pebble-ready https://paste.ubuntu.com/p/Sn8NSC82Kj/. We've seen 5 container restarts on the indico-celery container since the pod was created as part of a charm upgrade, and 5 container restarts on indico-nginx. It's not clear that the `socket.timeout` errors are directly tied to the container restarts given we've also seen them (per the paste above) on containers that haven't restarted, but it does seem like it could be related, and certainly seems worth investigation.

Revision history for this message
John A Meinel (jameinel) wrote :

Also filed: https://github.com/canonical/operator/issues/826
With a lot of digging details about how I think the Operator Framework is running during this.
(I think this is a "pebble is under too much load, and OF should handle socket.timeout better") bug.

Changed in juju:
importance: Undecided → High
status: New → Triaged
tags: added: pebble sidecar-charm
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.