tl;dr: If you `juju add-machine ssh:...` a machine then deploy a unit to it, you won't be able to `juju exec --unit` to it. However `juju exec --machine` works. Long description: I've deployed a LXD VM (essentially `lxc launch images:ubuntu/22.04 microk8s`), added openssh-server to it, injected `~/.local/share/juju/ssh/juju_id_rsa.pub` to the ubuntu@microk8s account and then `juju add-machine ssh:10.184.166.111` it. Juju sees it without any apparent issue: $ juju status Model Controller Cloud/Region Version SLA Timestamp microk8s overlord juju-lxd/default 3.1.0 unsupported 16:59:13-05:00 App Version Status Scale Charm Channel Rev Exposed Message microk8s active 1 microk8s stable 27 no Unit Workload Agent Machine Public address Ports Message microk8s/0* active idle 1 10.184.166.111 80,443,16443/tcp Machine State Address Inst id Base AZ Message 1 started 10.184.166.111 manual:10.184.166.111 ubuntu@22.04 Manually provisioned machine However, `juju exec --unit` keeps timing out: $ juju exec --debug --unit microk8s/0 -- hostname 17:01:21 INFO juju.cmd supercommand.go:56 running juju [3.1.0 924c9e190eb56d9a9324d9cc0dd9dd663c501ac4 gc go1.18.10] 17:01:21 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/22136/bin/juju", "exec", "--debug", "--unit", "microk8s/0", "--", "hostname"} 17:01:21 INFO juju.juju api.go:86 connecting to API addresses: [10.184.166.216:17070] 17:01:21 DEBUG juju.api apiclient.go:1167 successfully dialed "wss://10.184.166.216:17070/model/4968e5e9-fc42-4c24-86e0-b7c4f8b78717/api" 17:01:21 INFO juju.api apiclient.go:702 connection established to "wss://10.184.166.216:17070/model/4968e5e9-fc42-4c24-86e0-b7c4f8b78717/api" 17:01:21 INFO cmd common.go:375 Running operation 55 with 1 task 17:01:21 INFO cmd common.go:375 - task 56 on unit-microk8s-0 17:01:21 INFO cmd common.go:375 17:01:21 INFO cmd common.go:375 Waiting for task 56... 17:06:21 DEBUG juju.api monitor.go:35 RPC connection died ERROR timed out waiting for results from: unit microk8s/0 17:06:21 DEBUG cmd supercommand.go:548 error stack: github.com/juju/juju/cmd/juju/action.(*runCommandBase).handleTimeout:355: timed out waiting for results from: unit microk8s/0 github.com/juju/juju/cmd/juju/action.(*runCommandBase).processOperationResults:227: But `juju exec --machine` works OK: $ juju exec --wait 10s --debug --machine 1 -- hostname 17:11:21 INFO juju.cmd supercommand.go:56 running juju [3.1.0 924c9e190eb56d9a9324d9cc0dd9dd663c501ac4 gc go1.18.10] 17:11:21 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/22136/bin/juju", "exec", "--wait", "10s", "--debug", "--machine", "1", "--", "hostname"} 17:11:21 INFO juju.juju api.go:86 connecting to API addresses: [10.184.166.216:17070] 17:11:21 DEBUG juju.api apiclient.go:1167 successfully dialed "wss://10.184.166.216:17070/model/4968e5e9-fc42-4c24-86e0-b7c4f8b78717/api" 17:11:21 INFO juju.api apiclient.go:702 connection established to "wss://10.184.166.216:17070/model/4968e5e9-fc42-4c24-86e0-b7c4f8b78717/api" 17:11:21 INFO cmd common.go:375 Running operation 57 with 1 task 17:11:21 INFO cmd common.go:375 - task 58 on machine-1 17:11:21 INFO cmd common.go:375 17:11:21 INFO cmd common.go:375 Waiting for task 58... microk8s 17:11:23 DEBUG juju.api monitor.go:35 RPC connection died 17:11:23 INFO cmd supercommand.go:555 command finished There isn't much interesing in the controller logs: $ juju debug-log -m controller --level error --replay machine-0: 21:58:35 ERROR juju.mongo could not set the value of "/proc/sys/net/core/netdev_max_backlog" to "1000" because of: "/proc/sys/net/core/netdev_max_backlog" does not exist, will not set "1000" machine-0: 21:58:35 ERROR juju.mongo could not set the value of "/sys/kernel/mm/transparent_hugepage/enabled" to "never" because of: open /sys/kernel/mm/transparent_hugepage/enabled: permission denied machine-0: 21:58:35 ERROR juju.mongo could not set the value of "/sys/kernel/mm/transparent_hugepage/defrag" to "never" because of: open /sys/kernel/mm/transparent_hugepage/defrag: permission denied machine-0: 21:58:35 ERROR juju.mongo could not set the value of "/proc/sys/net/core/somaxconn" to "16384" because of: "/proc/sys/net/core/somaxconn" does not exist, will not set "16384" machine-0: 21:58:35 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [963919] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused machine-0: 21:58:40 ERROR juju.worker.dependency "lxd-container-provisioner" manifold worker returned unexpected error: container types not yet available machine-0: 21:58:40 ERROR juju.worker.dependency "broker-tracker" manifold worker returned unexpected error: no container types determined machine-0: 21:58:40 ERROR juju.worker.logger connection is shut down machine-0: 21:58:40 ERROR juju.worker.dependency "kvm-container-provisioner" manifold worker returned unexpected error: container types not yet available machine-0: 21:58:40 ERROR juju.worker.dependency "broker-tracker" manifold worker returned unexpected error: no container types determined machine-0: 21:58:40 ERROR juju.worker.dependency "lxd-container-provisioner" manifold worker returned unexpected error: container types not yet available unit-controller-0: 21:58:41 ERROR juju.worker.meterstatus error running "meter-status-changed": charm missing from disk machine-0: 22:10:31 ERROR juju.apiserver debug-log handler error: sending failed: write tcp 10.184.166.216:17070->10.184.166.1:47090: write: broken pipe machine-0: 22:10:32 ERROR juju.rpc error writing response: *tls.permanentError write tcp 10.184.166.216:17070->10.184.166.199:59234: write: connection reset by peer machine-0: 22:10:32 ERROR juju.rpc error writing response: *tls.permanentError write tcp 10.184.166.216:17070->10.184.166.199:59234: write: connection reset by peer machine-0: 22:10:32 ERROR juju.rpc error writing response: *tls.permanentError write tcp 10.184.166.216:17070->10.184.166.199:59234: write: connection reset by peer machine-0: 22:10:32 ERROR juju.rpc error writing response: *tls.permanentError write tcp 10.184.166.216:17070->10.184.166.199:59234: write: connection reset by peer machine-0: 22:10:32 ERROR juju.rpc error writing response: *tls.permanentError write tcp 10.184.166.216:17070->10.184.166.199:59234: write: connection reset by peer machine-0: 22:10:32 ERROR juju.rpc error writing response: *tls.permanentError write tcp 10.184.166.216:17070->10.184.166.199:59234: write: connection reset by peer machine-0: 22:10:32 ERROR juju.rpc error writing response: *tls.permanentError write tcp 10.184.166.216:17070->10.184.166.199:59234: write: connection reset by peer machine-0: 22:10:32 ERROR juju.rpc error writing response: *tls.permanentError write tcp 10.184.166.216:17070->10.184.166.199:59234: write: connection reset by peer machine-0: 22:10:32 ERROR juju.rpc error writing response: *tls.permanentError write tcp 10.184.166.216:17070->10.184.166.199:59234: write: connection reset by peer machine-0: 15:21:46 ERROR juju.mongo could not set the value of "/sys/kernel/mm/transparent_hugepage/defrag" to "never" because of: open /sys/kernel/mm/transparent_hugepage/defrag: permission denied machine-0: 15:21:46 ERROR juju.mongo could not set the value of "/proc/sys/net/core/somaxconn" to "16384" because of: "/proc/sys/net/core/somaxconn" does not exist, will not set "16384" machine-0: 15:21:46 ERROR juju.mongo could not set the value of "/proc/sys/net/core/netdev_max_backlog" to "1000" because of: "/proc/sys/net/core/netdev_max_backlog" does not exist, will not set "1000" machine-0: 15:21:46 ERROR juju.mongo could not set the value of "/sys/kernel/mm/transparent_hugepage/enabled" to "never" because of: open /sys/kernel/mm/transparent_hugepage/enabled: permission denied machine-0: 15:21:46 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [963919] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused machine-0: 15:21:50 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [963919] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused machine-0: 15:26:02 ERROR juju.mongo could not set the value of "/sys/kernel/mm/transparent_hugepage/enabled" to "never" because of: open /sys/kernel/mm/transparent_hugepage/enabled: permission denied machine-0: 15:26:02 ERROR juju.mongo could not set the value of "/sys/kernel/mm/transparent_hugepage/defrag" to "never" because of: open /sys/kernel/mm/transparent_hugepage/defrag: permission denied machine-0: 15:26:02 ERROR juju.mongo could not set the value of "/proc/sys/net/core/somaxconn" to "16384" because of: "/proc/sys/net/core/somaxconn" does not exist, will not set "16384" machine-0: 15:26:02 ERROR juju.mongo could not set the value of "/proc/sys/net/core/netdev_max_backlog" to "1000" because of: "/proc/sys/net/core/netdev_max_backlog" does not exist, will not set "1000" machine-0: 15:26:02 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [963919] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused machine-0: 15:26:06 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [963919] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused machine-0: 15:26:10 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [963919] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused machine-0: 15:52:14 ERROR juju.worker.dependency "environ-tracker" manifold worker returned unexpected error: model "microk8s" (9426dd00-5211-4881-8dcc-70e4addb6ca8): reading model config: model "9426dd00-5211-4881-8dcc-70e4addb6ca8": settings not found (not found) machine-0: 15:52:14 ERROR juju.worker.dependency "undertaker" manifold worker returned unexpected error: model "9426dd00-5211-4881-8dcc-70e4addb6ca8" not found (not found) machine-0: 09:59:49 ERROR juju.mongo could not set the value of "/proc/sys/net/core/somaxconn" to "16384" because of: "/proc/sys/net/core/somaxconn" does not exist, will not set "16384" machine-0: 09:59:49 ERROR juju.mongo could not set the value of "/proc/sys/net/core/netdev_max_backlog" to "1000" because of: "/proc/sys/net/core/netdev_max_backlog" does not exist, will not set "1000" machine-0: 09:59:49 ERROR juju.mongo could not set the value of "/sys/kernel/mm/transparent_hugepage/enabled" to "never" because of: open /sys/kernel/mm/transparent_hugepage/enabled: permission denied machine-0: 09:59:49 ERROR juju.mongo could not set the value of "/sys/kernel/mm/transparent_hugepage/defrag" to "never" because of: open /sys/kernel/mm/transparent_hugepage/defrag: permission denied machine-0: 09:59:50 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [963919] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused machine-0: 09:59:53 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [963919] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused machine-0: 09:59:58 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [963919] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused machine-0: 16:15:49 ERROR juju.apiserver debug-log handler error: sending failed: write tcp 10.184.166.216:17070->10.184.166.1:53632: write: broken pipe machine-0: 16:17:07 ERROR juju.apiserver debug-log handler error: sending failed: write tcp 10.184.166.216:17070->10.184.166.1:58112: write: broken pipe machine-0: 16:17:07 ERROR juju.apiserver debug-log handler error: sending failed: write tcp 10.184.166.216:17070->10.184.166.1:49726: write: broken pipe machine-0: 16:28:12 ERROR juju.apiserver debug-log handler error: sending failed: write tcp 10.184.166.216:17070->10.184.166.1:55592: write: broken pipe Those "broken pipe" seemed related but they don't correlate with failed `juju exec` as I've run 10s of failed/timed out exec. This has be discussed and initially triaged by @hmlanigan (thanks!) here https://chat.charmhub.io/charmhub/channels/juju/wyzbo4ezjtdiuriypeyxjf8mah Additional information: $ snap list juju Name Version Rev Tracking Publisher Notes juju 3.1.0 22136 3.1/stable canonical✓ -