We're seeing a number of containers stuck in pending then usual since upgrade from 1.20.11 to 1.20.12. This is causing deployment timeouts.
Looking at the logs, it's not clear why the containers are stuck in the pending state since there are no error messages. However, this has been recreated multiple times and saw as many as 5 systems in same state with relatively short period of time.
'1':
agent-state: started
agent-version: 1.20.12
containers:
1/lxc/0:
agent-state: pending
hardware: arch=amd64
instance-id: juju-machine-1-lxc-0
series: trusty
1/lxc/1:
agent-state: pending
hardware: arch=amd64
instance-id: juju-machine-1-lxc-1
series: trusty
dns-name: skookum.oil
$ grep "machine-1-lxc-0" juju_debug_log.txt
machine-0: 2014-11-24 17:45:30 DEBUG juju.state.apiserver apiserver.go:150 <- [61] machine-1 {"RequestId":55,"Type":"Provisioner","Request":"Life","Params":{"Entities":[{"Tag":"machine-1-lxc-0"}]}}
machine-0: 2014-11-24 17:45:30 DEBUG juju.state.apiserver apiserver.go:150 <- [61] machine-1 {"RequestId":57,"Type":"Provisioner","Request":"InstanceId","Params":{"Entities":[{"Tag":"machine-1-lxc-0"}]}}
machine-0: 2014-11-24 17:45:30 DEBUG juju.state.apiserver apiserver.go:150 <- [61] machine-1 {"RequestId":58,"Type":"Provisioner","Request":"Status","Params":{"Entities":[{"Tag":"machine-1-lxc-0"}]}}
machine-0: 2014-11-24 17:45:30 DEBUG juju.state.apiserver apiserver.go:150 <- [61] machine-1 {"RequestId":61,"Type":"Provisioner","Request":"InstanceId","Params":{"Entities":[{"Tag":"machine-1-lxc-0"}]}}
machine-0: 2014-11-24 17:45:30 DEBUG juju.state.apiserver apiserver.go:150 <- [61] machine-1 {"RequestId":63,"Type":"Provisioner","Request":"SetPasswords","Params":{"Changes":[{"Tag":"machine-1-lxc-0","Password":"NG6pZUlqkgbl4G1mrGISwXsR"}]}}
machine-0: 2014-11-24 17:45:31 INFO juju.state.apiserver.common password.go:98 setting password for "machine-1-lxc-0"
machine-0: 2014-11-24 17:45:31 DEBUG juju.state.apiserver apiserver.go:150 <- [61] machine-1 {"RequestId":64,"Type":"Provisioner","Request":"ProvisioningInfo","Params":{"Entities":[{"Tag":"machine-1-lxc-0"}]}}
machine-1: 2014-11-24 17:50:05 INFO juju.provisioner.lxc lxc-broker.go:100 started lxc container for machineId: 1/lxc/0, juju-machine-1-lxc-0, arch=amd64
machine-1: 2014-11-24 17:50:05 INFO juju.provisioner provisioner_task.go:482 started machine 1/lxc/0 as instance juju-machine-1-lxc-0 with hardware "arch=amd64", networks [], interfaces []
machine-0: 2014-11-24 17:50:05 DEBUG juju.state.apiserver apiserver.go:150 <- [61] machine-1 {"RequestId":71,"Type":"Provisioner","Request":"SetInstanceInfo","Params":{"Machines":[{"Tag":"machine-1-lxc-0","InstanceId":"juju-machine-1-lxc-0","Nonce":"machine-1:af1c3a7f-1e79-4d1b-8985-31d24cf2e9e4","Characteristics":{"Arch":"amd64"},"Networks":null,"Interfaces":null}]}}
machine-0: 2014-11-24 17:50:09 DEBUG juju.state.apiserver apiserver.go:157 -> [37] user-admin 19.922566583s {"RequestId":61,"Response":{"Deltas":[["machine","change",{"Id":"1/lxc/0","InstanceId":"juju-machine-1-lxc-0","Status":"pending","StatusInfo":"","StatusData":null,"Life":"alive","Series":"trusty","SupportedContainers":null,"SupportedContainersKnown":false,"HardwareCharacteristics":{"Arch":"amd64"},"Jobs":["JobHostUnits"],"Addresses":[]}]]}} AllWatcher["1"].Next
machine-0: 2014-11-24 18:48:50 DEBUG juju.state.apiserver apiserver.go:157 -> [84] user-admin 36.538273ms {"RequestId":3,"Response":{"EnvironmentName":"maas","Machines":{"0":{"Agent":{"Status":"started","Info":"","Data":{},"Version":"1.20.12","Life":"","Err":null},"AgentState":"started","AgentStateInfo":"","AgentVersion":"1.20.12","Life":"","Err":null,"DNSName":"reading.oil","InstanceId":"/MAAS/api/1.0/nodes/node-f64f188e-ae16-11e3-b194-00163efc5068/","InstanceState":"","Series":"precise","Id":"0","Containers":{},"Hardware":"arch=amd64 cpu-cores=8 mem=32768M tags=hw-ok,oil-slave-4,hardware-hp-proliant-DL320E","Jobs":["JobManageEnviron","JobHostUnits"],"HasVote":true,"WantsVote":true},"1":{"Agent":{"Status":"started","Info":"","Data":{},"Version":"1.20.12","Life":"","Err":null},"AgentState":"started","AgentStateInfo":"","AgentVersion":"1.20.12","Life":"","Err":null,"DNSName":"skookum.oil","InstanceId":"/MAAS/api/1.0/nodes/node-600e169a-ae98-11e3-b194-00163efc5068/","InstanceState":"","Series":"trusty","Id":"1","Containers":{"1/lxc/0":{"Agent":{"Status":"pending","Info":"","Data":{},"Version":"","Life":"","Err":null},"AgentState":"pending","AgentStateInfo":"","AgentVersion":"","Life":"","Err":null,"DNSName":"","InstanceId":"juju-machine-1-lxc-0","InstanceState":"","Series":"trusty","Id":"1/lxc/0","Containers":{},"Hardware":"arch=amd64","Jobs":["JobHostUnits"],"HasVote":false,"WantsVote":false},"1/lxc/1":{"Agent":{"Status":"pending","Info":"","Data":{},"Version":"","Life":"","Err":null},"AgentState":"pending","AgentStateInfo":"","AgentVersion":"","Life":"","Err":null,"DNSName":"","InstanceId":"juju-machine-1-lxc-1","InstanceState":"","Series":"trusty","Id":"1/lxc/1","Containers":{},"Hardware":"arch=amd64","Jobs":["JobHostUnits"],"HasVote":false,"WantsVote":false}},"Hardware":"arch=amd64 cpu-cores=4 mem=32768M tags=hw-ok,oil,hardware-dell-poweredge-R210,transient-error-1","Jobs":["JobHostUnits"],"HasVote":false,"WantsVote":false},"2":{"Agent":{"Status":"started","Inf
Juju log files attached.