nova-compute CrashLoopBackOff

Bug #1700774 reported by Miloslav Pansky
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla-kubernetes
New
Undecided
Unassigned

Bug Description

When I deploy nova-compute:

helm install --debug kolla-kubernetes/helm/service/nova-compute --namespace kolla --name nova-compute --values ./cloud.yaml

nova-compute pod get in CrashLoopBackOff state

20m 3m 4 kubelet, k8os01.devops.unidemo.local spec.containers{main} Normal Started (events with common reason combined)
  26m 10s 112 kubelet, k8os01.devops.unidemo.local spec.containers{main} Warning BackOff Back-off restarting failed container
  19m 10s 86 kubelet, k8os01.devops.unidemo.local Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "main" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=main pod=nova-compute-g6pmn_kolla(a4138340-5b45-11e7-bbe5-005056af5e96)"

docker ps -a | grep nova-compute

cfd666c6517a kolla/centos-source-nova-compute@sha256:7b6c58f8f7df7a44b732ddd9586a34fc1ece5c104c88ed3da247b4ba20737c4c "kolla_start" 18 seconds ago Up 17 seconds k8s_main_nova-compute-g6pmn_kolla_a4138340-5b45-11e7-bbe5-005056af5e96_10
a6d30c8f9705 kolla/centos-source-nova-compute@sha256:7b6c58f8f7df7a44b732ddd9586a34fc1ece5c104c88ed3da247b4ba20737c4c "kolla_start" 5 minutes ago Exited (0) 5 minutes ago k8s_main_nova-compute-g6pmn_kolla_a4138340-5b45-11e7-bbe5-005056af5e96_9
48944fa455d6 kolla/centos-source-nova-compute@sha256:7b6c58f8f7df7a44b732ddd9586a34fc1ece5c104c88ed3da247b4ba20737c4c "sh -ce '[ ! -d /var/" 29 minutes ago Exited (0) 29 minutes ago k8s_initialize-nova_nova-compute-g6pmn_kolla_a4138340-5b45-11e7-bbe5-005056af5e96_0
265d8e8fb1ee kolla/centos-source-kolla-toolbox@sha256:fe10cce8325402a8c0a175ec88f11be13c8ff46430f6ef22a30c962c817adcbb "sh -xec 'cp -a /conf" 29 minutes ago Exited (0) 29 minutes ago k8s_initialize-nova-compute_nova-compute-g6pmn_kolla_a4138340-5b45-11e7-bbe5-005056af5e96_0
bc8f9c923bf0 kolla/centos-source-kubernetes-entrypoint@sha256:e01591cfdb5def77ebc0f243b4d3fe7c94c19acac7ed41a73bdbca82fbcdd312 "/bin/sh -c /usr/loca" 31 minutes ago Exited (0) 30 minutes ago k8s_dependencies_nova-compute-g6pmn_kolla_a4138340-5b45-11e7-bbe5-005056af5e96_0
976a6f3cefb6 gcr.io/google_containers/pause-amd64:3.0 "/pause" 31 minutes ago Up 31 minutes

Container exited immediately after start but no error in log :-(

docker logs cfd666c6517a

INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ONCE
INFO:__main__:Copying service configuration files
INFO:__main__:Coping file from /var/lib/kolla/config_files/nova.conf to /etc/nova/nova.conf
INFO:__main__:Setting file /etc/nova/nova.conf owner to nova:nova
INFO:__main__:Setting file /etc/nova/nova.conf permission to 0600
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/log/kolla/nova
INFO:__main__:Setting permission for /var/log/kolla/nova/nova-compute.log
INFO:__main__:Setting permission for /var/lib/nova
INFO:__main__:Setting permission for /var/lib/nova/instances
INFO:__main__:Setting permission for /var/lib/nova/.cinderclient
INFO:__main__:Setting permission for /var/lib/nova/instances/locks
INFO:__main__:Setting permission for /var/lib/nova/instances/compute_nodes
INFO:__main__:Setting permission for /var/lib/nova/instances/locks/nova-storage-registry-lock
INFO:__main__:Setting permission for /var/lib/nova/.cinderclient/d41d8cd98f00b204e9800998ecf8427e
INFO:__main__:Setting permission for /var/lib/nova/.cinderclient/d41d8cd98f00b204e9800998ecf8427e/volume-human-id-cache
INFO:__main__:Setting permission for /var/lib/nova/.cinderclient/d41d8cd98f00b204e9800998ecf8427e/volume-uuid-cache
Running command: 'nova-compute'

Other pods in cluster are in Running state:

kubectl get pods -n kolla

NAME READY STATUS RESTARTS AGE
cinder-api-2967377306-pghrt 2/2 Running 0 44m
cinder-scheduler-0 1/1 Running 0 44m
glance-api-2272216545-mnvc2 1/1 Running 0 46m
glance-registry-1790312474-1bwfx 2/2 Running 0 46m
horizon-3565390796-wvdvd 1/1 Running 0 41m
keystone-594553528-t2d0l 1/1 Running 0 49m
mariadb-0 1/1 Running 0 55m
memcached-2750153393-sw6rn 1/1 Running 0 49m
neutron-dhcp-agent-st633 1/1 Running 0 39m
neutron-l3-agent-network-k5f7p 1/1 Running 0 39m
neutron-metadata-agent-network-vjgzb 1/1 Running 0 39m
neutron-openvswitch-agent-network-cw4h3 1/1 Running 0 39m
neutron-server-2216068945-wm2mt 2/2 Running 0 39m
nova-api-3047009415-chnbc 2/2 Running 0 37m
nova-compute-g6pmn 0/1 CrashLoopBackOff 10 33m
nova-conductor-0 1/1 Running 0 37m
nova-consoleauth-0 1/1 Running 0 37m
nova-libvirt-l5wb8 1/1 Running 0 33m
nova-novncproxy-3451654711-d4lls 2/2 Running 0 37m
nova-scheduler-0 1/1 Running 0 37m
openvswitch-ovsdb-network-fhzsd 1/1 Running 0 40m
openvswitch-vswitchd-network-bjckf 1/1 Running 0 40m
placement-api-3133989353-zd6rn 1/1 Running 0 37m
rabbitmq-0 1/1 Running 0 51m

I followed doc: https://docs.openstack.org/developer/kolla-kubernetes/deployment-guide.html

kolla-ansible branch stable/ocata (4.0.2)
kolla-ansible 0.6.0
docker 1.12.6

Images pulled from docker Hub with tag 4.0.0

kolla/centos-source-glance-api 4.0.0 27ab3cd401f7 3 months ago 914.4 MB
kolla/centos-source-glance-registry 4.0.0 eabee8c7f38b 3 months ago 856.7 MB
kolla/centos-source-nova-compute 4.0.0 b8c6e6becb6c 3 months ago 1.244 GB
kolla/centos-source-keystone 4.0.0 76cb102ddb52 3 months ago 889.4 MB
kolla/centos-source-nova-api 4.0.0 53286ccb1258 3 months ago 1.053 GB
kolla/centos-source-nova-placement-api 4.0.0 f4fe13f29cd5 3 months ago 1.054 GB
kolla/centos-source-nova-novncproxy 4.0.0 a387a47948bc 3 months ago 1.004 GB
kolla/centos-source-nova-scheduler 4.0.0 fb67f05804e0 3 months ago 978.9 MB
kolla/centos-source-nova-consoleauth 4.0.0 fb67f05804e0 3 months ago 978.9 MB
kolla/centos-source-nova-conductor 4.0.0 ccda5cf40636 3 months ago 978.9 MB
kolla/centos-source-cinder-api 4.0.0 7e2e472d937a 3 months ago 1.137 GB
kolla/centos-source-horizon 4.0.0 7870cfaadac1 3 months ago 1.012 GB
kolla/centos-source-cinder-scheduler 4.0.0 205ad64bcd0e 3 months ago 1.062 GB
kolla/centos-source-neutron-server 4.0.0 3abd1f4e5af7 3 months ago 948.5 MB
kolla/centos-source-neutron-metadata-agent 4.0.0 569a74e1ecb9 3 months ago 940.3 MB
kolla/centos-source-neutron-openvswitch-agent 4.0.0 7bad304ef1fa 3 months ago 940.3 MB
kolla/centos-source-neutron-dhcp-agent 4.0.0 7bad304ef1fa 3 months ago 940.3 MB
kolla/centos-source-neutron-l3-agent 4.0.0 7bad304ef1fa 3 months ago 940.3 MB
kolla/centos-source-openvswitch-db-server 4.0.0 b40cb390cc3f 3 months ago 442.2 MB
kolla/centos-source-mariadb 4.0.0 2aa99c21026f 3 months ago 810.5 MB
kolla/centos-source-openvswitch-vswitchd 4.0.0 622248ddf694 3 months ago 442.2 MB
kolla/centos-source-nova-libvirt 4.0.0 cec2f2d1d6c4 3 months ago 999.2 MB
kolla/centos-source-memcached 4.0.0 0cd964e59bc5 3 months ago 418.3 MB
kolla/centos-source-rabbitmq 4.0.0 7f485bb65215 3 months ago 477.4 MB
kolla/centos-source-kubernetes-entrypoint 4.0.0 38e1c49f0bf6 3 months ago 443.8 MB

Thanks for help or advice.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.