Deployment on 98 nodes with Ceilo on CentOS failed due to wrong ceph roles layout

Please link a logs

Have similar on Ubuntu:

http://paste.openstack.org/show/142420/

{"build_id": "2014-11-27_23-41-13", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "45", "auth_required": true, "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "51e66db7750e9c856ba128f35cfb6724895bf479", "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-11-27_23-41-13", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "45", "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "51e66db7750e9c856ba128f35cfb6724895bf479", "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "15a387462f7be50c4f87ad986d0c81535025c125"}}}, "fuellib_sha": "15a387462f7be50c4f87ad986d0c81535025c125"}

Guys, it is really confirmed for 5.1.1, as I remember we certified 5.1.1 only on 20 nodes.

We cannot address high bugs in 5.1.1 anyway, due to HCF

@Nastya, but you're right we do not support 100 nodes deployment in 5.1.x

Well, nodes *were* provisioned, though
2014-11-28T16:27:46 info: [421] All nodes are provisioned

will continue investigate the logs in #2

The deployment logs from astute are ok, I can see also glance report about successful image creation, but it came a bit later than astute expected: http://paste.openstack.org/show/142646/

Yes, Bogdan is right. Glance image should upload not more than 60 seconds. Glance image size ~ 16 mb. If this operation took more then 60 seconds - we have cinder performance degradation.

This bug similar with https://bugs.launchpad.net/fuel/+bug/1397254 (the same signs ).

I believe that this error not in Astute and we need discovery cinder performance.

MOS team, please analyse openstack logs - we need to understand why it took too much to upload an image into Glance. This could be related to Glance, Keystone, Glance backend...

The message
INFO glance.registry.api.v1.images [911143ba-bf4b-4847-a852-77385abd308a 97bea994c8cd4e16b8701d1dd776272f 18d3f86359684daa94bc83fc6f8868eb - - -] Successfully created image 4dc69685-c45f-4193-bd50-6c2743c9dd4a
Does not mean that the image was indeed uploaded, it just says that the record has been created in the database (that's what the registry service does). The actual uploading is made in glance-api service, and it seems like its last log message is

2014-11-28T17:18:46.437043+00:00 debug: 2014-11-28 17:18:46.450 26587 DEBUG glance.store.rbd [9ec256a4-29b6-4c14-a4dd-11c64a5d796f 97bea994c8cd4e16b8701d1dd776272f 18d3f86359684daa94bc83fc6f8868eb - - -] creating image 4dc69685-c45f-4193-bd50-6c2743c9dd4a with order 23 and size 13167616 add /usr/lib/python2.7/dist-packages/glance/store/rbd.py:332

I.e. glance calls to ceph and attempts to put data there. And it seems like there is no response. We need somebody to look into ceph logs to find out what's going on.

astute.yaml:
volumes_ceph: true
images_ceph: true

roles:
  role: primary-controller
  name: node-1
  role: controller
  name: node-2
  role: controller
  name: node-3
  role: compute
  name: node-4
  role: compute
  name: node-5
  role: cinder
  name: node-6
  role: cinder
  name: node-7

There are node ceph-osd nodes

There are not ceph-osd nodes

If comment #12 goes to wrong direction, then we have to investigate more. If there were no Ceph, then there was Swift (in HA we use Swift by default for Glance backend)

Mike, according to snapshot glance is configured to use ceph: see glance-api.conf, it has "default_store = rbd"

Also, the logs state that it was indeed attempting to put the image to ceph. If there is no ceph deployed, then something is wrong with the configuration.

According to #13, the configuration looks not valid

We should check how it was possible to deploy an env with ceph for volumes and images w/o ceph-osd roles assigned

Well, all of it was related to the https://bugs.launchpad.net/fuel/+bug/1397367/comments/2 actually. The original bug does not provide the logs

I suggest to resubmit the logs from #2 and #3 to the separate bug about ceph roles layout and close this one as invalid due to no logs