Fuel for OpenStack

[library] ERR in puppet.log: RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite

Bug #1328881 reported by Anastasia Palkina on 2014-06-11

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Committed	High	Dmitry Borodaenko	Fuel for OpenStack 5.1
	5.0.x	Invalid	High	Dmitry Borodaenko	Fuel for OpenStack 5.0.2

Bug Description

"build_id": "2014-06-11_00-30-36",
"mirantis": "yes",
"build_number": "250",
"ostf_sha": "67b61ed3788297fa5d985afec32498d8c0f812db",
"nailgun_sha": "f851199db57d9561b51c429956c056b463ad3f0b",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "e59cb325d99fba49085ed5b0b6cfb0d9a83d2fc8",
"astute_sha": "17b1afa5f0dc8f4fca5ed4eb03ec566fbfb5ed19",
"release": "5.1",
"fuellib_sha": "d9b8daa61e131e3c346e93d3870620897a71c593"

1. Create new environment (Ubuntu, HA mode)
2. Choose VLAN segmentation
3. Choose both ceph
4. Choose rados
5. Add 3 controller+ceph, 2 compute+ceph
6. Start deployment. It was successful
7. There are error in puppet.log on controllers (node-1,2,3):

Wed Jun 11 10:52:16 +0000 2014 /Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns (notice): [node-1][ERROR ] Traceback (most recent call last):
Wed Jun 11 10:52:16 +0000 2014 /Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns (notice): [node-1][ERROR ] File "/usr/lib/python2.7/dist-packages/ceph_deploy/util/decorators.py", line 10, in inner
Wed Jun 11 10:52:16 +0000 2014 /Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns (notice): [node-1][ERROR ] def inner(*args, **kwargs):
Wed Jun 11 10:52:16 +0000 2014 /Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns (notice): [node-1][ERROR ] File "/usr/lib/python2.7/dist-packages/ceph_deploy/conf.py", line 12, in write_conf
Wed Jun 11 10:52:16 +0000 2014 /Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns (notice): [node-1][ERROR ] line = self.fp.readline()
Wed Jun 11 10:52:16 +0000 2014 /Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns (notice): [node-1][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite

Tags:

Revision history for this message

Anastasia Palkina (apalkina) wrote on 2014-06-11:

fuel-snapshot-2014-06-11_11-34-05.tgz Edit (20.1 MiB, application/x-tar)

Andrew Woodward (xarses) on 2014-06-13

tags:

added: ceph

Sergii Golovatiuk (sgolovatiuk) on 2014-06-17

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Andrew Woodward (xarses)
importance:	Undecided → High

Revision history for this message

Dmitry Borodaenko (angdraug) wrote on 2014-06-17:

This error message is generated by ceph-deploy and is safely ignored by puppet. It does not affect deployment, and is likely to go away with new versions of ceph-deploy.

Changed in fuel:
importance:	High → Low
status:	New → Confirmed

Dmitry Ilyin (idv1985) on 2014-07-15

summary:

- ERR in puppet.log: RuntimeError: config file /etc/ceph/ceph.conf exists
- with different content; use --overwrite-conf to overwrite
+ [library] ERR in puppet.log: RuntimeError: config file
+ /etc/ceph/ceph.conf exists with different content; use --overwrite-conf
+ to overwrite

Ryan Moe (rmoe) on 2014-07-28

Changed in fuel:
importance:	Low → High
assignee:	Andrew Woodward (xarses) → Dmitry Borodaenko (dborodaenko)

Revision history for this message

Ryan Moe (rmoe) wrote on 2014-07-28:

In ceph-deploy 1.5.9 this error will cause OSD prepare to fail. ceph-deploy 1.2.7 used a context manger to decorate remote methods before calling them. The decorator would catch exceptions from the remote method and raise a new exception with a specific attribute set. The context manager would check for that specific attribute and then log (but not re-raise) the exceptions from the remote methods (see: https://github.com/ceph/ceph-deploy/blob/v1.2.7/ceph_deploy/util/context.py#L58-L63).

Revision history for this message

Dmitry Borodaenko (angdraug) wrote on 2014-07-28:

The problem would only happen if radosgw is enabled (radosgw.pp is the only manifest that modifies ceph.conf outside of primary mon), AND if ceph-osd role is assigned to secondary controllers. Here's what happens:

1) ceph-deploy config pull downloads ceph.conf from primary mon
2) all other ceph-deploy invokations done for secondary controller role are not affected since ceph.conf is not touched until:
3) radosgw.pp (running as part of controller role) modifies ceph.conf to update radosgw\host variable
4) there's no more ceph-deploy invokations done after radosgw.pp for controller, so controller deployment succeeds
5) ceph-osd role starts with ceph-deploy osd prepare which fails due to ceph.conf in /root and /etc/ceph being out of sync after radosgw.pp updates in (3)

The fix is to make sure /root/ceph.conf is updated after /etc/ceph/ceph.conf is modified before any other ceph-deploy commands.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-07-29: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/110168

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-07-30: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/110168
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c2401c2b8f28129b5e063e6c8050d3868ba5c02d
Submitter: Jenkins
Branch: master

commit c2401c2b8f28129b5e063e6c8050d3868ba5c02d
Author: Dmitry Borodaenko <email address hidden>
Date: Mon Jul 28 16:39:25 2014 -0700

create /root/ceph.conf symlink on all ceph nodes

    We need /root/ceph.conf to link to /etc/ceph/ceph.conf on all ceph nodes, not
    just primary monitor. With ceph-deploy 1.3+, if Puppet modifies
    /etc/ceph/ceph.conf and makes it differ from /root/ceph.conf (e.g. change host
    for radosgw), all subsequent ceph-deploy commands will fail.

Change-Id: I6254c35f546cd22837d29e952b8632759d5b40f8
Closes-Bug: #1328881