old entries are left as down

Bug #1492748 reported by Nobuto Murata
32
This bug affects 7 people
Affects Status Importance Assigned to Milestone
OpenStack Cinder Charm
Triaged
Medium
Unassigned
OpenStack Cinder-Ceph charm
Triaged
Medium
Unassigned
cinder (Juju Charms Collection)
Invalid
Medium
Unassigned
cinder-ceph (Juju Charms Collection)
Invalid
Medium
Unassigned

Bug Description

When deployed with cinder, cinder-ceph and ceph charm, cinder service-list shows "down" entries. Cinder volume service itself is working with no problem, but remained entries break external service status monitoring.

$ cinder service-list
+------------------+-----------------------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+-----------------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | juju-machine-14-lxc-1 | nova | enabled | up | 2015-09-06T11:39:45.000000 | None |
| cinder-scheduler | juju-machine-15-lxc-3 | nova | enabled | up | 2015-09-06T11:39:40.000000 | None |
| cinder-scheduler | juju-machine-16-lxc-2 | nova | enabled | up | 2015-09-06T11:39:35.000000 | None |
| cinder-volume | juju-machine-14-lxc-1@cinder-ceph | nova | enabled | up | 2015-09-06T11:39:40.000000 | None |
| cinder-volume | juju-machine-15-lxc-3 | nova | enabled | down | 2015-09-06T07:12:15.000000 | None |
| cinder-volume | juju-machine-15-lxc-3@cinder-ceph | nova | enabled | up | 2015-09-06T11:39:40.000000 | None |
| cinder-volume | juju-machine-16-lxc-2 | nova | enabled | down | 2015-09-06T07:12:46.000000 | None |
| cinder-volume | juju-machine-16-lxc-2@cinder-ceph | nova | enabled | up | 2015-09-06T11:39:36.000000 | None |
+------------------+-----------------------------------+------+---------+-------+----------------------------+-----------------+

Nobuto Murata (nobuto)
tags: added: cpec
Revision history for this message
Billy Olsen (billy-olsen) wrote :

Looks to be that the cinder-ceph charm sets the hostname to ceph in the cinder.conf file.

Changed in cinder-ceph (Juju Charms Collection):
status: New → Triaged
importance: Undecided → Medium
tags: added: openstack sts
Revision history for this message
Edward Hope-Morley (hopem) wrote :

The cinder-ceph related host name issue has been resolved by bug 1493931. See that LP for more but in short, affected users will need to tidy db entries for existing volumes.

James Page (james-page)
Changed in cinder (Juju Charms Collection):
status: New → Triaged
milestone: none → 16.04
Changed in cinder-ceph (Juju Charms Collection):
milestone: none → 16.04
Changed in cinder (Juju Charms Collection):
importance: Undecided → Low
Changed in cinder-ceph (Juju Charms Collection):
importance: Medium → Low
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

I am seeing this behavior with code that includes the fix for bug 1493931, so
this is not yet fixed.

Changed in cinder (Juju Charms Collection):
importance: Low → Medium
Changed in cinder-ceph (Juju Charms Collection):
importance: Low → Medium
Changed in cinder (Juju Charms Collection):
assignee: nobody → Jorge Niedbalski (niedbalski)
Changed in cinder-ceph (Juju Charms Collection):
assignee: nobody → Jorge Niedbalski (niedbalski)
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

OK, I am seeing this exact behavior on trusty-kilo.

The problem seems to be that because of the fix made for LP 1493931, the hostname
set on cinder.conf for the cinder Service is the service_name(), which on this case is cinder or cinder-ceph.

You can see this definition here: https://github.com/openstack/cinder/blob/c1870433ae72cbfacbf0909921ca18401f84653d/cinder/service.py#L243

My understanding is that this is required for HA deployments using a stateless
backend, in order to keep performing operations on the volumes, regardless of the
service location.

This is my current setup, the service is never updated since deployed,
because the report_state method is using the newly created service instance (which points to the host cinder).

mysql> select updated_at, host, report_count, topic from services;
+---------------------+-------------------------------+--------------+------------------+
| updated_at | host | report_count | topic |
+---------------------+-------------------------------+--------------+------------------+
| 2016-02-04 16:43:50 | juju-niedbalski-sec-machine-6 | 5 | cinder-scheduler |
| 2016-02-04 16:43:54 | juju-niedbalski-sec-machine-5 | 3 | cinder-scheduler |
| 2016-02-04 16:43:30 | juju-niedbalski-sec-machine-6 | 9 | cinder-volume |
| 2016-02-04 16:43:13 | juju-niedbalski-sec-machine-7 | 7 | cinder-volume |
| 2016-02-04 16:43:13 | juju-niedbalski-sec-machine-7 | 1 | cinder-scheduler |
| 2016-02-04 16:43:34 | juju-niedbalski-sec-machine-5 | 7 | cinder-volume |
| 2016-02-04 17:31:11 | cinder@cinder-ceph | 845 | cinder-volume |
| 2016-02-04 17:31:10 | cinder | 834 | cinder-scheduler |
+---------------------+-------------------------------+--------------+------------------+
8 rows in set (0.00 sec)

mysql>

However I am not entirely sure if this is approach is completely correct since after
creating a volume I can't see the reference to the host.

mysql> select host, id, size, provider_location from volumes;
+------+--------------------------------------+------+-------------------+
| host | id | size | provider_location |
+------+--------------------------------------+------+-------------------+
| NULL | fb7870b5-f0cf-4913-ac98-697c65351c15 | 10 | NULL |
+------+--------------------------------------+------+-------------------+
1 row in set (0.00 sec)

So, what's the correct fix for this? Delete the older entries on the DB after deployment?

Any suggestion?

James Page (james-page)
Changed in cinder (Juju Charms Collection):
milestone: 16.04 → 16.07
Changed in cinder-ceph (Juju Charms Collection):
milestone: 16.04 → 16.07
Changed in cinder (Juju Charms Collection):
assignee: Jorge Niedbalski (niedbalski) → nobody
Changed in cinder-ceph (Juju Charms Collection):
assignee: Jorge Niedbalski (niedbalski) → nobody
Liam Young (gnuoy)
Changed in cinder (Juju Charms Collection):
milestone: 16.07 → 16.10
Changed in cinder-ceph (Juju Charms Collection):
milestone: 16.07 → 16.10
James Page (james-page)
Changed in cinder (Juju Charms Collection):
milestone: 16.10 → 17.01
Changed in cinder-ceph (Juju Charms Collection):
milestone: 16.10 → 17.01
James Page (james-page)
Changed in charm-cinder:
importance: Undecided → Medium
status: New → Triaged
Changed in cinder (Juju Charms Collection):
status: Triaged → Invalid
Changed in charm-cinder-ceph:
importance: Undecided → Medium
status: New → Triaged
Changed in cinder-ceph (Juju Charms Collection):
status: Triaged → Invalid
Revision history for this message
Nobuto Murata (nobuto) wrote :
Ante Karamatić (ivoks)
tags: added: cpe-onsite
removed: cpec
Revision history for this message
Dariusz Smigiel (smigiel-dariusz) wrote :
Download full text (3.3 KiB)

It seems that there is an action for the charm which does exactly what Jorge described in #4
`juju run-action cinder/leader remove-services`

` removed: cinder@cinder-ceph-hdd,cinder@cinder-ceph-hdd,cinder@cinder-ceph-flash,cinder@cinder-ceph-flash,cinder@cinder-ceph-flash,cinder@cinder-ceph-hdd
`

From
```
+------------------+--------------------------+------+---------+-------+----------------------------+---------+-----------------+---------------+
| Binary | Host | Zone | Status | State | Updated_at | Cluster | Disabled Reason | Backend State |
+------------------+--------------------------+------+---------+-------+----------------------------+---------+-----------------+---------------+
| cinder-scheduler | cinder | nova | enabled | up | 2021-08-04T16:00:00.000000 | - | - | |
| cinder-volume | cinder@cinder-ceph-flash | nova | enabled | down | 2021-08-04T14:12:03.000000 | - | - | up |
| cinder-volume | cinder@cinder-ceph-flash | nova | enabled | down | 2021-08-04T14:12:08.000000 | - | - | up |
| cinder-volume | cinder@cinder-ceph-flash | nova | enabled | up | 2021-08-04T15:59:57.000000 | - | - | up |
| cinder-volume | cinder@cinder-ceph-hdd | nova | enabled | down | 2021-08-04T14:11:58.000000 | - | - | up |
| cinder-volume | cinder@cinder-ceph-hdd | nova | enabled | down | 2021-08-04T14:12:04.000000 | - | - | up |
| cinder-volume | cinder@cinder-ceph-hdd | nova | enabled | up | 2021-08-04T15:59:57.000000 | - | - | up |
+------------------+--------------------------+------+---------+-------+----------------------------+---------+-----------------+---------------+
```

to

```
Binary Host Zone Status State Updated At RPC Version Object Version Cluster
cinder-scheduler cinder nova enabled :-) 2021-08-04 16:54:27 3.12 1.38
cinder-volume cinder@cinder-ceph-hdd nova enabled :-) 2021-08-04 16:54:26 3.16 1.38
cinder-volume cinder@cinder-ceph-hdd nova enabled :-) 2021-08-04 16:54:26 3.16 1.38
cinder-volume cinder@cinder-ceph-hdd nova enabled :-) 2021-08-04 16:54:27 3.16 1.38
cinder-volume cinder@cinder-ceph-flash nova enabled :-) 2021-08-04 16:54:26 3.16 1.38
cinder-volume cinder@cinder-ceph-flash nova enabled :-) 2021-08-04 16:54:26 3.16 1.38
cinder-volume cinder@cinder-ceph-flash ...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.