Ceilometer client reconnection to Mongo prevents it from being able to rollback

Bug #1785996 reported by Nikita Konovalov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
Medium
Oleksiy Molchanov

Bug Description

Detailed bug description:
 When Ceilometer connects to MongoDB it issues a TTL update command. Rollback for this command is case of Mongo replica recovery is not possible. The following error will appear:
replSet error cannot rollback a collMod command: { collMod: "event", index: { expireAfterSeconds: 172800, keyPattern: { timestamp: -1 } } }

Steps to reproduce:
 1. Set up an environment with Ceilometer and MongoDB
 2. Restart Ceilometer services on one of the controllers.
 3. Disconnect one Mongo node from the network (or just stop Mongo on it) for 10 minutes.
 4. Restore connection to the Mongo nodes.
 5. It will not be able to restore the replica set.
Expected results:
 Mongo cluster restores all replica sets.
Actual result:
 The reconnected Mongo node will have a FATAL status.
Reproducibility:
 Every time
Workaround:
 Make a full sync for a FATAL node manually
Description of the environment:
- Operation system: any
- Versions of components: MOS 9.2
- Network model: any
- Related projects installed: Ceilometer
Additional information:
 Updating TTL every time a connection is established is not actually necessary. In the newer releases of Ceilometer the TTL update mechanics was moved to CLI rather then Ceilometer itself.

Revision history for this message
Alexander Rubtsov (arubtsov) wrote :

sla2 for 9.0-updates

Changed in mos:
importance: Undecided → Medium
assignee: nobody → MOS Maintenance (mos-maintenance)
milestone: none → 9.x-updates
tags: added: sla2
Changed in mos:
milestone: 9.x-updates → 9.2-mu-8
status: New → Confirmed
Changed in mos:
assignee: MOS Maintenance (mos-maintenance) → Oleksiy Molchanov (omolchanov)
Changed in mos:
status: Confirmed → In Progress
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/ceilometer (9.0/mitaka)

Fix proposed to branch: 9.0/mitaka
Change author: Oleksiy Molchanov <email address hidden>
Review: https://review.fuel-infra.org/39292

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/ceilometer (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/39292
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: c0566551dfec664c69e856e4553308e0ac1444f2
Author: Oleksiy Molchanov <email address hidden>
Date: Thu Sep 20 19:11:13 2018

Do not modify mongo collection TTL if it is not changed

Change-Id: I50f43f3f2b277e30bf2e32f01b497d783a541ee0
Closes-Bug: 1785996

Changed in mos:
status: In Progress → Fix Committed
Revision history for this message
Dmitry (dtsapikov) wrote :

Verified on 9.2+mu8

Changed in mos:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.