Ceilometer

compute agent no longer reconnects if libvirtd is restarted

Bug #1498013 reported by Guangyu Suo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
Medium
Guangyu Suo
Ceilometer 5.0.0 "liberty"

Bug Description

The libvirt inspector may can't inspect data from libvirt on some occasions after libvirtd is restarted, though we have reconnecting logic. It happens by chance. The wierd thing is that there is not any exception log when it occurs. The way to reproduce is to restart libvirtd during compute agent polling data. I am using libvirt version 1.2.11 on CentOS 6.5.

After some debuging and reseaching, I found when this occurs, it will raise libvirt.VIR_ERR_INTERNAL_ERROR instead of libvirt.VIR_ERR_SYSTEM_ERROR in reconnecting logic. This results compute agent no longer reconnects to libvirt, and fails to inspect data.

Also I found there is the same logic in nova/libvirt, and it has fixed this issue two years ago: https://bugs.launchpad.net/nova/+bug/1154473

The libvirt error code can be found here: https://libvirt.org/html/libvirt-virterror.html

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/225699

Changed in ceilometer:
assignee: nobody → Guangyu Suo (yugsuo)
status: New → In Progress
Guangyu Suo (yugsuo)
summary: - compute agent no longer reconnects if libvirt is restarted
+ compute agent no longer reconnects if libvirtd is restarted
gordon chung (chungg)
Changed in ceilometer:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/225699
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=7f290fcdb51eff0e70f5b2092257eb0f40f3ef5d
Submitter: Jenkins
Branch: master

commit 7f290fcdb51eff0e70f5b2092257eb0f40f3ef5d
Author: Guangyu Suo <email address hidden>
Date: Mon Sep 21 14:12:17 2015 +0000

    Fix reconnecting to libvirt

    The libvirt inspector may can't inspect data from libvirt on some occasions after
    libvirtd is restarted, though we have reconnecting logic. When this occurs, it will
    raise libvirt.VIR_ERR_INTERNAL_ERROR instead of libvirt.VIR_ERR_SYSTEM_ERROR in
    reconnecting logic. This results compute agent no longer reconnects to libvirt, and
    fails to inspect data any more. This patch is just updated to deal with INTERNAL_ERROR.

    Change-Id: Ifbd754cb050ec7e364d17fea3b32191da1e22ef5
    Closes-bug: #1498013

Changed in ceilometer:
status: In Progress → Fix Committed
gordon chung (chungg)
Changed in ceilometer:
milestone: none → liberty-rc1
Thierry Carrez (ttx)
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: liberty-rc1 → 5.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.