resource metadata not reflecting actual state of resource

Does nova emit notifications when the state of the VM changes?

Yes. Nova emits notifications when it does any update on a VM. So, creates, deletes, power offs, resizes, rebuilds, etc. all emit notifications.

We get the notifications for creating the VM (create.start and create.end). The question is whether we get notifications for the state of the VM changing while it is still being created.

Yes, but it depends on your Nova configuration:

'To enable notifications when the VM state changes, set configuration flag "notify_on_state_change" to "vm_state".' (https://wiki.openstack.org/wiki/SystemUsageData)

Check out the entry for 'compute.instance.update'.

Ah, OK, so maybe this is just an issue with our documentation not explaining all of the settings the deployer may want to configure.

I'm not sure I follow. Which setting(s) would you suggest deployers have?

The problem seems to be a byproduct of how MySQL decided to handle retrieving non-aggregated values from an aggregated query. SQLite was returning the last value when I used an ORDER BY before the grouping, MySQL was returning the first. It sounds like PostgreSQL would say it's an invalid query, though I haven't tested; that was the first thing some other teammates said immediately when I asked for some help. This still seems like a bug to me. =\

It was failing in Mongo because it was using $first on a query that was just ordering by project_id and user_id and not timestamp, AFAICT.

Based on your comment about the notify_on_state_change , I thought the data wasn't there at all because nova wasn't configured to send it. Does our documentation include instructions for setting notify_on_state_change?

If that's on and the queries still return the wrong values, then we need to fix the code as well (obviously).

Thanks for the clarification, I gotcha now. =]

Anyone using devstack should have this already set for them, if they go with the default (this is what I'm using): https://github.com/openstack-dev/devstack/blob/master/lib/nova#L502-L507. I haven't tried without this configuration, though I probably will just to be thorough.

I wasn't able to find it in the CM documentation, so it'd be good to add a bit about it.

It's in the documentation already:
http://docs.openstack.org/developer/ceilometer/install/manual.html

Aha! Sorry, I was looking in the wrong spot. Thanks, Julien!

HBase is currently overwriting historical metadata, so with that driver we can't get the latest resource state within a time period without addressing the schema issue there: https://bugs.launchpad.net/ceilometer/+bug/1217412

Fix proposed to branch: master
Review: https://review.openstack.org/44277

Fix proposed to branch: master
Review: https://review.openstack.org/45008

Reviewed: https://review.openstack.org/44277
Committed: http://github.com/openstack/ceilometer/commit/64f17d6552cde85e87ba394815fd27ceeebdb103
Submitter: Jenkins
Branch: master

commit 64f17d6552cde85e87ba394815fd27ceeebdb103
Author: Thomas Maddox <email address hidden>
Date: Tue Aug 20 18:10:17 2013 +0000

    Fix to return latest resource metadata

    Addresses the latest resource metadata not being returned in
    the MongoDB, SQLAlchemy, DB2, and HBase drivers. A schema
    change was required for HBase, because it was overwriting
    historical metadata.

    Closes-Bug: #1208547
    Related-Bug: #1201701
    Implements: blueprint hbase-meter-table-enhancement
    Change-Id: Ib09e21cbc7bbd45a6ecc321403e9947df837e14b

Not sure why, but Grizzly backport was done in separate bug 1229395

@Alan: Sorry, yeah it was a misunderstanding, I think. I'm relatively new to this, so I asked someone (I can't remember who) where how I should track the stable bug and they told me to open a new bug against Grizzly. I probably didn't do a very good job of explaining what was up, and I think I tagged this one too late.