OpenStack Compute (nova)

nova.task_log database records are never deleted

Series ussuri
Bug #1877189

Bug #1877189 reported by melanie witt on 2020-05-06

This bug affects 3 people

	Status	Importance	Assigned to
OpenStack Compute (nova)	Fix Released	Wishlist	melanie witt
Train	New	Undecided	Unassigned
Ussuri	New	Undecided	Unassigned
Victoria	New	Undecided	Unassigned
Wallaby	In Progress	Undecided	Unassigned

Bug Description

nova.task_log database records are related to the server usage audit log API [1] which is dependent on the [DEFAULT]/instance_usage_audit config option being set to True (it defaults to False). Enablement of the config option causes nova-compute to emit notifications which are consumed by OpenStack Telemetry. When the config option is enabled, records will be periodically created in the nova.task_log table. And there is no cleanup of the records in nova.

I started a ML thread about the server usage audit log API awhile back [2] and the conclusion was that deprecating or removing the API isn't practical as there are systems in the wild still using it, so the best action to take for now is to create a nova-manage command for cleaning up old nova.task_log records.

I brought up the proposal in a nova meeting [3] and in the nova channel [4] and the consensus was to treat it as a Wishlist bug so that it can be backported upstream.

So this is a bug for the work to add a new 'nova-manage db purge_task_log' command that offers a --before <datetime> option.

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-09-06.log.html#t2019-09-06T14:19:50
[2] http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009245.html
[3] http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-10-17-14.00.log.html#l-306
[4] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-10-17.log.html#t2019-10-17T14:54:39

Tags:

melanie witt (melwitt) on 2020-05-06

Changed in nova:
importance:	Undecided → Wishlist
status:	New → Confirmed

Revision history for this message

melanie witt (melwitt) wrote on 2021-03-12:

https://review.opendev.org/c/openstack/nova/+/780395 has been proposed

Changed in nova:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-18: Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/780395
Committed: https://opendev.org/openstack/nova/commit/bc71331bcaa92718fce06ac3d2ecda6a4c9d2d20
Submitter: "Zuul (22348)"
Branch: master

commit bc71331bcaa92718fce06ac3d2ecda6a4c9d2d20
Author: melanie witt <email address hidden>
Date: Thu Mar 11 23:47:05 2021 +0000

Add --task-log option to nova-manage db archive_deleted_rows

    The task_log table contains instance usage audit records if
    nova-compute has been configured with [DEFAULT]instance_usage_audit =
    True. This will be the case if OpenStack Telemetry is being used in the
    deployment, as the option causes nova to generate audit information
    that Telemetry then retrieves from the server usage audit log API [1].

    Historically, there has been no way to delete task_log table records
    other than manual database modification. Because of this, task_log
    records could pile up over time and operators are forced to perform
    manual steps to periodically truncate the table.

    This adds a --task-log option to the 'nova-manage db
    archive_deleted_rows' CLI to also archive task_log records while
    archiving the database. --task-log works in conjunction with --before
    if operators desire archving only records that are older than <date>.
    The 'updated_at' field is used by --task-log --before <date> to
    determine the age of a task_log record for archival.

Closes-Bug: #1877189

[1] https://docs.openstack.org/api-ref/compute/#server-usage-audit-log-os-instance-usage-audit-log

Change-Id: Ibed67854a693c930effd4dba7aca6cd03b65bd92

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-07-16: Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/801144

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-07-22: Related fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/801144
Committed: https://opendev.org/openstack/nova/commit/d093849c3c70e46bf409b1051454cbf70bc12a1f
Submitter: "Zuul (22348)"
Branch: master

commit d093849c3c70e46bf409b1051454cbf70bc12a1f
Author: melanie witt <email address hidden>
Date: Fri Jul 16 21:53:38 2021 +0000

Correct reno for archiving task_log database records

    The release note incorrectly stated that the Telemetry service calls
    the /os-instance_usage_audit_log REST API to retrieve instance usage
    data. Telemetry rather consumes the notifications that are only generated
    when [DEFAULT]instance_usage_audit=True.

Related-Bug: #1877189

Change-Id: Iea358090c42877bedfdcc4a04019f6dc406176c2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-07-28: Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/802766