kolla

kolla_set_configs --check fails for source directories

Bug #1890567 reported by Mark Goddard on 2020-08-06

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
kolla	Fix Released	Medium	Radosław Piliszek	kolla 11.0.0 "victoria"
Train	Fix Released	Medium	Unassigned	kolla 9.3.0 "Train"
Ussuri	Fix Released	Medium	Unassigned	kolla 10.2.0 "ussuri"
Victoria	Fix Released	Medium	Radosław Piliszek	kolla 11.0.0 "victoria"

Bug Description

There are two issues with kolla_set_configs --check:

1. We calculate the destination path incorrectly when comparing a file
   in a directory, due to passing arguments to os.path.relpath in the
   wrong order
2. For directories that have not changed, we also attempt to compare
   them as files, which fails when they are open()ed.

This affects the fluentd container, which specifies directories as the
source, without using a glob:

{
   "source": "/var/lib/kolla/config_files/output",
    "dest": "/etc/td-agent/output",
    "owner": "td-agent",
    "perm": "0600"
}

This results in an error like this:

INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
ERROR:__main__:Unexpected error:
Traceback (most recent call last):
  File "/usr/local/bin/kolla_set_configs", line 423, in main
    execute_config_check(config)
  File "/usr/local/bin/kolla_set_configs", line 409, in execute_config_check
    config_file.check()
  File "/usr/local/bin/kolla_set_configs", line 228, in check
    if os.path.isdir(source) and not self._cmp_dir(source, dest):
  File "/usr/local/bin/kolla_set_configs", line 209, in _cmp_dir
    if not self._cmp_file(full_path, dest_full_path):
  File "/usr/local/bin/kolla_set_configs", line 167, in _cmp_file
    with open(source) as f1, open(dest) as f2:
IsADirectoryError: [Errno 21] Is a directory: '/etc/td-agent/input/..'

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-08-06: Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.opendev.org/745105

Changed in kolla:
assignee:	nobody → Mark Goddard (mgoddard)
status:	New → In Progress

Mark Goddard (mgoddard) on 2020-08-06

Changed in kolla:
importance:	Undecided → Medium

OpenStack Infra (hudson-openstack) on 2020-08-07

Changed in kolla:
assignee:	Mark Goddard (mgoddard) → Radosław Piliszek (yoctozepto)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-08-28: Fix merged to kolla (master)

Reviewed: https://review.opendev.org/745105
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=c5320eb22386c34d83435fa0b9aa03b178996b52
Submitter: Zuul
Branch: master

commit c5320eb22386c34d83435fa0b9aa03b178996b52
Author: Mark Goddard <email address hidden>
Date: Thu Aug 6 10:50:25 2020 +0100

Fix kolla_set_configs --check with a directory

There are several issues with kolla_set_configs --check:

    1. We calculate the destination path incorrectly when comparing a file
       in a directory, due to passing arguments to os.path.relpath in the
       wrong order
    2. For directories that have not changed, we also attempt to compare
       them as files, which fails when they are open()ed.
    3. If the config JSON does not have a config_files key, it fails with a
       KeyError.

    The first two issues affect the fluentd container, which specifies
    directories as the source, without using a glob. The third affects OVN
    containers.

This patch fixes these issues.

Closes-Bug: #1890567

Change-Id: I8921befe51da4282121443849177a7ca5ebe8822

Changed in kolla:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-09-14: Fix proposed to kolla (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/751726

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-09-14: Fix proposed to kolla (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/751727

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-09-14: Fix proposed to kolla (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/751728

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-09-15: Fix merged to kolla (stable/stein)

Reviewed: https://review.opendev.org/751728
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=c6835b12a6f1e8f715aa5c69383a8d1836551496
Submitter: Zuul
Branch: stable/stein

commit c6835b12a6f1e8f715aa5c69383a8d1836551496
Author: Mark Goddard <email address hidden>
Date: Thu Aug 6 10:50:25 2020 +0100

Fix kolla_set_configs --check with a directory

There are several issues with kolla_set_configs --check:

    1. We calculate the destination path incorrectly when comparing a file
       in a directory, due to passing arguments to os.path.relpath in the
       wrong order
    2. For directories that have not changed, we also attempt to compare
       them as files, which fails when they are open()ed.
    3. If the config JSON does not have a config_files key, it fails with a
       KeyError.

    The first two issues affect the fluentd container, which specifies
    directories as the source, without using a glob. The third affects OVN
    containers.

This patch fixes these issues.

Closes-Bug: #1890567

Change-Id: I8921befe51da4282121443849177a7ca5ebe8822
(cherry picked from commit c5320eb22386c34d83435fa0b9aa03b178996b52)

tags:

added: in-stable-stein

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-09-16: Fix merged to kolla (stable/ussuri)

Reviewed: https://review.opendev.org/751726
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=c191e6fbd7ebf0e6f455dc8cfc34ecda64e33bc1
Submitter: Zuul
Branch: stable/ussuri

commit c191e6fbd7ebf0e6f455dc8cfc34ecda64e33bc1
Author: Mark Goddard <email address hidden>
Date: Thu Aug 6 10:50:25 2020 +0100

Fix kolla_set_configs --check with a directory

There are several issues with kolla_set_configs --check:

    1. We calculate the destination path incorrectly when comparing a file
       in a directory, due to passing arguments to os.path.relpath in the
       wrong order
    2. For directories that have not changed, we also attempt to compare
       them as files, which fails when they are open()ed.
    3. If the config JSON does not have a config_files key, it fails with a
       KeyError.

    The first two issues affect the fluentd container, which specifies
    directories as the source, without using a glob. The third affects OVN
    containers.

This patch fixes these issues.

Closes-Bug: #1890567

Change-Id: I8921befe51da4282121443849177a7ca5ebe8822
(cherry picked from commit c5320eb22386c34d83435fa0b9aa03b178996b52)

tags:

added: in-stable-ussuri

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-09-17: Fix merged to kolla (stable/train)

Reviewed: https://review.opendev.org/751727
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=797ecd6ddbadcc7a726ce1611c210667565d8a91
Submitter: Zuul
Branch: stable/train

commit 797ecd6ddbadcc7a726ce1611c210667565d8a91
Author: Mark Goddard <email address hidden>
Date: Thu Aug 6 10:50:25 2020 +0100

Fix kolla_set_configs --check with a directory

There are several issues with kolla_set_configs --check:

    1. We calculate the destination path incorrectly when comparing a file
       in a directory, due to passing arguments to os.path.relpath in the
       wrong order
    2. For directories that have not changed, we also attempt to compare
       them as files, which fails when they are open()ed.
    3. If the config JSON does not have a config_files key, it fails with a
       KeyError.

    The first two issues affect the fluentd container, which specifies
    directories as the source, without using a glob. The third affects OVN
    containers.

This patch fixes these issues.

Closes-Bug: #1890567

Change-Id: I8921befe51da4282121443849177a7ca5ebe8822
(cherry picked from commit c5320eb22386c34d83435fa0b9aa03b178996b52)

tags:

added: in-stable-train

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-11-24: Fix included in openstack/kolla 8.0.5

This issue was fixed in the openstack/kolla 8.0.5 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-01-07: Fix included in openstack/kolla 10.2.0

#10

This issue was fixed in the openstack/kolla 10.2.0 release.