kayobe

Data file path detection broken on Rocky Linux 9

  1. Series antelope
  2. Bug #2020135
Bug #2020135 reported by Pierre Riteau
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
kayobe
Status tracked in Bobcat
Antelope
In Progress
Medium
Pierre Riteau
Bobcat
Fix Released
Medium
Pierre Riteau
Xena
Won't Fix
Medium
Pierre Riteau
Yoga
Fix Committed
Medium
Pierre Riteau
Zed
Fix Committed
Medium
Pierre Riteau

Bug Description

I tried setting up a Kayobe dev environment on Rocky Linux 9.1 and 9.2 following instructions at https://docs.openstack.org/kayobe/latest/contributor/automated.html

It fails with:

$ ./dev/overcloud-deploy.sh
Deploying a kayobe development environment. This consists of a
single node OpenStack control plane.
Using Kayobe config from /home/rocky/kayobe/config/src/kayobe-config
Bootstrapping the Ansible control host
Installing galaxy role dependencies from kayobe
ERROR! The requirements file '/home/rocky/kayobe/kayobe/requirements.yml' does not exist.
Failed to install Ansible roles from /home/rocky/kayobe/kayobe/utils.py/../requirements.yml via Ansible Galaxy: returncode 1
Control host bootstrap failed - likely Ansible Galaxy flakiness. Sleeping 5 seconds before retrying

The problem originates from the detection of where Kayobe is installed.

On c8s, _get_base_path() in kayobe/utils.py would detect the data path using the egg link:

$ cat /home/centos/kayobe-venv/lib/python3.6/site-packages/kayobe.egg-link
/home/centos/kayobe

This works differently in Rocky Linux 9.1/9.2, there is no such kayobe.egg-link file:

$ ls -ld /home/rocky/kayobe-venv/lib/python3.9/site-packages/*kayobe*
-rw-r--r--. 1 rocky rocky 97 May 18 19:30 /home/rocky/kayobe-venv/lib/python3.9/site-packages/__editable__.kayobe-13.1.0.dev64.pth
-rw-r--r--. 1 rocky rocky 2411 May 18 19:30 /home/rocky/kayobe-venv/lib/python3.9/site-packages/__editable___kayobe_13_1_0_dev64_finder.py
drwxr-xr-x. 2 rocky rocky 182 May 18 19:30 /home/rocky/kayobe-venv/lib/python3.9/site-packages/kayobe-13.1.0.dev64.dist-info

Workaround:

export KAYOBE_DATA_FILES_PATH=/home/rocky/kayobe

See original description
Revision history for this message
Pierre Riteau (priteau) wrote :

Note this may affect earlier versions of Rocky Linux 9 and other distributions. It appears to be related to recent changes to editable installations in setuptools.

We don't use editable installs in CI so this could have been overlooked for a while.

Pierre Riteau (priteau)
summary: - Data file path detection broken on Rocky Linux 9.2
+ Data file path detection broken on Rocky Linux 9
description: updated
description: updated
Bartosz Bezak (bbezak)
Changed in kayobe:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Bartosz Bezak (bbezak) wrote (last edit ):

I've noticed the same on focal with kayobe stable/yoga and jammy with kayobe/master

ERROR! The requirements file '/home/ubuntu/kayobe/kayobe/requirements.yml' does not exist.
Failed to install Ansible roles from /home/ubuntu/kayobe/kayobe/utils.py/../requirements.yml via Ansible Galaxy: returncode 1

Centos 8 Stream stable/yoga works fine

Revision history for this message
Pierre Riteau (priteau) wrote :

This also affects Ubuntu 20.04 with stable/yoga.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kayobe (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kayobe/+/894831

Changed in kayobe:
status: Confirmed → In Progress
Revision history for this message
Pierre Riteau (priteau) wrote :

This is actually caused by updating pip, not setuptools.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kayobe (master)

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/894831
Committed: https://opendev.org/openstack/kayobe/commit/1847ad3f17a039f29e22f465508991fcb6e1cca7
Submitter: "Zuul (22348)"
Branch: master

commit 1847ad3f17a039f29e22f465508991fcb6e1cca7
Author: Pierre Riteau <email address hidden>
Date: Thu Sep 21 18:29:23 2023 +0200

    Fix data file path detection with new pip

    Using an editable installation of Kayobe fails on Rocky Linux 9 or
    Ubuntu with an error such as:

        ERROR! The requirements file '/home/rocky/kayobe/kayobe/requirements.yml' does not exist.
        Failed to install Ansible roles from /home/rocky/kayobe/kayobe/utils.py/../requirements.yml via Ansible Galaxy: returncode 1
        Control host bootstrap failed - likely Ansible Galaxy flakiness. Sleeping 5 seconds before retrying

    This is caused by recent changes to how pip manages editable
    installations. The egg-link file that Kayobe was using to find the
    source path does not exist anymore. Instead, there is a direct_url.json
    file under the kayobe dist-info directory that can be parsed.

    Change-Id: I9dd5b97dec93c0e5393a1e7d9640f85003651b56
    Closes-Bug: #2020135

Changed in kayobe:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kayobe (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kayobe/+/896379

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kayobe (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kayobe/+/896380

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kayobe (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kayobe/+/896381

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kayobe (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kayobe/+/896382

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kayobe (stable/xena)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/xena
Review: https://review.opendev.org/c/openstack/kayobe/+/896382
Reason: This branch of this repository is End of Life. To be able to delete the branch open patches needs to be abandoned.

Revision history for this message
Maksim Malchuk (mmalchuk) wrote :

Xena EOL

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to kayobe (master)

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/896877
Committed: https://opendev.org/openstack/kayobe/commit/cfaf571794296d6e0cfed2771f8046a8280ffb22
Submitter: "Zuul (22348)"
Branch: master

commit cfaf571794296d6e0cfed2771f8046a8280ffb22
Author: Pierre Riteau <email address hidden>
Date: Fri Sep 29 07:11:43 2023 +0200

    Work around mocking issues on stable branches

    Backports of I9dd5b97dec93c0e5393a1e7d9640f85003651b56 to stable
    branches are failing CI checks because several unit tests mock
    os.path.exists and would need to be updated. This does not affect master
    because test code changed in I4d9f96ec4cf3c6cd0d28dfe5ddb239d863498a72.

    Work around this issue by replacing os.path.exists by os.path.isfile. It
    is a better check anyway since we are reading the file.

    Change-Id: Ic53efd03cecbd53ad3e3b64b664e084f4e25be0e
    Related-Bug: #2020135

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kayobe (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/896380
Committed: https://opendev.org/openstack/kayobe/commit/77a0e36fbf430706951138fbc4d187417ece20d3
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 77a0e36fbf430706951138fbc4d187417ece20d3
Author: Pierre Riteau <email address hidden>
Date: Thu Sep 21 18:29:23 2023 +0200

    Fix data file path detection with new pip

    Using an editable installation of Kayobe fails on Rocky Linux 9 or
    Ubuntu with an error such as:

        ERROR! The requirements file '/home/rocky/kayobe/kayobe/requirements.yml' does not exist.
        Failed to install Ansible roles from /home/rocky/kayobe/kayobe/utils.py/../requirements.yml via Ansible Galaxy: returncode 1
        Control host bootstrap failed - likely Ansible Galaxy flakiness. Sleeping 5 seconds before retrying

    This is caused by recent changes to how pip manages editable
    installations. The egg-link file that Kayobe was using to find the
    source path does not exist anymore. Instead, there is a direct_url.json
    file under the kayobe dist-info directory that can be parsed.

    This also includes Ic53efd03cecbd53ad3e3b64b664e084f4e25be0e to work
    around mocking issues in unit tests.

    Change-Id: I9dd5b97dec93c0e5393a1e7d9640f85003651b56
    Closes-Bug: #2020135
    (cherry picked from commit 1847ad3f17a039f29e22f465508991fcb6e1cca7)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kayobe (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/896379
Committed: https://opendev.org/openstack/kayobe/commit/a44f06365a130d4322f0b18e4a3eeb083295da30
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit a44f06365a130d4322f0b18e4a3eeb083295da30
Author: Pierre Riteau <email address hidden>
Date: Wed Oct 4 09:56:22 2023 +0200

    Fix data file path detection with new pip

    Using an editable installation of Kayobe fails on Rocky Linux 9 or
    Ubuntu with an error such as:

        ERROR! The requirements file '/home/rocky/kayobe/kayobe/requirements.yml' does not exist.
        Failed to install Ansible roles from /home/rocky/kayobe/kayobe/utils.py/../requirements.yml via Ansible Galaxy: returncode 1
        Control host bootstrap failed - likely Ansible Galaxy flakiness. Sleeping 5 seconds before retrying

    This is caused by recent changes to how pip manages editable
    installations. The egg-link file that Kayobe was using to find the
    source path does not exist anymore. Instead, there is a direct_url.json
    file under the kayobe dist-info directory that can be parsed.

    This also includes Ic53efd03cecbd53ad3e3b64b664e084f4e25be0e to work
    around mocking issues in unit tests.

    Change-Id: I9dd5b97dec93c0e5393a1e7d9640f85003651b56
    Closes-Bug: #2020135
    (cherry picked from commit 1847ad3f17a039f29e22f465508991fcb6e1cca7)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kayobe (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/896381
Committed: https://opendev.org/openstack/kayobe/commit/390fbf1d416f64d9e75856ef17632c9ccd71483d
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 390fbf1d416f64d9e75856ef17632c9ccd71483d
Author: Pierre Riteau <email address hidden>
Date: Wed Oct 4 22:23:47 2023 +0200

    Fix data file path detection with new pip

    Using an editable installation of Kayobe fails on Rocky Linux 9 or
    Ubuntu with an error such as:

        ERROR! The requirements file '/home/rocky/kayobe/kayobe/requirements.yml' does not exist.
        Failed to install Ansible roles from /home/rocky/kayobe/kayobe/utils.py/../requirements.yml via Ansible Galaxy: returncode 1
        Control host bootstrap failed - likely Ansible Galaxy flakiness. Sleeping 5 seconds before retrying

    This is caused by recent changes to how pip manages editable
    installations. The egg-link file that Kayobe was using to find the
    source path does not exist anymore. Instead, there is a direct_url.json
    file under the kayobe dist-info directory that can be parsed.

    This also includes Ic53efd03cecbd53ad3e3b64b664e084f4e25be0e to work
    around mocking issues in unit tests and an amended version of
    Iecf1d469caaa777c5b253eb0e44dc00692197ef7 to import importlib.metadata
    or importlib_metadata depending on the Python version.

    Change-Id: I9dd5b97dec93c0e5393a1e7d9640f85003651b56
    Closes-Bug: #2020135
    (cherry picked from commit 1847ad3f17a039f29e22f465508991fcb6e1cca7)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.