tempestconf fails to cirros image due to SSL: CERTIFICATE_VERIFY_FAILED issue

Bug #2024208 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned

Bug Description

https://review.rdoproject.org/zuul/buildset/f81e4b4e20e0406889f4b95ce95139e6 cs9 wallaby periodic line jobs are failing while running python-tempestconf [1]
```
Downloading 'https://download.cirros-cloud.net/0.5.2/cirros-0.5.2-x86_64-disk.img' and saving as '/home/zuul/tempest/etc/cirros-0.5.2-x86_64-disk.img'
2023-06-16 13:35:57.878 262771 WARNING config_tempest.constants [-] <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)>, Retrying in 3 seconds.: urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)>
2023-06-16 13:36:01.091 262771 WARNING config_tempest.constants [-] <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)>, Retrying in 6 seconds.: urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)>
2023-06-16 13:36:07.245 262771 WARNING config_tempest.constants [-] <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)>, Retrying in 12 seconds.: urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)>
2023-06-16 13:36:19.407 262771 INFO config_tempest.constants [-] Downloading 'http://images.rdoproject.org/cirros/cirros-0.5.2-x86_64-disk.img' and saving as '/home/zuul/tempest/etc/cirros-0.5.2-x86_64-disk.img'
2023-06-16 13:36:19.513 262771 WARNING config_tempest.constants [-] HTTP Error 404: Not Found, Retrying in 3 seconds.: urllib.error.HTTPError: HTTP Error 404: Not Found
2023-06-16 13:36:22.623 262771 WARNING config_tempest.constants [-] HTTP Error 404: Not Found, Retrying in 6 seconds.: urllib.error.HTTPError: HTTP Error 404: Not Found
2023-06-16 13:36:28.731 262771 WARNING config_tempest.constants [-] HTTP Error 404: Not Found, Retrying in 12 seconds.: urllib.error.HTTPError: HTTP Error 404: Not Found
2023-06-16 13:36:40.844 262771 CRITICAL tempest [-] Unhandled error: tenacity.RetryError: RetryError[<Future at 0x7f3e13717af0 state=finished raised HTTPError>]
2023-06-16 13:36:40.844 262771 ERROR tempest Traceback (most recent call last):
2023-06-16 13:36:40.844 262771 ERROR tempest File "/usr/lib/python3.9/site-packages/config_tempest/services/image.py", line 182, in find_or_upload_image
2023-06-16 13:36:40.844 262771 ERROR tempest shutil.copyfile(image_source, image_dest)
2023-06-16 13:36:40.844 262771 ERROR tempest File "/usr/lib64/python3.9/shutil.py", line 264, in copyfile
2023-06-16 13:36:40.844 262771 ERROR tempest with open(src, 'rb') as fsrc:
2023-06-16 13:36:40.844 262771 ERROR tempest FileNotFoundError: [Errno 2] No such file or directory: '/opt/cache/files/cirros-0.5.2-x86_64-disk.img'
2023-06-16 13:36:40.844 262771 ERROR tempest
2023-06-16 13:36:40.844 262771 ERROR tempest During handling of the above exception, another exception occurred:
2023-06-16 13:36:40.844 262771 ERROR tempest
2023-06-16 13:36:40.844 262771 ERROR tempest Traceback (most recent call last):
2023-06-16 13:36:40.844 262771 ERROR tempest File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__
2023-06-16 13:36:40.844 262771 ERROR tempest result = fn(*args, **kwargs)
2023-06-16 13:36:40.844 262771 ERROR tempest File "/usr/lib/python3.9/site-packages/config_tempest/services/image.py", line 203, in _download_with_retry
2023-06-16 13:36:40.844 262771 ERROR tempest self._download_file(C.DEFAULT_IMAGES[self.retry_attempt], destination)
2023-06-16 13:36:40.844 262771 ERROR tempest File "/usr/lib/python3.9/site-packages/config_tempest/services/image.py", line 327, in _download_file
2023-06-16 13:36:40.844 262771 ERROR tempest f = self.retry_urlopen(url)
2023-06-16 13:36:40.844 262771 ERROR tempest File "/usr/lib/python3.9/site-packages/config_tempest/services/image.py", line 305, in f_retry
2023-06-16 13:36:40.844 262771 ERROR tempest return f(*args, **kwargs)
2023-06-16 13:36:40.844 262771 ERROR tempest File "/usr/lib/python3.9/site-packages/config_tempest/services/image.py", line 315, in retry_urlopen
2023-06-16 13:36:40.844 262771 ERROR tempest return urllib.request.urlopen(url)
2023-06-16 13:36:40.844 262771 ERROR tempest File "/usr/lib64/python3.9/urllib/request.py", line 214, in urlopen
2023-06-16 13:36:40.844 262771 ERROR tempest return opener.open(url, data, timeout)
2023-06-16 13:36:40.844 262771 ERROR tempest File "/usr/lib64/python3.9/urllib/request.py", line 523, in open
2023-06-16 13:36:40.844 262771 ERROR tempest response = meth(req, response)
2023-06-16 13:36:40.844 262771 ERROR tempest File "/usr/lib64/python3.9/urllib/request.py", line 632, in http_response
2023-06-16 13:36:40.844 262771 ERROR tempest response = self.parent.error(
2023-06-16 13:36:40.844 262771 ERROR tempest File "/usr/lib64/python3.9/urllib/request.py", line 561, in error
2023-06-16 13:36:40.844 262771 ERROR tempest return self._call_chain(*args)
2023-06-16 13:36:40.844 262771 ERROR tempest File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain
2023-06-16 13:36:40.844 262771 ERROR tempest result = func(*args)
2023-06-16 13:36:40.844 262771 ERROR tempest File "/usr/lib64/python3.9/urllib/request.py", line 641, in http_error_default
2023-06-16 13:36:40.844 262771 ERROR tempest raise HTTPError(req.full_url, code, msg, hdrs, fp)
2023-06-16 13:36:40.844 262771 ERROR tempest urllib.error.HTTPError: HTTP Error 404: Not Found
```

On doing curl on cirros image url, we got following:
```
❯ curl https://download.cirros-cloud.net/0.5.2/cirros-0.5.2-x86_64-disk.img
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
```

In our code, we used to set the image url from github [2] but somehow it is working fine:
```
- name: Set extra facts for tempestconf
      vars:
        default_cirros_image: /opt/cache/files/cirros-0.5.2-x86_64-disk.img
      set_fact:
        tempest_tempestconf_profile_extras:
          image: "{{ featureset_override['cirros_image'] | default(default_cirros_image) }}"
          flavor_min_mem: 128
          retry_image: true
          overrides:
            identity.v3_endpoint_type: public
            validation.ssh_key_type: 'ecdsa'
            image.http_image: https://github.com/cirros-dev/cirros/releases/download/0.5.2/cirros-0.5.2-x86_64-disk.img
```

Here is the generated profile [3].
```
create: true
debug: true
flavor_min_mem: 128
image: /opt/cache/files/cirros-0.5.2-x86_64-disk.img
network-id: 4e98d9e4-2a75-4433-856b-a581f5b2cfad
os-cloud: standalone
out: /home/zuul/tempest/etc/tempest.conf
overrides:
    auth.tempest_roles: Member
    identity.v3_endpoint_type: public
    image.http_image: https://github.com/cirros-dev/cirros/releases/download/0.5.2/cirros-0.5.2-x86_64-disk.img
    telemetry.alarm_granularity: '60'
    validation.ssh_key_type: ecdsa
retry_image: true
```

There is something wrong with tempestconf overrides may be[4] as cirros url is coming from tempestconf.
```
DEFAULT_IMAGES = [("https://download.cirros-cloud.net/0.5.2/"
                  "cirros-0.5.2-x86_64-disk.img"),
                  ("http://images.rdoproject.org/cirros
```

Links:
[1]. https://logserver.rdoproject.org/0b/0bc4d1bc5ee4f6af42bd2c3740b8732a45ce0d68/openstack-periodic-integration-stable1/periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby/c6623f4/logs/undercloud/var/log/tempest/tempestconf.log

[2]. https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/tasks/tempest.yml#L202

[3]. https://logserver.rdoproject.org/0b/0bc4d1bc5ee4f6af42bd2c3740b8732a45ce0d68/openstack-periodic-integration-stable1/periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby/c6623f4/logs/undercloud/home/zuul/tempest/etc/profile.yaml

[4]. https://opendev.org/openinfra/python-tempestconf/src/branch/master/config_tempest/constants.py#L30

Revision history for this message
chandan kumar (chkumar246) wrote :

The ssl cert issue for the cirros image url is fixed now: https://download.cirros-cloud.net/0.5.2/cirros-0.5.2-x86_64-disk.img and centos-9 wallaby got promoted on JUn 18, 2023.

Ronelle Landy (rlandy)
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.