tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_port_security_disable_security_group fails due to instances failing to retrieve public key via metadata service
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Incomplete
|
Undecided
|
Unassigned | ||
networking-ovn |
New
|
Undecided
|
Unassigned |
Bug Description
Running tempest test tempest.
After debugging, I managed to connect to the instance via virsh console and check that the instance had not the SSH key installed:
____ ____ ____
/ __/ __ ____ ____ / __ \/ __/
/ /__ / // __// __// /_/ /\ \
\___//_//_/ /_/ \____/___/
http://
login as 'cirros' user. default password: 'cubswin:)'. use 'sudo' for root.
tempest-
Password:
$ cat .ssh/authorized
$
Checking up in the console log, I can see the following:
cirros-ds 'net' up at 6.37
checking http://
successful after 1/20 tries: up 6.54. iid=i-0000006e
failed to get http://
warning: no ec2 metadata for public-keys
failed to get http://
warning: no ec2 metadata for user-data
found datasource (ec2, net)
So it looks like it is able to fetch the instance-id but not getting the public-key.
When I try to do it manually, it retrieves it successfully:
$ curl 169.254.
ssh-rsa AAAAB3NzaC1yc2E
Also, running the following command doesn't work:
$ sudo cirros-apply net -v
$ cat .ssh/authorized
$
If, instead I run the following command and reboot, it will get properly installed:
$ sudo cirros-per boot cirros-apply-net cirros-apply net && reboot
...
$ cat .ssh/authorized
ssh-rsa AAAAB3NzaC1yc2E
After checking the ovn metadata proxy log and also nova-metadata-api logs, I can see the requests and the 200 OK responses:
2018-04-19 22:31:37.383 24 INFO eventlet.
2018-04-19 22:31:38.800 24 INFO eventlet.
2018-04-19 22:31:49.148 24 INFO eventlet.
2018-04-19 22:31:49.387 24 INFO eventlet.
2018-04-19 22:31:50.225 24 INFO eventlet.
2018-04-19 22:31:50.482 24 INFO eventlet.
2018-04-19 22:31:50.612 24 INFO eventlet.
2018-04-19 22:31:50.793 24 INFO eventlet.
2018-04-19 22:31:51.039 24 INFO eventlet.
2018-04-19 22:31:51.197 24 INFO eventlet.
2018-04-19 22:31:51.475 24 INFO eventlet.
2018-04-19 22:31:51.579 24 INFO eventlet.
2018-04-19 22:31:51.672 24 INFO eventlet.
2018-04-19 22:31:51.735 24 INFO eventlet.
2018-04-19 22:31:51.930 24 INFO eventlet.
nova-api-
2018-04-19 22:31:38.852 5 INFO nova.metadata.
2018-04-19 22:31:51.474 5 INFO nova.metadata.
This is a TripleO (tripleo-ci-master) setup running 3 controllers and 1 compute.
Please, note that the test doesn't always fail... I ran it 10 times in a row and it failed with this error in 3 of them.
Even though it is an OVN environment, I suspect that it might have to do either with Nova or with the cirros image itself.
+------
| ID | Name | Status |
+------
| 671c2d8e-
| e4d8721c-
The test would use the image with id 671c2d8e-
I can't see any error traces in ovn-metadata-agent, ovn-controller nor neutron-server.
Thanks,
Daniel
affects: | networking-ovn → nova |
affects: | nova → networking-ovn |
summary: |
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_port_security_disable_security_group - fails due to metadata error + fails due to instances failing to retrieve public key via metadata + service |
tags: | added: metadata |
@dalvarez, can you try running:
$ sudo bash -x cirros-apply net -v
I believe it was bash, so may be you'll see what's going on there?
very weird.