Grenade jobs fail during verify with timeout

Bug #2039707 reported by Dan Smith
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
New
Undecided
Unassigned

Bug Description

Grenade jobs are occasionally failing with a timeout while running sync:

2023-10-10 12:33:49.688 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:verify_noapi:245 : timeout 30 ssh -v -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i /opt/stack/save/cinder_key.pem cirros@172.24.5.210 sync
2023-10-10 12:33:49.695 | OpenSSH_8.9p1 Ubuntu-3ubuntu0.4, OpenSSL 3.0.2 15 Mar 2022
2023-10-10 12:33:49.695 | debug1: Reading configuration data /etc/ssh/ssh_config
2023-10-10 12:33:49.695 | debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
2023-10-10 12:33:49.695 | debug1: /etc/ssh/ssh_config line 21: Applying options for *
2023-10-10 12:33:49.696 | debug1: Connecting to 172.24.5.210 [172.24.5.210] port 22.
2023-10-10 12:33:49.704 | debug1: fd 3 clearing O_NONBLOCK
2023-10-10 12:33:49.704 | debug1: Connection established.
2023-10-10 12:33:49.704 | debug1: identity file /opt/stack/save/cinder_key.pem type 2
2023-10-10 12:33:49.704 | debug1: identity file /opt/stack/save/cinder_key.pem-cert type -1
2023-10-10 12:33:49.704 | debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.4
2023-10-10 12:33:49.730 | debug1: Remote protocol version 2.0, remote software version dropbear_2020.81
2023-10-10 12:33:49.730 | debug1: compat_banner: no match: dropbear_2020.81
2023-10-10 12:33:49.730 | debug1: Authenticating to 172.24.5.210:22 as 'cirros'
2023-10-10 12:33:49.730 | debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
2023-10-10 12:33:49.730 | debug1: SSH2_MSG_KEXINIT sent
2023-10-10 12:33:49.730 | debug1: SSH2_MSG_KEXINIT received
2023-10-10 12:33:49.730 | debug1: kex: algorithm: curve25519-sha256
2023-10-10 12:33:49.730 | debug1: kex: host key algorithm: ssh-ed25519
2023-10-10 12:33:49.730 | debug1: kex: server->client cipher: <email address hidden> MAC: <implicit> compression: none
2023-10-10 12:33:49.730 | debug1: kex: client->server cipher: <email address hidden> MAC: <implicit> compression: none
2023-10-10 12:33:49.733 | debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
2023-10-10 12:33:49.801 | debug1: SSH2_MSG_KEX_ECDH_REPLY received
2023-10-10 12:33:49.801 | debug1: Server host key: ssh-ed25519 SHA256:xYF3/NWzL2emqPReb7/I97TimxDel9q1Zn9diUoVi3c
2023-10-10 12:33:49.801 | debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
2023-10-10 12:33:49.801 | Warning: Permanently added '172.24.5.210' (ED25519) to the list of known hosts.
2023-10-10 12:33:49.810 | debug1: rekey out after 134217728 blocks
2023-10-10 12:33:49.811 | debug1: SSH2_MSG_NEWKEYS sent
2023-10-10 12:33:49.811 | debug1: expecting SSH2_MSG_NEWKEYS
2023-10-10 12:33:49.811 | debug1: SSH2_MSG_NEWKEYS received
2023-10-10 12:33:49.811 | debug1: rekey in after 134217728 blocks
2023-10-10 12:33:49.811 | debug1: Will attempt key: /opt/stack/save/cinder_key.pem ECDSA SHA256:sVZ90mbZniy5dibywsKv1qPUtSmkxdpmJuRmxPnJWrk explicit
2023-10-10 12:33:49.811 | debug1: SSH2_MSG_EXT_INFO received
2023-10-10 12:33:49.811 | debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,ssh-rsa>
2023-10-10 12:33:49.815 | debug1: SSH2_MSG_SERVICE_ACCEPT received
2023-10-10 12:33:49.819 | debug1: Authentications that can continue: publickey,password
2023-10-10 12:33:49.820 | debug1: Next authentication method: publickey
2023-10-10 12:33:49.820 | debug1: Offering public key: /opt/stack/save/cinder_key.pem ECDSA SHA256:sVZ90mbZniy5dibywsKv1qPUtSmkxdpmJuRmxPnJWrk explicit
2023-10-10 12:33:49.825 | debug1: Server accepts key: /opt/stack/save/cinder_key.pem ECDSA SHA256:sVZ90mbZniy5dibywsKv1qPUtSmkxdpmJuRmxPnJWrk explicit
2023-10-10 12:33:49.873 | Authenticated to 172.24.5.210 ([172.24.5.210]:22) using "publickey".
2023-10-10 12:33:49.874 | debug1: channel 0: new [client-session]
2023-10-10 12:33:49.874 | debug1: Entering interactive session.
2023-10-10 12:33:49.874 | debug1: pledge: network
2023-10-10 12:33:49.877 | debug1: Sending environment.
2023-10-10 12:33:49.878 | debug1: channel 0: setting env LANG = "C.UTF-8"
2023-10-10 12:33:49.878 | debug1: channel 0: setting env LC_ALL = "C.UTF-8"
2023-10-10 12:33:49.878 | debug1: Sending command: sync
2023-10-10 12:34:19.690 | debug1: channel 0: free: client-session, nchannels 1
2023-10-10 12:34:19.690 | Killed by signal 15.
2023-10-10 12:34:19.693 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:verify_noapi:1 : exit_trap
2023-10-10 12:34:19.695 | + /opt/stack/new/grenade/functions:exit_trap:115 : local r=124
2023-10-10 12:34:19.698 | 1 verify_noapi /opt/stack/new/grenade/projects/70_cinder/resources.sh
2023-10-10 12:34:19.698 | 216 verify /opt/stack/new/grenade/projects/70_cinder/resources.sh
2023-10-10 12:34:19.698 | 308 main /opt/stack/new/grenade/projects/70_cinder/resources.sh
2023-10-10 12:34:19.698 | Exit code: 124
2023-10-10 12:34:22.145 | + inc/plugin:resources:123 : die 123 'Failed to run /opt/stack/new/grenade/projects/70_cinder/resources.sh verify pre-upgrade'
2023-10-10 12:34:22.147 | + /opt/stack/new/devstack/functions-common:die:264 : local exitcode=124
2023-10-10 12:34:22.149 | [Call Trace]

If this is close to the edge and that's why it only fails some of the time, we might just need to increase the timeout.

Tags: gate-failure
Dan Smith (danms)
tags: added: gate
tags: added: gate-failure
removed: gate
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.