Grenade jobs fail during verify with timeout
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
New
|
Undecided
|
Unassigned |
Bug Description
Grenade jobs are occasionally failing with a timeout while running sync:
2023-10-10 12:33:49.688 | + /opt/stack/
2023-10-10 12:33:49.695 | OpenSSH_8.9p1 Ubuntu-3ubuntu0.4, OpenSSL 3.0.2 15 Mar 2022
2023-10-10 12:33:49.695 | debug1: Reading configuration data /etc/ssh/ssh_config
2023-10-10 12:33:49.695 | debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/
2023-10-10 12:33:49.695 | debug1: /etc/ssh/ssh_config line 21: Applying options for *
2023-10-10 12:33:49.696 | debug1: Connecting to 172.24.5.210 [172.24.5.210] port 22.
2023-10-10 12:33:49.704 | debug1: fd 3 clearing O_NONBLOCK
2023-10-10 12:33:49.704 | debug1: Connection established.
2023-10-10 12:33:49.704 | debug1: identity file /opt/stack/
2023-10-10 12:33:49.704 | debug1: identity file /opt/stack/
2023-10-10 12:33:49.704 | debug1: Local version string SSH-2.0-
2023-10-10 12:33:49.730 | debug1: Remote protocol version 2.0, remote software version dropbear_2020.81
2023-10-10 12:33:49.730 | debug1: compat_banner: no match: dropbear_2020.81
2023-10-10 12:33:49.730 | debug1: Authenticating to 172.24.5.210:22 as 'cirros'
2023-10-10 12:33:49.730 | debug1: load_hostkeys: fopen /etc/ssh/
2023-10-10 12:33:49.730 | debug1: SSH2_MSG_KEXINIT sent
2023-10-10 12:33:49.730 | debug1: SSH2_MSG_KEXINIT received
2023-10-10 12:33:49.730 | debug1: kex: algorithm: curve25519-sha256
2023-10-10 12:33:49.730 | debug1: kex: host key algorithm: ssh-ed25519
2023-10-10 12:33:49.730 | debug1: kex: server->client cipher: <email address hidden> MAC: <implicit> compression: none
2023-10-10 12:33:49.730 | debug1: kex: client->server cipher: <email address hidden> MAC: <implicit> compression: none
2023-10-10 12:33:49.733 | debug1: expecting SSH2_MSG_
2023-10-10 12:33:49.801 | debug1: SSH2_MSG_
2023-10-10 12:33:49.801 | debug1: Server host key: ssh-ed25519 SHA256:
2023-10-10 12:33:49.801 | debug1: load_hostkeys: fopen /etc/ssh/
2023-10-10 12:33:49.801 | Warning: Permanently added '172.24.5.210' (ED25519) to the list of known hosts.
2023-10-10 12:33:49.810 | debug1: rekey out after 134217728 blocks
2023-10-10 12:33:49.811 | debug1: SSH2_MSG_NEWKEYS sent
2023-10-10 12:33:49.811 | debug1: expecting SSH2_MSG_NEWKEYS
2023-10-10 12:33:49.811 | debug1: SSH2_MSG_NEWKEYS received
2023-10-10 12:33:49.811 | debug1: rekey in after 134217728 blocks
2023-10-10 12:33:49.811 | debug1: Will attempt key: /opt/stack/
2023-10-10 12:33:49.811 | debug1: SSH2_MSG_EXT_INFO received
2023-10-10 12:33:49.811 | debug1: kex_input_ext_info: server-
2023-10-10 12:33:49.815 | debug1: SSH2_MSG_
2023-10-10 12:33:49.819 | debug1: Authentications that can continue: publickey,password
2023-10-10 12:33:49.820 | debug1: Next authentication method: publickey
2023-10-10 12:33:49.820 | debug1: Offering public key: /opt/stack/
2023-10-10 12:33:49.825 | debug1: Server accepts key: /opt/stack/
2023-10-10 12:33:49.873 | Authenticated to 172.24.5.210 ([172.24.5.210]:22) using "publickey".
2023-10-10 12:33:49.874 | debug1: channel 0: new [client-session]
2023-10-10 12:33:49.874 | debug1: Entering interactive session.
2023-10-10 12:33:49.874 | debug1: pledge: network
2023-10-10 12:33:49.877 | debug1: Sending environment.
2023-10-10 12:33:49.878 | debug1: channel 0: setting env LANG = "C.UTF-8"
2023-10-10 12:33:49.878 | debug1: channel 0: setting env LC_ALL = "C.UTF-8"
2023-10-10 12:33:49.878 | debug1: Sending command: sync
2023-10-10 12:34:19.690 | debug1: channel 0: free: client-session, nchannels 1
2023-10-10 12:34:19.690 | Killed by signal 15.
2023-10-10 12:34:19.693 | + /opt/stack/
2023-10-10 12:34:19.695 | + /opt/stack/
2023-10-10 12:34:19.698 | 1 verify_noapi /opt/stack/
2023-10-10 12:34:19.698 | 216 verify /opt/stack/
2023-10-10 12:34:19.698 | 308 main /opt/stack/
2023-10-10 12:34:19.698 | Exit code: 124
2023-10-10 12:34:22.145 | + inc/plugin:
2023-10-10 12:34:22.147 | + /opt/stack/
2023-10-10 12:34:22.149 | [Call Trace]
If this is close to the edge and that's why it only fails some of the time, we might just need to increase the timeout.
tags: | added: gate |
tags: |
added: gate-failure removed: gate |