Comment 8 for bug 1643911

On this job I've noticed that the memory corruption error [1] happends after (what I find are) suspicious iscsi errors and libvirt logging issues:

Feb 02 08:44:04 ubuntu-xenial-infracloud-chocolate-7072289 kernel: sd 3:0:0:1: [sda] Synchronizing SCSI cache
                                                                                     ^^^^^^^^^^^^^^^^^^^^^^^^
Feb 02 08:44:05 ubuntu-xenial-infracloud-chocolate-7072289 account-replicator[31349]: Beginning replication run
Feb 02 08:44:05 ubuntu-xenial-infracloud-chocolate-7072289 account-replicator[31349]: Replication run OVER
Feb 02 08:44:05 ubuntu-xenial-infracloud-chocolate-7072289 account-replicator[31349]: Attempted to replicate 1 dbs in 0.01254 seconds (79.75111/s)
Feb 02 08:44:05 ubuntu-xenial-infracloud-chocolate-7072289 account-replicator[31349]: Removed 0 dbs
Feb 02 08:44:05 ubuntu-xenial-infracloud-chocolate-7072289 account-replicator[31349]: 1 successes, 0 failures
Feb 02 08:44:05 ubuntu-xenial-infracloud-chocolate-7072289 account-replicator[31349]: no_change:1 ts_repl:0 diff:0 rsync:0 diff_capped:0 hashmatch:0 empty:0
Feb 02 08:44:05 ubuntu-xenial-infracloud-chocolate-7072289 sudo[21121]: stack : TTY=unknown ; PWD=/opt/stack/new/devstack ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf tgt-admin --show
Feb 02 08:44:05 ubuntu-xenial-infracloud-chocolate-7072289 sudo[21121]: pam_unix(sudo:session): session opened for user root by (uid=0)
Feb 02 08:44:06 ubuntu-xenial-infracloud-chocolate-7072289 iscsid[651]: Connection2:0 to [target: iqn.2010-10.org.openstack:volume-2c20350a-011b-45ac-bc99-e2a9089e423e, portal: 15.184.70.50,3260] through [iface: default] is shutdown.
                                                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 02 08:44:06 ubuntu-xenial-infracloud-chocolate-7072289 sudo[21121]: pam_unix(sudo:session): session closed for user root
Feb 02 08:44:06 ubuntu-xenial-infracloud-chocolate-7072289 sudo[21125]: stack : TTY=unknown ; PWD=/opt/stack/new/devstack ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf tgt-admin --force --delete iqn.2010-10.org.openstack:volume-2c20350a-011b-45ac-bc99-e2a9089e423e
Feb 02 08:44:06 ubuntu-xenial-infracloud-chocolate-7072289 sudo[21125]: pam_unix(sudo:session): session opened for user root by (uid=0)
Feb 02 08:44:07 ubuntu-xenial-infracloud-chocolate-7072289 sudo[21125]: pam_unix(sudo:session): session closed for user root
Feb 02 08:44:07 ubuntu-xenial-infracloud-chocolate-7072289 sudo[21146]: stack : TTY=unknown ; PWD=/opt/stack/new/devstack ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf tgt-admin --show
Feb 02 08:44:07 ubuntu-xenial-infracloud-chocolate-7072289 sudo[21146]: pam_unix(sudo:session): session opened for user root by (uid=0)
Feb 02 08:44:07 ubuntu-xenial-infracloud-chocolate-7072289 kernel: qbrd128337a-f9: port 2(tapd128337a-f9) entered disabled state
Feb 02 08:44:07 ubuntu-xenial-infracloud-chocolate-7072289 kernel: device tapd128337a-f9 left promiscuous mode
Feb 02 08:44:07 ubuntu-xenial-infracloud-chocolate-7072289 kernel: qbrd128337a-f9: port 2(tapd128337a-f9) entered disabled state
Feb 02 08:44:07 ubuntu-xenial-infracloud-chocolate-7072289 virtlogd[16661]: End of file while reading data: Input/output error
                                                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 02 08:44:07 ubuntu-xenial-infracloud-chocolate-7072289 kernel: qbrd128337a-f9: port 1(qvbd128337a-f9) entered disabled state
Feb 02 08:44:07 ubuntu-xenial-infracloud-chocolate-7072289 ovs-vsctl[21190]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=120 -- --if-exists del-port br-int qvod128337a-f9
Feb 02 08:44:08 ubuntu-xenial-infracloud-chocolate-7072289 sudo[21146]: pam_unix(sudo:session): session closed for user root
Feb 02 08:44:08 ubuntu-xenial-infracloud-chocolate-7072289 dnsmasq[18369]: read /opt/stack/data/neutron/dhcp/2d50cc29-221d-49c0-93b1-6022df8ce360/addn_hosts - 5 addresses
Feb 02 08:44:08 ubuntu-xenial-infracloud-chocolate-7072289 dnsmasq-dhcp[18369]: read /opt/stack/data/neutron/dhcp/2d50cc29-221d-49c0-93b1-6022df8ce360/host
Feb 02 08:44:08 ubuntu-xenial-infracloud-chocolate-7072289 dnsmasq-dhcp[18369]: read /opt/stack/data/neutron/dhcp/2d50cc29-221d-49c0-93b1-6022df8ce360/opts
Feb 02 08:44:08 ubuntu-xenial-infracloud-chocolate-7072289 virtlogd[16661]: Cannot open log file: '/var/log/libvirt/qemu/instance-0000001c.log': Device or resource busy
                                                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 02 08:44:08 ubuntu-xenial-infracloud-chocolate-7072289 virtlogd[16661]: End of file while reading data: Input/output error
Feb 02 08:44:08 ubuntu-xenial-infracloud-chocolate-7072289 kernel: qbr3e6aabb5-e7: port 2(tap3e6aabb5-e7) entered disabled state
Feb 02 08:44:08 ubuntu-xenial-infracloud-chocolate-7072289 kernel: device tap3e6aabb5-e7 left promiscuous mode
Feb 02 08:44:08 ubuntu-xenial-infracloud-chocolate-7072289 kernel: qbr3e6aabb5-e7: port 2(tap3e6aabb5-e7) entered disabled state
Feb 02 08:44:09 ubuntu-xenial-infracloud-chocolate-7072289 kernel: device tap3e6aabb5-e7 entered promiscuous mode
Feb 02 08:44:09 ubuntu-xenial-infracloud-chocolate-7072289 kernel: qbr3e6aabb5-e7: port 2(tap3e6aabb5-e7) entered forwarding state
Feb 02 08:44:09 ubuntu-xenial-infracloud-chocolate-7072289 kernel: qbr3e6aabb5-e7: port 2(tap3e6aabb5-e7) entered forwarding state
Feb 02 08:44:09 ubuntu-xenial-infracloud-chocolate-7072289 dnsmasq-dhcp[18369]: DHCPRELEASE(tapa680c36a-5d) 10.1.0.13 fa:16:3e:65:86:a0 unknown lease
Feb 02 08:44:09 ubuntu-xenial-infracloud-chocolate-7072289 dnsmasq[18369]: read /opt/stack/data/neutron/dhcp/2d50cc29-221d-49c0-93b1-6022df8ce360/addn_hosts - 4 addresses
Feb 02 08:44:09 ubuntu-xenial-infracloud-chocolate-7072289 dnsmasq-dhcp[18369]: read /opt/stack/data/neutron/dhcp/2d50cc29-221d-49c0-93b1-6022df8ce360/host
Feb 02 08:44:09 ubuntu-xenial-infracloud-chocolate-7072289 dnsmasq-dhcp[18369]: read /opt/stack/data/neutron/dhcp/2d50cc29-221d-49c0-93b1-6022df8ce360/opts
Feb 02 08:44:09 ubuntu-xenial-infracloud-chocolate-7072289 kernel: sd 4:0:0:1: [sdb] Synchronizing SCSI cache
Feb 02 08:44:10 ubuntu-xenial-infracloud-chocolate-7072289 kernel: IPv6: ADDRCONF(NETDEV_UP): rfp-5a68bbb5-a: link is not ready
Feb 02 08:44:10 ubuntu-xenial-infracloud-chocolate-7072289 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): rfp-5a68bbb5-a: link becomes ready
Feb 02 08:44:11 ubuntu-xenial-infracloud-chocolate-7072289 virtlogd[16661]: End of file while reading data: Input/output error
Feb 02 08:44:11 ubuntu-xenial-infracloud-chocolate-7072289 iscsid[651]: Connection3:0 to [target: iqn.2010-10.org.openstack:volume-3b79dc82-3efc-4312-87bb-39d78dd4f475, portal: 15.184.70.50,3260] through [iface: default] is shutdown.

[1] http://logs.openstack.org/24/425924/2/check/gate-tempest-dsvm-neutron-dvr-ubuntu-xenial/4dd4d3c/logs/syslog.txt.gz#_Feb_02_08_44_35