OpenStack Compute (nova)

Comment 0 for bug 2004555

Revision history for this message

Jan Wasilewski (janwasilewski) wrote on 2023-02-02: [ussuri] Wrong volume attachment - volumes overlapping when connected through iscsi on host

Hello OpenStack Security Team,

I’m writing to you, as we faced a serious security breach in OpenStack functionality(correlated a bit with libvirt, iscsi and huawei driver). I was going through OSSA documents and correlated libvirt notes, but I couldn't find something similar. It is not related to https://security.openstack.org/ossa/OSSA-2020-006.html

In short: we observed that newly created cinder volume(1GB size) was attached to compute node instance, but an instance recognized it as a 115GB volume, which(this 115GB volume) in fact was connected to another instance on the same compute node.

# dpkg -l | grep multipath
ii multipath-tools 0.8.3-1ubuntu2 amd64 maintain multipath block device access

# dpkg -l | grep iscsi
ii libiscsi7:amd64 1.18.0-2 amd64 iSCSI client shared library
ii open-iscsi 2.0.874-7.1ubuntu6.2 amd64 iSCSI initiator tools

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"

Instance OS: Debian-11-amd64

[2. Test scenario]
Already created instance with two volumes attached. First - 10GB for root system, second - 115GB used as vdb. Recognized by compute node as vda - dm-11, vdb - dm-9:

# virsh domblklist 90fas439-fc0e-4e22-8d0b-6f2a18eee5c1
Target Source
----------------------
vda /dev/dm-11
vdb /dev/dm-9

# multipath -ll
(...)
36e00084100ee7e7ed6ad25d900002f6b dm-9 HUAWEI,XSG1
size=115G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 14:0:0:4 sdm 8:192 active ready running
  |- 15:0:0:4 sdo 8:224 active ready running
  |- 16:0:0:4 sdl 8:176 active ready running
  `- 17:0:0:4 sdn 8:208 active ready running
(...)
36e00084100ee7e7ed6acaa2900002f6a dm-11 HUAWEI,XSG1
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 14:0:0:3 sdq 65:0 active ready running
  |- 15:0:0:3 sdr 65:16 active ready running
  |- 16:0:0:3 sdp 8:240 active ready running
  `- 17:0:0:3 sds 65:32 active ready running

Creating a new instance, with the same OS guest system and 10GB root volume. After successful deployment, creating a new volume(1GB) size and attaching it to newly created instance. We can see after that:

# multipath -ll
(...)
36e00084100ee7e7ed6ad25d900002f6b dm-9 HUAWEI,XSG1
size=115G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 14:0:0:10 sdao 66:128 failed faulty running
  |- 14:0:0:4 sdm 8:192 active ready running
  |- 15:0:0:10 sdap 66:144 failed faulty running
  |- 15:0:0:4 sdo 8:224 active ready running
  |- 16:0:0:10 sdan 66:112 failed faulty running
  |- 16:0:0:4 sdl 8:176 active ready running
  |- 17:0:0:10 sdaq 66:160 failed faulty running
  `- 17:0:0:4 sdn 8:208 active ready running

This way at instance we were able to see a new drive - not 1GB, but 115GB -> so it seems it was incorrectly attached and this way we were able to destroy some data on that volume.

Additionaly we were able to see many errors like that in compute node logs:

# dmesg -T | grep dm-9
[Fri Jan 27 13:37:42 2023] blk_update_request: critical target error, dev dm-9, sector 62918760 op 0x1:(WRITE) flags 0x8800 phys_seg 2 prio class 0
[Fri Jan 27 13:37:42 2023] blk_update_request: critical target error, dev dm-9, sector 33625152 op 0x1:(WRITE) flags 0x8800 phys_seg 6 prio class 0
[Fri Jan 27 13:37:46 2023] blk_update_request: critical target error, dev dm-9, sector 66663000 op 0x1:(WRITE) flags 0x8800 phys_seg 5 prio class 0
[Fri Jan 27 13:37:46 2023] blk_update_request: critical target error, dev dm-9, sector 66598120 op 0x1:(WRITE) flags 0x8800 phys_seg 5 prio class 0
[Fri Jan 27 13:37:51 2023] blk_update_request: critical target error, dev dm-9, sector 66638680 op 0x1:(WRITE) flags 0x8800 phys_seg 12 prio class 0
[Fri Jan 27 13:37:56 2023] blk_update_request: critical target error, dev dm-9, sector 66614344 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[Fri Jan 27 13:37:56 2023] blk_update_request: critical target error, dev dm-9, sector 66469296 op 0x1:(WRITE) flags 0x8800 phys_seg 24 prio class 0
[Fri Jan 27 13:37:56 2023] blk_update_request: critical target error, dev dm-9, sector 66586472 op 0x1:(WRITE) flags 0x8800 phys_seg 3 prio class 0
(...)

Unfortunately we do not know what is a perfect test-scenario for it as we could face such issue in less than 2% of our tries, but it looks like a serious security breach.

Additionally we observed that linux kernel is not fully clearing a device allocation(from volume detach), so some of drives names are visible in an output, i.e. lsblk command. Then, after new volume attachment we can see such names(i.e. sdao, sdap, sdan and so on) are reusable by that drive and wrongly mapped by multipath/iscsi to another drive and this way we hit an issue.
Our question is why linux kernel of compute node is not removing devices allocation and this way is leading to a scenario like that? Maybe this can be a solution here.

Thanks in advance for your help and understanding. In case when more details is needed, do not hesitate to contact me.

Hello OpenStack Security Team,
 
I’m writing to you, as we faced a serious security breach in OpenStack functionality(correlated a bit with libvirt, iscsi and huawei driver). I was going through OSSA documents and correlated libvirt notes, but I couldn't find something similar. It is not related to https://security.openstack.org/ossa/OSSA-2020-006.html

[1. Test environment]
Compute node: OpenStack Ussuri configured with Huawei dorado as a storage backend(configuration driver is available here: https://docs.openstack.org/cinder/rocky/configuration/block-storage/drivers/huawei-storage-driver.html)
Packages:
v# dpkg -l | grep libvirt
ii  libvirt-clients                       6.0.0-0ubuntu8.16                                    amd64        Programs for the libvirt library
ii  libvirt-daemon                        6.0.0-0ubuntu8.16                                    amd64        Virtualization daemon
ii  libvirt-daemon-driver-qemu            6.0.0-0ubuntu8.16                                    amd64        Virtualization daemon QEMU connection driver
ii  libvirt-daemon-driver-storage-rbd     6.0.0-0ubuntu8.16                                    amd64        Virtualization daemon RBD storage driver
ii  libvirt-daemon-system                 6.0.0-0ubuntu8.16                                    amd64        Libvirt daemon configuration files
ii  libvirt-daemon-system-systemd         6.0.0-0ubuntu8.16                                    amd64        Libvirt daemon configuration files (systemd)
ii  libvirt0:amd64                        6.0.0-0ubuntu8.16                                    amd64        library for interfacing with different virtualization systems
ii  nova-compute-libvirt                  2:21.2.4-0ubuntu1                                    all          OpenStack Compute - compute node libvirt support
ii  python3-libvirt                       6.1.0-1                                              amd64        libvirt Python 3 bindings

# dpkg -l | grep qemu
ii  ipxe-qemu                             1.0.0+git-20190109.133f4c4-0ubuntu3.2                all          PXE boot firmware - ROM images for qemu
ii  ipxe-qemu-256k-compat-efi-roms        1.0.0+git-20150424.a25a16d-0ubuntu4                  all          PXE boot firmware - Compat EFI ROM images for qemu
ii  libvirt-daemon-driver-qemu            6.0.0-0ubuntu8.16                                    amd64        Virtualization daemon QEMU connection driver
ii  qemu                                  1:4.2-3ubuntu6.23                                    amd64        fast processor emulator, dummy package
ii  qemu-block-extra:amd64                1:4.2-3ubuntu6.23                                    amd64        extra block backend modules for qemu-system and qemu-utils
ii  qemu-kvm                              1:4.2-3ubuntu6.23                                    amd64        QEMU Full virtualization on x86 hardware
ii  qemu-system-common                    1:4.2-3ubuntu6.23                                    amd64        QEMU full system emulation binaries (common files)
ii  qemu-system-data                      1:4.2-3ubuntu6.23                                    all          QEMU full system emulation (data files)
ii  qemu-system-gui:amd64                 1:4.2-3ubuntu6.23                                    amd64        QEMU full system emulation binaries (user interface and audio support)
ii  qemu-system-x86                       1:4.2-3ubuntu6.23                                    amd64        QEMU full system emulation binaries (x86)
ii  qemu-utils                            1:4.2-3ubuntu6.23                                    amd64        QEMU utilities

# dpkg -l | grep nova
ii  nova-common                           2:21.2.4-0ubuntu1                                    all          OpenStack Compute - common files
ii  nova-compute                          2:21.2.4-0ubuntu1                                    all          OpenStack Compute - compute node base
ii  nova-compute-kvm                      2:21.2.4-0ubuntu1                                    all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt                  2:21.2.4-0ubuntu1                                    all          OpenStack Compute - compute node libvirt support
ii  python3-nova                          2:21.2.4-0ubuntu1                                    all          OpenStack Compute Python 3 libraries
ii  python3-novaclient                    2:17.0.0-0ubuntu1                                    all          client library for OpenStack Compute API - 3.x

# dpkg -l | grep multipath
ii  multipath-tools                       0.8.3-1ubuntu2                                       amd64        maintain multipath block device access

# dpkg -l | grep iscsi
ii  libiscsi7:amd64                       1.18.0-2                                             amd64        iSCSI client shared library
ii  open-iscsi                            2.0.874-7.1ubuntu6.2                                 amd64        iSCSI initiator tools

# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"

Instance OS:  Debian-11-amd64

[2. Test scenario]
Already created instance with two volumes attached. First - 10GB for root system, second - 115GB used as vdb. Recognized by compute node as vda - dm-11, vdb - dm-9:

# virsh domblklist 90fas439-fc0e-4e22-8d0b-6f2a18eee5c1
 Target   Source
----------------------
 vda      /dev/dm-11
 vdb      /dev/dm-9

# multipath -ll
(...)
36e00084100ee7e7ed6ad25d900002f6b dm-9 HUAWEI,XSG1
size=115G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 14:0:0:4  sdm  8:192  active ready running
  |- 15:0:0:4  sdo  8:224  active ready running
  |- 16:0:0:4  sdl  8:176  active ready running
  `- 17:0:0:4  sdn  8:208  active ready running
(...)
36e00084100ee7e7ed6acaa2900002f6a dm-11 HUAWEI,XSG1
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 14:0:0:3  sdq  65:0   active ready running
  |- 15:0:0:3  sdr  65:16  active ready running
  |- 16:0:0:3  sdp  8:240  active ready running
  `- 17:0:0:3  sds  65:32  active ready running

# multipath -ll
(...)
36e00084100ee7e7ed6ad25d900002f6b dm-9 HUAWEI,XSG1
size=115G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 14:0:0:10 sdao 66:128 failed faulty running
  |- 14:0:0:4  sdm  8:192  active ready running
  |- 15:0:0:10 sdap 66:144 failed faulty running
  |- 15:0:0:4  sdo  8:224  active ready running
  |- 16:0:0:10 sdan 66:112 failed faulty running
  |- 16:0:0:4  sdl  8:176  active ready running
  |- 17:0:0:10 sdaq 66:160 failed faulty running
  `- 17:0:0:4  sdn  8:208  active ready running

This way at instance we were able to see a new drive - not 1GB, but 115GB -> so it seems it was incorrectly attached and this way we were able to destroy some data on that volume.

Additionaly we were able to see many errors like that in compute node logs:

Unfortunately we do not know what is a perfect test-scenario for it as we could face such issue in less than 2% of our tries, but it looks like a serious security breach.

Additionally we observed that linux kernel is not fully clearing a device allocation(from volume detach), so some of drives names are visible in an output, i.e. lsblk command. Then, after new volume attachment we can see such names(i.e. sdao, sdap, sdan and so on) are reusable by that drive and wrongly mapped by multipath/iscsi to another drive and this way we hit an issue. 
Our question is why linux kernel of compute node is not removing devices allocation and this way is leading to a scenario like that? Maybe this can be a solution here.

Thanks in advance for your help and understanding. In case when more details is needed, do not hesitate to contact me.