Fuel for OpenStack

O_DIRECT writes not works for XFS on disks with 4K sectors

  1. Series 5.0.x
  2. Bug #1316266
Bug #1316266 reported by Roman Sokolkov
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Vladimir Kozhukalov
Fuel for OpenStack 5.0
4.1.x
Fix Committed
High
Vladimir Kuklin
Fuel for OpenStack 4.1.1
5.0.x
Fix Committed
High
Vladimir Kozhukalov
Fuel for OpenStack 5.0

Bug Description

Description:
File injection fails with http://paste.openstack.org/show/79000/ in /var/log/nova-all.log

Environment:
- Fuel 4.1A
- HA + Neutron GRE
- CentOS
- Cisco C240 M3S servers + LSI Logic / Symbios Logic MegaRAID SAS 2208

Steps to reproduce:

- Use disk with 4K sectors for "Virtual Storage"(/var/lib/nova).
- Deploy Cluster on CentOS (Simple or HA no matter)
- try to create VM with file injection: nova boot --flavor 1 --image TestVM --file file=openc VM1

Expected result:
- VM status ACTIVE

Actual result:
- VM status ERROR

Details:
Disk has logical sector=512b and physical=4096b
XFS uses sector size = 4096 bytes .
And unfortunately in that case XFS unable to provide O_DIRECT writes. (that are used by guestfs - "cache=none" option)
I've found fresh patch in mailing lists that works around this issue http://oss.sgi.com/archives/xfs/2014-01/msg00255.html .

To check issue:

guestfish -a /var/lib/nova/instances/627a3ce2-0f4a-4ba3-9f4b-2859861fbef1/disk -i

libguestfs: error: guestfs_launch failed.
See http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
and/or run 'libguestfs-test-tool'.

or

dd if=/var/lib/nova/instances/627a3ce2-0f4a-4ba3-9f4b-2859861fbef1/disk of=/dev/null iflag=direct
dd: reading `/var/lib/nova/instances/627a3ce2-0f4a-4ba3-9f4b-2859861fbef1/disk': Invalid argument
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000131719 s, 0.0 kB/s

# parted /dev/sdj print
Model: LSI MR9271-8i (scsi)
Disk /dev/sdj: 1797GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number Start End Size File system Name Flags
 1 17.4kB 25.2MB 25.1MB primary bios_grub
 2 25.2MB 235MB 210MB primary boot
 3 235MB 445MB 210MB ext2 primary boot
 4 445MB 1797GB 1796GB primary lvm

# xfs_info /var/lib/nova/
meta-data=/dev/mapper/vm-nova isize=256 agcount=32, agsize=13705216 blks
         = sectsz=4096 attr=2, projid32bit=0
data = bsize=4096 blocks=438566912, imaxpct=5
         = sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=214144, version=2
         = sectsz=4096 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0

Workaround:

On live system:
- recreate XFS with sector size=512

/etc/init.d/openstack-nova-compute stop
cd /var/lib/
cp -a nova nova2
umount nova
mkfs.xfs -f -s size=512 /dev/vm/nova
mount -a
cp -a nova2/* nova
/etc/init.d/openstack-nova-compute start

In fuel:
use ext4 for "virtual storage"

In nailgun/fixtures/openstack.yaml set file_system: "ext4" for virtual storage volume.

Roman Sokolkov (rsokolkov)
tags: added: nailgun
Mike Scherbakov (mihgen)
Changed in fuel:
milestone: none → 5.0
tags: added: backports-4.1.1
Changed in fuel:
assignee: nobody → Vladimir Kozhukalov (kozhukalov)
Vladimir Kuklin (vkuklin)
Changed in fuel:
importance: Undecided → Medium
milestone: 5.0 → 5.1
tags: added: release-notes
Mike Scherbakov (mihgen)
Changed in fuel:
milestone: 5.1 → 5.0
importance: Medium → High
Vladimir Kozhukalov (kozhukalov)
Changed in fuel:
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/92849

Changed in fuel:
status: Triaged → In Progress
Sergii Golovatiuk (sgolovatiuk)
Changed in fuel:
assignee: Vladimir Kozhukalov (kozhukalov) → Sergii Golovatiuk (sgolovatiuk)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/92849
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5128535954a7aa1c456fc3ae31396141660f0383
Submitter: Jenkins
Branch: master

commit 5128535954a7aa1c456fc3ae31396141660f0383
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu May 8 18:45:53 2014 +0400

    Set xfs sector size into 512

    By default mkfs.xfs sets sector size equal to
    physical block size which makes it impossible to have
    O_DIRECT mode. Enforcing xfs to use 512 byte size
    fixes this problem but decreases fs performance a bit.

    Change-Id: I4b38ef148c16baacfba12ed07534a3761b3cecf1
    Related-Bug: #1316266

Revision history for this message
Roman Sokolkov (rsokolkov) wrote :

I'm using latest 5.0 iso and noticed that anaconda failed to create XFS filesystem

post-partiton.log
mkfs.xfs -f -s 512 /dev/mapper/image-glance
+ mkfs.xfs -f -s 512 /dev/mapper/image-glance
unknown option -s 512
....

Options should be:

"-f -s size=512"

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/93535

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/93535
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=358ead05d6743ac5cbdb5437868d45e9a6707a97
Submitter: Jenkins
Branch: master

commit 358ead05d6743ac5cbdb5437868d45e9a6707a97
Author: Mike Scherbakov <email address hidden>
Date: Wed May 14 09:54:37 2014 +0400

    Fixed incorrect params to mkfs.xfs

    Thanks to Roman Sokolkov for regression report.
    Related-Bug: #1316266

    Change-Id: Ifd51891d18d2c61724b169785cc59debd52b4db0

Mike Scherbakov (mihgen)
Changed in fuel:
status: In Progress → Fix Committed
Sergii Golovatiuk (sgolovatiuk)
Changed in fuel:
assignee: Sergii Golovatiuk (sgolovatiuk) → Vladimir Kozhukalov (kozhukalov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/93726

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/93726
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=96a614f0a0ed85482cb47bc3491d8699736b74e8
Submitter: Jenkins
Branch: master

commit 96a614f0a0ed85482cb47bc3491d8699736b74e8
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu May 15 17:37:11 2014 +0400

    Fixed incorrect params to mkfs.xfs for ubuntu

    Thanks to Roman Sokolkov for regression report.
    Related-Bug: #1316266

    Change-Id: Id67ab7f1d47d62c3e3d82a4bca4fa538132d25db

Revision history for this message
Meg McRoberts (dreidellhasa) wrote :

Added note to "Fixed Issues" in 5.0 Release Notes.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (stable/4.1)

Related fix proposed to branch: stable/4.1
Review: https://review.openstack.org/96851

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: stable/4.1
Review: https://review.openstack.org/96855

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: stable/4.1
Review: https://review.openstack.org/96856

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (stable/4.1)

Reviewed: https://review.openstack.org/96851
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=66825b50f279e281c4d2ef1c2cb4899a1c47a085
Submitter: Jenkins
Branch: stable/4.1

commit 66825b50f279e281c4d2ef1c2cb4899a1c47a085
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu May 8 18:45:53 2014 +0400

    Set xfs sector size into 512

    By default mkfs.xfs sets sector size equal to
    physical block size which makes it impossible to have
    O_DIRECT mode. Enforcing xfs to use 512 byte size
    fixes this problem but decreases fs performance a bit.

    Change-Id: I4b38ef148c16baacfba12ed07534a3761b3cecf1
    Related-Bug: #1316266

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/96855
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=94a06e8d64d33de6c50cc18cf17ba66cb692669e
Submitter: Jenkins
Branch: stable/4.1

commit 94a06e8d64d33de6c50cc18cf17ba66cb692669e
Author: Mike Scherbakov <email address hidden>
Date: Wed May 14 09:54:37 2014 +0400

    Fixed incorrect params to mkfs.xfs

    Thanks to Roman Sokolkov for regression report.
    Related-Bug: #1316266

    Change-Id: Ifd51891d18d2c61724b169785cc59debd52b4db0

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/96856
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=270bf259e3e109fbb8921e6c2cc992e1103fb06a
Submitter: Jenkins
Branch: stable/4.1

commit 270bf259e3e109fbb8921e6c2cc992e1103fb06a
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu May 15 17:37:11 2014 +0400

    Fixed incorrect params to mkfs.xfs for ubuntu

    Thanks to Roman Sokolkov for regression report.
    Related-Bug: #1316266

    Change-Id: Id67ab7f1d47d62c3e3d82a4bca4fa538132d25db

Revision history for this message
Meg McRoberts (dreidellhasa) wrote :

Documented as fixed in 5.0 and 4.1.1 Release Notes

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.