Include ext4 defragmenting tool

Thank you for taking the time to request this package and helping to make Ubuntu better. Unfortunately you have not provided enough information for a developer to start packaging this application. Please make sure this bug meets the guidelines at https://wiki.ubuntu.com/UbuntuDevelopment/NewPackages. An example of a complete package request is available at https://wiki.ubuntu.com/UbuntuDevelopment/NewPackages/ExamplePackageRequest.

I would love to see any with a GUI that gives a report :)

I think :) this would go into e2fsprogs.

We won't be including any such tool until it's stable and integrated into e2fsprogs upstream. It would be far, far worse to ship a broken version of such a tool than to ship no defragmentation utility at all. (Besides, our kernel doesn't have the online defragmentation patch yet; I gather this is quite recent work.)

While I'm going to leave this bug open since otherwise I imagine we'll just get another copy of it, please do not make further requests here; this is entirely the wrong place for this kind of request. The work needs to happen upstream and *then* be integrated into Ubuntu.

The online defrag patch isn't ready for prime-time yet. It's getting much better, but it's still not at the point where I'm comfortable merging it into the mainline kernel yet, and the userspace API's are still not yet set into stone. So the userspace component is not going into e2fsprogs upstream until I'm satisfied the kernel piece is ready for mainline. The current status is that I gave the ext4 defrag developers some comments, and I haven't heard back from them in a few weeks. Probably time for me to ping them again....

Is there any other tracker we can use to see the status of this integration?

Just a quick comment if I may...

Reading the many discussions/forums etc. concerning ext4, I believe one of the MAIN reasons that folk are looking to get a defrag tool running is to 'refresh' files on their disk having moved from ext3 to ext4, rather than worrying about the (minimal) benefits of a defrag itself. Indeed, that is why I came across this thread because I upgraded from Ubuntu 8.10 to 9.04, switched to ext4 and then wanted to refresh all of my files to the new format - apparently the most effective way to do this is using a defrag tool.

Maybe if there was an alternate (and 'safe/easy') way to refresh all files on the disk to ext4 most folk would be happy and maybe wait for 9.10KK to get a completely stable defrag installation?

I don't think there is any simpler way to do this than to use the defrag itself. I'm ok with waiting until it's been tested enough to become part of e2fsprogs which likely won't be around until 9.10. That's ok though.

I know this is a big wishlist item, but I doubt it'll become part of 9.04 since it's been released and that will be a big testing item.

Is there any update on the state of defrag with ext4 in 9.10?

On Fri, 2009-08-14 at 09:04 +0000, Mark Shuttleworth wrote:

> Is there any update on the state of defrag with ext4 in 9.10?
>
See Ted Ts'o's comments above.

Scott
--
Scott James Remnant
<email address hidden>

Actually, the defrag stuff is in the current kernel now and e4defrag is checked into the e2fsprogs repo. I've tried it and it works.

On Fri, 2009-08-14 at 10:20 +0000, Brian Rogers wrote:

> Actually, the defrag stuff is in the current kernel now and e4defrag is
> checked into the e2fsprogs repo. I've tried it and it works.
>
Good to know; has there been a release yet?

Scott
--
Scott James Remnant
<email address hidden>

No release yet. e4defrag comes after the latest tag, v1.41.8.

The defragmentation support in ext4 is still a work in progress. The current e4defrag essentially creates a new file that is the same size is the existing file, checks to see if it is less fragmented, and if so, invokes a ioctl which atomically substitutes the blocks from the new file to the new file, while preserving the contents of the file.

It however, doesn't have any concept trying to make the file system better from a global point of view, in terms of (a) making sure the free space is not fragmented so as to avoid the file system becoming fragmented again in the near future, and (b) using a global file system wide strategy to move files aside to "make room" for a large fragmented file. It also hasn't undergone enough testing that it's really a good idea to recommend its use in production settings.

There are kernel patches that will provide more functionality to enable a more intelligent userspace program, but (a) those patches still need to be evaluated, so they are not yet in mainline, (b) the enhancements to improve e4defrag to take advange of these new ioctls have not yet been written, and that work is non-trivial, and (c) there is still work needed to improve ext4's base allocation algorithms to enable the current e4defrag to work more efficiently and to prevent fragmentation of the free space to avoid fragmentation problems on full filesystems in the first place.

I've been working on (c), and there are patches in the ext4 patch queue that address the last point. But quite frankly, e4defrag work is somewhat lower priority compared to another things that that we have been working on as of late, such as, integrating the 64-bit block number support into e2fsrpogs' mainline, fixing direct I/O into preallocated blocks, and so on. Some of these items are less important to Ubuntu, as a desktop distribution but they are important to the enterprise server companies and distributions that have been donating engineering effort to ext4. If Canonical is interested in pushing ext4 defragmentation support and contributing some real kernel and userspace engineering support to accelerate this project, please contact me directly. Otherwise it will keep chugging along, but it will be a while before it is really something I would recommend for end-user use.

Thanks Ted. Even if it's not ready for heavy production use, it sounds reasonable to have it included so adventurous users can test it out. Perhaps that will catalyze work on tools to visualise the level of fragmentation, and in turn tools to improve the defrag.

As long as adventurous users realize that e4defrag might destroy their data, sure. There has been some discussion about initializing requiring e4defrag to check for the existence of an environment variable, "I_KNOW_E4DEFRAG_MAY_DESTORY_MY_DATA_AND_WILL_DO_BACKUPS_FIRST" before it will run.

One of the big differences between filesystem utilities and other programs is that failures with say, a cr*p proprietary video driver that causes a system crash whenever you exit a game is after you reboot, little harm is done. But if e4defrag ends up destroying data, users tend to get awfully cranky.

Eric Sandeen from Red Hat has contributed code to a test suite to provide very basic defragmentation testing, but making sure e4defrag does the right thing under all circumstances (i.e., sparse files, files with preallocation space, files which I/O is taking place at the same time as the defragmentation, etc.) still needs to be added.

As far as tools to visualize the level of fragmentation, Andreas Dilger from Sun Microsystems has already donated a tool to help visualize the level of free-space fragmentation, e2freefrag, which is already in the git mainline sources.

One thing comes to my mind with respect to people trying this out. I recall reading that e4defrag was going be the proposed way to convert files to using extents after migrating from ext3. But I get "Operation not supported" when I try to defrag files older than the date of migration, so it looks like e4defrag doesn't work on non-extents files yet (that or something strange is going on).

Testers should be aware of this so they won't expect to be able to migrate to ext4 then immediately run e4defrag and have it work. If you migrated from ext3, the older files won't defrag.

You can convert an individual file to use extents by using "chattr +e file1 [...]"; in theory the ioctl used by chattr +e could be added to e4defrags. This functionality was added to chattr after e2fsprogs 1.41.6. The kernel code to support the ioctl has been in the kernel for a long time, but it hasn't gotten much testing. The same thing applies in terms of potential for data loss. It's been lightly tested, but hasn't received a major shakedown, mainly because it hasn't been a high priority. If someone would be willing to do some heavy testing of what happens if the file is sparse, or is being written to at the time, etc., for both chattr +e and e4defrag, or better yet could write some test code which could be added to a regression test suite, please contact the linux-ext4 mailing list at vger.kernel.org. The test effort would be greatly appreciated.

I was trying to follow Ted's guidance on this and couldn't get chattr to add the extent option to the files. Ted told me by email to do this:

 tune2fs -O extents,uninit_bg,dir_index /dev/yourfilesystem
 e2fsck -pDf /dev/yourfilesystem

(it required booting from a USB stick so I could access the filesystem while not running from it). The correct e2fsprogs is in Karmic.

chattr +e works for me on up-to-date Karamic.

This feature request is now almost one year old.
xfs support extents and defrag since long time, but it doesn't support shrink operations.
ext4 support shrink and extents, but doesn't have defrag.
What will come first? Defrag on EXT4 or shrink on XFS?

From a user perspective, unfortunately, we don't need 10 filesystems doing each one, one different things. We need one filesystem doing all these 10 things.

@Theodore: why you wrote "crap *proprietary* video driver"? Frankly, but really frankly, as long as I have seen "crap video drivers" have been usually opensource. Even the Intel - which is not reverse-engineered - is crap compared to nvidia driver and nvidia tools.

@maxadamo,

E2defrag should hopefully be releasable in Karmic+1. There are still a lot of bugs that are still being fixed in the defrag code, some of which could cause data loss. They work fine if the system isn't under stress, sure, but acid test is to make sure things work OK even when the system is under memory pressure and swapping heavily, or when the file is being actively modified at the point where the defrag takes place. The kernel patches are there, and e2defrag is in the upstream development tree (which hasn't been released yet). So it will happen, but it's always been a matter of time.

The Intel driver has never crashed on me. The Nvidia driver was crashing regularly --- users were reporting regular crashes while using it during the Januty development cycle. Unfortunately, many Ubuntu users are Windows refugees, so they see nothing wrong with the system locking up whenever they exited a 3D game (I think it was World of Goo in one case, if memory serves correctly). That is simply totally unacceptable.

I'm old school, which means job #1 for file systems and video drivers is that they don't lose data and they don't cause your system to crash. It's not about "number of file system features", or "hundreds of FPS when playing game X".

I've recently been looking into this a little more. The kernel support is there. The development is there. It sounds like it's still planned to be available in 10.04.

I did some testing and reading. A migration to a fresh Ext4 partition seems to yield better performance than the e4defrag. Of course ymmv.

If you want defrag for performance, it may be a good idea to check and see if your file system that would benefit from this.
sudo e2fsck -nfv /dev/sdax
The worst I had was 1.1% file fragmentation and 0.6% on the same 100MB partition. Considering this is where I drop all my custom kernels and do a lot of altering here, that's pretty darned low. I wouldn't even consider wasting my time with a defrag of ext4 unless I'm running over 3.0% which is still pretty dang low.

I am excited to see how this and the other features progress. Ext4 has been awesome when it comes to my servers.

@Michael, I hope I had at least 3% of fragmentation )

Just performed a check on my 1TB partition (I use it in my Gentoo home server).
The partition is one year and one month old
917GB total, 871GB used, 37GB free, 10GB reserved
it is used as storage for audio and video files and for rtorrent.

$ sudo /sbin/tune2fs -l /dev/sda1
tune2fs 1.41.9 (22-Aug-2009)
...
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
...
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 965
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Flex block group size: 16

$ sudo fsck.ext4 -nv /dev/sda1
e2fsck 1.41.9 (22-Aug-2009)
...
   43007 inodes used (0.07%)
   15990 non-contiguous files (37.2%)
...
   39260 regular files
    3736 directories

37.2% is very high level of fragmentation I suppose, so defragmentation support is much awaited

do it.... please :-)
This issue is still there from one year and I would like to migrate my XFS filesystem.
This feature was added to Veritas filesystem, during the pleistocene epoch, and a little bit later, after the the firt glaciation, was seen first time in XFS.

On Wed, 2010-02-10 at 02:08 +0000, Michael Lustfield wrote:

> I've recently been looking into this a little more. The kernel support
> is there. The development is there. It sounds like it's still planned to
> be available in 10.04.
>
There is no plan for this to be made available until e2fsprogs upstream
feels sufficiently happy about the tool to include it as standard in
their releases.

Scott
--
Scott James Remnant
<email address hidden>

When I take a look at this thread, I see a lot of ppl expecting too much of a defragmentation tool. Please keep in mind that almost all linux file systems will try to avoid fragmentation by themselves.

If you're really concerned or running a file server with a lot of small files (like e-mail), take a look at this article:
http://www.h-online.com/open/features/Fragmentation-746701.html

@Serhii
To measure *real* fragmentation, use their tool, posted in the german article: http://www.heise.de/open/artikel/Fragmentierung-224380.html - see their per script. I'm pretty sure tune2fs doesn't exclude device files etc.

Everyone else who is migrating to ext4 should stay with cp -a (or similar) until Ubuntu ships e2fsprogs.

@ov3rcl0ck: pelase don't change the bug status without at least explaining why you're doing it. (Triaged is the correct status here)

@bmhm: I am not pretty sure if you're right.
First, I would state that this linux is not a kind of thing where everything is magic, and so fa and so on, and file self-manage fragmentation and filesystem never fragment.
Veritas and XFS, have a defrag tool from a long time.
As far fragmentation is concerned, if I am not wrong, all journaling filesystems have fragmentation issues. And Linux is not different.
What you're saying (again: if I am not wrong) was true for paging filesystems, where each file had an offset to the end. Furthermore, a paging filesystem almost full, was fragmenting as well (because all offset were going to be filled).
I suspect the structure of modern filesystems is completely different and we have fragmentation issue.

I recently migrated from XFS to ext4 and the FS is pretty good.
What I am missing is:
1. such tool to defragment FS
2. xfsdump, which was a very good tool for backups.

yes, every widely used "normal" filesystem fragments.
maxadamo, there actually do exist the program "dump" - I use that to get fast backups.

Does anyone know if Akira Fujita on the LinuxCon (http://events.linuxfoundation.org/2010/linuxcon-japan/fujita) enlighted something about plans and future of ext4 online defrag?

Here are the slides: http://events.linuxfoundation.org/slides/2010/linuxcon_japan/linuxcon_jp2010_fujita.pdf

Cool, the e4defrag built with those patches seems to work fine. The check function (-c) on my root partition reported a fragmentation score of zero (I guess it must be an average) but did find a number of fragmented files, eg

<Fragmented files> now/best size/ext
1. /var/log/pm-powersave.log 8/1 4 KB
Total/best extents 8/1
 Average size per extent 4 KB
 Fragmentation score 98
 [0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag]

Running e4defrag on this file resulted in:

<File> now/best size/ext
/var/log/pm-powersave.log 1/1 32 KB
 Total/best extents 1/1
 Average size per extent 32 KB
 Fragmentation score 0
 [0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag]

If I run it on a directory, it reports a number of 'successes' and 'failures'. I suspect the failures are for files that are locked (wpa_supplicant was one such file), ie not all files can be defragged online.

The "relevant files defrag" always reports a 100% failure rate for some reason.

Where do you get the e4defrag source?

The source for e4defrag is in e2fsprogs source with:

git clone http://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git

Build/install instructions are in the INSTALL file. Note that e4defrag is built in the misc folder but doesn't install with the rest when you do a sudo make install.

There are also patches for the relevant file defragmentation mentioned in the slides (comment #32), ie at http://marc.info/?l=linux-ext4&m=128272678710600&w=2. The first three are for the kernel (it looks like they are based on 2.6.34 and didn't cleanly apply to my 2.6.37-rc1 kernel so I didn't apply them, which is likely why -r kept failing for me) and the last two are for e2fsprogs. I guess you don't need them if you are just trying out the file/directory defrag. From the slides, it looks like file/dir defrag just allocates a contiguous space for the defrag file and moves it into this space.

BTW, I don't recommend running sudo make install on e2fsprogs. When I attempted to reboot, the system didn't recognise the new fsck and hence would only mount root in read-only mode. (Oddly enough, fsck still ran fine.) I had to boot from a live CD, chroot to my broken partition, and reinstall e2fsprogs and util-linux using apt-get. e4defrag still runs fine by itself.

For the time being the following two commands can at least reduce the fragmentation of files:
  cp -a
  e2fsck -D

I found another tool for online defragmentation - shake-fs, see: http://vleu.net/shake/
There are packages for Ubuntu and Debian at http://www.vleu.net/apt/ and https://launchpad.net/~un-brice/+archive/ppa

Shake is a defragmenter that runs in userspace, without the need of patching the kernel and while the system is used (for now, on GNU/Linux only).
There is nothing magic in that : it just works by rewriting fragmented files. But it has some heuristics that could make it more efficient than other tools, including defrag (see ck.kolivas.org/apps/defrag/) and, maybe, xfs_fsr.

For an impression of the fragmentation of the free space of the filesystem:
usedblocks /dev/...

AFAICS e4defrag is now included in e2fsprogs of precise as of a few months ago, so I guess this can be closed for Ubuntu?

Which is indeed the case
$ apt-cache policy e2fsprogs
e2fsprogs:
  Installed: 1.42~WIP-2011-10-16-1ubuntu1
  Candidate: 1.42~WIP-2011-10-16-1ubuntu1
  Version table:
 *** 1.42~WIP-2011-10-16-1ubuntu1 0
        500 http://ftp-stud.hs-esslingen.de/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status
$ dpkg -L e2fsprogs | grep e4defrag
/usr/sbin/e4defrag
/usr/share/man/man8/e4defrag.8.gz

I guess precise isn't precisly released.

Does this mean that as usual, we won't get this for the last stable release (currently oneiric) nor for the last LTS (currently Lucid)? Will it get backported or we really must rush with the OS upgrade if we want it?

Oh, and concerning ext4 fragmentation, I find it strange that so much fragmentation as cited above... The most fragmented ext4 FS I have is about 8%, and strangely, another one which is almost full and its content moderately often changed, stays low at about 2% even though I've expected much more.

I believe such a low degree of fragmentation I have is due to Extents, and I have specifically enabled them for my ext4 filesystems. Perhaps in those cases with higher ext4 fragmentation extents are turned off? I'm not sure are they enabled by default, but I believe they are disabled if the FS is upgraded from ext3, so I'd suggest anyone to manually enable it (in fstab, perhaps also with tune2fs); I'd recommend rebooting to single mode immediately after that, and then manually fsck all filesystems for which extents were turned on. And remember, backup often ;)

Also, when extents are turned on, perhaps running the Defrag script or Shake would help lower the fragmentation...

As always, if you want new features, you upgrade to the new release.

Also extents are enabled by default, but if the fs was upgraded from ext3, then all of the files that already existed will not be using them. You can see whether a file is using extents with lsattr.