Amd64 ubuntu build hogs memory due to libs built with excessive alignment requirement

Bug #24692 reported by Phillip Susi
56
This bug affects 5 people
Affects Status Importance Assigned to Milestone
NVIDIA Drivers Ubuntu
Incomplete
Undecided
Unassigned
Ubuntu
Confirmed
High
Ubuntu AMD64 Porting Team

Bug Description

I started a thread on this subject on the ubuntu-devel mailing list, but I
figured I should create a bug to track it here. It seems that there are a
number of badly built libraries in the amd64 build that hog tons of memory.
This results in a freshly booted ubuntu desktop system using 500 out of 1000
megs of ram. A number of processes that should be rather small end up using
~100 megs of ram, for instance, on my system, clock-applet uses 111 megs of ram.

I believe that I have found the cause of the problem to be a number of libraries
that, for some reason, were built with a 1 MB alignment requirement, so when
they are mapped into memory, what should be a small 8 KB mapping becomes a 1 MB
mapping. It appears typical that each library gets 2 of these 1 MB mappings in
each process that loads it, and a number of processes are loading ~40 libraries
with this problem, so ~80 megs of ram is being wasted per process.

One example lib has this objdump:

/usr/lib/libpanel-applet-2.so.0.1.10: file format elf64-x86-64
/usr/lib/libpanel-applet-2.so.0.1.10
architecture: i386:x86-64, flags 0x00000150:
HAS_SYMS, DYNAMIC, D_PAGED
start address 0x0000000000006310

Program Header:
   LOAD off 0x0000000000000000 vaddr 0x0000000000000000 paddr
0x0000000000000000 align 2**20
        filesz 0x000000000000cd8c memsz 0x000000000000cd8c flags r-x
   LOAD off 0x000000000000d000 vaddr 0x000000000010d000 paddr
0x000000000010d000 align 2**20
        filesz 0x00000000000018e0 memsz 0x0000000000001960 flags rw-
DYNAMIC off 0x000000000000d470 vaddr 0x000000000010d470 paddr
0x000000000010d470 align 2**3
        filesz 0x0000000000000420 memsz 0x0000000000000420 flags rw-
EH_FRAME off 0x000000000000c094 vaddr 0x000000000000c094 paddr
0x000000000000c094 align 2**2
        filesz 0x00000000000002ac memsz 0x00000000000002ac flags r--
  STACK off 0x0000000000000000 vaddr 0x0000000000000000 paddr
0x0000000000000000 align 2**3
        filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw-

The 2**20 should be more like 2**3. I'm not sure where/how this package was
built, but it seems it was done wrong, and these packages will all need to be
rebuilt with the correct alignment.

Revision history for this message
Matt Zimmerman (mdz) wrote :
Revision history for this message
Phillip Susi (psusi) wrote :

I am not sure why this was marked as invalid. The discussion on the developer
list kind of stalled, but as far as I can see, the conclusion was that it isn't
as bad as it seems, most of that memory actually isn't being wasted, it's just
reported as such.

This issue still needs resolved because at the very least, it gives the
appearence that a lot of memory is being wasted, which can cause the casual user
to look at ubuntu and decide not to use it because it is "wasting tons of memory".

The tool chains on build daemons or wherever the problem is needs to be fixed so
as not to set the rediculous 1 MB alignment requirement.

Revision history for this message
Andrew Ash (ash211) wrote :

Does the bug still exist in the latest version of (k)ubuntu, Dapper Drake 6.06.1 ? Please try Dapper and give us an update if you can. Thanks.

Revision history for this message
Andrew Ash (ash211) wrote :

Also, can you reproduce it on Edgy?

Revision history for this message
Phillip Susi (psusi) wrote : Re: [Bug 24692] Re: Amd64 ubuntu build hogs memory due to badly built libs

Andrew Ash wrote:
> Also, can you reproduce it on Edgy?

Yes. I just looked through the mem maps of a running xmms process that
is listed as taking 114 MB of Virtual Memory. I found several offending
libs:

/usr/lib/libdrm.so.2.0.0 has a 20 bit alignment requirement in its
program header according to objdump -x. This causes a 1 MB mapping to
be made even though the entire file size is only 31 KB.

/usr/lib/libvorbis.so.0.3.1 is another offender causing a 1 MB mapping
with a total file size of only 172 KB.

/usr/lib/libXfixes.so.3.1.0 is using a meg mapping for a 21 kb lib
/usr/lib/libXi.so.6.0.0 uses a meg for only 33k
/usr/lib/libz.so.1.2.3 wastes an extra meg for an 89 KB lib

You get the idea, there are a lot of them like this. If you see a
"2**20" in the objdump of the file, it has this problem.

Revision history for this message
Phillip Susi (psusi) wrote : Re: Amd64 ubuntu build hogs memory due to badly built libs

I just came up with the following test for effected files and results for /lib. It came up with many duplicates due to the multiple symbolic names for each library though.

root@ubuntu:/lib# for file in `ls *.so.*` ; do if objdump -x $file | grep -q -e '2\*\*20' ; then echo $file ; fi ; done
ld-linux-x86-64.so.2
libacl.so.1
libacl.so.1.1.0
libanl.so.1
libatm.so.1
libatm.so.1.0.0
libattr.so.1
libattr.so.1.1.0
libblkid.so.1
libblkid.so.1.0
libbrlapi.so.0.4
libbrlapi.so.0.4.1
libBrokenLocale.so.1
libbz2.so.1
libbz2.so.1.0
libbz2.so.1.0.3
libcap.so.1
libcap.so.1.10
libcfont.so.0
libcfont.so.0.0.0
libcidn.so.1
libcom_err.so.2
libcom_err.so.2.1
libconsole.so.0
libconsole.so.0.0.0
libcrypt.so.1
libc.so.6
libctutils.so.0
libctutils.so.0.0.0
libdal-0.3.so.0
libdal-0.3.so.0.0.0
libdb1.so.2
libdb.so.2
libdevmapper.so.1.01
libdevmapper.so.1.02
libdiscover.so.1
libdiscover.so.1.0.0
libdl.so.2
libe2p.so.2
libe2p.so.2.3
libevms-2.5.so.0
libevms-2.5.so.0.5
libext2fs.so.2
libext2fs.so.2.4
libgcc_s.so.1
libhandle.so.1
libhandle.so.1.0.3
libhistory.so.4
libhistory.so.4.3
libhistory.so.5
libhistory.so.5.1
libiw.so.28
libm.so.6
libncurses.so.5
libncurses.so.5.5
libncursesw.so.5
libncursesw.so.5.5
libnsl.so.1
libnss_compat.so.2
libnss_dns.so.2
libnss_files.so.2
libnss_hesiod.so.2
libnss_nisplus.so.2
libnss_nis.so.2
libpamc.so.0
libpamc.so.0.79
libpam_misc.so.0
libpam_misc.so.0.79
libpam.so.0
libpam.so.0.79
libparted-1.6.so.12
libparted-1.6.so.12.0.9
libparted-1.6.so.13
libparted-1.6.so.13.11.1
libparted-1.7.so.1
libparted-1.7.so.1.0.0
libpopt.so.0
libpopt.so.0.0.0
libpthread.so.0
libreadline.so.4
libreadline.so.4.3
libreadline.so.5
libreadline.so.5.1
libreiserfs-0.3.so.0
libreiserfs-0.3.so.0.0.0
libresolv.so.2
librt.so.1
libselinux.so.1
libsepol.so.1
libslang.so.1
libslang.so.1.4.9
libslang.so.2
libslang.so.2.0.6
libss.so.2
libss.so.2.0
libsysfs.so.1
libsysfs.so.1.0.3
libsysfs.so.2
libsysfs.so.2.0.0
libthread_db.so.1
libusb-0.1.so.4
libusb-0.1.so.4.4.4
libusplash.so.0
libutil.so.1
libuuid.so.1
libuuid.so.1.2
libvolume_id.so.0
libvolume_id.so.0.65.0
libwrap.so.0
libwrap.so.0.7.6

Revision history for this message
Andrew Ash (ash211) wrote :

I'm sure that info will be handy for the devs. I've subscribed the AMD64 team to this bug, since I don't really have the power or knowledge to fix problems. If anything new comes up, keep us posted.

Thanks

Revision history for this message
Phillip Susi (psusi) wrote :

I don't think this should still be marked as Needs Info, so I am marking as Confirmed.

Revision history for this message
Andrew Ash (ash211) wrote :

Has any work been done on fixing this in feisty?

Revision history for this message
Stéphane Loeuillet (leroutier) wrote :

In current gutsy, /lib is ok (no such problem) but /usr/lib still has libs with this problem.
Some beeing Firewire related (libavc/rom1394), the closed source nvidia driver too.

All impacted packages on my system :
libavc1394-0
libfaac0
libgdbm3
libgnomecups1.0-1
liblzo1
libmng1
libsidplay1
libsoundtouch1c2
libsp1c2
libsphinx2g0
nvidia-glx-new

I suppose those are packages that haven't seen an update in a while, so they were not rebuilt. Or no one reported the problem to their owner.

Revision history for this message
Mendez (ubuntu-mendez) wrote :

Should these be reported as bugs to the individual packages?

I appear to have 31 incorrect ones in /usr/lib on my system.

Revision history for this message
Jim Qode (jimqode) wrote :

In current Hardy alpha 6 the problem in /lib seems to be fixed completely. However still some libraries in /usr/lib have it:

libavc1394.so.0
libavc1394.so.0.3.0
libBPM.so.1
libBPM.so.1.0.0
libgdbm_compat.so.3
libgdbm_compat.so.3.0.0
libgdbm.so.3
libgdbm.so.3.0.0
libmng.so.1
libmng.so.1.1.0.9
librom1394.so.0
librom1394.so.0.3.0
libsidplay.so.1
libsidplay.so.1.0.3
libSoundTouch.so.1
libSoundTouch.so.1.0.0
libtar.so.0
libtar.so.0.0.0

Revision history for this message
Ryan Hamilton (rthamilt) wrote :

I stumbled across this bug looking for something else, but I looked at system monitor on my AMD64 Gusty machine and it seems to report garbage numbers. Screenshot of sys-monitor attached. I'm guessing its somewhat related?

Revision history for this message
Jonathan Thurn (jthurn) wrote :

Ryan,

You're looking for bug no. 148325, "Certain processes show incorrect memory usage when using xgl," a known xserver-xgl issue.

Best,
Jonathan T.

Revision history for this message
Rocko (rockorequin) wrote :

I just ran the test in /usr/lib with Ubutun 8.04 (kernel 2.6.24-19-generic amd64) and get a couple more libs than Jim Quode. The extra ones seem mostly to be the nvidia drivers. I've removed the symlinks from the output:

libavc1394.so.0.3.0
libBPM.so.1.0.0
libcuda.so.173.14.05
libfame-0.9.so.1.0.0
libgdbm_compat.so.3.0.0
libgdbm.so.3.0.0
libGLcore.so.173.14.05
libGL.so.173.14.05
liblzo.so.1.0.0
libmng.so.1.1.0.9
libnvidia-cfg.so.173.14.05
libnvidia-tls.so.173.14.05
librom1394.so.0.3.0
libsidplay.so.1.0.3
libSoundTouch.so.1.0.0
libtar.so.0.0.0
libXvMCNVIDIA.so.173.14.05

Revision history for this message
Simon Ruggier (simon80) wrote :

for file in `find /usr/lib -type f -name '*.so.*'`; do if objdump -x $file | grep -q -e '2\*\*20' ; then echo $file ; fi ; done

Run the above to avoid seeing symlinks in the output. On my 8.10 desktop with fglrx installed, this outputs:
/usr/lib/libfglrx_pp.so.1.0
/usr/lib/libifp.so.4.1.4
/usr/lib/libavc1394.so.0.3.0
/usr/lib/librom1394.so.0.3.0
/usr/lib/libGL.so.1.2
/usr/lib/libfglrx_gamma.so.1.0
/usr/lib/libXvBAW.so.1.0
/usr/lib/libglitz-glx.so.1.0.0
/usr/lib/libgdbm_compat.so.3.0.0
/usr/lib/libglitz.so.1.0.0
/usr/lib/libfglrx_dm.so.1.0
/usr/lib/libfglrx_tvout.so.1.0
/usr/lib/libmng.so.1.1.0.9
/usr/lib/libgdbm.so.3.0.0

Revision history for this message
draco (draco31-fr) wrote :

I'm on Jaunty too.
Here is my output :
draco@12:13:03 [0] <0>:~/tmp$ for file in `find /usr/lib -type f -name '*.so.*'`; do if objdump -x $file | grep -q -e '2\*\*20' ; then echo $file ; fi ; done | sort
/usr/lib/libavc1394.so.0.3.0
/usr/lib/libcuda.so.180.44
/usr/lib/libGLcore.so.180.44
/usr/lib/libGL.so.180.44
/usr/lib/libmac.so.2.0.0
/usr/lib/libmng.so.1.1.0.9
/usr/lib/libnvidia-tls.so.180.44
/usr/lib/librom1394.so.0.3.0
/usr/lib/libvdpau_nvidia.so.180.44
/usr/lib/libvdpau.so.180.44
/usr/lib/libvdpau_trace.so.180.44
/usr/lib/libXvMCNVIDIA.so.180.44
/usr/lib/nvidia/libnvidia-cfg.so.180.44
/usr/lib/tls/libnvidia-tls.so.180.44
/usr/lib/xorg/modules/extensions/libglx.so.180.44

Most of those libs are from the NVidia drivers (cuda, nvpau ...).

Revision history for this message
Poppafuze (poppafuze) wrote :

The above one-liners do not pick up on files that end in ".so". Moreover, there are two such files that are not in fact shared objects....they're ASCII text. Also, since some of the vendor-specific objects are discovered to reintroduce the problem, it would be good to look in their directories.

The following version examines all shared objects for the alignment of interest in the directories of interest, and emits the name of the package that owns it (or whatever dpkg-query's error string is for unowned files). I've evolved and tested this over a few hours, and it seems to pick up everything.

warning: This version is not for the faint of CPU. It will run at least 6 minutes on a modern machine before emitting anything at all (due to the queuing of sort). It has not been optimized in any way, but it does spread across CPUs and I did like the self-regulating nature of its CPU usage.

for file in `find {/usr,}/lib{32,} -type f`; do if file $file | grep -q -e 'shared object'; then if objdump -x $file | grep -q -e '2\*\*20'; then dpkg-query -S `echo $file`; fi; fi; done | grep -o -e '.*:' | sort | uniq

Output to follow....

Revision history for this message
Poppafuze (poppafuze) wrote :

For a recently updated Jaunty:

for file in `find {/usr,}/lib{32,} -type f`; do if file $file | grep -q -e 'shared object'; then if objdump -x $file | grep -q -e '2\*\*20'; then dpkg-query -S `echo $file`; fi; fi; done | grep -o -e '.*:' | sort | uniq

diversion by xorg-driver-fglrx from:
diversion by xorg-driver-fglrx to:
gkrellm-volume:
gkrellmwho2:
libavc1394-0:
libglitz1:
libglitz-glx1:
libmng1:
libnet0:
libphysfs-1.0-0:
sun-java6-bin:
xorg-driver-fglrx:
xorg-driver-fglrx, libgl1-mesa-glx:
xserver-xorg-core, xorg-driver-fglrx:

Some had quite a few files.

Revision history for this message
Poppafuze (poppafuze) wrote :

For a recently updated Hardy, the output was:

libgdbm3:
libtext-iconv-perl:
...and a number of non-deb .sos from VMware server.

Revision history for this message
Poppafuze (poppafuze) wrote :

For a recently updated Karmic, the output was:

libglitz1:
libglitz-glx1:
sun-java6-bin:

Revision history for this message
Per Ångström (autark) wrote :

This is my output for Lucid Lynx Alpha 2+:

libglitz1:
libglitz-glx1:
nvidia-173:
nvidia-current:

Revision history for this message
rusivi2 (rusivi2-deactivatedaccount) wrote :

Thank you for posting.

Is this still an issue in Maverick?

Changed in nvidia-drivers-ubuntu:
status: New → Incomplete
Revision history for this message
Brian Rogers (brian-rogers) wrote :

I think it is still an issue. If I try to use a 64-bit livecd in a VM with 512 MB of RAM, the VM can't run very long before the OOM killer comes into play.

I just tried with today's daily build. Running the live session, plus Firefox, plus the installer (but not doing anything in any of them) was enough to make it start killing processes.

Revision history for this message
Brian Rogers (brian-rogers) wrote :

Or, high memory usage is currently a problem. Not sure if it's because of this, or if this can cause much extra memory usage.

I'll investigate.

Revision history for this message
Phillip Susi (psusi) wrote :

It is now even worse in Maverick. The alignment has increased from 2**20 to 2**21.

Revision history for this message
Simon Ruggier (simon80) wrote :

Usage of virtual address space is not the same as usage of physical pages. A process using up 100+ MB of virtual memory isn't actually a problem. For comparison, on my machine, a bit of mental arithmetic shows that the top few processes shown by System Monitor have over 8 GB of memory mapped, whereas the real usage is around 1.4GB.

If you are trying to measure the physical memory usage of processes, you should use the "Resident Memory" column instead, as far as I can tell.

Changed in ubuntu:
status: Confirmed → Invalid
Revision history for this message
draco (draco31-fr) wrote :

The aligment of 2**21 is already used in some Lucid packages currently.

Here the command I use :
for file in `find {/usr,}/lib{32,} -xdev -type f -iname '*.so.*'`; do if file $file | grep -q -e 'shared object'; then if objdump -x "$file" | egrep -q '2\*\*[0-9]{2}'; then apt-file search "$file" ; fi; fi; done | sort

Changed in ubuntu:
status: Invalid → Confirmed
Revision history for this message
ruediix@gmail.com (ruedii) wrote :

Simon Ruggier is correct. Virtual addressing has nothing to do with physical address space used.

This "empty" space doesn't even take up swap space, let alone virtual memory. It does take up some page tables but that's it.

Having the extra virtual address space there is actually a good thing. It reduces vulnerability to buffer overflows, and similar issues.

However, I must agree that 1MB is a bit big, especially for libraries that are only a few kilobytes. Generally 4KB-64KB of space is enough to prevent issues. However, considering the Linux Userland virtual address space on AMD64 is currently 128TB, I don't think there is any shortage of virtual address space in the near future.

There may be other reasons for providing this extra space that I don't know about. Some of these libraries might allocate additional memory for certain things, and this would provide plenty of space for them to do so while keeping the same block of memory.

Revision history for this message
Phillip Susi (psusi) wrote :

The space can not be "allocated" and made use of at runtime; it is permanently off limits. It also does nothing for buffer overflows since they are done on the stack.

Revision history for this message
ruediix@gmail.com (ruedii) wrote :
Download full text (3.2 KiB)

Yes, but the space is also unallocated.
Allocation of physical memory is done far smaller memory blocks, hence the "zero space" is just empty allocation space. It's not using physical RAM. This is because memory mapping is done through sparse allocation on 64bit mapping, where the virtual memory layer is then mapped to a physical memory layer, in small pages, with most of those pages simply referencing nothing.

It's much like non existing phone numbers don't need lines and buildings to go to them. Unused sparse memory allocation is pretty much the same. The program is just like a business getting all the phone numbers in a 100 number range and then only using 25 of them. So long as there are enough phone numbers to go around it doesn't make there be any more lines being used.

Sure, it's a problem. There is no benefit, and it wastes resources to map in this manner. However, it doesn't hemorrhage memory like you would think.

More importantly, if any 32bit libraries are built in this manner, 32bit program virtual memory space is a very precious thing when running memory demanding 32bit legacy programs, such as games under wine that require a single 4GB flat memory, and wasting the shared virtual memory space of a 32bit program dramatically reduces the flat memory space accessible by such programs. This is probably a far bigger concern than 64bit virtual address space, which there is honestly so much of (64TB of allocation space in the current standard mapping, more on some large servers.)

It would be far more beneficial to map to 4K or 64K depending on the size of the library. Only insanely big libraries should be mapped to the 1MB barrier.

I can list the real consiquences of 1MB alignment of small libraries, and they are pretty bad:
1. Cache Prefetch performance issues.
2. Bulk Swap Transfers: If the shared libraries for a program aren't in contiguous space, they can't be swapped in bulk as one chunk.
3. Increased VM overhead.

You are right, there is no reason for the 1MB alignment and it wastes some resources. However, VM handling, particularly the D_Paged function which enables sparse memory mapping and dynamic paging, should prevent it from using nearly as much physical memory as you think.

Personally, I think that the memory alignment should be to 4KB, 64KB or 2MB depending on object size on systems with 4KB pages, and 64KB or 2MB on systems with 64KB pages. It should be using a waste factor calculation that weighs waste factor over increased allocation costs.

The reason for the option 2MB size is for large page allocation. Specifically, if a program is using hundreds of megabytes of memory, it is better to allocate them in 2MB aligned blocks so that the kernel can reallocate them dynamically into huge pages. This allows reduced VM overhead in both page indexing and processor overhead, reduces memory map fragmentation on both the physical and virtual layers, provides for faster cache transfers on modern CPUs, and faster transfers in and out of swap.

I agree that there is absolutely no reason to use a 1MB alignment. It is too big to be beneficial for small allocations, and too small to be beneficial for larg...

Read more...

Revision history for this message
Phillip Susi (psusi) wrote :

Even on large libraries, a 2 M alignment does not help. If a library maps a section larger than 2 M then it will still be able to use large pages for that mapping, even if its alignment requirement for the section is only 1 byte. The alignment requirement just reserves extra address space that goes unused ( and won't be converted to a large page since part of that 2 M space maps to nothing ).

Basically there is absolutely no reason to ever force an alignment larger than 4k.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.