console-kit-daemon leaks memory

I'm using Ubuntu 9.10 (consolekit - 0.3.1-0ubuntu2) and have noticed that this daemon continually sucks up more and more physical memory over time, even when the machine is not being used.

It starts around 3.7MB, but then increases. After about 1-2 days it reaches 200MB, I presume it would continue indefinitely but I usually reboot at that point.

I am also trying to chase down another memory issue that massively impacts performance. The box has 2GB of RAM, but something somewhere seems to be using it, but it isn't accounted for anywhere. free -m indicates that it has been used (ignoring cache and buffers of course!), as does System Monitor. But adding up all the processes' Memory (in S.M.) or all the RES/RSS in ps or top falls many hundreds of MBs short, and this gap increases. Quite quickly it gets to the point where the box is swapping constantly, when the number/type/use of apps shouldn't require anything like it. I do not have any reason to assume this is related, but mention it just in case.

Last night, not long after a reboot and before going to bed: (10MB)
root 891 1.0 0.4 122652 9840 ? Ssl 21:38 0:23 /usr/sbin/console-kit-daemon

This morning after box being unused all last night: (100MB)
root 891 2.2 5.2 221388 108180 ? Ssl Jan25 13:33 /usr/sbin/console-kit-daemon

pmap seems to indicate something weird... libraries are being mapped with 2MB chunks each with no permissions against them.

colin@iluvatar:~$ sudo pmap -d 891
891: /usr/sbin/console-kit-daemon
Address Kbytes Mode Offset Device Mapping
0000000000400000 128 r-x-- 0000000000000000 008:00001 console-kit-daemon
000000000061f000 4 r---- 000000000001f000 008:00001 console-kit-daemon
0000000000620000 4 rw--- 0000000000020000 008:00001 console-kit-daemon
0000000001af3000 36980 rw--- 0000000000000000 000:00000 [ anon ]
00007f4778000000 33188 rw--- 0000000000000000 000:00000 [ anon ]
00007f477a069000 32348 ----- 0000000000000000 000:00000 [ anon ]
00007f4780000000 38020 rw--- 0000000000000000 000:00000 [ anon ]
00007f4782521000 27516 ----- 0000000000000000 000:00000 [ anon ]
00007f4785ead000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4785eae000 10240 rw--- 0000000000000000 000:00000 [ anon ]
00007f47868ae000 88 r-x-- 0000000000000000 008:00001 libgcc_s.so.1
00007f47868c4000 2044 ----- 0000000000016000 008:00001 libgcc_s.so.1
00007f4786ac3000 4 r---- 0000000000015000 008:00001 libgcc_s.so.1
00007f4786ac4000 4 rw--- 0000000000016000 008:00001 libgcc_s.so.1
00007f4786ade000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786adf000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786aef000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786af0000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b00000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b01000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b11000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b12000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b22000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b23000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b33000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b34000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b44000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b45000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b55000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b56000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b66000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b67000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b77000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b78000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b88000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b89000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b99000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b9a000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786baa000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786bab000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786bbb000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786bbc000 64 rw--- 0000000000000000 000:00000 [ anon ]
0000...

Can you reproduce this with the latest release or from git? Thanks. Might be good to try to get some valgrind info too.

(In reply to comment #2)
> Can you reproduce this with the latest release or from git? Thanks. Might be
> good to try to get some valgrind info too.

eek, a bit beyond me! I don't suppose there is a deb anywhere that I can use on Ubuntu 0910 64bit? Compiling from source is only going to get me in trouble!

Hello?? I have downloaded the latest source, but can't even work out how to compile it and there are no instructions. Doesn't seem to be the usual ./configure ,,, make ... make install process....

seriously, I need help here! After only 1 day and 6 hours uptime:

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  959 root 20 0 405m 291m 1864 S 13 14.5 82:21.37 console-kit-dae
18974 colin 20 0 751m 148m 13m S 1 7.4 16:25.05 firefox
 2242 root 20 0 199m 141m 2160 S 1 7.0 7:33.82 polkitd
27313 colin 20 0 784m 121m 14m S 0 6.1 2:08.98 cairo-dock
16807 root 20 0 290m 113m 12m S 3 5.6 26:06.31 Xorg
17198 colin 20 0 359m 86m 5608 S 0 4.3 2:55.99 polkit-gnome-au

Console-kit-daemon, polkitd and polkit-gnome-authentication-agent-1 are consuming over 0.5GB between them!

I would love to try the latest version, but simply can't work out how to compile it, what compilation tools I need, and what other dependencies there are...

please can you help, as this is crippling my machine daily.

Forget it.... its perfectly clear from all your other tickets that this is your modus operandi, its a shame such a great project has such an unresponsive, discourtious dev involved.

I'm running Ubuntu 9.10 x64 with consolekit 0.3.1-0ubuntu2. This workstation has been setting idle fresh from a reboot without logging in for the past two days, and I appear to be suffering from the same issue as this bug report describes.

 18:21:05 up 2 days, 2:54, 4 users, load average: 1.07, 1.52, 1.75

Mem: 2056992k total, 1989884k used, 67108k free, 94448k buffers
Swap: 6024332k total, 78180k used, 5946152k free, 300644k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 1192 root 20 0 753m 558m 1932 S 10 27.8 248:28.92 console-kit-dae

My pmap looks very similar, only larger numbers.

I am familiar with valgrind, but I haven't dug into how consolekit works. If someone could give me some quick guidance on how to profile consolekit. I'll have 20 or 30 minutes tomorrow evening to try to dig into why this is happening.

At least one memory leak is easy to reproduce by running a loop like

  ssh-copy-id localhost
  while true; do ssh localhost true; done

and watching console-kit-daemon’s resident memory usage go up by tens of kilobytes per second in `top`.

(I’m running consolekit 0.4.1-4 on Ubuntu maverick amd64.)

Created an attachment (id=37533)
patches for a few bad memory leaks

These three patches fix most of the really bad (per-session) memory leaks I found with valgrind. (Apply with `git am`.)

There is still one worrisome leak within dbus-glib that might or might not be ConsoleKit’s fault:

==4766== 2,400 bytes in 150 blocks are definitely lost in loss record 997 of 1,005
==4766== at 0x4C284A8: malloc (vg_replace_malloc.c:236)
==4766== by 0x6282514: g_malloc (gmem.c:134)
==4766== by 0x6297749: g_slice_alloc (gslice.c:836)
==4766== by 0x6298F62: g_slist_append (gslist.c:229)
==4766== by 0x4E394C1: dbus_g_connection_register_g_object (dbus-gobject.c:2373)
==4766== by 0x40F127: register_session (ck-session.c:147)
==4766== by 0x40F577: ck_session_new_with_parameters (ck-session.c:1249)
==4766== by 0x408FC7: collect_parameters_cb (ck-manager.c:1621)
==4766== by 0x40DF4F: job_completed (ck-session-leader.c:390)
==4766== by 0x59EB69D: g_closure_invoke (gclosure.c:766)
==4766== by 0x5A02DA8: signal_emit_unlocked_R (gsignal.c:3252)
==4766== by 0x5A04525: g_signal_emit_valist (gsignal.c:2983)

But at least it’s much smaller than the ones I fixed.

Review of attachment 37533:

Thanks for tracking these down! The review tool doesn't cope very well with having two different patches for the same file apparently so I'm not sure this review will look as intended. Apart from the unref issues I mentioned this looks good.

::: src/ck-manager.c
@@ +1256,1 @@
 }

This isn't right. Functions shouldn't generally unref passed in parameters as a side effect.

@@ +1273,1 @@
 }

Also not right for the same reason.

@@ +1287,3 @@
         log_seat_session_removed_event (manager, seat, ssid);
+
+ g_free (ssid);

We don't need to take a new ref here since we already have one in this context.

> This isn't right. Functions shouldn't generally unref passed in parameters as
> a side effect.

I’m pretty sure I needed to do it that way, because ck_session_leader_collect_parameters stashes the callback and the leader away in the pending_jobs queue rather than running the callback immediately. With your modified version of my patch that you committed, is there anything to prevent the leader from being freed while the job is still in the queue?

(In reply to comment #11)
> your modified version of my patch that you committed, is there anything to
> prevent the leader from being freed while the job is still in the queue?

The hash table holds a ref. So I don't think it should happen.

(In reply to comment #12)
> The hash table holds a ref. So I don't think it should happen.

But still, what stops the hash table entry from being removed while the job is still in the queue?

Quite frankly, the patch look like it makes a whole lot of sense.

Meanwhile, users see their servers die or hang because the daemon eats memory like crazy. Please either apply the patch, or find some other way to stop the leak. Thank you.

(In reply to comment #14)
> Please either apply the patch, or find some other way to stop the leak.

William already applied a modified version of my patch for 0.4.2. The leak should already be gone.
http://cgit.freedesktop.org/ConsoleKit/commit/?id=7b9212fa6aff55420c58f2cacd0a941762920337

In the discussion that followed, I was just worried that his modifications introduced a potential use-after-free crash, because the JobData created by ck_session_leader_collect_parameters isn’t holding a reference to its leader. If there is no way for a CkSessionLeader to be freed with entries remaining in its pending_jobs queue (previously this was prevented by the missing unref in create_session_for_sender), then that isn’t a concern and this can be closed. Otherwise, perhaps this presentation makes it clearer why the extra ref and unref are okay?

--- a/src/ck-session-leader.c
+++ b/src/ck-session-leader.c
@@ -409,6 +409,7 @@ job_completed (CkJob *job,
 static void
 job_data_free (JobData *data)
 {
+ g_object_unref (data->leader);
         g_free (data);
 }

@@ -428,7 +429,7 @@ ck_session_leader_collect_parameters (CkSessionLeader *session_leader,
         ret = FALSE;

         data = g_new0 (JobData, 1);
- data->leader = session_leader;
+ data->leader = g_object_ref (session_leader);
         data->done_cb = done_cb;
         data->user_data = user_data;
         data->context = context;

(In reply to comment #15)
> + g_object_unref (data->leader);
> …
> - data->leader = session_leader;
> + data->leader = g_object_ref (session_leader);

William, do you have any comments on this follow-up patch (comment #15)? Or even just a reason that a CkSessionLeader can’t be freed with entries remaining in its pending_jobs queue, and hence the patch wouldn’t be necessary?

I see this bug hasn't been worked on for a while. Issue is still present!

root@ccs:/home/milz04# pmap -d 1452
1452: /usr/sbin/console-kit-daemon --no-daemon
Address Kbytes Mode Offset Device Mapping
08048000 116 r-x-- 0000000000000000 008:00001 console-kit-daemon
08065000 4 r---- 000000000001c000 008:00001 console-kit-daemon
08066000 4 rw--- 000000000001d000 008:00001 console-kit-daemon
08892000 981868 rw--- 0000000000000000 000:00000 [ anon ]
b6500000 132 rw--- 0000000000000000 000:00000 [ anon ]
b6521000 892 ----- 0000000000000000 000:00000 [ anon ]
b663b000 28 r--s- 0000000000000000 008:00001 gconv-modules.cache
b6642000 4 ----- 0000000000000000 000:00000 [ anon ]
b6643000 10240 rw--- 0000000000000000 000:00000 [ anon ]
b7043000 4 ----- 0000000000000000 000:00000 [ anon ]
b7044000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7054000 4 ----- 0000000000000000 000:00000 [ anon ]
b7055000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7065000 4 ----- 0000000000000000 000:00000 [ anon ]
b7066000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7076000 4 ----- 0000000000000000 000:00000 [ anon ]
b7077000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7087000 4 ----- 0000000000000000 000:00000 [ anon ]
b7088000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7098000 4 ----- 0000000000000000 000:00000 [ anon ]
b7099000 64 rw--- 0000000000000000 000:00000 [ anon ]
b70a9000 4 ----- 0000000000000000 000:00000 [ anon ]
b70aa000 64 rw--- 0000000000000000 000:00000 [ anon ]
b70ba000 4 ----- 0000000000000000 000:00000 [ anon ]
b70bb000 64 rw--- 0000000000000000 000:00000 [ anon ]
b70cb000 4 ----- 0000000000000000 000:00000 [ anon ]
b70cc000 64 rw--- 0000000000000000 000:00000 [ anon ]
b70dc000 4 ----- 0000000000000000 000:00000 [ anon ]
b70dd000 64 rw--- 0000000000000000 000:00000 [ anon ]
b70ed000 4 ----- 0000000000000000 000:00000 [ anon ]
b70ee000 64 rw--- 0000000000000000 000:00000 [ anon ]
b70fe000 4 ----- 0000000000000000 000:00000 [ anon ]
b70ff000 64 rw--- 0000000000000000 000:00000 [ anon ]
b710f000 4 ----- 0000000000000000 000:00000 [ anon ]
b7110000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7120000 4 ----- 0000000000000000 000:00000 [ anon ]
b7121000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7131000 4 ----- 0000000000000000 000:00000 [ anon ]
b7132000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7142000 4 ----- 0000000000000000 000:00000 [ anon ]
b7143000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7153000 4 ----- 0000000000000000 000:00000 [ anon ]
b7154000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7164000 4 ----- 0000000000000000 000:00000 [ anon ]
b7165000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7175000 4 ----- 0000000000000000 000:00000 [ anon ]
b7176000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7186000 4 ----- 0000000000000000 000:00000 [ anon ]
b7187000 64 rw--...