playback crashes on LInux with 'Out of memory'

Bug #1198222 reported by ska
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MOVED - Percona Playback
New
Undecided
Unassigned

Bug Description

[In:Percona Playback Documentation]
Hi,

I am currently trying to run Playback tool but every time it gets killed by OOM due to memory exhaustion. I have tried the version, which comes with Debian Squeeze(0.6-179.squeeze) and the source code:
http://www.percona.com/downloads/Percona-Playback/Percona-Playback-0.6/

The result is the same. Moreover, if I run Playback on the server itself, sometimes MariaDB gets killed by OOM which is pretty bad.

OS details
Linux wkwvdb-fra2 2.6.32-5-amd64 #1 SMP Fri May 10 08:43:19 UTC 2013 x86_64 GNU/Linux

and I was testing it against MariaDB 5.5:
whatever@my_hidden_ip:~$ dpkg -l | grep mari
ii libmariadbclient18 5.5.31+maria-1~squeeze MariaDB database client library
ii libmysqlclient18 5.5.31+maria-1~squeeze Virtual package to satisfy external depends
ii mariadb-client-5.5 5.5.31+maria-1~squeeze MariaDB database client binaries
ii mariadb-client-core-5.5 5.5.31+maria-1~squeeze MariaDB database core client binaries
ii mariadb-common 5.5.31+maria-1~squeeze MariaDB database common files (e.g. /etc/mysql/conf.d/mariadb.cnf)
ii mariadb-server 5.5.31+maria-1~squeeze MariaDB database server (metapackage depending on the latest version)
ii mariadb-server-5.5 5.5.31+maria-1~squeeze MariaDB database server binaries
ii mariadb-server-core-5.5 5.5.31+maria-1~squeeze MariaDB database core server files

I used the following line to start playback:

./percona-playback --input-plugin tcpdump --tcpdump-file /data/db1.tcpdump --tcpdump-mode accurate --db-plugin libmysqlclient --mysql-host my_hidden_ip --mysql-username root --mysql-schema imdb --mysql-port 3306

The syslog has the following:

Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.002307] lt-percona-play invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.005245] lt-percona-play cpuset=/ mems_allowed=0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.007048] Pid: 5047, comm: lt-percona-play Not tainted 2.6.32-5-amd64 #1
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.009311] Call Trace:
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.010304] [<ffffffff810b6708>] ? oom_kill_process+0x7f/0x23f
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.012217] [<ffffffff810b6c2c>] ? __out_of_memory+0x12a/0x141
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.014165] [<ffffffff810b6d83>] ? out_of_memory+0x140/0x172
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.016026] [<ffffffff810baae8>] ? __alloc_pages_nodemask+0x4ec/0x5fc
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.018278] [<ffffffff810cd034>] ? handle_mm_fault+0x27a/0x80f
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.020411] [<ffffffff810d2372>] ? do_brk+0x227/0x307
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.022230] [<ffffffff81300186>] ? do_page_fault+0x2e0/0x2fc
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.024201] [<ffffffff812fe025>] ? page_fault+0x25/0x30
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.026074] Mem-Info:
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.027076] Node 0 DMA per-cpu:
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.028456] CPU 0: hi: 0, btch: 1 usd: 0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.030230] Node 0 DMA32 per-cpu:
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.031598] CPU 0: hi: 186, btch: 31 usd: 30
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.033402] active_anon:124262 inactive_anon:124371 isolated_anon:0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.033403] active_file:69 inactive_file:174 isolated_file:0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.033403] unevictable:0 dirty:1 writeback:1 unstable:0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.033404] free:1998 slab_reclaimable:634 slab_unreclaimable:1087
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.033405] mapped:61 shmem:0 pagetables:1814 bounce:0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.043148] Node 0 DMA free:4036kB min:60kB low:72kB high:88kB active_anon:5744kB inactive_anon:5988kB active_file:16kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15368kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:0kB slab_reclaimable:56kB slab_unreclaimable:32kB kernel_stack:8kB pagetables:48kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.055187] lowmem_reserve[]: 0 994 994 994
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.057034] Node 0 DMA32 free:3956kB min:4000kB low:5000kB high:6000kB active_anon:491304kB inactive_anon:491496kB active_file:260kB inactive_file:696kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1018060kB mlocked:0kB dirty:4kB writeback:4kB mapped:236kB shmem:0kB slab_reclaimable:2480kB slab_unreclaimable:4316kB kernel_stack:528kB pagetables:7208kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.069915] lowmem_reserve[]: 0 0 0 0
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.071390] Node 0 DMA: 1*4kB 6*8kB 9*16kB 10*32kB 7*64kB 4*128kB 2*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4036kB
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.075570] Node 0 DMA32: 477*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3964kB
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.107716] 2501 total pagecache pages
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.109426] 2243 pages in swap cache
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.110934] Swap cache stats: add 2608278, delete 2606035, find 60756/96530
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.113557] Free swap = 0kB
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.114871] Total swap = 1044216kB
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.120817] 262139 pages RAM
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.122226] 5393 pages reserved
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.123611] 125 pages shared
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.124970] 254553 pages non-shared
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.126417] Out of memory: kill process 5047 (lt-percona-play) score 42951 or a child
Jul 5 16:50:17 wkwvdb-fra2 kernel: [1293580.129494] Killed process 5047 (lt-percona-play)

I also ran playback with strace. It sends queries to server but seems not to properly free the memory.

Do I do something wrong or is it a bug?

Have a nice weekend,
Sergey

Tags: doc
Revision history for this message
ska (kasmanaft) wrote :

and the dump has been created using

tcpdump -i any port 3306 -w example.dump

as explained in http://www.percona.com/doc/percona-playback/plugins/tcpdump/index.html

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.