Ubuntu
vsftpd package

vsftp processes crashes Ubuntu by using all of the available memory

Bug #1406605 reported by Onur Boyaci
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
vsftpd (Ubuntu)
New
Undecided
Unassigned

Bug Description

We've started using vsftpd on an Amazon EC2 instance like 2 months ago.
Server is getting very high concurrent ftp traffic.
But when the concurrency gets higher, vsftpd (standalone version) processes start to eat up more and more memory (totally all vsftpd processes together eat up all of the real avialable memory (30GB) of the instance and causes it to go out of service.
I may give you more details, but the most important vsftpd.conf settings are as follows:

vsftpd seems to limiting the # of clients (given by max_clients=1000) since I see "Connection refused: too many sessions." errors loged in the ftp log file.
There doesn't seem to be any other strange errors apart from lots of
[15684.883021] type=1326 audit(1419948989.671:348978): auid=4294967295 uid=65534 gid=65534 ses=4294967295 pid=29979 comm="vsftpd"
sig=31 syscall=37 compat=0 ip=0x7f531bf67757 code=0x0
erros in the kernel logs.

/etc/vsftpd.conf
listen=YES
anonymous_enable=NO
local_enable=YES
pasv_enable=YES
port_enable=YES
pasv_min_port=10000
pasv_max_port=20000
pasv_address=Publix IP of the Amazon instance
allow_writeable_chroot=YES
userlist_enable=YES
userlist_deny=NO
max_clients=1000
max_per_ip=3
idle_session_timeout=30
data_connection_timeout=120
seccomp_sandbox=NO

vsftpd version : 3.0.2-1ubuntu2.14.04.1
OS: Ubuntu 14.04.1 LTS, 3.13.0-43-generic #72-Ubuntu, x86_64

I'm quite sure that vsftpd processes use all of the real memory by inspecting
        the output of ps -C vsftpd -o user,rss,size,pid,cmd.
i.e. total rss of the vsftpd processes add upto 30GB when the system is about to run OOM.

Revision history for this message
Onur Boyaci (onurboyaci71) wrote :
Download full text (4.1 KiB)

Sorry there are lots of

INFO: task vsftpd:31878 blocked for more than 120 seconds. errors in the kernel logs also:

Dec 30 12:09:06 localhost kernel: [ 841.204124] INFO: task vsftpd:31878 blocked for more than 120 seconds.
Dec 30 12:09:06 localhost kernel: [ 841.206412] Not tainted 3.13.0-43-generic #72-Ubuntu
Dec 30 12:09:06 localhost kernel: [ 841.208721] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 30 12:09:06 localhost kernel: [ 841.211745] vsftpd D ffff88078fc33480 0 31878 878 0x00000000
Dec 30 12:09:06 localhost kernel: [ 841.211749] ffff8806d8dd9d40 0000000000000086 ffff880754cf4800 ffff8806d8dd9fd8
Dec 30 12:09:06 localhost kernel: [ 841.211753] 0000000000013480 0000000000013480 ffff880754cf4800 ffffffff81cdb7a0
Dec 30 12:09:06 localhost kernel: [ 841.211756] ffffffff81cdb7a4 ffff880754cf4800 00000000ffffffff ffffffff81cdb7a8
Dec 30 12:09:06 localhost kernel: [ 841.211760] Call Trace:
Dec 30 12:09:06 localhost kernel: [ 841.211769] [<ffffffff81725539>] schedule_preempt_disabled+0x29/0x70
Dec 30 12:09:06 localhost kernel: [ 841.211772] [<ffffffff817273a5>] __mutex_lock_slowpath+0x135/0x1b0
Dec 30 12:09:06 localhost kernel: [ 841.211777] [<ffffffff811a2929>] ? __kmalloc+0x1e9/0x230
Dec 30 12:09:06 localhost kernel: [ 841.211779] [<ffffffff8172743f>] mutex_lock+0x1f/0x2f
Dec 30 12:09:06 localhost kernel: [ 841.211784] [<ffffffff8161db61>] copy_net_ns+0x71/0x130
Dec 30 12:09:06 localhost kernel: [ 841.211788] [<ffffffff8108f899>] create_new_namespaces+0xf9/0x180
Dec 30 12:09:06 localhost kernel: [ 841.211791] [<ffffffff8108f993>] copy_namespaces+0x73/0xa0
Dec 30 12:09:06 localhost kernel: [ 841.211795] [<ffffffff81065b16>] copy_process.part.26+0x9a6/0x16b0
Dec 30 12:09:06 localhost kernel: [ 841.211797] [<ffffffff810669f5>] do_fork+0xd5/0x340
Dec 30 12:09:06 localhost kernel: [ 841.211800] [<ffffffff81066ce6>] SyS_clone+0x16/0x20
Dec 30 12:09:06 localhost kernel: [ 841.211804] [<ffffffff81731a09>] stub_clone+0x69/0x90
Dec 30 12:09:06 localhost kernel: [ 841.211806] [<ffffffff817316ad>] ? system_call_fastpath+0x1a/0x1f
Dec 30 12:09:06 localhost kernel: [ 841.211809] INFO: task vsftpd:31886 blocked for more than 120 seconds.
Dec 30 12:09:06 localhost kernel: [ 841.214584] Not tainted 3.13.0-43-generic #72-Ubuntu
Dec 30 12:09:06 localhost kernel: [ 841.216778] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 30 12:09:06 localhost kernel: [ 841.220142] vsftpd D ffff88078fc13480 0 31886 878 0x00000000
Dec 30 12:09:06 localhost kernel: [ 841.220146] ffff88069cff7d40 0000000000000082 ffff880662351800 ffff88069cff7fd8
Dec 30 12:09:06 localhost kernel: [ 841.220149] 0000000000013480 0000000000013480 ffff880662351800 ffffffff81cdb7a0
Dec 30 12:09:06 localhost kernel: [ 841.220152] ffffffff81cdb7a4 ffff880662351800 00000000ffffffff ffffffff81cdb7a8
Dec 30 12:09:06 localhost kernel: [ 841.220156] Call Trace:
Dec 30 12:09:06 localhost kernel: [ 841.220160] [<ffffffff81725539>] schedule_preempt_disabled+0x29/0x70
Dec 30 12:09:06 localhost kernel: [ 841.220163] [<ffffffff817273a5>] __mutex_lock_slo...

Read more...

Revision history for this message
Onur Boyaci (onurboyaci71) wrote :

vsftpd uses ext4-formatted filesystem over the Provisioned IOPS (SSD) of type gp2 volume.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.