LRU manager thread might run too long on server shutdown

With diagnostics for bug 1622952 added:

innodb.innodb-log-file-size-1 w4 [ fail ] Found warnings/errors in server log file!
(...)
Program terminated with signal SIGABRT, Aborted.
#0 0xb7740c31 in __kernel_vsyscall ()
[Current thread is 1 (Thread 0xa10feb40 (LWP 3095))]
#0 0xb7740c31 in __kernel_vsyscall ()
#1 0xb7444ce1 in __pthread_kill (threadid=2702175040, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:58
#2 0x08713d5c in my_write_core (sig=6) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/mysys/stacktrace.c:422
#3 0x08352c0f in handle_fatal_signal (sig=6) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/sql/signal_handler.cc:236
#4 <signal handler called>
#5 0xb7740c31 in __kernel_vsyscall ()
#6 0xb70c3e89 in __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#7 0xb70c53e7 in __GI_abort () at abort.c:89
#8 0x0898a29c in fil_space_get_by_id (id=0) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/innobase/fil/fil0fil.cc:493
#9 0x0898ca13 in fil_space_get_space (id=0) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/innobase/fil/fil0fil.cc:1533
#10 0x0898ccaa in fil_space_get_size (id=0) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/innobase/fil/fil0fil.cc:1633
#11 0x08930bec in buf_flush_try_neighbors (space=0, offset=7, flush_type=BUF_FLUSH_LRU, n_flushed=0, n_to_flush=100) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/innobase/buf/buf0flu.cc:1248
#12 0x089310c5 in buf_flush_page_and_try_neighbors (bpage=0xb09ea6d0, flush_type=BUF_FLUSH_LRU, n_to_flush=100, count=0xa10fdf70) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/innobase/buf/buf0flu.cc:1394
#13 0x08931786 in buf_flush_LRU_list_batch (buf_pool=0xa11c248, max=100, limited_scan=true, n=0xa10fdf70) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/innobase/buf/buf0flu.cc:1591
#14 0x08931aca in buf_do_LRU_batch (buf_pool=0xa11c248, max=100, limited_scan=true, n=0xa10fdf70) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/innobase/buf/buf0flu.cc:1663
#15 0x08931fea in buf_flush_batch (buf_pool=0xa11c248, flush_type=BUF_FLUSH_LRU, min_n=100, lsn_limit=0, limited_lru_scan=true, n=0xa10fdf70) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/innobase/buf/buf0flu.cc:1794
#16 0x08932446 in buf_flush_LRU (buf_pool=0xa11c248, min_n=100, limited_scan=true, n=0xa10fdf70) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/innobase/buf/buf0flu.cc:1959
#17 0x08932ec0 in buf_flush_LRU_tail () at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/innobase/buf/buf0flu.cc:2264
#18 0x0893495c in buf_flush_lru_manager_thread (arg=0x0) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/innobase/buf/buf0flu.cc:2863
#19 0xb743e2b5 in start_thread (arg=0xa10feb40) at pthread_create.c:333
#20 0xb717f16e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:114

...

Thread 6 (Thread 0xa8e9db40 (LWP 3108)):
#0 0xb7740c31 in __kernel_vsyscall ()
#1 0xb7177ab5 in select () at ../sysdeps/unix/syscall-template.S:84
#2 0x087c79f6 in os_thread_sleep (tm=100000) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/innobase/os/os0thread.cc:298
#3 0x08896c64 in innobase_shutdown_for_mysql () at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/innobase/srv/srv0start.cc:3042
#4 0x0872df2b in innobase_end (hton=0x9f706d0, type=HA_PANIC_CLOSE) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/innobase/handler/ha_innodb.cc:4038
#5 0x0825a18e in ha_finalize_handlerton (plugin=0xa0a7330) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/sql/handler.cc:641
#6 0x08423448 in plugin_deinitialize (plugin=0xa0a7330, ref_check=true) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/sql/sql_plugin.cc:945
#7 0x0842390b in reap_plugins () at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/sql/sql_plugin.cc:1024
#8 0x0842595d in plugin_shutdown () at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/sql/sql_plugin.cc:1722
#9 0x0823bc0d in clean_up (print_message=true) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/sql/mysqld.cc:1959
#10 0x0823b7a1 in unireg_end () at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/sql/mysqld.cc:1827
#11 0x0823b671 in kill_server (sig_ptr=0x0) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/sql/mysqld.cc:1755
#12 0x0823b6a2 in kill_server_thread (arg=0xa8eff280) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/sql/mysqld.cc:1778
#13 0x08a2007f in pfs_spawn_thread (arg=0x9bf00948) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-xenial-32bit/storage/perfschema/pfs.cc:1860
#14 0xb743e2b5 in start_thread (arg=0xa8e9db40) at pthread_create.c:333
#15 0xb717f16e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:114

Thread 6 is very late in the shutdown, at the point where the buffer pool is supposed to be clean and all the tablespaces are already closed. Yet the LRU manager thread is trying to flush something, possibly because logs_empty_and_mark_files_at_shutdown() does not wait for it to quit.

There may or may not be further underlying issues (such as buffer pool page dirtied too late in shutdown process), which will still be visible after this one is fixed.